arXiv:2611.24971v1 Announce Type: new Abstract: In-context learning (ICL) improves large language model performance at inference time without updating model parameters. However, these gains are not free: in practical deployments, longer prompts directly increase inference cost and latency. Under a fixed token budget, prompt construction therefore becomes a resource-allocation problem. While prior work has often focused on how many examples to include in ICL, we study the complementary question of whether the available budget is better spent on a few long demonstrations or on a larger number of shorter ones. We study this aspect in question answering (QA), by organizing candidate demonstrations into length-based bins and constructing prompts from examples within the same length range under fixed budgets. Experiments on two QA benchmarks (\textit{Beat the AI} and \textit{Natural Questions}), using six instruction-tuned LLMs, show that shorter demonstrations are generally the most effective use of prompt space. This effect is especially strong on \textit{Beat the AI}, and remains visible on NQ, although gains are smaller and saturate earlier in the long-context setting. An additional natural language inference experiment on ANLI R3 finds that one of the two shorter bins performs best for five of six models, although longer demonstrations remain competitive. Overall, our results highlight demonstration granularity as an important factor in prompt design.
The ROI of Small Examples: Optimizing In-Context Learning Under Token Budgets
Shorter in-context learning demonstrations generally make the most effective use of a fixed prompt token budget, according to arXiv paper 2611.24971v1, which tested length-binned prompts on the Beat the AI and Natural Questions QA benchmarks with six instruction-tuned LLMs. The advantage was strongest on Beat the AI and remained visible but smaller and earlier-saturating on Natural Questions in long-context settings, while an ANLI R3 natural language inference experiment found one of the two shorter bins best for five of six models, with longer demonstrations still competitive. The authors frame prompt construction under a fixed budget as a resource-allocation problem and identify demonstration granularity as a key factor in prompt design.
When building few-shot prompts under a token budget, bin candidate demonstrations by length and fill the budget with shorter examples first; shorter demos beat longer ones across six models on Beat the AI and NQ.
more in playbook: Controlling LLM costRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.