{"slug": "same-prompt-same-endpoint-different-score", "title": "Same prompt, same endpoint, different score", "summary": "A test of a small prompt edit on 100 single-call tool-calling cases from UC Berkeley's BFCL V4 benchmark found the edit reduced invented optional arguments from 14 to 7 across three runs but produced overall scores too close to confirm a real improvement, according to the CodeCut experiment. Running deepseek/deepseek-v4-flash-0731 through OpenRouter with temperature=0, top_p=1, and max_tokens=1024, the author found Prompt B scored 81, 81, and 82 versus Prompt A's 79, 82, and 79 on the pinned deepinfra/fp8 provider. Pinning the provider did not make results repeatable, and wafer/fast averaged 77.0 while every other provider averaged at least 80.0.", "body_md": "## Table of Contents\n\n## Introduction\n\nImagine this scenario: You change one sentence, keep everything else fixed, and the result improves.\n\nBut is it real improvement, or is it just noise?\n\nA small gain does not always come from the prompt. It can also come from the provider, the hosted endpoint, or the test sample.\n\nIn this article, I test a small prompt edit on 100 tool-calling cases and repeat the same setup to separate the prompt effect from normal run-to-run noise.\n\n## TL;DR\n\n- The small prompt edit **helped with the mistake it targeted** , but the overall score was too close to show that the new prompt was better.\n- The same pinned setup still **produced different scores across runs** . Because the prompt gain was small, it was hard to tell whether the prompt or the run caused it.\n\n### Stay Current with CodeCut\n\nEasy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.\n\n## How I Ran It\n\nI tested the prompts on 100 single-call tool-calling cases from [BFCL V4](https://gorilla.cs.berkeley.edu/blogs/17_bfcl_v4_prompt_variation.html), a UC Berkeley benchmark for function calling.\n\nEach case had a known correct tool call. A response counted as correct only when it used the right function and filled the arguments with accepted values.\n\nI ran `deepseek/deepseek-v4-flash-0731` through [OpenRouter](https://openrouter.ai/). Every request used `temperature=0`, `top_p=1`, and `max_tokens=1024`.\n\nThe goal was to see whether a small prompt change could improve tool-call accuracy. Prompt A was the original prompt. Prompt B changed one part of the prompt to reduce a recurring mistake.\n\n```\nPrompt A:\nYou are a helpful assistant with access to tools. Call the appropriate function.\n\nPrompt B:\nYou are a helpful assistant with access to tools. Call the appropriate function.\nIf a parameter is not specified in the request, omit it rather than guessing a value.\n```\n\nI ran two checks:\n\n- Run Prompt A multiple times with each provider pinned, to see how much the result changed even when the prompt stayed the same.\n- Run Prompt A and Prompt B on the same pinned provider, `deepinfra/fp8` , to see whether Prompt B looked better under the same setup.\n\n## The Results\n\n### Prompt B Helped With Invented Optional Arguments\n\nOne recurring mistake was extra arguments. The user asked for a tool call, but the model sometimes filled in optional fields that were never mentioned. Prompt B was meant to reduce that.\n\n```\nuser asks: \"Get API tokens for user 12345\"\n\nbefore Prompt B:\nget_shareable_api_tokens(user_id=\"12345\",\n                         include_revoked=true)  # not requested by the user\n\nafter Prompt B:\nget_shareable_api_tokens(user_id=\"12345\")\n```\n\nPrompt B made fewer of these mistakes. Across three runs, invented optional arguments fell from 14 to 7.\n\n### The Overall Score Gain Was Not Significant Enough\n\nPrompt B fixed the targeted mistake more often, but that was not enough to make the overall result clearly better.\n\nFor the prompt comparison, I used one pinned provider, `deepinfra/fp8`, the same 100 cases, and three runs per prompt.\n\n| Prompt | Run 0 | Run 1 | Run 2 | \n|---|---|---|---|\n| Prompt A | 79 | 82 | 79 | \n| Prompt B | 81 | 81 | 82 | \n\nPrompt B looked a little better, but the gap was too small to call it a real improvement. It fixed some of Prompt A’s failures, but also made new mistakes, so the overall gain stayed small.\n\n### Pinning the Provider Did Not Make the Results Repeatable\n\nDoes pinning the provider make the results repeatable? To test that, I ran Prompt A three times on the same 100 cases. The model, scorer, decoding settings, and provider stayed fixed.\n\nIt did not. Every pinned provider still produced different scores across the three runs.\n\n### One Provider Was Clearly Weaker\n\nWere some providers better than others?\n\nIn this experiment, yes.\n\nThe provider scores were not all the same. `wafer/fast` was the outlier, averaging 77.0 while every other provider averaged at least 80.0.\n\n### Quantization Did Not Predict Quality\n\nDid lower precision hurt performance?\n\nI expected `fp4` to be weaker because it uses fewer bits than `fp8`. Fewer bits can make serving cheaper or faster, but it can also lose detail.\n\nThe result did not match that expectation. `relace/fp4` was slightly above the two `fp8` providers.\n\nThis does not prove that fp4 is better. It only shows that the quantization label was not enough to predict which provider is better.\n\n### Input Price Did Not Predict Quality\n\nDid price predict quality?\n\nNot here.\n\n`together` had the highest input price, but it was not the top scorer. `wafer/fast` also cost more than several providers that scored higher.\n\n## Key Takeaways\n\nThis does not mean prompt edits are useless. Prompt B did reduce the mistake it was written for.\n\nThe problem is the size of the effect. When the prompt change is small, the measured gain can get mixed with provider choice, endpoint variation, and the test sample. A larger prompt change may show a clearer effect.\n\nFor small prompt edits, I would check three things:\n\n- **Run the current prompt more than once** so you know how much the score moves before changing anything.\n- **Keep the provider fixed** when comparing prompts. Keep the prompt fixed when comparing providers.\n- Look at both **the targeted mistake and the overall result** . A prompt can fix one behavior while leaving the full score mostly unchanged.\n\nIf you want to move beyond manual prompt edits, the [DSPy guide](https://codecut.ai/dspy-auto-optimize-llm-classifier/) walks through optimizing an LLM classifier with examples and evaluation metrics.\n\n## Run the Experiment\n\nThe source code and recorded results are in the companion folder on GitHub: [prompt-ab-test-phantom-gains](https://github.com/khuyentran1401/codecut-blog/tree/main/prompt-ab-test-phantom-gains).\n\nTo generate the article tables from the recorded results:\n\n```\ncd notebooks/prompt-ab-test-phantom-gains\nscripts/run_prompt_ab_eval.sh\n```\n\nTo rerun the API experiment from scratch, set `OPENROUTER_API_KEY` in the repo root `.env`, then run:\n\n```\ncd notebooks/prompt-ab-test-phantom-gains\nscripts/run_prompt_ab_eval.sh --rerun\n```\n\nThe script prints the same provider comparison and prompt A/B tables shown above.\n\n## References\n\n- **[So you want to use OpenRouter?](https://mmoustafa.com/blog/so-you-want-to-use-openrouter/)** (Mo Moustafa, 2026): the reported TAU-Bench Airline spread across providers of one model, which motivated testing provider variation on this task set.\n- **[Provider Failover vs Model Fallbacks Explained](https://openrouter.ai/blog/insights/reliability-failover)** (OpenRouter): the distinction between provider-layer failover and model fallback, and why`allow_fallbacks: False` matters for pinning.\n- **[Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference)** (Thinking Machines, 2025): the batch-invariance explanation for temperature-0 nondeterminism and the 1,000-prompt endpoint test.\n- **[BFCL V4: Format Sensitivity](https://gorilla.cs.berkeley.edu/blogs/17_bfcl_v4_prompt_variation.html)** (Mao et al., UC Berkeley, 2025): the format-sensitivity case selection used to build this experiment.\n- **[Berkeley Function Calling Leaderboard dataset](https://github.com/ShishirPatil/gorilla)** (UC Berkeley, Apache-2.0): the source cases, ground-truth answers, and AST-style matching rule reimplemented for this experiment.\n\n## Related Tutorials\n\n- [Enforce Structured Outputs from LLMs with PydanticAI](https://codecut.ai/enforce-structured-outputs-from-llms-with-pydanticai/) : Shows how to validate LLM outputs with Pydantic models instead of relying only on prompt instructions.\n- [Make Your Coding Agent Answers Easier to Scan with i-have-adhd](https://codecut.ai/i-have-adhd-coding-agent-answers/) : Another small instruction-change experiment, focused on whether coding-agent answers become easier to scan.\n- [Before You Upgrade the Model, Try Thinking Mode](https://codecut.ai/before-you-upgrade-the-model-try-thinking-mode/)\n- [Stop Treating Your RAG Grounding Score as a Safety Net](https://codecut.ai/your-rag-checker-may-miss-errors-that-matter-most/)\n\n### Stay Current with CodeCut\n\nEasy-to-digest articles on Python, AI, and open-source tools. Delivered twice a week.", "url": "https://wpnews.pro/news/same-prompt-same-endpoint-different-score", "canonical_source": "https://codecut.ai/prompt-tweak-score-gain/", "published_at": "2026-09-22 09:24:00+00:00", "updated_at": "2026-09-22 09:53:52.458280+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-research"], "entities": ["CodeCut", "BFCL V4", "UC Berkeley", "deepseek/deepseek-v4-flash-0731", "OpenRouter", "deepinfra/fp8", "wafer/fast"], "alternates": {"html": "https://wpnews.pro/news/same-prompt-same-endpoint-different-score", "markdown": "https://wpnews.pro/news/same-prompt-same-endpoint-different-score.md", "text": "https://wpnews.pro/news/same-prompt-same-endpoint-different-score.txt", "jsonld": "https://wpnews.pro/news/same-prompt-same-endpoint-different-score.jsonld"}}