{"slug": "testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers", "title": "Testing LLMs Like Software: A Promptfoo Deep Dive for QA Engineers", "summary": "Promptfoo, an open-source evaluation and red-teaming framework, addresses the challenge of testing non-deterministic LLM outputs by treating prompts as versioned, testable code. The framework enables QA engineers to define correctness beyond simple assertions, using deterministic checks, model-pinned providers, and CI gates to catch regressions like hallucinations or jailbreaks before deployment.", "body_md": "Want the full 46-page handbook?Promptfoo for QA: The Complete Engineer's Handbook (2026 Edition)by Himanshu Agarwal covers everything below in production depth — runnable code in every chapter, RAG/agent/red-team playbooks, and a full capstone repo.\n\nGet it here:[https://himanshuai.gumroad.com/l/Promptfoo-for-QA-The-2026-Engineers-Handbook]\n\nYour LLM feature passed in the playground. Then it hallucinated a refund policy, leaked a system prompt in a jailbreak attempt, or silently degraded when the model vendor pushed a point release — and nothing in your test suite caught it.\n\nThis isn't a testing gap you can patch with more `assert equals()`\n\n. It's a category error. Traditional QA assumes a function is deterministic: same input, same output, every time. LLMs break that assumption at the root. The same prompt can return three different answers across three calls, and *all three can be correct*. So \"correctness\" itself has to be redefined before you write a single test.\n\nIf you've spent 5–15 years writing test automation, this is the part that trips people up. You already know how to build a harness, mock a dependency, wire a CI gate. What's new is the *assertion layer* — deciding what \"pass\" even means when the system under test is a probability distribution over tokens.\n\nPromptfoo, the open-source eval and red-teaming framework, is built specifically for this gap. This article walks through how to actually use it at a production-engineering level — not \"hello world\" prompt testing, but the workflows you'll run on Monday: versioned evals, RAG faithfulness checks, agent regression suites, OWASP-mapped red teaming, and CI gates that block a bad deploy before it ships.\n\nThe first mistake most teams make is treating prompt testing as a one-off script — a Jupyter notebook someone runs before a demo. That doesn't scale, and it doesn't survive a model swap, a temperature change, or a teammate editing the system prompt six months from now.\n\nThe fix is to treat every prompt as versioned, testable code, with a config that lives in source control next to the application it powers.\n\nA minimal Promptfoo config already forces you to think like an engineer, not a prompt tinkerer:\n\n```\n# promptfooconfig.yaml\ndescription: \"Refund policy assistant — regression suite\"\n\nproviders:\n  - openai:gpt-4.1-2025-04-14\n  - anthropic:claude-sonnet-4-6\n  - vertex:gemini-2.5-pro\n\nprompts:\n  - file://prompts/refund_assistant.txt\n\ntests:\n  - file://tests/refund_cases.yaml\n```\n\nNotice the model strings are pinned to dated releases, not floating aliases like `gpt-4.1-latest`\n\n. This single habit prevents the most common silent regression in production LLM systems: a vendor updates the underlying model, your eval suite quietly starts scoring against a different distribution, and nobody notices until a customer does. Pin versions. Bump them deliberately, with a changelog entry, the same way you'd bump a dependency in `package.json`\n\n.\n\nNot everything in an LLM response needs a judge model to grade it. A huge percentage of real-world failure modes — malformed JSON, missing required fields, banned phrases, latency blowouts, cost overruns — are checkable with plain deterministic logic. Use these first. They're free, fast, and don't introduce a second LLM's bias into your pass/fail decision.\n\n```\ntests:\n  - vars:\n      query: \"I want a refund for order #4471\"\n    assert:\n      - type: contains\n        value: \"return window\"\n      - type: not-contains\n        value: \"I cannot help with that\"\n      - type: is-json\n      - type: regex\n        value: '\"order_id\":\\s*\"4471\"'\n      - type: latency\n        threshold: 2500\n      - type: cost\n        threshold: 0.02\n```\n\nFor anything the built-in assertion types can't express, drop into JavaScript or Python and write the check yourself:\n\n``` js\n      - type: javascript\n        value: |\n          const parsed = JSON.parse(output);\n          return parsed.refund_eligible === true && parsed.confidence > 0.8;\n```\n\nThe engineering discipline here matters more than the specific syntax: **exhaust deterministic checks before reaching for a model-graded one.** Every model-graded assertion you add is a second LLM call, a second cost line item, and a second source of nondeterminism in your test suite. Treat it as a scarce resource, not a default.\n\nFor genuinely fuzzy criteria — tone, helpfulness, whether a response \"sounds like our brand\" — you need an LLM to grade another LLM's output. Promptfoo supports this through `llm-rubric`\n\n, `similar`\n\n, and `factuality`\n\nassertion types.\n\n```\n      - type: llm-rubric\n        value: |\n          The response should acknowledge the customer's frustration\n          without admitting fault, and should not promise a specific\n          refund timeline.\n        provider: anthropic:claude-sonnet-4-6\n```\n\nHere's the part most tutorials skip: **the judge model has its own failure modes.** LLM judges systematically favor longer responses, are biased toward outputs that resemble their own writing style, and can be gamed by responses that use confident, authoritative language regardless of correctness. If you use the same model family as both the system under test and the judge, you inherit correlated blind spots — the judge may rate a hallucination highly simply because it \"sounds like\" something that model would say.\n\nTwo mitigations that actually hold up in production:\n\nRun llm-rubric assertions multiple times (Promptfoo supports repeat runs) and treat a single pass as noise, not signal. Three-out-of-three or four-out-of-five is a real signal.\n\nOnce you're running the same test suite across OpenAI, Claude, Gemini, Mistral, Llama, and Qwen, you'll notice something uncomfortable: there is no universal \"best model.\" One model wins on factual grounding, another on latency, another on cost-per-1k-tokens, another on instruction-following for your specific domain vocabulary.\n\nDon't pick a model on vibes or a public leaderboard that tested a completely different task distribution. Build a weighted scorecard against *your* task:\n\n```\ntests:\n  - vars:\n      query: \"Summarize this 4-page compliance doc in 3 bullets\"\n    assert:\n      - type: llm-rubric\n        value: \"Summary must not omit any regulatory deadline mentioned in the source\"\n        weight: 3\n      - type: latency\n        threshold: 3000\n        weight: 1\n      - type: cost\n        threshold: 0.015\n        weight: 1\n      - type: g-eval\n        criteria: \"Faithfulness to source document\"\n        weight: 2\n```\n\nWeights force a conversation your team needs to have anyway: is a 400ms latency win worth a 2% drop in factual accuracy for a compliance summarizer? That's a product decision disguised as a testing parameter — Promptfoo just makes it explicit and repeatable instead of a Slack argument.\n\nRun this matrix on a schedule (weekly is usually enough), because provider behavior drifts even when the version string doesn't change — quietly updated safety filters, routing changes, or backend optimizations can shift output distribution without any changelog at all.\n\nRAG systems fail in ways that a generic \"did it answer the question\" test completely misses. Four specific failure modes need their own assertions:\n\n**Faithfulness** — does the generated answer only use information present in the retrieved context, or did the model fill gaps with parametric knowledge? Test this by deliberately retrieving *incomplete* context and checking the model refuses to fabricate the missing piece rather than confidently guessing.\n\n**Groundedness** — can every claim in the output be traced back to a specific retrieved chunk? This is different from faithfulness: a response can be faithful (nothing contradicts the context) while still including ungrounded elaboration (nothing in the context supports it either).\n\n**Attribution** — if your system cites sources, do the citations actually point to the chunks that contain the claim, or is the model attaching plausible-looking citations to unrelated passages? This is a silent trust-destroyer in production and almost never gets tested explicitly.\n\n**Hallucination under retrieval failure** — what happens when retrieval returns zero relevant chunks, or chunks below a similarity threshold? The correct behavior is an explicit \"I don't have enough information,\" not a fluent, confident, made-up answer. Write tests that force this exact scenario:\n\n```\ntests:\n  - vars:\n      query: \"What is our policy on refunds for digital goods purchased in 2019?\"\n      retrieved_context: \"\"   # simulate empty retrieval\n    assert:\n      - type: llm-rubric\n        value: \"Must explicitly state it cannot find relevant information. Must NOT invent a policy.\"\n      - type: not-contains\n        value: \"our policy is\"\n```\n\nIf your RAG eval suite doesn't have at least one test category dedicated to *forced retrieval failure*, you don't actually know how your system behaves in the one scenario guaranteed to happen in production — the day your index is stale or the query genuinely has no match.\n\nAgentic systems compound LLM nondeterminism with tool-call sequencing, state management, and multi-turn error recovery. Testing them as a black box (\"did the final answer look right\") misses where they actually break. Split your suite into three layers:\n\n**Tool correctness** — given a task, does the agent select the right tool with the right arguments? This is checkable deterministically by intercepting the tool call, not just the final text output:\n\n``` js\n      - type: javascript\n        value: |\n          const toolCalls = context.toolCalls || [];\n          const refundCall = toolCalls.find(t => t.name === 'issue_refund');\n          return refundCall && refundCall.args.amount <= context.vars.order_total;\n```\n\n**Business-rule adherence** — does the agent respect constraints that live outside the model's training entirely, like \"never issue a refund over ₹50,000 without escalation\"? These are exactly the kind of hard constraints that should never be trusted to the model's judgment alone — enforce them as deterministic assertions on tool arguments, and separately test that the agent correctly triggers the escalation path instead of silently failing or retrying.\n\n**Failure recovery** — feed the agent a tool error (timeout, malformed response, permission denied) mid-task and check it recovers gracefully: retries appropriately, informs the user, or escalates — instead of looping, hallucinating a success message, or crashing the conversation state. This is the single most under-tested behavior in agent systems shipped today, and it's usually the first thing that breaks in front of a real customer.\n\nBuild these as separate test files, not one giant end-to-end suite. A tool-correctness regression and a business-rule regression have completely different root causes and completely different owners on your team; a single monolithic suite just tells you \"something broke\" without telling you what.\n\nSecurity testing for LLM applications isn't optional hardening — it's baseline QA, the same way you wouldn't ship a REST API without testing for injection. Promptfoo's red-teaming module maps directly to the OWASP LLM Top 10, and each category deserves a real test suite, not a single example prompt:\n\n```\nredteam:\n  plugins:\n    - harmful:privacy\n    - prompt-injection\n    - excessive-agency\n    - pii\n  strategies:\n    - jailbreak\n    - prompt-injection\n    - multi-turn\n```\n\nAlign the severity scoring with NIST AI RMF categories (govern, map, measure, manage) if your organization needs an audit trail — banking, healthcare, and insurance deployments increasingly require this mapping for compliance sign-off, and retrofitting it after launch is far more painful than building it into the initial test plan.\n\nRun red-team suites on a separate cadence from your functional regression suite — they're slower, higher-cost (many require multi-turn adversarial generation), and often need human review of borderline results. Weekly or pre-release is usually the right cadence, not on every commit.\n\nAn eval suite that only runs manually is a suite that gets skipped under deadline pressure. The whole point is a gate that can't be bypassed by accident.\n\nGitHub Actions:\n\n```\nname: LLM Eval Gate\non: [pull_request]\njobs:\n  promptfoo-eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: npm install -g promptfoo\n      - run: promptfoo eval --config promptfooconfig.yaml --output results.json\n      - run: promptfoo eval --output results.json --fail-on-error\n```\n\nThe critical design decision isn't the YAML syntax — it's setting an explicit pass threshold and failing the build below it, the same way you'd fail a build on a code-coverage drop:\n\n```\n      - run: |\n          SCORE=$(jq '.results.stats.successes / .results.stats.totalTests' results.json)\n          if (( $(echo \"$SCORE < 0.95\" | bc -l) )); then\n            echo \"Eval pass rate $SCORE below 0.95 threshold\"\n            exit 1\n          fi\n```\n\nFor Jenkins or Azure DevOps, the pattern is identical — run the eval as a pipeline stage, parse the JSON output, fail the stage on threshold breach. What changes across teams isn't the tooling, it's the threshold and the categories you gate on. A conservative starting point: gate hard on deterministic assertions and red-team critical findings (100% pass required), and gate softer on llm-rubric scores (95% with a manual-review path for the remainder). Loosening this over time is easy; tightening it after a bad release has already shipped is a much harder conversation to have with your team.\n\nA few failure patterns show up repeatedly once a suite grows past a few hundred test cases, worth naming so you don't rediscover them the hard way:\n\nTest suites rot faster than code. A prompt that passed 300 cases six months ago will silently drift out of relevance as the underlying product changes — refund policies update, new product lines launch, edge cases the original author never considered start showing up in production logs. Treat your eval dataset as a living artifact: feed real production failures back into it continuously, not just at initial build time.\n\nFlaky model-graded assertions get muted, not fixed. The instinct when an llm-rubric check fails intermittently is to loosen the rubric until it stops failing. Resist this — intermittent failure on a well-specified rubric usually means the *system under test* has genuine nondeterministic behavior worth investigating, not that the test is wrong.\n\nCost compounds invisibly. A red-team suite with multi-turn adversarial strategies across six providers can run into real money fast. Budget it explicitly, and don't run the full matrix on every commit — reserve that for release candidates.\n\nEverything above is really one idea applied six different ways: **an LLM application is a system with probabilistic components, and your job is to build a test harness that quantifies that probability instead of pretending it doesn't exist.** Deterministic assertions handle what's checkable with certainty. Model-graded assertions handle what isn't, with enough repetition and cross-model judging to make the signal trustworthy. RAG and agent testing target the specific failure modes those architectures introduce. Red teaming assumes an adversary is already in your test data. And none of it matters if it isn't wired into CI where a regression can actually block a deploy.\n\nThis is the same engineering discipline you've applied to every system you've tested for the last decade. The primitives are just different.\n\nThis article covers the surface.Promptfoo for QA: The Complete Engineer's Handbook (2026 Edition)by Himanshu Agarwal goes end-to-end across 46 pages and 11 chapters — full runnable configs, a complete enterprise capstone repo (PDF → RAG → Agent → Eval → Security Dashboard → CI/CD), and copy-paste CI gates aligned to OWASP LLM Top 10 and NIST AI RMF.\n\nBuy the full handbook:[https://himanshuai.gumroad.com/l/Promptfoo-for-QA-The-2026-Engineers-Handbook]", "url": "https://wpnews.pro/news/testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers", "canonical_source": "https://dev.to/himanshuai/testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers-5hgl", "published_at": "2026-07-12 15:32:39+00:00", "updated_at": "2026-07-12 15:43:58.001321+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-safety", "mlops", "ai-products"], "entities": ["Promptfoo", "Himanshu Agarwal", "OpenAI", "Anthropic", "Google Vertex AI", "GPT-4.1", "Claude Sonnet 4", "Gemini 2.5 Pro"], "alternates": {"html": "https://wpnews.pro/news/testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers", "markdown": "https://wpnews.pro/news/testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers.md", "text": "https://wpnews.pro/news/testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers.txt", "jsonld": "https://wpnews.pro/news/testing-llms-like-software-a-promptfoo-deep-dive-for-qa-engineers.jsonld"}}