{"slug": "the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab", "title": "The AI Crash Test: adversarial LLM testing you can audit in the Network tab", "summary": "A developer built The AI Crash Test, a browser-based tool for adversarial LLM testing that uses deterministic predicate grading instead of an LLM judge. The tool allows users to audit results via the Network tab, with the API key never touching the server. In a recent run, claude-haiku-4-5 showed 0% vulnerability across seven attack kinds.", "body_md": "*A browser tool that points your own API key at an adversarial battery and grades every answer with pure predicates — no LLM judge, and your key never touches my server.*\n\nThe first time I ran it against a real model, it told me the model was ~29% vulnerable.\n\nThat number was wrong. And the tool proved it was wrong — to me, in public — because of exactly one design decision.\n\nEvery verdict in The AI Crash Test is a deterministic predicate over the model's answer string: exact match, regex, a number check, an injection canary, a must-refuse rule. No model grades another model. So when the report flagged ~29% vulnerable, it also showed the fail card for every miss — prompt, expected, actual, side by side. Three of those cards didn't show a broken model. They showed a broken grader: false positives in my own code. I fixed the graders; the real number was 0%.\n\nThat's the whole pitch. An auditable grader has bugs you can catch in public. A vibes-based, LLM-as-judge arena just hands you a number and asks you to trust it.\n\nLLM red-teaming is a crowded, mature space. garak (NVIDIA), PyRIT (Microsoft), and promptfoo all do far more than this — more probes, more scale, more integrations. Browser tools that adversarially test with your own key exist too; most lean on an LLM judge.\n\nSo this isn't a new category, and I won't pretend it is. The narrow thing that's mine is an intersection: **browser-based BYOK + deterministic no-judge grading + a provably shared engine with a longitudinal drift board.** Distinctive engineering and discipline, not a market-novel product. If you want heavy artillery, go use garak. If you want a result you can reproduce byte-for-byte and a key that goes straight to the provider and never touches my server, read on.\n\n**1. Deterministic grading — no LLM in the grade path.** Every grade is a pure function of the answer string, run in an open-source engine called gradecore. Run a mock model through it twice and the score is byte-identical. No temperature, no judge drift, no \"the grader was having a bad day.\"\n\n**2. BYOK, never-touches.** The browser calls the provider directly with your key. crashkit's server receives only the answers — the grade request has no key field at all.\n\nDon't take my word for it. Open DevTools → Network, run a battery with your key, and search the panel for the key itself. It lights up only on the request to the provider (e.g. `api.anthropic.com`\n\n, in the `x-api-key`\n\nheader) — never in the `/api/grade`\n\ncall. I verified this live before writing this; you can reproduce it in about thirty seconds.\n\nThe honest caveat: this only works where the provider allows direct browser calls. Anthropic (with the dangerous-direct-browser-access header) and Gemini work; OpenAI-direct is often CORS-blocked. Stating the limit is part of the point.\n\ngradecore isn't a crashkit-only toy. It's the same deterministic engine behind my live model-drift board, which tracks 16 LLMs over time. Same code, two jobs: the board is longitudinal monitoring; The AI Crash Test is on-demand adversarial testing.\n\nAnd it's the *same* engine, not a lookalike. Run the board's frozen suite through gradecore and the `suite_hash`\n\ncomes out identical, byte for byte — faithful extraction, not a reimplementation. (To be clear: crashkit uses gradecore, not my whole eval stack.)\n\nEight tasks across seven attack kinds: prompt-injection, tool-abuse, spec-violation, refusal-calibration, unsafe-compliance, hallucination-bait, and consistency. You get a severity-weighted vulnerability report and a fail card for every miss.\n\nA real result from a recent run: **claude-haiku-4-5 resisted 100% — 0% vulnerable across all seven kinds**, deterministically graded, reproducible.\n\nI built this solo, self-taught, in under a year — and yes, with heavy AI assistance in the loop. The design decision I care about is the one you can check without trusting me: open the Network tab, run it twice, read the graders. Prove it — don't take my word for it.", "url": "https://wpnews.pro/news/the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab", "canonical_source": "https://dev.to/agentdev9/the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab-1b29", "published_at": "2026-07-23 04:57:51+00:00", "updated_at": "2026-07-23 05:29:31.680153+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-tools", "developer-tools"], "entities": ["The AI Crash Test", "gradecore", "Anthropic", "Gemini", "OpenAI", "NVIDIA", "Microsoft", "claude-haiku-4-5"], "alternates": {"html": "https://wpnews.pro/news/the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab", "markdown": "https://wpnews.pro/news/the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab.md", "text": "https://wpnews.pro/news/the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab.txt", "jsonld": "https://wpnews.pro/news/the-ai-crash-test-adversarial-llm-testing-you-can-audit-in-the-network-tab.jsonld"}}