{"slug": "the-9-4-you-can-t-afford-to-verify-evaluating-sdar-and-the-finops-of-trying", "title": "The ~+9.4% You Can't Afford to Verify: Evaluating SDAR (and the FinOps of Trying)", "summary": "A developer evaluating the SDAR (Stable Distillation with Adaptive Regularization) paper found that the reported ~9.4% improvement on ALFWorld is less interesting than the claim that SDAR avoids training instability. To verify this, three training arms are needed: plain GRPO, naive GRPO+OPSD, and SDAR, with the second arm being the expensive but crucial one. The cost of running the cheapest credible three-arm comparison on a single p4d.24xlarge instance ranges from $2,300 to $9,900 depending on spot pricing and debugging overhead.", "body_md": "Recap.\n\n[Part 1]framed the supervision problem.\n\n[Part 2]architected the four-model system on AWS and counted the hardware.\n\n[Part 3]put the gate on the page in fifteen lines of PyTorch. Now the question that decides whether any of it mattered:does the gate actually earn its keep - and what would it cost to know?This is the close. A verification design, the stability instrumentation that separates a real evaluation from a lucky one, and the FinOps reality that is the reason this whole series is a blueprint instead of a benchmark.\n\nIt's tempting to reduce the paper to a single number. The reported gains over plain GRPO are real and worth stating - **their numbers, not mine**: roughly +9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop accuracy.\n\nBut the headline percentage is the *less* interesting claim. The one that matters is the second: SDAR **avoids the training instability that naive GRPO+OPSD falls into**. The whole point of the gate [Part 3](https://dev.to/shoaibalimir/the-whole-paper-fits-in-one-sigmoid-implementing-the-sdar-gate-1f4k) was to keep the teacher's noisy rejections from destabilizing training. So a verification that only measures final task success has tested half the paper. You have to measure stability too - and most reproductions don't.\n\nThat two-pronged claim dictates the experiment.\n\nYou cannot prove SDAR's thesis with a before/after. You need three training runs, identical except for the supervision:\n\n| Arm | What it is | What it proves |\n|---|---|---|\nA. GRPO |\nPlain agentic RL, no teacher | The baseline the +9.4% is measured against |\nB. Naive GRPO+OPSD |\nTeacher distillation, no gate\n|\nThe instability SDAR claims to fix |\nC. SDAR |\nGated distillation\n|\n\nDrop arm B and you can show SDAR beats GRPO, but you've quietly deleted the paper's actual contribution - there's no evidence the *gate* did anything a plain teacher wouldn't. Arm B is the expensive arm nobody wants to run and the one that makes the result credible.\n\n| Metric | What it measures | Why it's here |\n|---|---|---|\nTask success rate |\n% of episodes that hit the goal | The headline; compares all three arms |\nTurn efficiency |\nSteps taken vs optimal | Catches \"succeeds but flails\" agents |\nGate-activation rate |\nFraction of tokens amplified vs softened | SDAR-specific sanity check - a dead or saturated gate (\n|\n\nThat last row is the one to obsess over. SDAR's selling point lives or dies in the variance, not the mean.\n\nFinal accuracy is a lagging indicator. By the time arm B's success rate looks bad, you've already paid for a diverged run. The win is catching instability *while it happens* and killing the job - which is both good science and direct cost control.\n\nHere's what to stream off every training run:\n\n```\nWATCH FOR DIVERGENCE  (kill early = save money)\n  per-turn loss variance ── spikes ─────────► instability onset\n  gate-value histogram   ── drifts to 0/1 ──► τ wrong, gate saturating\n  gradient norm          ── blows up ───────► λ too high, teacher noise winning\n  KL-to-reference        ── runs away ──────► actor drifting off-policy\n```\n\nOn AWS, this is a standard observability wiring job:\n\nThe honest version of the stability claim isn't \"SDAR scored higher.\" It's \"here are arm B and arm C on the same axes, and B's variance climbs while C's stays flat.\" That picture is worth more than any percentage.\n\nTime for the number that shaped the entire series. To run the three-arm comparison at the *cheapest credible* scale - one environment (ALFWorld), Qwen2.5-3B with LoRA, on a single `p4d.24xlarge`\n\n(8× A100 80GB):\n\n| Line item | On-demand | Spot (~50–70% off) |\n|---|---|---|\n| One converging run (~2–3 days × 8 GPU) | $1,570 – $2,360 | $550 – $1,100 |\n| Three arms (A + B + C) | $4,720 – $7,080 | $1,650 – $3,300 |\n| + debugging & false starts (~+40%) | ~$6,600 – $9,900 | ~$2,300 – $4,600 |\n| Idle node left running one weekend | $1,573 |\n- |\n\n*Public list pricing, US regions - verify yours; AWS cut P4/P5 by up to 45% in mid-2025.*\n\nSo even the minimal honest verification is **low-thousands on spot, near five figures on-demand** - before you touch the second and third environments or scale past 3B. That is not a side-project line item, and pretending otherwise would be the dishonest move. Hence: I built and explained the mechanism; I am not buying the benchmark.\n\nThe cost levers, if you do run it:\n\n`λ`\n\nschedule position `λ`\n\nis a compute knob tooA technique is only as good as knowing when to skip it. SDAR's dense, gated supervision earns its complexity on **long-horizon, multi-turn** tasks where trajectory reward is hopelessly coarse. It buys you much less when:\n\nThat last bullet is, honestly, my own situation - and naming it is more useful than pretending otherwise.\n\nFour parts, zero training runs, and a deliberate refusal to wave around a number I couldn't stand behind. What you have instead:\n\nThat's a blueprint. Someone with GPU budget - a lab, a sponsor, a pile of credits - could pick this up and run the three-arm comparison tomorrow, and they'd know exactly what to measure and what it'll cost. The benchmarked reproduction is genuine future work, gated on compute, and I'd rather say that plainly than fake a convergence curve.\n\nIf you're that someone, the design is yours. And if you run it, I want to see the arm-B-vs-arm-C variance plot - that's the picture this whole series was pointing at.\n\n*That's the series. If it was useful, the implementation details in Part 3 are the part most worth stress-testing - tell me where the reconstruction is wrong.*", "url": "https://wpnews.pro/news/the-9-4-you-can-t-afford-to-verify-evaluating-sdar-and-the-finops-of-trying", "canonical_source": "https://dev.to/shoaibalimir/the-94-you-cant-afford-to-verify-evaluating-sdar-and-the-finops-of-trying-ffj", "published_at": "2026-07-18 15:44:24+00:00", "updated_at": "2026-07-18 15:59:32.189741+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "mlops"], "entities": ["SDAR", "GRPO", "OPSD", "ALFWorld", "Qwen2.5-3B", "LoRA", "AWS", "p4d.24xlarge"], "alternates": {"html": "https://wpnews.pro/news/the-9-4-you-can-t-afford-to-verify-evaluating-sdar-and-the-finops-of-trying", "markdown": "https://wpnews.pro/news/the-9-4-you-can-t-afford-to-verify-evaluating-sdar-and-the-finops-of-trying.md", "text": "https://wpnews.pro/news/the-9-4-you-can-t-afford-to-verify-evaluating-sdar-and-the-finops-of-trying.txt", "jsonld": "https://wpnews.pro/news/the-9-4-you-can-t-afford-to-verify-evaluating-sdar-and-the-finops-of-trying.jsonld"}}