{"slug": "five-comments-that-redesigned-my-llm-verification-pipeline", "title": "Five Comments That Redesigned My LLM Verification Pipeline", "summary": "A developer redesigned their LLM verification pipeline after commenters Alexey and Manuel pointed out that deterministic code checks could catch garbage scenarios before any LLM call. The new pipeline layers code checks first, then LLM on residual ambiguity, and humans only where models diverge, achieving zero-cost rejection of four garbage scenarios in a 38-scenario test.", "body_md": "**Agent Determinism Illusions (Part 6)**\n\nWhere this fits:[Part 5]closed the experimental arc with an honest answer — no clean fix for the 75% false-negative wall. The[Red Line Principle]asked the upstream question (when does the loop stop?). This part takes thedownstreamturn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Fiveinsightsfrom overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed.\n\nSix experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: **there's no clean solution** to LLM output verification.\n\nBut after those posts went live, commenters saw something I didn't — not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline.\n\n§§1–4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). §5 is a design claim — flagged as such in place.\n\n[Alexey's comment](https://dev.to/alex_spinov/comment/3ai7e) identified the most fundamental design flaw in my experiments:\n\n\"G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM.\"\n\n[Manuel](https://dev.to/tecnomanu/comment/3aj7c) added the constructive direction:\n\n\"Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual.\"\n\nI went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4):\n\n| ID | Output | Type | Could code catch it? |\n|---|---|---|---|\n| G1 | \"I am a little duck, quack quack\" | nonsense | ✅ very short + no keywords |\n| G2 | \"。\" (a period) | pure punctuation | ✅ punctuation ratio > 50% |\n| G3 | \"TODO\" | placeholder | ✅ keyword blacklist |\n| G4 | \"0 passed in 0.00s (no tests collected)\" | zero-test pass | ✅ regex `0 passed` + `no tests`\n|\n\n**All four garbage scenarios can be caught deterministically, at zero cost, before any LLM call.**\n\nWhy didn't I do this? Because I defaulted to treating \"verification\" as \"ask the LLM.\" My experiment design was: Phase Gate (form check) → LLM (content check). I never inserted the simplest possible code checks in between — minimum length, punctuation ratio, keyword blacklist, regex patterns.\n\nThis omission rippled through the entire series:\n\n```\n                    ┌─────────────────┐\n          input ──→ │  Layer 0 (code) │  shape / existence\n                    │                 │  empty? punctuation? placeholder? zero tests?\n                    └────────┬────────┘\n                             │ pass\n                             ▼\n                    ┌─────────────────┐\n                    │  Layer 1 (code) │  contract match\n                    │                 │  minLen, keywords, blacklist\n                    └────────┬────────┘\n                         pass│\n              ┌──────────────┴──────────────┐\n              │ fail                        │ pass\n              ▼                             ▼\n           REJECT                  ┌─────────────────┐\n                                   │  Layer 2 (LLM)  │  semantic residual only\n                                   └────────┬────────┘\n                        unanimous│          │divergence (e.g. 2–1)\n                                 ▼          ▼\n                            AUTO-PASS   ┌─────────────────┐\n                                        │  Layer 3 human  │\n                                        └─────────────────┘\n```\n\nEach of L0/L1 can early-exit to REJECT. Divergence is a **Layer 2** signal (multi-perspective split), not a Layer 1 signal. If Layer 0 catches it, the LLM never sees it.\n\nI implemented this pipeline as a Python prototype and ran it on both the P1 (8-scenario) and P4 (30-sample) test sets. The results:\n\n**P1 test set:**\n\n| Metric | Original P1 (LLM only, v2) | Layered + calibrated prompt (Experiment F) |\n|---|---|---|\n| LLM calls needed (single judge / sample) | 8 (100%) | 4 (50%) |\n| Garbage caught by L0/L1 | 0 | 4/4 (100%) |\n| False positives | 0 | 0 |\n| False negatives | 3 (75%) | 0 |\n\n*(FN→0 here is layering **plus** the calibrated prompt — not layering alone. §2 separates the two effects. Call counts here are **one judge call per sample**. When §2 multiplies by three perspectives, it says so.)*\n\nRerun: `python forge-verify-layered-prototype.py`\n\n(needs `ANTHROPIC_*`\n\nfor Layer 2; `SKIP_LLM=1`\n\nfor L0/L1 only). Numbers above are from a full run with Layer 2 enabled.\n\n**P4 test set:**\n\n| Category | Samples | Caught by L0 | Caught by L1 | Reaches L2 | Zero-cost catch rate |\n|---|---|---|---|---|---|\n| correct | 10 | 0 | 0 | 10 | 0% (should all go to LLM) |\n| garbage | 10 | 3 |\n5 |\n2 | 80% |\n| edge | 10 | 0 | 2 | 8 | 20% |\n\n**Overall: single-judge LLM calls reduced 33% (30→20). Zero false positives from deterministic layers.**\n\n**This does not move Part 5's wall.** On the P1 set, the original 75% FN (3/4 legitimate rejects) went to 0 FN *after* L0/L1 removed all four garbage cases from the LLM's input — the LLM only judged the four legitimate scenarios, and with a calibrated prompt it didn't reject them. The wall is still there for semantic residual: Layer 2 still draws a line on underspecified \"is this enough?\" questions. Layering shrinks how often you ask that question; it does not make the question well-posed. If you read the FN→0 cell as \"we fixed the wall,\" you've misread the table.\n\nThe two garbage samples that made it through to Layer 2 (G08: \"I cannot parse this command\", G10: incomplete translation) are genuinely ambiguous — they *should* reach the LLM. That's correct behavior, not a leak.\n\n[Alexey's second comment](https://dev.to/alex_spinov/comment/3ai7e) pointed out a measurement problem:\n\n\"A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices.\"\n\nAll experiments P1-P4 used symmetric precision-recall metrics. F1 gives FP and FN equal weight. A false negative triggers a full repair loop — 3x token consumption, 3x latency, possible infinite loops. A false positive is one-shot contamination.\n\nI ran a dedicated cost-weight analysis (`scripts/cost-weight-optimization.py`\n\n) that takes P3b's 5 prompt variants and evaluates them across 5 cost ratios, to show how the \"optimal\" choice shifts.\n\n| Prompt | FP | FN | F1 | WCost(1:1) | WCost(3:1) | WCost(5:1) | WCost(10:1) |\n|---|---|---|---|---|---|---|---|\n| v1 extreme strict | 0 | 4 | 0 | 4 | 12 |\n20 |\n40 |\n| v2 strict (P1 baseline) | 0 | 3 | 0 | 3 | 9 |\n15 |\n30 |\n| v3 balanced | 0 | 0 | 100 | 0 |\n0 |\n0 |\n0 |\n| v4 lenient | 0 | 0 | 100 | 0 |\n0 |\n0 |\n0 |\n| v5 extreme lenient | 1 | 0 | 86 | 1 |\n1 |\n1 |\n1 |\n\nUnder symmetric F1, v3 (100) and v5 (86) are far apart. Under weighted cost at 3:1, v5 (cost=1) **beats v2** (cost=9) — v5 let one piece of garbage through, but because it never rejected valid work, its total cost is far lower than the strict prompt. v3 (cost=0) still wins outright; the useful flip is **v5 vs v2**, not “v5 ties v3.”\n\n**Read this table as a ranking-flip demo, not as a production recommendation.** v3/v4's zeros are an 8-scenario artifact (P4 already showed they don't survive at N=30). The load-bearing claim is the *shift in relative ranking under cost weight* — especially that thrift can prefer a slightly leaky prompt over a zero-FP / high-FN one — not that F1's winner changes on this tiny set (v3 stays on top whenever FN=FP=0).\n\n*Call counts in this table = samples reaching an LLM × **3 perspectives** (Strict/Balanced/Lenient), matching P3-style voting cost. §1's Experiment F table counts **one** judge call per sample. Same pipeline; different billing unit.*\n\n| Strategy | WCost(1:1) | WCost(3:1) | WCost(10:1) | LLM calls (×3 perspectives) |\n|---|---|---|---|---|\n| P3b v2 (unlayered) | 3 | 9 |\n30 |\n8×3=24\n|\n| P3b v3 (unlayered) | 0 | 0 |\n0 |\n8×3=24\n|\n| P1 layered + v3 | 0 | 0 |\n0 |\n4×3=12 (−50%)\n|\n| P4 unlayered (estimate) | 4 | 8 |\n22 |\n30×3=90\n|\n| P4 layered (Experiment F residual) | 1 | 3 |\n10 |\n20×3=60 (−33%)\n|\n\nLayering doesn't change that v3's cost is 0 (it already has FP=FN=0 on the 8-scenario set). But it changes two things that the raw cost number doesn't capture:\n\nFor v2 (the strict prompt from P1), the effect is more instructive. v2 has FN=3. Layering saves calls on garbage but doesn't reduce FN on the legitimate set:\n\nThis exposes the boundary of layering: it reduces the LLM's *workload*, not its *bias*. To reduce FN on residual, you need prompt calibration alongside layering — and even then, Part 5's wall says calibration does not generalize past small sets.\n\nI ran a continuous scan from costFN:costFP = 1:1 to 15:1. On the P3b 8-scenario set, **v3 dominates every ratio** — because FP=FN=0 yields zero weighted cost at any weight. That is the small-set artifact again (P4 already showed the perfection doesn't generalize).\n\nWhat *does* move is the **gap narrative**: at 1:1, F1 makes v3 look far ahead of v5 (100 vs 86). At 3:1, weighted costs are 0 vs 1 — v3 still wins, but the moral of the story is no longer “balance beats thrift”; it is “any FN>0 gets expensive fast, so a one-FP leak can beat a three-FN strict prompt (v5 vs v2).” At 10:1, every strategy with FN>0 collapses relative to zero-FN prompts *on this set*.\n\n**Symmetric metrics hide relative rankings that matter under cost.** F1 dramatizes v3 ≫ v5. Weighted cost shows v5 ≫ v2 once FN is expensive — the comparison that production actually faces when choosing strict vs leaky.\n\n**On this 8-scenario set, the F1 winner (v3) remains the weighted-cost winner.** Do not read the section as “the optimum flips away from v3 at 3:1.” It does not. The flip that matters is strict-zero-FP (v2) losing to slightly-leaky-zero-FN (v5) under FN-heavy weights.\n\n**v3/v4's zero errors are an 8-scenario artifact.** P4 already showed the advantage disappears at 30 samples. Treat zeros as a demo substrate, not a deployable operating point.\n\n**Layering doesn't reduce bias, but it shrinks how often bias is invoked.** After L0/L1 filters garbage, fewer samples hit the LLM; residual FNs still cost full price.\n\n**Drive FN→0 where rules apply; accept the wall on semantic residual.** Above cost ratio ~5:1, strategies with FN>0 on *garbage/contract* work are unsustainable — use L0/L1 + a non-strict residual prompt. On underspecified “is this enough?” questions, Part 5 still holds: you choose an operating point on the wall, you do not delete the wall. Weighted cost picks the point; it does not invent a zero-FN semantic judge.\n\nP3's multi-perspective voting experiment found a pattern I described but misinterpreted. My original framing:\n\n\"In split-vote scenarios, the majority was always wrong. Majority voting can't correct for systematic bias.\"\n\n[Dipankar](https://dev.to/dipankar_sarkar/comment/3aiii) flipped the interpretation:\n\n\"Vote disagreement itself is the most valuable signal. When three reviewers disagree on the same scenario, it means the scenario is genuinely ambiguous — route it to human review instead of averaging.\"\n\nRe-examining P3's data through this lens:\n\n| Scenario | Strict | Balanced | Lenient | Majority | Correct? |\n|---|---|---|---|---|---|\n| L1 (excerpt) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |\n| L2 (summary) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |\n| L3 (one chapter) | REJ | REJ | PASS | REJ (2-1) | ✗ FN |\n| G3 (TODO) | REJ | REJ | PASS | REJ (2-1) | ✓ |\n\nMajority voting was wrong on 3 of 4 split scenarios. But if I use divergence as the control signal:\n\nCaveat: divergence-routing fixes **split** errors. It does **not** fix unanimous systematic bias — if all three perspectives share the same wrong line (Part 5's wall), auto-execute still ships the wrong call. Dipankar's move measures uncertainty; it does not delete the wall.\n\nDipankar wasn't proposing a \"better multi-perspective voting algorithm.\" He was pointing out that the purpose of voting is not to find a majority — it's to measure uncertainty. I missed this distinction when writing P3.\n\nOperational rule (now implemented in forge-verify's layer 3):\n\n```\nif max(PASS, REJECT) / N < threshold (default 0.8)\n    → mark as UNCLEAR, write to human review queue\n    → do NOT majority-vote\n```\n\n[Mike Czerwinski](https://dev.to/jugeni/comment/3ahff) named the architectural limit I'd been circling without stating:\n\n\"Stacking more symbolic checks on top doesn't grow that reach, it just adds more places for the same blind spot to hide... 'Ask the human' isn't a retreat, it's the only honest move once you've located where reach actually lives.\"\n\nThe verification layer has reach into symbolic events (file exists, exit 0) but not into semantic correctness — the blind spot doesn't shrink, it moves. P4 reported 83.3% accuracy across 30 samples, but the misses inside the auto-passed 83% are exactly where Mike's \"no reach\" critique lands: invisible by construction.\n\n[xm_dev_2026](https://dev.to/xm_dev_2026/comment/3ajod) showed where this bites hardest in production — fixed-percentage audits:\n\n\"Fixed-percentage audits feel 'fair' but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way.\"\n\nMy original mitigation had been \"5-10% random audit.\" This isn't a parameter-tuning problem — it's a design principle problem. Fixed sampling assumes errors are uniformly distributed. Real production errors are long-tailed.\n\nI ran a **simulation** (`scripts/adaptive-sampling-sim.py`\n\n) — synthetic verification streams with controlled error distributions, not production logs — then compared fixed-rate sampling against adaptive sampling (200-trial averages). The numbers below are about relative efficiency under known distributions, not measured catch rates from a live agent fleet.\n\nThe adaptive formula:\n\n```\naudit_rate = base_rate × risk_weight / confidence^1.5\n```\n\nHigh-risk items get a higher audit rate; high-confidence items get a lower one.\n\n**Uniform (errors spread evenly):**\n\n| Strategy | Audit rate | Catch rate | Efficiency |\n|---|---|---|---|\n| Fixed 10% | 10.0% | 12% | 0.111 |\n| Adaptive(combined) | 7.8% | 18% | 0.247 |\n\nNo significant gap — without a long-tail structure, random sampling is adequate.\n\n**Long-tail burst (90% of errors in 10% of the stream — xm_dev_2026's scenario):**\n\n| Strategy | Audit rate | Long-tail catch rate |\nEfficiency |\n|---|---|---|---|\n| Fixed 5% | 5.0% | 5% |\n0.120 |\n| Fixed 10% | 10.0% | 10% |\n0.119 |\n| Fixed 20% | 20.0% | 20% |\n0.118 |\n| Adaptive(confidence) | 8.7% | 24% |\n0.321 |\n| Adaptive(risk) | 6.0% | 15% |\n0.264 |\nAdaptive(combined) |\n12.8% |\n65% |\n0.543 |\n\nAdaptive(combined) uses a slightly higher audit rate than Fixed 10% (12.8% vs 10.0%) and catches **65% of long-tail errors** vs Fixed 10%'s 10% — roughly 6× on catch rate, not an equal-budget comparison. For equal budget, see the Mixed table below (10.0% vs 10.1% → 10% vs 56% catch).\n\n**Distribution drift (Mike's \"input shift erases visibility\"):**\n\n| Strategy | Audit rate | Catch rate |\n|---|---|---|\n| Fixed 10% | 10.0% (flat) | 11% (constant) |\n| Adaptive(combined) | 21.5% (ramps up) | 48% |\n\nFixed sampling's catch rate stays flat as the error rate drifts upward. Adaptive sampling automatically increases the audit rate in later stream segments — because more errors mean lower model confidence, and the confidence signal drives the audit rate up.\n\n**Mixed (uniform noise + periodic long-tail bursts — most realistic, 200-trial average):**\n\n| Strategy | Audit rate | Long-tail catch rate |\nEfficiency |\n|---|---|---|---|\n| Fixed 5% | 5.0% | 5% |\n0.095 |\nFixed 10% |\n10.0% |\n10% |\n0.094 |\n| Fixed 20% | 20.0% | 20% |\n0.094 |\n| Adaptive(confidence) | 7.8% | 19% |\n0.230 |\n| Adaptive(risk) | 6.0% | 15% |\n0.148 |\nAdaptive(combined) |\n10.1% |\n56% |\n0.324 |\n\nKey row: **Fixed 10% and Adaptive(combined) have nearly identical audit rates (10.0% vs 10.1%), but long-tail catch rate goes from 10% to 56% — a 5.6x improvement at the same cost.**\n\nP2's consistency test showed that binary LLM judgments on identical input are highly stable (N=10, 0 divergence). So the \"confidence\" signal doesn't come from within-model variance — it comes from P3's multi-perspective divergence:\n\nP2 and P3 were independent experiments, but their outputs combine into a coherent adaptive sampling *story*. This isn't parameter tweaking — it's a design principle difference. Fixed sampling pursues \"uniform global coverage.\" Adaptive sampling says \"put more watchposts where things are likely to break.\" In the simulation, the latter uses audit budget **~3–4× more efficiently** (Mixed: efficiency 0.324 vs 0.094).\n\n**Production caveat:** the “confidence” signal here is **cross-prompt divergence**, not within-model variance (P2 showed the latter is ~0). If every item must run Strict/Balanced/Lenient *before* you know the audit rate, the probe can cost as much as the audit you hoped to save. Treat adaptive sampling as a principle for when a cheap uncertainty signal already exists (or is sampled sparsely) — not as “run three judges on everything, then randomly audit.” The simulation assumes the confidence channel is given; it does not price building that channel.\n\nThroughout P1-P4, all LLM review experiments output free-text \"reason\" fields. [Manuel](https://dev.to/tecnomanu/comment/3aj7c) identified the structural problem and the fix in one sentence:\n\n\"Treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence.\"\n\n[Alexey](https://dev.to/alex_spinov/comment/3ai7e) sharpened the architectural split:\n\n\"Deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue.\"\n\nMy experiments had this blind spot:\n\n```\nP1, scenario L1 (model REJECT):\n\"The research brief should cover the core mechanisms of the loop engine,\nbut the file only has a short excerpt...\"\n\nP1, scenario L3 (model REJECT):\n\"The task requires three chapters, but the output only contains one.\"\n```\n\nThese are impression judgments. You can't code-verify whether \"a short excerpt\" is enough.\n\nThe proposed output format:\n\n```\nAssertion 1: \"File line count = 3, expected > 20\"        → code-verifiable\nAssertion 2: \"File contains 1/3 required keywords\"        → code-verifiable\nAssertion 3: \"Content structure completeness < threshold\" → semantic judgment\n```\n\nAssertions 1-2 are deterministic — code can confirm whether the model's claim is true. Assertion 3 is the actual semantic judgment, preserve for Layer 2.\n\nThis creates a cascade: when a deterministic assertion is code-verified and found inconsistent with the actual file → explicit hallucination signal → mark as UNCLEAR → escalate. No human judgment required in the loop — the code flow triggers automatically.\n\n**Scope note:** unlike §§1–4, this section is a design claim, not a separate A/B in Experiment F. The prototype implements evidence-shaped L2 output; it does not measure whether assertion format alone reduces hallucination rate versus free-text reasons. Treat the cascade above as an engineering pattern pending that measurement.\n\n| Comment | My blind spot | Replacement |\n|---|---|---|\n| Alexey + Manuel | Fed everything to the same LLM reviewer | L0/L1 filter deterministically; LLM handles residual |\n| Alexey (2nd) | Symmetric FP/FN metrics | Weighted cost (FN×3) shifts optimal operating point |\n| Dipankar | Split votes averaged by majority | Divergence = UNCLEAR → human, no majority |\n| Mike + xm_dev_2026 | Fixed 5-10% audit rate | Adaptive sampling by confidence × risk |\n| Manuel + Alexey (2nd) | Narrative \"reason\" field | Evidence-quoted reviewer + deterministic assertions |\n\nCombined, these form a layered verification architecture — not a closed one: L0/L1 handle deterministic filtering (Alexey+Manuel), L2 LLM quotes exact failing evidence (Manuel+Alexey), divergence escalates to L3 human review (Dipankar), audit rate adapts by confidence (Mike+xm_dev_2026), and system thresholds are selected by weighted cost (Alexey 2nd). Each layer narrows what the next sees; none closes the semantic residue.\n\nThis article doesn't claim to have solved anything. It just puts the design decisions I made and the corrections the community provided side by side.\n\nThe full pipeline has been implemented in forge-verify's `content-verify.mjs`\n\n(**ReqForge product repo**, not this blog tree — the blog ships the Python prototype `forge-verify-layered-prototype.py`\n\n). File-by-file results show which layer stopped each sample. Early-exit example (L1 blacklist — L2/L3 never run):\n\n```\n  📄 src/api/register.ts\n  ❌ REJECT @ L1: contains blacklisted keyword: FIXME\n    └ L0: PASS\n    └ L1: REJECT — blacklisted keyword: FIXME\n```\n\nDivergence example (L0/L1 pass; L2 split → L3 human, no majority vote):\n\n```\n  📄 docs/brief.md\n  ⚠ UNCLEAR @ L3: split vote → human queue\n    └ L0: PASS\n    └ L1: PASS\n    └ L2: [REJECT/REJECT/PASS] PASS=1 REJ=2\n    └ L3: UNCLEAR — do not majority-vote\n```\n\nLayer 0/1 checks are zero-cost code. Layer 2 only runs on the residual. Layer 3 divergence detection prevents false majority decisions.\n\nAn earlier draft appended a long apology for a fabricated “directional failure” claim in a Part 3 comment. That thread became its own experiment (20×3×600) and then a correction stack (comment wrong → apology v1 wrong on DS4 → v2 numbers). Under the harness label, DS4 still 100% misses on qwen3/gemma3; deepseek is 13%/67%/20% catch/PARSE/miss. Post-hoc, DS4 is partly task ambiguity (10→10); clean L0/L1 wins remain DF6/DS9 value mismatch. Full write-up: forthcoming aside. Scripts: `directional-failure-v2.py`\n\n/ `scripts/results-v2/`\n\n.\n\n**Series navigation (Agent Determinism Illusions):**\n\nPublished parts: [dev.to/zxpmail](https://dev.to/zxpmail). Scripts: [GitHub](https://github.com/zxpmail/blog/tree/main/agent-determinism-illusions/scripts).\n\n*Experiment F prototype (this repo): forge-verify-layered-prototype.py (Python, runnable with or without API)*\n\n`scripts/forge-verify/content-verify.mjs`\n\n(not vendored here)*Previous: The Red Line Principle*\n\n**Which comment did I miss?** If you've hit a verification failure mode that the L0/L1/L2/L3 pipeline doesn't catch, drop it in the comments — I'll run it through Experiment F and report what each layer does with it.", "url": "https://wpnews.pro/news/five-comments-that-redesigned-my-llm-verification-pipeline", "canonical_source": "https://dev.to/zxpmail/five-comments-that-redesigned-my-llm-verification-pipeline-388f", "published_at": "2026-07-21 06:33:26+00:00", "updated_at": "2026-07-21 06:59:24.425817+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-agents"], "entities": ["Alexey", "Manuel"], "alternates": {"html": "https://wpnews.pro/news/five-comments-that-redesigned-my-llm-verification-pipeline", "markdown": "https://wpnews.pro/news/five-comments-that-redesigned-my-llm-verification-pipeline.md", "text": "https://wpnews.pro/news/five-comments-that-redesigned-my-llm-verification-pipeline.txt", "jsonld": "https://wpnews.pro/news/five-comments-that-redesigned-my-llm-verification-pipeline.jsonld"}}