{"slug": "the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through", "title": "The calibration bug that taught our fraud agent to wave fraud through", "summary": "A developer built an agentic fraud investigation system on TigerGraph for the Hacker House Goa challenge, using a LangGraph agent that queries the graph through an MCP server while keeping all regulator-facing decisions in an LLM-free core package enforced by an AST test. Working from 590,742 IEEE-CIS card transactions and 5,565 closed investigations, the agent produces provisional actions and approval routes before requesting more evidence, then revises them, and runs end to end without an API key.", "body_md": "*Building an agentic fraud investigator on TigerGraph — and the two measurement mistakes that silently inverted it.*\n\n**All 18 architecture diagrams, interactive:** [FraudGraph Blueprints](https://claude.ai/artifact/VioVT79phoeLTjBKkMWRvb):\n\nsystem, agent loop, MCP path, schema, GraphRAG, policy rules, all twenty\n\nbenchmark cases, evaluation and failure modes.\n\n*Three triggers, one LangGraph agent, TigerGraph reached through the MCP server, and a deterministic core the LLM cannot reach.*\n\n*The one conditional edge that makes it an agent: gather more evidence, or act.*\n\nTigerGraph's Hacker House Goa challenge hands you six months of card\n\ntransactions — 590,742 of them, from the IEEE-CIS dataset — and takes away the\n\none column everyone reaches for. There is no `isFraud` flag. Every transaction\n\ncarries a **risk score** from the bank's own model instead, and the dataset\n\nREADME is blunt about what that score is worth:\n\nAbove 0.7, most flagged transactions turn out to be legitimate. Some fraud\n\nscores near zero.\n\nWhat you get instead of labels is four months of **closed investigations** —\n\n5,565 of them, confirmed fraud and cleared false alarms, with the analyst's\n\nnotes. And twenty new alerts to decide.\n\nSo the task isn't classification. It's investigation: work out what kind of\n\nfraud this is, how far it goes, what to do about it, **and when you know enough to act**.\n\nThat last clause is the whole thing.\n\nBuried in the answer format is a requirement that quietly determines the\n\narchitecture:\n\nThe next best action and required approval route recorded: **before** any\n\nadditional evidence is requested, **after** any additional evidence is\n\nreceived.\n\nYou cannot produce that pair honestly from a single-shot pipeline. There is no\n\n\"before\" unless the agent genuinely commits to a recommendation under\n\nuncertainty, *then* decides what it needs, *then* revises. So the topology is:\n\n```\nretrieve → believe → is this defensible?\n                          │ no\n                          ▼\n              PROVISIONAL action + approval route\n                          │\n              pick the question worth asking\n                          │\n                      response\n                          │\n                          ▼\n              REVISED action + what changed\n```\n\nEverything else follows from wanting that loop to be honest.\n\n*The model sits outside the decision boundary; people sit on the approval boundary.*\n\nEvery decision a bank would have to justify to a regulator lives in a\n\n`core/` package that **cannot import an LLM**. Not \"does not\" — cannot. A test\n\nwalks the AST of every file in that directory and fails the build if an import\n\nof `anthropic`, `openai`, `langchain` or anything with `llm` in the name ever\n\nappears:\n\n``` python\ndef test_core_cannot_import_an_llm():\n    banned = {\"anthropic\", \"openai\", \"google\", \"langchain\", \"langgraph\", ...}\n    for path in (ROOT / \"src\" / \"fraudagent\" / \"core\").glob(\"*.py\"):\n        tree = ast.parse(path.read_text())\n        ...\n    assert not offenders, f\"core/ must stay LLM-free: {offenders}\"\n```\n\nThat covers the Bayesian ledger, the policy engine, the approval routing, the\n\nstopping rule and the episode reconstruction. The model plans which graph\n\nprimitives to run, synthesises evidence into prose, writes the case summary and\n\nthe SAR narrative. If it proposes an action anyway, the policy decision object\n\ndrops it before it can reach an answer file.\n\nThe practical consequence, which I like more than the principle: the agent runs\n\nend to end with **no API key at all**. Verdicts, probabilities, actions and\n\napproval routes are bit-identical; only the wording changes. Anyone can clone\n\nthe repo and reproduce the twenty answer files without buying anything.\n\n*Similar past cases found by meaning and by shared entities inside one GSQL query, then fused.*\n\nThe obvious answer is \"traversal\", and that's true — but the reason this project\n\nbelongs on a graph database with vectors *in the same store* is a query that\n\nneeds both at once.\n\nCase memory is the requirement: prior closed cases must measurably change what\n\nthe agent concludes. The naive implementations both fail in the same\n\ncharacteristic way:\n\nNeither is what an analyst means by \"have we seen this before\". So fuse them, in\n\none GSQL statement, inside the database:\n\n```\ncandidates = vectorSearch({ClosedCase.narrative_emb}, query_vec, k * 8);\ncandidates = SELECT c FROM candidates:c WHERE c.opened_at < as_of\n             ACCUM @@vec_score += (c -> c.@distance);\n\nlinked_d = SELECT c FROM seed_dev:v -(CC_DEVICE_OF:e)- ClosedCase:c\n           WHERE c.opened_at < as_of\n           ACCUM @@shared += (c -> 1);\n\nresult = SELECT c FROM (candidates UNION linked_t UNION linked_c UNION linked_d):c\n         ACCUM c.@score += 0.55 * @@vec_score.get(c)\n                         + 0.45 * (1.0 - exp(-0.6 * @@shared.get(c)))\n         ORDER BY c.@score DESC LIMIT k;\n```\n\nTwo details do real work. The **saturating** term on the graph leg means the\n\nfirst shared entity is worth a lot and the fourth almost nothing — with a linear\n\nterm, one busy device profile dominates every ranking. And the `as_of` guard on\n\n**both** legs: a memory system that can retrieve a case opened after the one\n\nit's reasoning about isn't a memory system, it's a leak, and every number\n\ndownstream of it is fiction.\n\nThere's a second embedding space too, and it's the one I'd defend hardest.\n\n`V1`–` V339` are Vesta's engineered features: real signal, no names, and utterly\n\nhopeless as LLM context. 339 unnamed columns isn't evidence, it's noise with a\n\nschema. Compressed by incremental PCA to 24 components and concatenated with\n\nrank-normalised count and time-delta columns, they become a **64-dimensional behaviour vector** stored on every \n\n`Transaction`. Now \"which confirmed-fraud\nHere's the part I'd want to read in someone else's write-up.\n\nThe plan was to measure likelihood ratios instead of guessing them: replay all\n\n5,565 closed investigations through the same extractor the agent uses, under\n\neach case's own `as_of`, and count. Real numbers from real outcomes.\n\nThe first run produced this:\n\n| signal | P(s \\| fraud) | P(s \\| cleared) | LR | \n|---|---|---|---|\n| `known_device_and_region` | 0.610 | 0.188 | **3.30** | \n| `amount_typical` | 0.394 | 0.150 | **2.63** | \n| `device_new_on_account` | 0.134 | 0.484 | 0.28 | \n| `amount_anomalous` | 0.078 | 0.125 | 0.62 | \n\nRead that carefully. It says a transaction on a **known device, in a known region, for a typical amount** is over three times more likely to be fraud. And\n\nWhich is, taken at face value, insane. And the agent believed it completely —\n\n93% held-out accuracy, beautiful Brier score, and on the benchmark it confidently\n\ncalled quiet, ordinary transactions fraud while clearing the obvious ones.\n\nThe bug isn't in the modelling. It's in what the closed-case file *is*.\n\nEvery **cleared** case in that history got there by scoring high on the bank's\n\nmodel. It's a travel trip, a new phone, an unusual but genuine purchase — an\n\ninvestigation that happened *because* the transaction looked strange. Every\n\n**confirmed** case got there because a cardholder phoned up about a charge they\n\ndidn't recognise, and those are frequently small and unremarkable.\n\nSo inside the closed-case file, \"looks anomalous\" genuinely does predict\n\n*cleared*. The measurement was correct. The population was wrong.\n\nThe fix is a third stratum the file doesn't contain: **ordinary transactions from the same four months that no investigation ever touched.** Because months\n\nAdding untouched negatives helped, and did not fix it. `known_device_and_region`\n\ncame down to LR 1.56 — still pointing the wrong way.\n\nThe remaining confound is the card, not the transaction. Confirmed-fraud cases\n\nsit disproportionately on **long-lived, busy cards**: more history, therefore\n\nmore established devices and regions, therefore \"the device and region are\n\nalready established\" correlates with fraud through a variable that has nothing\n\nto do with fraud.\n\nThe fix is a **matched control design**. For each confirmed-fraud case, draw\n\nanother transaction **on the same card** as its control. The card-level confound\n\ncancels exactly, and the question the calibration answers becomes the question\n\nan analyst actually asks:\n\nWhat is different about *this* transaction compared with the rest of this\n\ncard's behaviour?\n\nWith both corrections in, held-out fraud-versus-not accuracy went from **0.93 to 0.57**, and most of the likelihood ratios collapsed to about 1.\n\nMy first instinct was that I'd broken something. I hadn't. That number is the\n\ndata answering the question I'd finally asked correctly.\n\nEvery confirmed case in this history was found by a cardholder ringing up. Fraud\n\nnever entered that file *because it looked anomalous* — it entered because\n\nsomeone noticed a charge. So once you remove the trigger as a feature, which you\n\nmust because it is perfectly confounded with the outcome, the graph signals\n\ngenuinely cannot separate confirmed fraud from ordinary activity on the same\n\ncard. Most fraud in this dataset is small and unremarkable. That is the point.\n\nThe 0.93 was the artefact. The 0.57 is the finding.\n\nWhat the history *can* teach is which typology a fraud is, and it teaches that\n\nstrongly:\n\n| signal | CNP-new-device | out-of-region | account takeover | \n|---|---|---|---|\n| `device_new_on_account` | **0.85** | 0.001 | 0.07 | \n| `match_flag_anomaly` | 0.001 | **0.56** | 0.47 | \n| `online_burst_2_to_4` | **0.43** | 0.001 | 0.04 | \n| `known_device_and_region` | 0.14 | **0.86** | 0.75 | \n\nSo the architecture follows the evidence rather than the other way round:\n\nfraud-versus-not is carried by the trigger prior and by the structural typology\n\nmatchers; *which* typology is carried by the measured table. I'd rather ship\n\nthat with the 0.57 written on the tin than ship the 0.93 and let the reliability\n\ncurve look excellent right up until the agent clears a real fraud for being\n\nunremarkable.\n\nThree lessons I'd take anywhere, not just here:\n\nRelated, and worth thirty seconds. If you measure \"is this device new to the\n\ncard\" against the card's whole history up to the alert, the **earlier legs of the same fraud episode** are in that history. The first fraudulent transaction\n\nSo anomaly is measured as **suspect window against established baseline**, with\n\nthe baseline cut off 72 hours before the alert. Seventy-five percent of\n\nclosed-case episodes open within 29 hours of their alert, so the window catches\n\nthe tail without swallowing ordinary history.\n\nThe dataset drops a deliberate hint:\n\nNot every fraud pattern present in the data is documented.\n\nFive typologies are described. Nine closed cases are marked `undocumented` —\n\nconfirmed fraud the bank's own analysts could not categorise. Their narratives\n\nsplit cleanly into two signatures, and both turn up in the graded twenty.\n\n**U1 — sub-threshold authorisation structuring.** Four online purchases inside\n\nroughly forty minutes, each amount parked just beneath a $500 authorisation\n\nceiling, about $1,900 in total. It's invisible to per-transaction scoring\n\nbecause *every individual leg looks completely ordinary*. It only exists as a\n\nproperty of the burst. **HHG-006** in the benchmark is one: $478.95, $456.96,\n\n$488.04, $482.12 in thirty minutes, $1,906.07 total — against five closed cases\n\nbetween $1,871 and $1,922.\n\n**U2 — a shared-device ring behind an anonymising proxy.** One device profile\n\nacross a run of unrelated cardholders in a single month, marked new on every\n\naccount it touches. **HHG-014** is one of these; the analyst request that\n\ntriggers it even says so.\n\nThe detection subtlety on U2 is the bit I'd put on a slide. A \"device profile\"\n\nin this dataset is a `DeviceInfo | OS | browser | screen` string — a model, not\n\na serial number — and **92% of online transactions sit on a profile shared by three or more cards**. Counting cards per device finds nothing but noise, and a\n\nWhat separates a real hub is the **anonymous-proxy ratio**:\n\n| device profile | cards | via anonymous proxy | ratio | \n|---|---|---|---|\n| `SM-G935F \\ | Android 7.0 \\ | chrome 62.0 \\ | 1920x1080` | \n| `Windows \\ | Windows 10 \\ | chrome 65.0 \\ | 1920x1080` | \n| `Trident/7.0 \\ | Windows 10 \\ | ie 11.0 \\ | 1920x1080` | \n\nEvery card that ever touched the ring device arrived anonymously. That is not a\n\nhousehold sharing a tablet.\n\n*Evidence becomes numbers through measured likelihood ratios; numbers become actions only through the written policy.*\n\nWhen belief won't support a defensible action, the agent has to pick one\n\nevidence action. Picking the strongest is wrong; picking the best information\n\nper unit of **friction** is right — and friction is not money. Pinging a\n\ncardholder at two in the morning has a real cost even when the API call is free.\n\nEach candidate is scored by expected reduction in Shannon entropy over the\n\nhypothesis space, divided by its friction cost, and filtered through the policy\n\n**first** — because the policy constrains *asking*, not only acting. That lets\n\nthe agent produce a sentence I'm fond of:\n\nThe highest-value next evidence is customer validation, but the contact-fatigue\n\nlimit forbids a third contact this week, so I am escalating to an analyst\n\ninstead.\n\nThat single sentence demonstrates uncertainty handling, policy compliance, next\n\nbest action and explainability at once.\n\nAnd the stopping rule is two-sided. Anyone can stop when confident. The second\n\ncondition is the one that matters:\n\nNo permitted evidence action carries enough information per unit of customer\n\nfriction to justify it. Further investigation would not change the decision, so\n\nthe case goes to a human with what is known.\n\nThat's a different and more defensible claim than \"I'm sure\", and it's what a\n\nreal fraud desk does.\n\n*HHG-018: a disputed charge the card has paid 21 times. Asked first, recognised, closed without a block.*\n\n*Rings you can't see from one card: gated so a popular browser string never becomes a false ring.*\n\nAfter the first complete build I read ten other teams' write-ups for the same\n\nbrief. Two disagreements with my answers were worth checking against the data.\n\n**HHG-019 is a ring you can't see from one card.** It is a risk-model alert, and\n\nthis history's risk-model alerts are almost always false alarms (900 of 900\n\ncleared). But ask what happened on *other* cards: the flagged purchase's device —\n\na profile only three cards had ever used — bought $100.00 and $100.06 on two other\n\ncards that week. `peer_purchases.gsql` is that two-hop question. The gate matters\n\nmore than the query: on a generic browser string (\"chrome 66.0\", 175 cards)\n\nordinary shoppers with similar baskets would make a false ring, so it only fires\n\non a rare device. Its weight is measured, not chosen. It fired on 31\n\nconfirmed-fraud closed cases and no cleared ones, a likelihood ratio of 22.4,\n\nthe strongest in the model.\n\nMy first version of it made two mistakes, and the October holdout caught both.\n\nIt treated the peer cards as *connected cards*, which switched on R6 and the\n\nfiling rule — and filed 19 reports on holdout cases the bank itself never\n\nreported. And it folded in \"did not fire\" on every other case, which nudged\n\neach one away from card-not-present fraud and cost nineteen correct pattern\n\ncalls. Both fixes are about what the evidence actually proves. Peers move the\n\nprobability; they don't prove a shared origin, so R6 and the report still need\n\none. And the absence of a rare hub is not a finding, so the signal is\n\nfired-only. HHG-019 now ends as fraud at 0.95, card blocked, and no report:\n\n$99.92 on one card with a documented pattern meets none of §3a's conditions.\n\n**HHG-018 is a subscription the cardholder forgot.** The disputed $39.08 charge\n\nappears on the card 21 times since July, every one to three weeks. R7 exists for\n\nexactly this. The first fix — \"four identical charges over six weeks is a habit\"\n\n— was wrong, and the October backtest said so: it fired on 15 disputes, 14 of\n\nthem confirmed fraud, because a fraud episode repeats an amount too, in a burst\n\n(one had 21 charges 2.8 days apart). The rule that survives needs a quarter of\n\nsteady history: at least eight charges over 90 days or more, a median gap of 5\n\nto 35 days. A burst can't fake that; HHG-018 clears it easily. The simulated\n\nreply follows R7's own premise — the cardholder recognises the charge — and the\n\nanswer file says so.\n\nThe lesson is the same one as the calibration trap: measure the change against\n\ndata it could be wrong on, not only the case it was written for.\n\n*Every graph call goes through the official TigerGraph MCP server first, with a per-call fallback.*\n\nThe agent never holds a database connection of its own. It starts the official\n\n`tigergraph-mcp` server over stdio and calls `tigergraph__run_installed_query`,\n\n`get_node`, `add_node` and friends — 1,492 calls on the last full run, zero\n\nfallbacks. A gateway falls back to a direct connection per call, so a flaky\n\ntool call degrades one read, not a case; `GRAPH_STRICT=1` turns that into a hard\n\nfailure when you'd rather know.\n\nTigerGraph's GDS library earns its place once you point it at the right\n\nsubgraph. WCC over the full card co-occurrence network returns one 6,125-card\n\nblob, held together by popular browser strings. Over the anonymous-proxy slice\n\n(1,725 edges) it isolates the HHG-014 ring as a 54-card component, with the\n\ncase's card second by PageRank. That result is all-time structure, so it is\n\ncited next to the time-bounded evidence and never weighed.\n\n`card_id` isn't in the data.`transactions.csv` has no card column, but the\n\ncase pack and the closed cases are expressed entirely as `C01234-K1`. A card is\n\nthe `(customer_id, card2..card6)` tuple — but the `K` index isn't derivable from\n\nit, and every obvious ordering tops out near 50% agreement with the labels. The\n\nanswer was to stop guessing and *pin* it from the 14,975 transactions the\n\ndataset itself labels. 100% agreement, asserted by a verification step. Get this\n\nwrong and every card-scoped query silently breaks on half the cases.\n\n**Reported probability is capped at 0.97.** `fraud_probability` is explicitly\n\nscored for calibration, and reporting 1.00 isn't confidence — it's a missing\n\nerror bar. The ceiling sits above the 0.85 action threshold, so it never changes\n\nan action.\n\n**Every decision is hash-chained.** Edit a recorded rationale after the fact and\n\n`verify()` tells you which link broke.\n\n**A write isn't a write until it reads back.** After writing the case subgraph\n\nthe agent walks it with `case_chain` and checks the transactions, connected cards\n\nand SAR landed. `written_to_graph` means read back, not attempted.\n\n**The model may only cite what it was given.** Any LLM-written sentence that\n\nnames a case, card or transaction absent from its brief is discarded for the\n\ntemplate. Every evidence item says whether it is a graph fact, an inference, a\n\nmodel score, policy text or a simulated reply.\n\n*11 legitimate, 9 fraud, 2 SARs, 5 decisions changed after asking one question.*\n\n*Measured on July–September, tested on an October holdout the calibration never saw.*\n\nThe other diagrams (schema, case lifecycle, policy rules, console, monitoring, failure modes) are on the [interactive page](https://claude.ai/artifact/VioVT79phoeLTjBKkMWRvb).\n\n`as_of` would let\nthem count as evidence.\n\n```\npython -m pip install -r requirements.txt\nmake ingest\npython benchmark/run_20.py\npython eval/audit_answers.py\npython ui/server.py\n```\n\nNo API key, no database, no build step. The model writes the prose; the graph\n\nmakes the decisions.", "url": "https://wpnews.pro/news/the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through", "canonical_source": "https://dev.to/prayant_mohanty_77d03552f/the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through-onc", "published_at": "2026-09-25 04:59:42+00:00", "updated_at": "2026-09-25 05:28:57.602304+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "mlops", "artificial-intelligence"], "entities": ["TigerGraph", "LangGraph", "MCP", "IEEE-CIS", "Hacker House Goa"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through", "markdown": "https://wpnews.pro/news/the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through.md", "text": "https://wpnews.pro/news/the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through.txt", "jsonld": "https://wpnews.pro/news/the-calibration-bug-that-taught-our-fraud-agent-to-wave-fraud-through.jsonld"}}