The calibration bug that taught our fraud agent to wave fraud through

A developer built an agentic fraud investigation system on TigerGraph for the Hacker House Goa challenge, using a LangGraph agent that queries the graph through an MCP server while keeping all regulator-facing decisions in an LLM-free core package enforced by an AST test. Working from 590,742 IEEE-CIS card transactions and 5,565 closed investigations, the agent produces provisional actions and approval routes before requesting more evidence, then revises them, and runs end to end without an API key.

Building an agentic fraud investigator on TigerGraph — and the two measurement mistakes that silently inverted it. All 18 architecture diagrams, interactive: FraudGraph Blueprints https://claude.ai/artifact/VioVT79phoeLTjBKkMWRvb : system, agent loop, MCP path, schema, GraphRAG, policy rules, all twenty benchmark cases, evaluation and failure modes. Three triggers, one LangGraph agent, TigerGraph reached through the MCP server, and a deterministic core the LLM cannot reach. The one conditional edge that makes it an agent: gather more evidence, or act. TigerGraph's Hacker House Goa challenge hands you six months of card transactions — 590,742 of them, from the IEEE-CIS dataset — and takes away the one column everyone reaches for. There is no isFraud flag. Every transaction carries a risk score from the bank's own model instead, and the dataset README is blunt about what that score is worth: Above 0.7, most flagged transactions turn out to be legitimate. Some fraud scores near zero. What you get instead of labels is four months of closed investigations — 5,565 of them, confirmed fraud and cleared false alarms, with the analyst's notes. And twenty new alerts to decide. So the task isn't classification. It's investigation: work out what kind of fraud this is, how far it goes, what to do about it, and when you know enough to act . That last clause is the whole thing. Buried in the answer format is a requirement that quietly determines the architecture: The next best action and required approval route recorded: before any additional evidence is requested, after any additional evidence is received. You cannot produce that pair honestly from a single-shot pipeline. There is no "before" unless the agent genuinely commits to a recommendation under uncertainty, then decides what it needs, then revises. So the topology is: retrieve → believe → is this defensible? │ no ▼ PROVISIONAL action + approval route │ pick the question worth asking │ response │ ▼ REVISED action + what changed Everything else follows from wanting that loop to be honest. The model sits outside the decision boundary; people sit on the approval boundary. Every decision a bank would have to justify to a regulator lives in a core/ package that cannot import an LLM . Not "does not" — cannot. A test walks the AST of every file in that directory and fails the build if an import of anthropic , openai , langchain or anything with llm in the name ever appears: python def test core cannot import an llm : banned = {"anthropic", "openai", "google", "langchain", "langgraph", ...} for path in ROOT / "src" / "fraudagent" / "core" .glob " .py" : tree = ast.parse path.read text ... assert not offenders, f"core/ must stay LLM-free: {offenders}" That covers the Bayesian ledger, the policy engine, the approval routing, the stopping rule and the episode reconstruction. The model plans which graph primitives to run, synthesises evidence into prose, writes the case summary and the SAR narrative. If it proposes an action anyway, the policy decision object drops it before it can reach an answer file. The practical consequence, which I like more than the principle: the agent runs end to end with no API key at all . Verdicts, probabilities, actions and approval routes are bit-identical; only the wording changes. Anyone can clone the repo and reproduce the twenty answer files without buying anything. Similar past cases found by meaning and by shared entities inside one GSQL query, then fused. The obvious answer is "traversal", and that's true — but the reason this project belongs on a graph database with vectors in the same store is a query that needs both at once. Case memory is the requirement: prior closed cases must measurably change what the agent concludes. The naive implementations both fail in the same characteristic way: Neither is what an analyst means by "have we seen this before". So fuse them, in one GSQL statement, inside the database: candidates = vectorSearch {ClosedCase.narrative emb}, query vec, k 8 ; candidates = SELECT c FROM candidates:c WHERE c.opened at < as of ACCUM @@vec score += c - c.@distance ; linked d = SELECT c FROM seed dev:v - CC DEVICE OF:e - ClosedCase:c WHERE c.opened at < as of ACCUM @@shared += c - 1 ; result = SELECT c FROM candidates UNION linked t UNION linked c UNION linked d :c ACCUM c.@score += 0.55 @@vec score.get c + 0.45 1.0 - exp -0.6 @@shared.get c ORDER BY c.@score DESC LIMIT k; Two details do real work. The saturating term on the graph leg means the first shared entity is worth a lot and the fourth almost nothing — with a linear term, one busy device profile dominates every ranking. And the as of guard on both legs: a memory system that can retrieve a case opened after the one it's reasoning about isn't a memory system, it's a leak, and every number downstream of it is fiction. There's a second embedding space too, and it's the one I'd defend hardest. V1 – V339 are Vesta's engineered features: real signal, no names, and utterly hopeless as LLM context. 339 unnamed columns isn't evidence, it's noise with a schema. Compressed by incremental PCA to 24 components and concatenated with rank-normalised count and time-delta columns, they become a 64-dimensional behaviour vector stored on every Transaction . Now "which confirmed-fraud Here's the part I'd want to read in someone else's write-up. The plan was to measure likelihood ratios instead of guessing them: replay all 5,565 closed investigations through the same extractor the agent uses, under each case's own as of , and count. Real numbers from real outcomes. The first run produced this: | signal | P s \| fraud | P s \| cleared | LR | |---|---|---|---| | known device and region | 0.610 | 0.188 | 3.30 | | amount typical | 0.394 | 0.150 | 2.63 | | device new on account | 0.134 | 0.484 | 0.28 | | amount anomalous | 0.078 | 0.125 | 0.62 | Read that carefully. It says a transaction on a known device, in a known region, for a typical amount is over three times more likely to be fraud. And Which is, taken at face value, insane. And the agent believed it completely — 93% held-out accuracy, beautiful Brier score, and on the benchmark it confidently called quiet, ordinary transactions fraud while clearing the obvious ones. The bug isn't in the modelling. It's in what the closed-case file is . Every cleared case in that history got there by scoring high on the bank's model. It's a travel trip, a new phone, an unusual but genuine purchase — an investigation that happened because the transaction looked strange. Every confirmed case got there because a cardholder phoned up about a charge they didn't recognise, and those are frequently small and unremarkable. So inside the closed-case file, "looks anomalous" genuinely does predict cleared . The measurement was correct. The population was wrong. The fix is a third stratum the file doesn't contain: ordinary transactions from the same four months that no investigation ever touched. Because months Adding untouched negatives helped, and did not fix it. known device and region came down to LR 1.56 — still pointing the wrong way. The remaining confound is the card, not the transaction. Confirmed-fraud cases sit disproportionately on long-lived, busy cards : more history, therefore more established devices and regions, therefore "the device and region are already established" correlates with fraud through a variable that has nothing to do with fraud. The fix is a matched control design . For each confirmed-fraud case, draw another transaction on the same card as its control. The card-level confound cancels exactly, and the question the calibration answers becomes the question an analyst actually asks: What is different about this transaction compared with the rest of this card's behaviour? With both corrections in, held-out fraud-versus-not accuracy went from 0.93 to 0.57 , and most of the likelihood ratios collapsed to about 1. My first instinct was that I'd broken something. I hadn't. That number is the data answering the question I'd finally asked correctly. Every confirmed case in this history was found by a cardholder ringing up. Fraud never entered that file because it looked anomalous — it entered because someone noticed a charge. So once you remove the trigger as a feature, which you must because it is perfectly confounded with the outcome, the graph signals genuinely cannot separate confirmed fraud from ordinary activity on the same card. Most fraud in this dataset is small and unremarkable. That is the point. The 0.93 was the artefact. The 0.57 is the finding. What the history can teach is which typology a fraud is, and it teaches that strongly: | signal | CNP-new-device | out-of-region | account takeover | |---|---|---|---| | device new on account | 0.85 | 0.001 | 0.07 | | match flag anomaly | 0.001 | 0.56 | 0.47 | | online burst 2 to 4 | 0.43 | 0.001 | 0.04 | | known device and region | 0.14 | 0.86 | 0.75 | So the architecture follows the evidence rather than the other way round: fraud-versus-not is carried by the trigger prior and by the structural typology matchers; which typology is carried by the measured table. I'd rather ship that with the 0.57 written on the tin than ship the 0.93 and let the reliability curve look excellent right up until the agent clears a real fraud for being unremarkable. Three lessons I'd take anywhere, not just here: Related, and worth thirty seconds. If you measure "is this device new to the card" against the card's whole history up to the alert, the earlier legs of the same fraud episode are in that history. The first fraudulent transaction So anomaly is measured as suspect window against established baseline , with the baseline cut off 72 hours before the alert. Seventy-five percent of closed-case episodes open within 29 hours of their alert, so the window catches the tail without swallowing ordinary history. The dataset drops a deliberate hint: Not every fraud pattern present in the data is documented. Five typologies are described. Nine closed cases are marked undocumented — confirmed fraud the bank's own analysts could not categorise. Their narratives split cleanly into two signatures, and both turn up in the graded twenty. U1 — sub-threshold authorisation structuring. Four online purchases inside roughly forty minutes, each amount parked just beneath a $500 authorisation ceiling, about $1,900 in total. It's invisible to per-transaction scoring because every individual leg looks completely ordinary . It only exists as a property of the burst. HHG-006 in the benchmark is one: $478.95, $456.96, $488.04, $482.12 in thirty minutes, $1,906.07 total — against five closed cases between $1,871 and $1,922. U2 — a shared-device ring behind an anonymising proxy. One device profile across a run of unrelated cardholders in a single month, marked new on every account it touches. HHG-014 is one of these; the analyst request that triggers it even says so. The detection subtlety on U2 is the bit I'd put on a slide. A "device profile" in this dataset is a DeviceInfo | OS | browser | screen string — a model, not a serial number — and 92% of online transactions sit on a profile shared by three or more cards . Counting cards per device finds nothing but noise, and a What separates a real hub is the anonymous-proxy ratio : | device profile | cards | via anonymous proxy | ratio | |---|---|---|---| | SM-G935F \ | Android 7.0 \ | chrome 62.0 \ | 1920x1080 | | Windows \ | Windows 10 \ | chrome 65.0 \ | 1920x1080 | | Trident/7.0 \ | Windows 10 \ | ie 11.0 \ | 1920x1080 | Every card that ever touched the ring device arrived anonymously. That is not a household sharing a tablet. Evidence becomes numbers through measured likelihood ratios; numbers become actions only through the written policy. When belief won't support a defensible action, the agent has to pick one evidence action. Picking the strongest is wrong; picking the best information per unit of friction is right — and friction is not money. Pinging a cardholder at two in the morning has a real cost even when the API call is free. Each candidate is scored by expected reduction in Shannon entropy over the hypothesis space, divided by its friction cost, and filtered through the policy first — because the policy constrains asking , not only acting. That lets the agent produce a sentence I'm fond of: The highest-value next evidence is customer validation, but the contact-fatigue limit forbids a third contact this week, so I am escalating to an analyst instead. That single sentence demonstrates uncertainty handling, policy compliance, next best action and explainability at once. And the stopping rule is two-sided. Anyone can stop when confident. The second condition is the one that matters: No permitted evidence action carries enough information per unit of customer friction to justify it. Further investigation would not change the decision, so the case goes to a human with what is known. That's a different and more defensible claim than "I'm sure", and it's what a real fraud desk does. HHG-018: a disputed charge the card has paid 21 times. Asked first, recognised, closed without a block. Rings you can't see from one card: gated so a popular browser string never becomes a false ring. After the first complete build I read ten other teams' write-ups for the same brief. Two disagreements with my answers were worth checking against the data. HHG-019 is a ring you can't see from one card. It is a risk-model alert, and this history's risk-model alerts are almost always false alarms 900 of 900 cleared . But ask what happened on other cards: the flagged purchase's device — a profile only three cards had ever used — bought $100.00 and $100.06 on two other cards that week. peer purchases.gsql is that two-hop question. The gate matters more than the query: on a generic browser string "chrome 66.0", 175 cards ordinary shoppers with similar baskets would make a false ring, so it only fires on a rare device. Its weight is measured, not chosen. It fired on 31 confirmed-fraud closed cases and no cleared ones, a likelihood ratio of 22.4, the strongest in the model. My first version of it made two mistakes, and the October holdout caught both. It treated the peer cards as connected cards , which switched on R6 and the filing rule — and filed 19 reports on holdout cases the bank itself never reported. And it folded in "did not fire" on every other case, which nudged each one away from card-not-present fraud and cost nineteen correct pattern calls. Both fixes are about what the evidence actually proves. Peers move the probability; they don't prove a shared origin, so R6 and the report still need one. And the absence of a rare hub is not a finding, so the signal is fired-only. HHG-019 now ends as fraud at 0.95, card blocked, and no report: $99.92 on one card with a documented pattern meets none of §3a's conditions. HHG-018 is a subscription the cardholder forgot. The disputed $39.08 charge appears on the card 21 times since July, every one to three weeks. R7 exists for exactly this. The first fix — "four identical charges over six weeks is a habit" — was wrong, and the October backtest said so: it fired on 15 disputes, 14 of them confirmed fraud, because a fraud episode repeats an amount too, in a burst one had 21 charges 2.8 days apart . The rule that survives needs a quarter of steady history: at least eight charges over 90 days or more, a median gap of 5 to 35 days. A burst can't fake that; HHG-018 clears it easily. The simulated reply follows R7's own premise — the cardholder recognises the charge — and the answer file says so. The lesson is the same one as the calibration trap: measure the change against data it could be wrong on, not only the case it was written for. Every graph call goes through the official TigerGraph MCP server first, with a per-call fallback. The agent never holds a database connection of its own. It starts the official tigergraph-mcp server over stdio and calls tigergraph run installed query , get node , add node and friends — 1,492 calls on the last full run, zero fallbacks. A gateway falls back to a direct connection per call, so a flaky tool call degrades one read, not a case; GRAPH STRICT=1 turns that into a hard failure when you'd rather know. TigerGraph's GDS library earns its place once you point it at the right subgraph. WCC over the full card co-occurrence network returns one 6,125-card blob, held together by popular browser strings. Over the anonymous-proxy slice 1,725 edges it isolates the HHG-014 ring as a 54-card component, with the case's card second by PageRank. That result is all-time structure, so it is cited next to the time-bounded evidence and never weighed. card id isn't in the data. transactions.csv has no card column, but the case pack and the closed cases are expressed entirely as C01234-K1 . A card is the customer id, card2..card6 tuple — but the K index isn't derivable from it, and every obvious ordering tops out near 50% agreement with the labels. The answer was to stop guessing and pin it from the 14,975 transactions the dataset itself labels. 100% agreement, asserted by a verification step. Get this wrong and every card-scoped query silently breaks on half the cases. Reported probability is capped at 0.97. fraud probability is explicitly scored for calibration, and reporting 1.00 isn't confidence — it's a missing error bar. The ceiling sits above the 0.85 action threshold, so it never changes an action. Every decision is hash-chained. Edit a recorded rationale after the fact and verify tells you which link broke. A write isn't a write until it reads back. After writing the case subgraph the agent walks it with case chain and checks the transactions, connected cards and SAR landed. written to graph means read back, not attempted. The model may only cite what it was given. Any LLM-written sentence that names a case, card or transaction absent from its brief is discarded for the template. Every evidence item says whether it is a graph fact, an inference, a model score, policy text or a simulated reply. 11 legitimate, 9 fraud, 2 SARs, 5 decisions changed after asking one question. Measured on July–September, tested on an October holdout the calibration never saw. The other diagrams schema, case lifecycle, policy rules, console, monitoring, failure modes are on the interactive page https://claude.ai/artifact/VioVT79phoeLTjBKkMWRvb . as of would let them count as evidence. python -m pip install -r requirements.txt make ingest python benchmark/run 20.py python eval/audit answers.py python ui/server.py No API key, no database, no build step. The model writes the prose; the graph makes the decisions.