{"slug": "fhir-validation-doesn-t-tell-you-what-to-do-next-so-i-built-a-decision-layer", "title": "FHIR Validation Doesn't Tell You What to Do Next, So I Built a Decision Layer", "summary": "A developer built a FHIR decision layer that sits on top of standard R4 conformance validation, sending flattened resource state to TypeSafe AI's Jev decision model to return accept, route, flag or human-review actions. On 204 synthetic records, the first live run underperformed hand-written rules on routing (0.814 versus 0.922) and missed notifiable diseases, both traced to a serializer bug and a poorly framed routing question; after fixes, Jev beat the rules on all three decisions on the same records, though it still misses the latency target by a wide margin. The proof of concept runs only on synthetic data and is not connected to any production system.", "body_md": "A FHIR resource can pass validation and still be the wrong thing to act on. I built a small decision layer for that gap: it takes parsed FHIR R4 resources, sends a flattened state to TypeSafe AI's Jev model, and turns the answer into accept, route, flag or human review.\n\nOn 204 synthetic records, the first live run lost routing to my hand-written rules, 0.814 to 0.922, and missed too many notifiable diseases. Both failures came from my own setup. One was a serializer bug and the other was a badly framed routing question. After fixing both, Jev scored above the rules on all three decisions, on the same records. It still misses my latency target by a wide margin.\n\nThis is a proof of concept. It runs on synthetic data and it is not connected to any production system.\n\nHere is part of a Patient that parses as FHIR R4:\n\n```\n{\n  \"resourceType\": \"Patient\",\n  \"id\": \"invalid-nik\",\n  \"identifier\": [{\"system\": \"nik\", \"value\": \"317301010190000\"}],\n  \"name\": [{\"family\": \"Wijaya\"}]\n}\n```\n\nThe NIK is Indonesia's national ID, and it has 16 digits. This one has 15. A Condition can have the same problem in a different form: well-formed, coded as dengue, with `verificationStatus` set to `refuted`. The patient does not have dengue, and the resource is still structurally correct.\n\nConformance checking answers one question: does this resource meet the spec? A pipeline has to answer a different one: what should happen to this resource now?\n\n|  | Conformance | Decision layer | \n|---|---|---|\n| Question | Does it meet the spec? | What do we do with it? | \n| Output | Pass or fail, with issues | An action and a confidence | \n| Defined by | Spec and profiles | Workflow and risk tolerance | \n| When unsure | Does not apply | Sends it to a person | \n\nProfiles can cover more of this than people expect. A FHIRPath invariant could enforce a 16-digit NIK, so that example does not need a model. The boundary I care about is not rules versus AI. It is *is this correct?* versus *what happens next?*\n\nJev is a decision model. It takes a structured state and a fixed question, and returns a typed answer with a confidence. It does not generate text. The project uses its three primitives: **Choice** picks one option from a list, **Score** returns a number on a scale, and **Noul** answers a statement with a probability that it is true.\n\nThe service is FastAPI with Pydantic v2, and it makes three decisions:\n\n| Decision | Primitive | Output | \n|---|---|---|\n| Is this resource good enough to accept automatically? | Score, plus Noul for the NIK | Score 0 to 100, then accept or review | \n| Which pipeline should get this Bundle? | Choice | One of five fixed categories | \n| Must this Condition be reported? | Noul | Probability, plus a Flag when confirmed | \n\nEvery serializer parses the resource before it extracts anything. A resource that fails parsing raises an error, so no Jev call happens. This is lighter than full profile and terminology validation, but the order is the design: conformance first, decisions on top.\n\n``` python\n# src/jev_fhir/serializer/patient.py\ndef serialize(self, resource: dict[str, Any]) -> dict[str, Any]:\n    \"\"\"Validate a Patient and produce a missing-field-safe flat state.\"\"\"\n    Patient.parse_obj(resource)\n\n    identifier = first_mapping(resource.get(\"identifier\"))\n    identifier_value = identifier.get(\"value\") if identifier is not None else None\n\n    return {\n        \"resource_type\": self.resource_type(),\n        \"has_identifier\": identifier is not None,\n        \"identifier_value_length\": len(identifier_value)\n        if isinstance(identifier_value, str)\n        else None,\n        # remaining demographic fields omitted\n    }\n```\n\nThe model never receives the NIK itself, only its length. This is the state produced from the full version of that Patient:\n\n```\n{\n  \"resource_type\": \"Patient\",\n  \"has_identifier\": true,\n  \"identifier_system\": \"nik\",\n  \"identifier_value_length\": 15,\n  \"has_name\": true,\n  \"name_family\": \"Wijaya\",\n  \"has_birth_date\": true,\n  \"has_gender\": true,\n  \"has_address\": true,\n  \"address_country\": \"ID\",\n  \"has_telecom\": false,\n  \"telecom_count\": 0,\n  \"field_completeness\": 8,\n  \"field_total\": 10\n}\n```\n\nA small state keeps each call cheap and easy to inspect. The demo UI shows this exact state next to every decision, which made debugging much easier later. It is not a privacy solution, though. `name_family` is still in there, and real data would need a proper review.\n\nThresholds turn a model output into something the pipeline can do. The router has a confidence floor, and anything below it becomes `unknown`:\n\n```\n# src/jev_fhir/modules/bundle_router.py\nstate = self._serializer.serialize(bundle)\nresult = await self._jev_client.choice(state, ROUTE_QUESTION, ROUTE_OPTIONS)\ncategory = result.choice if result.confidence >= confidence_minimum else \"unknown\"\n```\n\nThe quality scorer treats the NIK as a gate. A high completeness score cannot override it:\n\n```\n# src/jev_fhir/modules/quality_scorer.py\npasses_nik_gate = nik_valid is not False\naction = (\n    \"auto_accept\"\n    if score_result.score >= threshold and passes_nik_gate\n    else \"review_needed\"\n)\n```\n\nNotifiable detection uses two thresholds, which gives it a band for human review:\n\n```\n# src/jev_fhir/modules/notifiable_detector.py\nif result.probability >= confirmed_threshold and result.answer:\n    status = \"confirmed_notifiable\"\n    is_notifiable = True\n    # builds the FHIR Flag here\nelif result.probability >= review_threshold:\n    status = \"review_needed\"\n    is_notifiable = False\n    flag_resource = None\nelse:\n    status = \"not_notifiable\"\n    is_notifiable = False\n    flag_resource = None\n```\n\nThe review band is the part conformance has no equivalent for. The system can say it is not sure, and that outcome has a defined place to go.\n\nI compared Jev with a transparent rule baseline for each decision. The rules are deliberately simple: a field-count score, a length check for the NIK, routing on resource types and LOINC signals, and an exact ICD-10 lookup.\n\nThe dataset has 204 labelled fixtures, including a hard set built to separate semantic judgment from exact matching. Labels are never computed from the logic under test. The test suite uses a mock Jev client and never calls the live API, and the benchmark runs against the live model only when I ask it to.\n\n| Decision | Fixtures | Jev, first run | Jev, final | Rules | My target | \n|---|---|---|---|---|---|\n| Quality (accuracy) | 71 | 0.676 | **0.662** | 0.549 | ≥ 0.85, missed | \n| Routing (accuracy) | 102 | 0.814 | **0.951** | 0.922 | ≥ 0.90, met | \n| Notifiable (accuracy) | 31 | 0.839 | **0.935** | 0.677 | none set | \n| Notifiable (recall) | 31 | 0.737 | **0.895** | 0.579 | ≥ 0.95, missed | \n| Notifiable (precision) | 31 | 1.000 | **1.000** | 0.846 | ≥ 0.80, met | \n\nThe rest of this article explains how the first-run column became the final one.\n\nMy first live run scored 0.839 on notifiable detection, with a recall of 0.737. The score alone gave no hint of why.\n\nThe cause showed up while I was reviewing the demo. The state panel for a text-only dengue case showed `code_value`, `code_display` and `code_system` all null, with no disease name anywhere. My serializer read `code.coding` and never read `code.text`, so the model was judging a blank. Three of the five misses had the disease name only in `code.text`, such as \"Demam berdarah dengue (DBD)\" and \"TB paru\".\n\nThe fix is small:\n\n```\n# src/jev_fhir/serializer/condition.py\n+CODE_TEXT_MAX_LENGTH = 200\n\n         code = resource.get(\"code\")\n+        code_text = code.get(\"text\") if isinstance(code, dict) else None\n         onset = resource.get(\"onsetDateTime\")\n\n             \"code_display\": first_coding_value(code, \"display\"),\n+            \"code_text\": code_text[:CODE_TEXT_MAX_LENGTH] if isinstance(code_text, str) else None,\n             \"clinical_status\": first_coding_code(resource.get(\"clinicalStatus\")),\n```\n\nThe 200-character limit keeps a long note from inflating token use and limits how much free text leaves the system. I kept the rule baseline code-only, so the comparison stayed fair. I also kept the mock code-only, because teaching it to read text would make mock results look like model evidence.\n\nI then compared the two live reports row by row. Exactly three notifiable rows changed, and they were the three I expected:\n\n| Fixture | Confidence before | Confidence after | Result | \n|---|---|---|---|\n| `dbd_text_only` | 0.47 | 0.90 | wrong → correct | \n| `tb_paru_text_only` | 0.50 | 0.89 | wrong → correct | \n| `condition_dbd_text_only` | 0.49 | 0.89 | wrong → correct | \n\nAccuracy went from 0.839 to 0.935, and recall went from 0.737 to 0.895. The earlier score was measuring my pipeline, not the model. Without the row-level comparison, I could not have told that apart from noise, because quality accuracy moved from 0.676 to 0.648 between the same two runs on rows the fix does not touch.\n\nThe two remaining misses, `B20` HIV and typhoid `A01.0`, have proper codes. Both landed in the review band at about 0.77, so a person would see them. All 19 notifiable cases were either confirmed or sent to review, and none was dismissed.\n\nMy first explanation for the routing loss was that routing is mostly clear logic. I had written that down, but I had not measured it. So I compared the flat state of every Bundle Jev got right and wrong.\n\nAll 12 of its encounter errors followed one pattern:\n\n| Conditions in the Bundle | Bundles | Jev correct | \n|---|---|---|\n| 1 | 6 | 6 | \n| 2 | 2 | 1 | \n| 3 or more | 11 | 0 | \n\nBoth live runs showed this, and 100 of the 102 routing decisions were identical across the runs. This was a systematic failure, not noise.\n\nTwo things in my setup could explain it. The question was \"What type of clinical data does this FHIR Bundle primarily contain?\". One Encounter with eight Conditions primarily contains Conditions, and no category exists for that, so `unknown` was a fair answer. The same word would also explain the other errors, because all seven misrouted mixed Bundles went to the pipeline for their most common resource type.\n\nThe state was also uneven. It had direct flags for labs, immunizations and medications, which scored at 1.00 confidence, but nothing direct for encounters. `dominant_resource_type` said `Condition`.\n\nI changed both:\n\n```\n# src/jev_fhir/modules/bundle_router.py\n-ROUTE_QUESTION = \"What type of clinical data does this FHIR Bundle primarily contain?\"\n+ROUTE_QUESTION = \"Which processing pipeline should handle this FHIR Bundle?\"\n\n# src/jev_fhir/serializer/bundle.py\n             \"contains_medication\": \"MedicationDispense\" in resource_types,\n+            \"contains_encounter\": \"Encounter\" in resource_types,\n```\n\nTo see which change mattered, I ran the flag on its own first:\n\n| Live routing run | Accuracy | 3+ Condition Bundles correct | \n|---|---|---|\n| Original | 0.814 | 0 of 11 | \n| Flag only | 0.863 | 4 of 11 | \n| Flag and new question | 0.951 | 11 of 11 | \n\nFourteen rows improved and none got worse. The question did most of the work, which I did not expect. It was one sentence, and it had been costing more than the missing data.\n\nThe rule baseline stayed the same, and a test confirms it does not use the new field. In the final run Jev was right on three Bundles the rules got wrong, and the reverse never happened. That margin is small, so I would not read it as a clear win. The rules scored 0.922 with no model at all.\n\nFive mixed Bundles are still wrong, and three of those are at 0.87 confidence or higher, so the floor does not catch them. One of them, `vitals_only`, has a label I have not settled. The rules' 0.95 on generated Bundles also needs care: the rule checks `has_encounter_reference`, and my generator links every generated encounter Bundle that way, so the two share a signal.\n\nJev did best where the decision depends on more than one field. On the 14 hard Condition fixtures it scored 0.929 against 0.357 for the rules. The rules reported a refuted dengue case and a cholera record marked `entered-in-error` as real. Jev confirmed neither, and both went to review. It also caught dengue and tuberculosis coded only in SNOMED, which an ICD-10 lookup cannot see.\n\nSome of that gap is cheap to close. Prefix matching would handle subcodes like `A15.0`, and checking `verificationStatus` would handle the refuted case. A serious rule set would do both.\n\nFor routing, the rules remain the sensible default. They are simple and free, and a model can act as a second opinion, with disagreements going to a person.\n\nQuality scoring is the weakest result. Of 45 records that needed review, Jev let 14 through and the rules let 32 through. But Jev also held 10 of 26 good records, and the rules held none.\n\nA threshold is only useful if confidence means something, so the benchmark buckets decisions by confidence and compares that with observed accuracy. This is the top bucket:\n\n| Decision | Count | Mean confidence | Observed accuracy | \n|---|---|---|---|\n| Notifiable | 17 | 0.844 | 1.000 | \n| Routing | 84 | 0.982 | 0.964 | \n| Quality | 24 | 0.825 | 0.667 | \n\nNotifiable looks trustworthy here, and routing improved after the question change, from 0.918 in this bucket. Quality does not: a confidence of 0.82 was right about two times in three, so I would not automate anything on that threshold yet.\n\n**Latency.** My target was a p50 under 30 ms. The live p50 was about 235 ms for routing and notifiable, and 458 ms for quality, where a Patient takes two calls. The p95 ranged from 263 ms to 510 ms. These are client-side round trips from my machine, so they include the network, and I have not separated model time. The fast-decision premise is not proven by this data.\n\n**Cost target.** The full run used 147,401 tokens for about $0.006. That is about $0.03 per 1,000 records, three times my $0.01 target. It is still very cheap.\n\n**Evaluation quality.** No human has approved the labels yet. 124 are pending review, and the other 80 are correct by construction because a generator produced them. The hard sets are small. I also made both fixes after seeing these same fixtures fail, so 0.935 and 0.951 are not held-out results. They need confirming on data the system has not seen.\n\nThe first step is human review of the labels, because every number here depends on them. After that I want a held-out set for the notifiable and routing decisions, and a latency measurement that separates the network from the model. Two gaps from this work are still open: `ObservationSerializer` drops `code.text` the same way the Condition serializer did, and all the serializers read only the first `coding`.\n\nThe result I would keep is the shape of the system, more than any single score: parse first, flatten to a state a person can inspect, use a threshold per decision, give uncertain cases a lane, and keep a baseline that can beat the model. That baseline did beat it, and that is why I went looking. Before blaming a model, I now check what it actually received and what I actually asked.\n\n*I work on clinical data systems at a research unit, mostly FHIR R4, HL7 v2, and HAPI FHIR. This is a proof of concept for a typed decision layer over FHIR records: [https://github.com/budityw23/fhir_jev](https://github.com/budityw23/fhir_jev).*", "url": "https://wpnews.pro/news/fhir-validation-doesn-t-tell-you-what-to-do-next-so-i-built-a-decision-layer", "canonical_source": "https://dev.to/budiwidhiyanto/fhir-validation-doesnt-tell-you-what-to-do-next-so-i-built-a-decision-layer-1gff", "published_at": "2026-11-01 07:03:10+00:00", "updated_at": "2026-11-01 07:22:37.454818+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "developer-tools"], "entities": ["TypeSafe AI", "Jev", "FastAPI", "Pydantic", "FHIR"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fhir-validation-doesn-t-tell-you-what-to-do-next-so-i-built-a-decision-layer", "markdown": "https://wpnews.pro/news/fhir-validation-doesn-t-tell-you-what-to-do-next-so-i-built-a-decision-layer.md", "text": "https://wpnews.pro/news/fhir-validation-doesn-t-tell-you-what-to-do-next-so-i-built-a-decision-layer.txt", "jsonld": "https://wpnews.pro/news/fhir-validation-doesn-t-tell-you-what-to-do-next-so-i-built-a-decision-layer.jsonld"}}