FHIR Validation Doesn't Tell You What to Do Next, So I Built a Decision Layer

A developer built a FHIR decision layer that sits on top of standard R4 conformance validation, sending flattened resource state to TypeSafe AI's Jev decision model to return accept, route, flag or human-review actions. On 204 synthetic records, the first live run underperformed hand-written rules on routing (0.814 versus 0.922) and missed notifiable diseases, both traced to a serializer bug and a poorly framed routing question; after fixes, Jev beat the rules on all three decisions on the same records, though it still misses the latency target by a wide margin. The proof of concept runs only on synthetic data and is not connected to any production system.

A FHIR resource can pass validation and still be the wrong thing to act on. I built a small decision layer for that gap: it takes parsed FHIR R4 resources, sends a flattened state to TypeSafe AI's Jev model, and turns the answer into accept, route, flag or human review. On 204 synthetic records, the first live run lost routing to my hand-written rules, 0.814 to 0.922, and missed too many notifiable diseases. Both failures came from my own setup. One was a serializer bug and the other was a badly framed routing question. After fixing both, Jev scored above the rules on all three decisions, on the same records. It still misses my latency target by a wide margin. This is a proof of concept. It runs on synthetic data and it is not connected to any production system. Here is part of a Patient that parses as FHIR R4: { "resourceType": "Patient", "id": "invalid-nik", "identifier": {"system": "nik", "value": "317301010190000"} , "name": {"family": "Wijaya"} } The NIK is Indonesia's national ID, and it has 16 digits. This one has 15. A Condition can have the same problem in a different form: well-formed, coded as dengue, with verificationStatus set to refuted . The patient does not have dengue, and the resource is still structurally correct. Conformance checking answers one question: does this resource meet the spec? A pipeline has to answer a different one: what should happen to this resource now? | | Conformance | Decision layer | |---|---|---| | Question | Does it meet the spec? | What do we do with it? | | Output | Pass or fail, with issues | An action and a confidence | | Defined by | Spec and profiles | Workflow and risk tolerance | | When unsure | Does not apply | Sends it to a person | Profiles can cover more of this than people expect. A FHIRPath invariant could enforce a 16-digit NIK, so that example does not need a model. The boundary I care about is not rules versus AI. It is is this correct? versus what happens next? Jev is a decision model. It takes a structured state and a fixed question, and returns a typed answer with a confidence. It does not generate text. The project uses its three primitives: Choice picks one option from a list, Score returns a number on a scale, and Noul answers a statement with a probability that it is true. The service is FastAPI with Pydantic v2, and it makes three decisions: | Decision | Primitive | Output | |---|---|---| | Is this resource good enough to accept automatically? | Score, plus Noul for the NIK | Score 0 to 100, then accept or review | | Which pipeline should get this Bundle? | Choice | One of five fixed categories | | Must this Condition be reported? | Noul | Probability, plus a Flag when confirmed | Every serializer parses the resource before it extracts anything. A resource that fails parsing raises an error, so no Jev call happens. This is lighter than full profile and terminology validation, but the order is the design: conformance first, decisions on top. python src/jev fhir/serializer/patient.py def serialize self, resource: dict str, Any - dict str, Any : """Validate a Patient and produce a missing-field-safe flat state.""" Patient.parse obj resource identifier = first mapping resource.get "identifier" identifier value = identifier.get "value" if identifier is not None else None return { "resource type": self.resource type , "has identifier": identifier is not None, "identifier value length": len identifier value if isinstance identifier value, str else None, remaining demographic fields omitted } The model never receives the NIK itself, only its length. This is the state produced from the full version of that Patient: { "resource type": "Patient", "has identifier": true, "identifier system": "nik", "identifier value length": 15, "has name": true, "name family": "Wijaya", "has birth date": true, "has gender": true, "has address": true, "address country": "ID", "has telecom": false, "telecom count": 0, "field completeness": 8, "field total": 10 } A small state keeps each call cheap and easy to inspect. The demo UI shows this exact state next to every decision, which made debugging much easier later. It is not a privacy solution, though. name family is still in there, and real data would need a proper review. Thresholds turn a model output into something the pipeline can do. The router has a confidence floor, and anything below it becomes unknown : src/jev fhir/modules/bundle router.py state = self. serializer.serialize bundle result = await self. jev client.choice state, ROUTE QUESTION, ROUTE OPTIONS category = result.choice if result.confidence = confidence minimum else "unknown" The quality scorer treats the NIK as a gate. A high completeness score cannot override it: src/jev fhir/modules/quality scorer.py passes nik gate = nik valid is not False action = "auto accept" if score result.score = threshold and passes nik gate else "review needed" Notifiable detection uses two thresholds, which gives it a band for human review: src/jev fhir/modules/notifiable detector.py if result.probability = confirmed threshold and result.answer: status = "confirmed notifiable" is notifiable = True builds the FHIR Flag here elif result.probability = review threshold: status = "review needed" is notifiable = False flag resource = None else: status = "not notifiable" is notifiable = False flag resource = None The review band is the part conformance has no equivalent for. The system can say it is not sure, and that outcome has a defined place to go. I compared Jev with a transparent rule baseline for each decision. The rules are deliberately simple: a field-count score, a length check for the NIK, routing on resource types and LOINC signals, and an exact ICD-10 lookup. The dataset has 204 labelled fixtures, including a hard set built to separate semantic judgment from exact matching. Labels are never computed from the logic under test. The test suite uses a mock Jev client and never calls the live API, and the benchmark runs against the live model only when I ask it to. | Decision | Fixtures | Jev, first run | Jev, final | Rules | My target | |---|---|---|---|---|---| | Quality accuracy | 71 | 0.676 | 0.662 | 0.549 | ≥ 0.85, missed | | Routing accuracy | 102 | 0.814 | 0.951 | 0.922 | ≥ 0.90, met | | Notifiable accuracy | 31 | 0.839 | 0.935 | 0.677 | none set | | Notifiable recall | 31 | 0.737 | 0.895 | 0.579 | ≥ 0.95, missed | | Notifiable precision | 31 | 1.000 | 1.000 | 0.846 | ≥ 0.80, met | The rest of this article explains how the first-run column became the final one. My first live run scored 0.839 on notifiable detection, with a recall of 0.737. The score alone gave no hint of why. The cause showed up while I was reviewing the demo. The state panel for a text-only dengue case showed code value , code display and code system all null, with no disease name anywhere. My serializer read code.coding and never read code.text , so the model was judging a blank. Three of the five misses had the disease name only in code.text , such as "Demam berdarah dengue DBD " and "TB paru". The fix is small: src/jev fhir/serializer/condition.py +CODE TEXT MAX LENGTH = 200 code = resource.get "code" + code text = code.get "text" if isinstance code, dict else None onset = resource.get "onsetDateTime" "code display": first coding value code, "display" , + "code text": code text :CODE TEXT MAX LENGTH if isinstance code text, str else None, "clinical status": first coding code resource.get "clinicalStatus" , The 200-character limit keeps a long note from inflating token use and limits how much free text leaves the system. I kept the rule baseline code-only, so the comparison stayed fair. I also kept the mock code-only, because teaching it to read text would make mock results look like model evidence. I then compared the two live reports row by row. Exactly three notifiable rows changed, and they were the three I expected: | Fixture | Confidence before | Confidence after | Result | |---|---|---|---| | dbd text only | 0.47 | 0.90 | wrong → correct | | tb paru text only | 0.50 | 0.89 | wrong → correct | | condition dbd text only | 0.49 | 0.89 | wrong → correct | Accuracy went from 0.839 to 0.935, and recall went from 0.737 to 0.895. The earlier score was measuring my pipeline, not the model. Without the row-level comparison, I could not have told that apart from noise, because quality accuracy moved from 0.676 to 0.648 between the same two runs on rows the fix does not touch. The two remaining misses, B20 HIV and typhoid A01.0 , have proper codes. Both landed in the review band at about 0.77, so a person would see them. All 19 notifiable cases were either confirmed or sent to review, and none was dismissed. My first explanation for the routing loss was that routing is mostly clear logic. I had written that down, but I had not measured it. So I compared the flat state of every Bundle Jev got right and wrong. All 12 of its encounter errors followed one pattern: | Conditions in the Bundle | Bundles | Jev correct | |---|---|---| | 1 | 6 | 6 | | 2 | 2 | 1 | | 3 or more | 11 | 0 | Both live runs showed this, and 100 of the 102 routing decisions were identical across the runs. This was a systematic failure, not noise. Two things in my setup could explain it. The question was "What type of clinical data does this FHIR Bundle primarily contain?". One Encounter with eight Conditions primarily contains Conditions, and no category exists for that, so unknown was a fair answer. The same word would also explain the other errors, because all seven misrouted mixed Bundles went to the pipeline for their most common resource type. The state was also uneven. It had direct flags for labs, immunizations and medications, which scored at 1.00 confidence, but nothing direct for encounters. dominant resource type said Condition . I changed both: src/jev fhir/modules/bundle router.py -ROUTE QUESTION = "What type of clinical data does this FHIR Bundle primarily contain?" +ROUTE QUESTION = "Which processing pipeline should handle this FHIR Bundle?" src/jev fhir/serializer/bundle.py "contains medication": "MedicationDispense" in resource types, + "contains encounter": "Encounter" in resource types, To see which change mattered, I ran the flag on its own first: | Live routing run | Accuracy | 3+ Condition Bundles correct | |---|---|---| | Original | 0.814 | 0 of 11 | | Flag only | 0.863 | 4 of 11 | | Flag and new question | 0.951 | 11 of 11 | Fourteen rows improved and none got worse. The question did most of the work, which I did not expect. It was one sentence, and it had been costing more than the missing data. The rule baseline stayed the same, and a test confirms it does not use the new field. In the final run Jev was right on three Bundles the rules got wrong, and the reverse never happened. That margin is small, so I would not read it as a clear win. The rules scored 0.922 with no model at all. Five mixed Bundles are still wrong, and three of those are at 0.87 confidence or higher, so the floor does not catch them. One of them, vitals only , has a label I have not settled. The rules' 0.95 on generated Bundles also needs care: the rule checks has encounter reference , and my generator links every generated encounter Bundle that way, so the two share a signal. Jev did best where the decision depends on more than one field. On the 14 hard Condition fixtures it scored 0.929 against 0.357 for the rules. The rules reported a refuted dengue case and a cholera record marked entered-in-error as real. Jev confirmed neither, and both went to review. It also caught dengue and tuberculosis coded only in SNOMED, which an ICD-10 lookup cannot see. Some of that gap is cheap to close. Prefix matching would handle subcodes like A15.0 , and checking verificationStatus would handle the refuted case. A serious rule set would do both. For routing, the rules remain the sensible default. They are simple and free, and a model can act as a second opinion, with disagreements going to a person. Quality scoring is the weakest result. Of 45 records that needed review, Jev let 14 through and the rules let 32 through. But Jev also held 10 of 26 good records, and the rules held none. A threshold is only useful if confidence means something, so the benchmark buckets decisions by confidence and compares that with observed accuracy. This is the top bucket: | Decision | Count | Mean confidence | Observed accuracy | |---|---|---|---| | Notifiable | 17 | 0.844 | 1.000 | | Routing | 84 | 0.982 | 0.964 | | Quality | 24 | 0.825 | 0.667 | Notifiable looks trustworthy here, and routing improved after the question change, from 0.918 in this bucket. Quality does not: a confidence of 0.82 was right about two times in three, so I would not automate anything on that threshold yet. Latency. My target was a p50 under 30 ms. The live p50 was about 235 ms for routing and notifiable, and 458 ms for quality, where a Patient takes two calls. The p95 ranged from 263 ms to 510 ms. These are client-side round trips from my machine, so they include the network, and I have not separated model time. The fast-decision premise is not proven by this data. Cost target. The full run used 147,401 tokens for about $0.006. That is about $0.03 per 1,000 records, three times my $0.01 target. It is still very cheap. Evaluation quality. No human has approved the labels yet. 124 are pending review, and the other 80 are correct by construction because a generator produced them. The hard sets are small. I also made both fixes after seeing these same fixtures fail, so 0.935 and 0.951 are not held-out results. They need confirming on data the system has not seen. The first step is human review of the labels, because every number here depends on them. After that I want a held-out set for the notifiable and routing decisions, and a latency measurement that separates the network from the model. Two gaps from this work are still open: ObservationSerializer drops code.text the same way the Condition serializer did, and all the serializers read only the first coding . The result I would keep is the shape of the system, more than any single score: parse first, flatten to a state a person can inspect, use a threshold per decision, give uncertain cases a lane, and keep a baseline that can beat the model. That baseline did beat it, and that is why I went looking. Before blaming a model, I now check what it actually received and what I actually asked. I work on clinical data systems at a research unit, mostly FHIR R4, HL7 v2, and HAPI FHIR. This is a proof of concept for a typed decision layer over FHIR records: https://github.com/budityw23/fhir jev https://github.com/budityw23/fhir jev .