{"slug": "your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context", "title": "Your Clinical AI Doesn’t Need a Million Tokens. It Needs Better Context", "summary": "A metadata-driven context layer between clinical event streams and large language models can cut per-event inference from roughly 1 million tokens to about 1,000 by pre-computing per-subject running state and per-cohort baselines instead of sending raw sensor data to the model, according to an analysis of a decentralized obesity trial using continuous glucose monitors. The analysis calculates that 100 matched subjects over a 14-day window at a reading every five minutes yields about 4,000 readings per subject, and that trimming to a subject's last 24 hours plus 50 subjects of light cohort data still leaves roughly 46,000 tokens per event. The proposed layer stores generic subject_state and cohort baseline tables keyed by endpoint, so the model receives only the event, the subject's current state, the matching baseline and the protocol rule, with a human clinical team retaining the decision on whether a reading is an adverse event.", "body_md": "Picture a decentralized obesity trial. Each subject wears a continuous glucose monitor, a connected scale and an activity band, so the study sees glucose every five minutes alongside weight and movement. At 3am, one glucose reading crosses a threshold and notification pops up.\n\nA human decides what happens next. In a trial that is the clinical team, an investigator or medical monitor who judges whether the reading is an adverse event, whether to contact the participant, whether the protocol calls for a step. The software never makes that call. Its job is narrower and comes first: is this reading normal for this subject, is it unusual against similar subjects, does it need a human’s eyes now or is it just noise? It surfaces the ones that matter, fast, with the context to act on.\n\nThe trouble is the model replies come back slow and heavy, the output feels a little less in your control every week, and the one that really stings: you can’t be sure the token bill survives your user base scaling.\n\nWhat your product actually needs is simpler: the right inference in seconds, defensible on paper.\n\nThe tempting move is to give the whole problem to a large model. Take the event, pull the subject’s recent history, pull comparable subjects for context, and let the model reason over all of it. That approach demos well, then falls over in production, and the thing that breaks it is tokens.\n\nSay you want real cohort context, 100 matched subjects across a 14-day window. At a reading every five minutes that’s roughly 4,000 readings per subject, so you’re pushing on the order of a million tokens into the model for a single event. That doesn’t fit in most context windows. Where it does, you’re paying for it and waiting on it every time an event fires.\n\nTrim it hard, down to the subject’s last 24 hours plus 50 subjects of light cohort data, and you’re still around 46,000 tokens per event. That’s an improvement, but you’re still doing it thousands of times a day, and the latency alone rules it out for anything that has to feel real-time.\n\nThe fix is to stop making the model do the aggregating. You put a metadata-driven layer between the event stream and the model, one that already holds the context and keeps it current as data arrives.\n\nThe layer keeps a small running summary for each subject, and a set of pre-computed baselines for each matched cohort across the population, so “normal for someone like this at this time of day” becomes a lookup rather than a computation. When an event fires, you don’t ship raw data. You ship the event, the subject’s current state, the matching baseline and the protocol rule, which together come to about a thousand tokens of signal that has already been thought through.\n\nThe layer itself knows nothing about glucose. It knows how to keep a small running state per subject and a baseline per cohort for whatever signal you point it at, and each endpoint hands it a short definition that says what to keep and what counts as worth a clinician’s attention.\n\nTake the obesity study. One of its endpoints is continuous glucose from the CGM. Its definition names the per-subject state the layer maintains for it, updated on every reading:\n\nThe same definition names the cohort baseline that each reading is compared against, which is recomputed on a schedule rather than on every event:\n\nAt event time the assessment is a comparison, not a computation. A reading at the stratum’s P98 with a steep positive slope and a low recent time-in-range is a different call than the same number sitting at P60 with a flat slope. The model sees those few derived facts, not the raw series.\n\nThe storage underneath is generic. The same two tables hold every signal, keyed by endpoint, so nothing about glucose is baked into the schema:\n\n```\n-- one row per (endpoint, subject); the state shape comes from the definitionsubject_state (  endpoint_id   text,  subject_id    text,  state         jsonb,        -- the fields the definition declares  updated_at    timestamptz,  primary key (endpoint_id, subject_id))-- one row per (endpoint, stratum, time bucket)cohort_baseline (  endpoint_id   text,  stratum_id    text,  tod_bucket    smallint,  bands         jsonb,        -- percentiles / thresholds from the definition  n_subjects    int,  updated_at    timestamptz,  primary key (endpoint_id, stratum_id, tod_bucket))\n```\n\nThe versioning matters as much as the values. Every aggregate carries updated_at, and in practice a definition version too, so you can replay exactly which baseline a past decision was made against. That is what turns “the model decided” into “here is the state the model decided from.”\n\nPoint the layer at a different endpoint and you write a different definition. The shape stays the same, only the fields change. Here are the same slots filled for three other endpoints, each onboarded as a new study without touching the engine:\n\nThe definition is also where study and customer variation lives. Two sponsors can watch the same signal with different thresholds and different cohort rules without forking the code. The shape of that registry is a post of its own, but the point here is that the layer extends by configuration, not by rewriting.\n\nHere’s the same event costed three ways. Assumptions: CGM sampling every five minutes, a 500-subject trial, and roughly 1,000 events a day. Cost figures are illustrative at $3 per million input tokens, so treat the ratios as the point, not the dollars.\n\nThe lean approach against the middle layer is about a 46x gap on tokens. Against the full version it stops being a cost comparison at all, because the full version doesn’t run.\n\nSaving money is the least interesting reason to build it this way. The ones that matter:\n\nThis is, at heart, a compiled-AI pattern. You compile the domain knowledge into deterministic, versioned precomputation ahead of time, the rolling state and the cohort bands, so at request time the model only does the last bounded step: read a small, known set of inputs and judge whether to surface them to a clinician. The heavy, ambiguous work lives in code you can read, not in a prompt you hope behaves. In healthcare that is the difference between a system you can sign off on and one you can’t.\n\nIt also lines up with the data-integrity bar the industry already runs on. Defined, versioned aggregates are attributable, contemporaneous and reproducible by construction, which is most of what ALCOA++ and 21 CFR Part 11 ask for. A reviewer can see exactly what the model saw and why it decided as it did; feed raw data straight to a model and you have none of that.\n\nThere’s a split forming in the industry. Some clinical-data platforms are building their own models and betting sponsors will adopt them. Others put a thin layer over the frontier models everyone already pays for and compete on how good that layer is.\n\nI think the second approach wins, for a fairly dull reason. Sponsors already pay for a frontier model or two, and they have little appetite for another model, another vendor and another line on the invoice when the general models keep improving on their own. A homegrown model has to chase the frontier forever. The layer doesn’t. It only has to turn a raw event into something a clinician can act on and an auditor can follow, and that is where the real work sits anyway, the aggregates, the retrieval, the plumbing that hands the model exactly what it needs and nothing more.\n\nIt also happens to be the part you own outright, instead of renting it from whoever trained this year’s best model.\n\nSo the real work here isn’t the model reading the data. It’s the layer in front of it that decides what the model ever sees, and keeps that cheap enough to run on every event and clear enough to defend to an auditor. Get that layer right and the model becomes the easy part.\n\n1. Battelino T, Danne T, Bergenstal RM, et al. Clinical Targets for Continuous Glucose Monitoring Data Interpretation: Recommendations From the International Consensus on Time in Range. [Diabetes Care. 2019;42(8):1593–1603.](https://doi.org/10.2337/dci19-0028)\n\n2. U.S. FDA. Clinical Decision Support Software — Guidance for Industry and FDA Staff. [September 2022.](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software)\n\n3. IMDRF. [Software as a Medical Device (SaMD): Key Definitions. 2013.](https://www.imdrf.org/documents/software-medical-device-samd-key-definitions)\n\n4. U.S. FDA. Data Integrity and Compliance With Drug CGMP: Questions and Answers — Guidance for Industry. [December 2018.](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/data-integrity-and-compliance-drug-cgmp-questions-and-answers)\n\n5. UK MHRA. [‘GXP’ Data Integrity Guidance and Definitions. 2018.](https://www.gov.uk/government/publications/guidance-on-gxp-data-integrity)\n\n6. U.S. FDA. 21 CFR Part 11 — Electronic Records; Electronic Signatures. [Code of Federal Regulations, Title 21, Part 11.](https://www.ecfr.gov/current/title-21/part-11)\n\n7. U.S. FDA. Decentralized Clinical Trials for Drugs, Biological Products, and Devices — Guidance for Industry. [2024.](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/decentralized-clinical-trials-drugs-biological-products-and-devices)\n\n[Your Clinical AI Doesn’t Need a Million Tokens. It Needs Better Context](https://pub.towardsai.net/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context-a30a628424ca) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context", "canonical_source": "https://pub.towardsai.net/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context-a30a628424ca?source=rss----98111c9905da---4", "published_at": "2026-11-13 21:01:02+00:00", "updated_at": "2026-11-13 21:22:29.409730+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-agents"], "entities": ["continuous glucose monitor", "subject_state", "cohort baseline"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context", "markdown": "https://wpnews.pro/news/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context.md", "text": "https://wpnews.pro/news/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context.txt", "jsonld": "https://wpnews.pro/news/your-clinical-ai-doesnt-need-a-million-tokens-it-needs-better-context.jsonld"}}