This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
A prescription reader for a neighbourhood medical shop in India. It runs on the shop's own laptop, with no network connection at all.
The shopkeeper photographs a handwritten prescription. The system reads it, works out what was actually prescribed, and decides for itself which lines are safe to fill and which ones need a human to look at the paper.
The person who would pay for this is the chemist himself. He currently re-reads every prescription by hand because he cannot afford to get one wrong, and he is doing it at 9pm after a fourteen-hour day.
I should be upfront that the interesting part of this project is not the reading. It is the deciding when not to trust the reading.
The whole pipeline runs from one command on the shop's machine:
python3 scripts/run_page.py prescription.jpg
A synthetic page, end to end:
--- OCR (1.77s wall, 1.64s engine, 11 observations -> 9 lines) ---
[conf 1.00] Dr. R.Deshmukh MBBSReg2019/4482
[conf 1.00] Pt: Mr. Suresh Wankhede 42/M
[conf 1.00] Ox: HTN, T2OM
[conf 1.00] Rx
[conf 1.00] 1. Tab Telma 40mg 1-0-0 x 30 days
[conf 1.00] 2. Tab Glycomet GP2 1-0-1 x 30 days
[conf 1.00] 3. Cap Pan 40 1-0-0 x15 days
[conf 1.00] 4. Inj. Insulin 0-0-1 x 10 days
[conf 1.00] Adv: review after 15 days
--- reconciled prescription (57.1s wall) ---
diagnosis: HTN, T2DM
[0] Telma 40mg Tab sig=1-0-0 days=30 conf=0.70
[1] Glycomet GP2 Tab sig=1-0-1 days=30 conf=0.70
[2] Pan 40 Cap sig=1-0-0 days=15 conf=0.60
[3] Insulin Inj sig=0-0-1 days=10 conf=0.60
--- gate: review ---
- line 1 (Telma): model confidence 0.70 is below 0.80
- line 2 (Glycomet): model confidence 0.70 is below 0.80
- line 3 (Pan): model confidence 0.60 is below 0.80
- line 4 (Insulin): model confidence 0.60 is below 0.80
total: 58.9s (OCR 1.8s + model 57.1s)
Look at the third OCR line and then the diagnosis line underneath it. Those are the same characters read two different ways: the OCR returned Ox: HTN, T2OM, and the model worked out that it was Dx: HTN, T2DM. That is the one thing here the model does genuinely better than any regex I could write.
The other thing worth noticing is that on this page, every line goes to a human. That is entirely the confidence threshold, not the interesting checks, because the model was candid enough to score its own repairs at 0.60–0.70. The structural checks are what earn their place on the poisoned set further down, where the model is not candid and hands back 0.95 on answers that are wrong.
On the timing: 57s is the steady state. The first page after the model has been idle takes about 80s, because gemma3:4b has to be loaded into memory first. I have not tried to hide that number. On an eight-core laptop with no GPU it is the real price of not sending the page anywhere.
I want to be exact about the limits of that link, because the limits happen to be the argument.
It replays the real recorded OCR transcript from the page above — not a mock-up of one, the actual Vision output, including the Ox: HTN, T2OM line. The gate, the reorder forecast and the Hindi counselling text all execute live against it. If you want to see the gate refuse something, it is the line 3 (Pan) entry in the "Why" column.
What it does not do is run gemma3:4b. Render's free tier is a Linux container with 512MB of RAM and no GPU, and the model is 3.4GB. So the reading step is a deterministic fixture, and the page says so on itself rather than pretending.
Two consequences, stated rather than left for you to find:
"method": "damped-mean", not TabPFN. Torch does not fit there either; requirements-ml.txt is what switches TabPFN on, and the fallback announces itself in that field — which is the whole reason the field exists.demo/counselling-hindi.mp3
That last point is the project's own thesis turned back on itself, so it seemed like the wrong thing to make an exception for.
The pieces worth reading:
app/gate.py — the deterministic safety checks, including the two that mattered mostapp/gemma.py — the local model call, with the two API gotchas commented so nobody re-breaks themapp/tabpfn_engine.py — refill forecasting from the shop's own dispense historyscripts/selfcheck.py — 53 regression checks, runnable with plain python3, no model, no setupscripts/eval.py — the poisoned-name eval harnessdata/eval_poisoned.json — the 20 test lines and their ground trutheval_base_v2.json and eval_primed_v2.json — the two runs quoted below, committed so the claims are checkable
I started by trying to get a vision model to read the handwriting. That was the wrong instinct, and the community had already measured why: a dedicated OCR engine is roughly 300x faster than a general-purpose vision model at reading, with fewer character errors (The Dedicated OCR Engine Lost to the General Purpose Model by hexisteme).
So the architecture splits the job:
page image
-> macOS Vision OCR (reading: 1.8s/page)
-> gemma3:4b via Ollama (meaning: what is this medicine?)
-> deterministic gate (safety: may this be filled?)
-> human review (only what the gate refuses)
The OCR reads. The model understands. The gate decides. Only one of those three is allowed to be wrong quietly, and it is not the model.
While I was still validating the OCR path, I rendered a test prescription in a handwriting font and read it back. Vision read it flawlessly and reported confidence 1.00 on every line.
One of the lines it read perfectly was this:
Ox: HTN, T2OM
That is Dx: HTN, T2DM with two letters misread. Nothing about that line was illegible. It was simply wrong, and the OCR was certain.
That taught me the number I had been planning to trust measures "were the pixels legible", not "is this thing real". So I threw it out as a safety signal and went looking for what could replace it.
Confidence scores being useless for this is not a new observation, and I did not discover it. sohakanu documented it precisely while building Scribe: median confidence 0.95 when right, 0.95 when wrong (Confidence Is Theater). Their fix was cross-model verification — reading each field twice with models that fail differently. raju_dandigam makes the same point from the other direction in A Confidence Score Is Not a Probability.
I took a different route, because a pharmacy has something a form does not: a fixed vocabulary. The shop knows which forty medicines it actually stocks. That is checkable in code, for free.
I built a test set of twenty lines, each a realistic OCR corruption of a real Indian brand: 0 misread as O, 1 as l, 5 as S.
Then I ran them through gemma3:4b and sorted every answer into one of four buckets:
| bucket | meaning |
|---|---|
| repaired | returned the true medicine |
| refused | declined to name one — safe |
| echoed_garbage | accepted the corrupted string as a medicine — dangerous |
| silent_wrong | invented a different medicine — dangerous |
The last two are tracked together, because from the patient's side they are the same failure: a wrong medicine dispensed with confidence.
Nine of twenty lines came back wrong. All nine came back at 0.95 confidence.
Eighteen of twenty answers scored 0.90 or above. Nine of those eighteen were wrong. The confidence column carried no signal whatsoever, which means a threshold gate built on it would have passed nine wrong prescriptions straight through to the counter.
Two failure modes did the damage, and I only understood the second one by reading the eval output line by line.
It substituted generics for brands. Asked about Glycomet 6P2, it answered Metformin. For Pan 4O it answered Panadol. For Atorva 1O, Atorvastatin.
These are not cosmetic swaps. Glycomet GP2 contains glimepiride as well as metformin. Panadol is paracetamol; Pan is pantoprazole. A generic substitution is a different prescription.
And it is invisible in the JSON — except in one place. The name it returned is not anywhere on the page.
So I check exactly that. The longest word in the returned drug name gets compared against the raw OCR text with an OCR-sized edit tolerance:
def _grounding_problems(line: RxLine) -> list[str]:
"""Does the medicine the model named actually appear on the page?"""
if not line.drug or not line.raw:
return []
drug_tokens = {t for t in tokens(normalise(line.drug)) if len(t) >= 3}
if not drug_tokens:
return []
source_tokens = tokens(normalise(line.raw))
anchor = max(drug_tokens, key=len)
if any(closely_matches(anchor, source) for source in source_tokens):
return []
return [
f"'{line.drug}' does not appear on the page as written "
f"(OCR line reads: {line.raw.strip()!r})"
]
Checking that a value actually appears in its source is a known and documented technique — michael_maurice lays out the full discipline, including the crucial repairable/non-repairable split, in Schema-Valid Isn't Correct. I am applying it to brand names rather than invoice totals.
That one check caught all five substitutions the model made.
With grounding in place, four of the nine were still getting through. All four were the same shape:
thyronorm-o-for-zero strength='5Omcg'
ecosprin-s-for-five strength='7S'
amlopres-s-for-five strength='2.S'
insulin-units-word strength='1O units'
Every one was the same thing. The drug name was correct, and it was genuinely on the page, so grounding had nothing to say about it. The strength was the corruption: 5O should be 50, 7S should be 75, 2.S should be 2.5, 1O units should be 10 units.
And my gate had accepted all four, because I was checking that the strength field was non-empty. Not that it was a strength.
Two regexes closed the gap. One for what a dose looks like, and one for the Indian combination markers that are legitimate strengths without being numbers at all:
_DOSE_RE = re.compile(r"^\d+(?:\.\d+)?(?:mg|mcg|mgs|g|gm|ml|iu|units|unit|%)?$")
_SUFFIX_RE = re.compile(r"^[a-z]{1,3}\d+$") # GP2 in Glycomet GP2
def _strength_problems(strength: str | None) -> list[str]:
if not strength:
return []
compact = re.sub(r"\s+", "", strength)
if _DOSE_RE.match(compact.lower()) or _SUFFIX_RE.match(compact.lower()):
return []
guess = _repair_digits(compact)
if guess != compact.lower() and _DOSE_RE.match(guess):
return [
f"strength {strength!r} does not parse as a strength "
f"(looks like {guess!r} with a letter misread for a digit)"
]
return [f"strength {strength!r} does not parse as a strength"]
Four misses went to zero. A wrong dose and a wrong drug are the same amount of wrong.
A word on false alarms, because a gate that cries wolf is a gate people click through. Three lines that the eval scores as safe still get sent to review: one where the model refused outright, one where it put x 30 days in the sig field, and the deliberately-illegible control case. I read all three. None is a correct answer being wrongly held back — they are the gate doing its job on a field my eval was not scoring.
Then, while reading the demo output above, I found one that was.
The strength check was flagging the GP2 in Glycomet GP2 as unparseable. GP2 is a combination marker — metformin plus glimepiride — and my regex only accepted digits with an optional unit. It was rejecting a correct answer, and no test caught it, because every test I had written encoded the same wrong assumption the code did. It surfaced because a human read the output.
That is now scripts/selfcheck.py: 53 checks covering the gate, the formulary, the counselling text and the end-to-end path, runnable with plain python3, no model and nothing to install. Two of those checks exist purely because of how the fix went wrong the first time — 1O units repaired to 10unit5 (the s in "units" was being read as a 5), and xyz acquired a confident repair hint for no reason at all. A wrong suggestion on a dosage is worse than no suggestion.
The same wrong assumption turned out to be sitting in the hosted demo's fixture too, a day later: the fixture had Glycomet GP2 with a null strength, so the gate reported "tablet with no strength" on a line whose strength was right there in the name. The real extractor returns GP2 in the strength field. The fixture was less realistic than the model it was standing in for, and it took reading the deployed page to notice.
| base model | with the shop's own correction history | |
|---|---|---|
| wrong answers | 9/20 (45%) | 6/20 (30%) |
| correct | 9/20 | 13/20 |
| filled with no human looking | 7/20 | 11/20 |
| wrong answers reaching the counter | 0/20 | 0/20 |
The model is still wrong a third of the time. None of it reaches the patient, because the gate does not accept the model's word for anything it can check itself.
Every line the pharmacist corrects is written to a local SQLite table on the shop's machine. Those corrections are the shop's own vocabulary: in this shop, this handwriting, "Telrna" is Telma.
I fed 18 of them back into the prompt as context and re-ran the eval. It improved things: 45% wrong down to 30%, and four more lines filled without a human.
But it also introduced three regressions I would have missed if I had only looked at the totals:
glycomet-g-to-6 silent_wrong -> echoed_garbage ("Glycomet 6P2" -> "Glycomet", GP2 dropped)
telmikind-h repaired -> echoed_garbage ("Telmikind-H" -> "Telmikind", -H dropped)
montck-dropped-e silent_wrong -> echoed_garbage ("Montck LC" -> "Montck LC")
Given the shop's vocabulary, the model starts snapping to the nearest familiar name and silently drops suffixes that change what the medicine is. Telmikind-H contains a diuretic. Telmikind does not.
So the correction history is not a free win. It is a trade: better recovery of corrupted strengths, at the cost of a new way to be confidently wrong. The gate catches all three anyway, which is the point of having a gate that does not care how confident the model sounds.
This is also the honest reason the corrections matter. A hosted API cannot learn that this shop's handwriting turns Telma into Telrna, because the corrections are the most private thing in the building. They stay on the machine, and they are what a fine-tune would later be built from — via Tinker, so the shop never needs a GPU.
Three reasons, in the order they actually matter to the shopkeeper.
The data cannot leave. A prescription carries a patient's name, age, diagnosis and medicines. That is the most sensitive record the shop holds. The whole point is that the model runs where the data already is. Shipping it to a hosted database to prove the model is local would give away the thing being protected.
The vocabulary is local knowledge. Knowing that this shop stocks forty specific brands, and that in this handwriting Telrna means Telma, is not general knowledge. It is not in anyone's training data. It exists only in that shop, and it is the difference between a system that works and a system that is confidently wrong. Owning the model is what makes the correction loop worth building.
Volume makes per-call pricing cruel. This is not a chat app. It is a shop doing a couple of hundred pages a day, forever. A per-page API bill is a recurring cost that grows with the shop's success. A local model costs electricity and a laptop the shop already owns — and the slowest thing in the pipeline is the part I did not pay for.
There is a fourth reason I did not expect. Running locally meant I could measure the model's failures on the shop's real material, as many times as I wanted, for free. The 20-line eval that this whole post is about cost nothing to run eleven times. That is not really possible when every run is a billed API call.
And a fifth, which is the one this post is really about. Running locally meant I could read the raw output. The generic-for-brand substitution at 0.95 confidence is not visible in an aggregate score. Neither was the false alarm I had shipped in my own gate. Both were found by looking at what actually came back, one line at a time, on a machine I owned.
The hosted demo makes the same point from the other side. Every capability I had to switch off to make it fit on a free container — the model itself, the forecaster, the voice — is a capability that works when the software sits next to the data and the patient.
The bug I am least proud of, and the one that taught me the most: the false positive above was invisible to every test I had, because I had written the tests from the same wrong assumption as the code. Here is the session where reading the real output caught it.
Best Use of Gemma. The understanding layer is gemma3:4b, served locally through Ollama, running on the CPU of an eight-core laptop with no GPU. It is the only reason the "generic becomes brand" substitution is visible at all: the model reads Glycomet 6P2 and reaches for the only metformin brand it has ever heard of. Every safety check in the system exists because Gemma is useful and unreliable in a specific, measurable way — and because it runs locally, I could measure exactly how unreliable, eleven times over, for nothing.
Best Use of TabPFN. The shop's second problem is knowing what to reorder, and it is a tabular forecasting problem of exactly the kind a chemist cannot answer from memory: a dozen medicines, three months of dispense counts, and spiky demand around festivals. app/tabpfn_engine.py builds a weekly series per medicine and fits TabPFN walk-forward — never training on a week it is about to predict — then ranks by urgency, weighted for the chronic medicines that must never run out.
It forecasts demand per medicine, not refills per patient. That is deliberate: the dispenses table has no patient column, because it is the only table that could leave the machine without carrying anyone's medical history with it. The richer signal would have cost the privacy claim the whole project rests on.
One detail I would keep in any system like this: without requirements-ml.txt the forecaster silently falls back to a damped moving average, so every row carries the method that produced it — "method": "tabpfn (53 shop rows)". A fallback can never be mistaken for a forecast. That field is also how I discovered that tabpfn was not installed at all while the code claimed it was, which is a lesson about the difference between integrating a library and running one.
Best Use of ElevenLabs. The last mile of a prescription is the part the pharmacist has already said fifty times before lunch: what to take, when to take it, for how long, and what to do if something goes wrong. That explanation is spoken, not read. Many of the people this shop serves would rather hear it than read a label, and by the fiftieth time the pharmacist is no longer saying all of it.
eleven_multilingual_v2 renders that script in Hindi from prompts/counsel.txt, and the parts he drops are exactly the parts written into the prompt: never take two doses at once to catch up on a missed one, come back to the shop with the strip if anything is unclear, and see a doctor immediately for a rash, swelling or difficulty breathing. It is a real render — 485 characters in, 36.8 seconds of Hindi out, at one credit per character — and it is checked into the repo as demo/counselling-hindi.mp3 so it can be listened to rather than taken on trust.
The design decision I would defend is that the audio is opt-in per request and off by default. It is the only step in the pipeline that sends anything off the machine, and what it sends necessarily contains the medicine names — a patient cannot be told to take something the sentence refuses to name. So the text stays local and the pharmacist asks for the speech. The hosted demo has no key at all, for the same reason.
Best Use of Render. The hosted demo above is deployed from a Render blueprint — render.yaml lives in the repository, so the service is described in version control rather than clicked together in a dashboard, and a stranger can reproduce the deploy from the repo alone.
The interesting part was making a free-tier container honest. There is no GPU and 512MB of RAM, so instead of pretending, the deployment was designed around the constraint: requirements.txt and requirements-ml.txt are separate files so the deployable dependency set stays small enough to build without dragging in torch, app/main.py never imports the macOS Vision wrapper (app/ocr.py) so the Linux container boots, and DEMO_MODE=true replays the real recorded OCR transcript while the gate, the forecaster and the counselling text all execute for real. The page tells you which parts are fixtures and why, which is the only reason I would put a demo link in a post about not trusting things.
Thanks to sohakanu, michael_maurice, raju_dandigam and hexisteme — this project would have taken a much longer route to a worse place without their write-ups.
All names, prescriptions and corrections in this post are synthetic. No real patient data was used, and no page left the machine.