TiffinTally: a WhatsApp-to-prep-sheet tool for tiffin cooks, and an honest look at its evals

A developer built TiffinTally, an open-source tool that parses WhatsApp chat exports from home tiffin cooks into kitchen prep sheets and dues ledgers, handling Hinglish orders, cancellations, dietary notes and payment claims while masking personal data before any hosted-model call. The project reports strict-match accuracy of 35/50 rows (Wilson 95% interval 56.2–80.9) and lenient match of 47/50 (83.8–97.9) for its offline regex rules on a frozen synthetic test set, while explicitly marking all model runs as NOT RUN and retracting earlier model accuracy figures whose per-row outputs were never saved. The developer also removed five regex patterns fitted to test rows and documented template leakage between train and test splits in docs/KNOWN_LEAKS.md.

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 2, theme: Side Hustle. Home tiffin cooks take orders on WhatsApp, usually late in the evening, in Hinglish: "Kal lunch 2, dinner skip." "Aaj raat 1 thali, bina pyaaz lahsun." "Roommate left, cancel one thali from tomorrow." I ran those three through the shipped rules heuristic extract , sender Ravi on 17 October 2026. The first is a lunch of 2 and a dinner cancel. The second is one dinner with both no onion and no garlic . The third does not name a meal, so it is held for review unscoped decrement instead of cancelling every lunch. Someone has to turn that chat into a count of plates, a list of special requests and a record of who owes what. A missed cancellation wastes groceries; a missed extra order loses a customer. You paste a WhatsApp export. TiffinTally: Parses the chat and extracts order events meal, quantity, new or cancel, dietary notes, payments . Applies cutoff times and re-statements "kal lunch 2" followed by "kal lunch 3 kar do" . Builds a kitchen prep sheet and a dues ledger. A payment mentioned in chat stays a claim until the cook confirms it. Masks phone numbers, addresses and names before any message goes to a hosted model. I have not put it in front of a cook yet, so I make no claim about real users. Most of my time went into finding out what I could not claim. What is measured today is only the offline regex rules, on the 50 rows of the frozen synthetic test set. Strict match meal, quantity, action, mods, payment, secondary customer is correct on 35 of those 50 rows, with a Wilson 95 percent interval of 56.2 to 80.9 points. Lenient match meal, quantity, action is correct on 47 of 50, interval 83.8 to 97.9. The table with accuracy percentages, intervals and the source file for every number is RESULTS.md , generated by scripts/publish results.py from saved per-row outputs. What is not measured: the base model, a few-shot model, my fine-tuned adapter and a hosted Gemma run. All are marked NOT RUN in RESULTS.md . I do not claim any model beats the rules, and I have not run Gemma at all, so no Gemma number exists anywhere in this repository. Why I trust even those numbers only a little. The data is synthetic. The labels come from the templates that generated the text. The test split shares sibling templates with the training split. I documented this in docs/KNOWN LEAKS.md , and the intervals do not correct for it. I had earlier reported model accuracy figures. The per-row outputs were never saved and the adapter cannot be re-scored, so I retracted them. The old logs are kept under evals/retracted/ . An audit found five regex patterns I had fitted to test rows. I removed them, and the regex score went down. A case that used to fail. The rules used to read "don't send lunch" as one new lunch order. They now cancel it, same as the Hinglish line: python from tiffintally.extract.base import heuristic extract heuristic extract "don't send lunch", "Ravi" action=cancel, meal=lunch, qty=0 heuristic extract "kal lunch mat bhejna", "Ravi" action=cancel, meal=lunch, qty=0 Both lines are what that command prints today, 17 October 2026. I have not measured how any model does on English negation, so I make no claim either way. A meal that is not named "cancel one thali" is still sent to review rather than guessed. Live: https://tiffintally-lbn2.onrender.com https://tiffintally-lbn2.onrender.com . It runs in demo mode, so nothing a visitor pastes is written to disk, and with no model key configured the app answers with its on-device rules and the page says so. The stills and the video below are a local recording of this commit, on that same rules path. The names in the paste are made up. The recording does not press "Confirm received" on the payment claim. Captioned recording, about three minutes, no audio: tiffintally-demo.mp4 https://github.com/Vedant817/tiffintally/blob/main/docs/demo/tiffintally-demo.mp4 download https://raw.githubusercontent.com/Vedant817/tiffintally/main/docs/demo/tiffintally-demo.mp4 , captions https://github.com/Vedant817/tiffintally/blob/main/docs/demo/tiffintally-demo.srt . Run it locally: git clone https://github.com/Vedant817/tiffintally.git cd tiffintally uv sync --all-extras --dev uv run tiffintally serve Open http://127.0.0.1:8000 http://127.0.0.1:8000 , paste an export and read the prep sheet. With no keys set, the app uses the offline rules and the page says so "Rules regex only" . When a model answers, the page names the model that actually answered. When it falls back, it says that too. /benchmarks runs a metric scorer over four hand-written sentences. It needs gold labels, so it demonstrates the scorer rather than correcting live output, and one of the four sentences does not pass. A home-kitchen tool that turns a pasted WhatsApp export into a prep sheet and a dues ledger. Paste an export, select the delivery date and meal price, then review uncertain or late messages. The ledger follows the same daily policy as the sheet. Payments in chat remain claims until the cook confirms receipt. Opening and closing balances can be carried between chats using CSV. Exporting separate chats remains manual work; no WhatsApp integration or first cook session is claimed. Live: https://tiffintally-lbn2.onrender.com https://tiffintally-lbn2.onrender.com The deployed instance runs in demo mode: sheets, reviews and payment confirmations stay in memory for your browser session and are never written to disk, so anything you paste into it is discarded when you close the tab. No model key is configured there, so it answers with the local rules path and says so in the UI rather than pretending a hosted model replied. Stack: FastAPI, Pydantic, uv, pytest. On 18 October 2026, EXTRACTOR BACKEND=off uv run --frozen pytest -q in this tree reported 368 passed and 4 expected failures xfailed . That number is from the local run. The GitHub Actions result for the commit is the check on the pull request, not this sentence. Extraction backends, chosen by environment variables: Offline regex rules default, no network . Gemma through Google AI Studio when GOOGLE API KEY is set, default id gemma-4-26b-a4b-it . No run of it has been measured, so I claim nothing about its accuracy. A Qwen2.5-3B LoRA adapter on Modal. The Qwen base carries a research licence, so it is non-commercial; I treat it as an experiment. The adapter is not published and not measured in this release. If it is ever distributed, its licence is separate from this repository's MIT licence, and "Built with Qwen" attribution is required. Any OpenAI-compatible endpoint, such as local Ollama. It is wired in but I have not measured it. The default hosted id used to be a Gemma 2 9B that the API had retired. It answered 404, which my client treats as terminal, so every call silently fell back to the rules while the page still named Gemma. The id is fixed, the parser now skips the "thinking" part Gemma 4 puts before its JSON, and a failed call shows the rules label plus the HTTP error instead of a model name. Model output is untrusted. Messages are numbered in the batch and replies are matched by index. A message the model skips falls back to the rules and is flagged for review. Quantities are bounded and prompt-injection text is flagged. Privacy. Nothing is written to disk by default. The opt-in SQLite mode TIFFINTALLY PERSIST=1 is experimental: names come back as tokens after a restart, and the cutoff, date, price and corrections are not restored. While it is on, one pseudonymizer per upload replaces every sender name, every roommate named in a message, phone numbers, email addresses and flat or building numbers with sequential tokens, in the stored lines and in the customer, secondary customer and payment note columns. A street name with no flat number in front of it is not matched, so street names can remain. A test pastes an export and reads the SQLite file every table, plus the WAL and shared-memory bytes and every file written under data/ for that session, including data/private/ . That is the disk the test covers. It does not say that no name can ever reach disk. Reconciliation and the ledger are plain deterministic code, so counts and money never depend on a model. Why open matters here The rules path runs on a laptop with no internet, so a cook's customer list never has to leave the kitchen. An open-weight model behind a local endpoint extends that. If a hosted model is used, the masked text goes to that provider and spends the owner's quota. The deployed demo has no model key, so it uses the on-device rules; the blueprint documents EXTRACTOR BACKEND=off to make that explicit. I am not claiming that value is set in the Render dashboard. Open weights also let me swap models without rewriting the app. I have not measured how much any of this saves. What is next A small set of real, human-written cook messages with hand labels. Until that exists, nothing here says how the app performs on real chats. The kitchen board keeps one corner radius, one accent, and no drop shadow, so it reads as a docket. That is the same point as @kiwibreaksme on why generated UIs look off https://dev.to/kiwibreaksme/why-ai-generated-uis-look-off-and-the-one-principle-that-fixes-it-4j20 and Steven Gonsalvez on the landing-page default https://dev.to/stevengonsalvez/impeccable-stop-your-ai-making-everything-look-like-a-saas-landing-page-2a4d . Leaving the chat on the laptop unless a model key is set is the same bet as Srijan on not sending the data to a cloud API https://dev.to/srijan bhai/stop-leaking-data-to-cloud-apis-self-hosting-deepseek-locally-with-ollama-4701 . This entry is for the overall Side Hustle prize, and for Best Use of Render. Best Use of Render The app is deployed and live at https://tiffintally-lbn2.onrender.com https://tiffintally-lbn2.onrender.com on Render's free tier, from a render.yaml blueprint in the repository: runtime: python , dependencies resolved from the committed lockfile with uv sync --frozen --no-dev , a /health endpoint, and demo mode with TIFFINTALLY PERSIST=0 . Platform health checks are off on the free plan healthCheckPath is empty in render.yaml . Every pull request also gets its own deployment from .github/workflows/preview.yml , which waits until the preview answers on /health before it reports ready. That is the whole claim: a rules-only, nothing-persisted demo on a free instance. No hosted model key is configured there, and I am not claiming a model result from it.