{"slug": "managing-a-small-local-ai-budget-mac-m2-16gb", "title": "Managing a small local AI budget (Mac M2 16gb)", "summary": "Millfolio's local AI budget on a Mac M2 with 16GB RAM uses a batch-oriented design where the model's judgment is computed once and stored as tags, not run at query time. Tags come in three flavors—string, reference, and AI—with AI tags being the only costly inference, which is batched and deduplicated to minimize GPU load. The system prioritizes laptop usability by queuing jobs and yielding to interactive questions.", "body_md": "# Managing a small local AI budget\n\nIn the [first post](/blog/send-the-program-to-your-data) I described millfolio’s split: a frontier model writes a small program over an aliased schema, and a local model reads your actual files on-device. I ended with a promise to write about the unglamorous part — running the local model in batch mode, with priorities, so your laptop stays usable. This is that post.\n\n## The problem: the model doesn’t scale with your records\n\nA couple of years of bank and card statements is thousands of transactions. The local model is good at reading one of them — “is this a subscription?” is exactly the kind of targeted question it answers well. What it cannot do is answer that question thousands of times, per user question, on a laptop GPU. Even at a fraction of a second per record, a question like “what did I spend on subscriptions last year?” would turn into minutes of inference — and the *next* question would do all that work again.\n\nSo the design rule became: **the model’s judgment is a thing you compute once and store, not a thing you run at question time.**\n\n## The answer: tags, in three flavors\n\nEvery transaction gets tags at index time, from a plain-text rule file you can open and edit (`categories.txt`\n\n— one rule per line, the file is the source of truth). The three flavors are ordered by cost:\n\n**String tags** — the cheap, common case. A tag is a list of case-insensitive substring keywords, with optional excludes that veto a match:\n\n```\nphone = verizon, at&t, t-mobile, mint mobile\ngroceries = whole foods, trader joe, safeway, costco\n```\n\nPure string matching, effectively free, and it covers most of a real spending history — merchant strings are repetitive.\n\n**Reference tags** — tags built out of other tags. A rule can reference `@tag`\n\nnames instead of keywords, so you can compose groups without repeating anybody’s keyword list:\n\n```\nfood = @groceries, @restaurant\n```\n\nStill deterministic and free — a reference tag fires when a referenced tag is already on the row. Reference cycles are harmless (tags are only ever added, so evaluation just converges), and a reference is allowed to point at an AI tag, which makes the third flavor composable too.\n\n**AI tags** — the fuzzy tail no keyword list captures. The rule is a yes/no question, and the on-device model is the judge:\n\n```\nsubscriptions : is this a recurring subscription charge?\n```\n\nThis is the only flavor that costs inference — which is exactly why it’s a *stored* tag and not a query-time call.\n\nAt question time, all three look identical: the generated program filters on `.tags`\n\n, and the answer over thousands of records is a string comparison, not a model call. There’s also a privacy dividend: the frontier model is told the tag *names* and scope notes (so it writes programs that filter on them), but never the keywords — your actual merchant strings stay on the device.\n\n## Batches, and knowing when to nap\n\nAI tags still have to be computed once, and “once” means a backfill over every existing record plus whatever arrives later. That work runs through a single on-device work orchestrator: a disk-backed queue that survives restarts, draining one job at a time — indexing and AI-tag backfill can never fight each other for the engine, and both yield to an interactive question.\n\nInside a backfill job, the classification is batched and deduplicated:\n\n- Many descriptions go to the model in\n**one prompt**, and it answers`1: yes, 2: no, …`\n\n— one inference call classifies a whole slice. - Exact-duplicate descriptions are\n**collapsed before the model sees them**: a monthly charge that appears 24 times is classified once and the verdict fans out. On real statements this saves a large fraction of the work (the Stats page keeps a running counter of records classified vs. records covered). - A small\n**ledger** records how far each rule has been materialized, keyed on a monotonic insertion generation — so a finished tag never re-runs over old records; new records get picked up incrementally.\n\nBetween slices, the backfiller naps — and the nap length is the **priority** setting: high is a 100ms breath, medium is 1.2s, low is 5s. That one knob is how you decide whether the machine is “get it done” or “I’m working, take all night.” You can also pause the background processing indefinitely.\n\n## How fast/slow is it?\n\nThe following stats were generated by running `mill run https://raw.githubusercontent.com/millfolio/vault/5915bf52f17b9baf2c4fe71695dae03e05a9ee16/privacy-box/eval/local_classifier_eval.mojo`\n\non a Mac M2 16GB. (The program prints only aggregates — no merchant strings or amounts — and runs on any millfolio install; the live progress lines it streams while running need millfolio ≥ 0.4.49.)\n\nThe raw output is below\n\n```\nModel: Qwen/Qwen2.5-3B-Instruct\nQuestion: \"For each snippet: answer 'yes' if it is a grocery store or supermarket purchase (food shopping), answer 'no' for everything else (dining out, restaurants, coffee shops, and any non-grocery charge). Every answer must be exactly 'yes' or 'no' — never 'none', never anything else.\" vs the keyword tag 'groceries' as reference.\n\nData: 2930 transaction rows, 988 distinct descriptions (3.0x dedup); classified 400 (balanced: 29 tagged + 371 untagged) covering 2228 rows\n\nML (row-level, model vs keyword rule):\n  agreement 70.6%   precision 39.0%   recall 100.0%   F1 56.2%\n  confusion: TP 419  FP 654  FN 0  TN 1155   base rate 18.8%\n  (keyword tags are imperfect truth — a disagreement is model-vs-rule, not a certified model error; the sample is BALANCED, so base rate + agreement reflect the sample, not the vault)\n  answer mix: 111 yes · 25 no · 264 none · 0 other\n\nOperational:\n  400 descriptions in ~40 model calls (batched ~10/call), 260.9s total\n  1.5 distinct/s  ·  8.5 rows/s effective after dedup fan-out  ·  ~652ms per distinct description\n```\n\n(A note on the `answer mix`\n\nline: `none`\n\nis the batch protocol’s does-not-apply escape hatch — the model uses it as its negative for most rows. It parses cleanly and counts as a “no.”)\n\nThe `groceries`\n\nkeyword tag — a curated list of supermarket brands — is the reference label, and Qwen2.5-3B-Instruct gets the equivalent yes/no question over the same records. The sample is deliberately balanced (every tagged description, padded with untagged ones), classification is batched ten descriptions per model call, and recurring charges are deduplicated exactly the way the real tag backfill does it: 2,930 transaction rows collapse to 988 distinct descriptions, a 3× saving before the model sees anything.\n\nThe result, over 400 distinct descriptions covering 2,228 rows: **100% recall and 39% precision** against the keyword rule (F1 56%). The recall number shows that the model caught every one of the 419 rows my keyword list tags as groceries, without ever seeing the list. The precision number is more interesting than it looks: the 654 “false positives” are rows where the model says groceries and my dozen-brand keyword list says nothing — a pool that mixes genuine model errors with grocers my list simply doesn’t know. That asymmetry is the whole argument for the hybrid design: deterministic rules give you a precise, auditable head; the model gives you coverage of the tail; and the preview UI exists because neither should be trusted blind.\n\nOperationally: ~650 ms per distinct description, 1.5 distinct/s, and 8.5 rows/s effective once deduplication fans each verdict back out — so a 2,900-row vault gets a full AI pass in minutes of background time, which is exactly the budget the backfill scheduler is designed around.\n\n## Trust, but preview\n\nA tag system is only as good as your confidence in the rules, so the UI never makes you save one blind. When you create or edit a tag, you get a dry-run before anything persists:\n\n- For a\n**string or reference tag**, the edited rules are evaluated over your stored transactions and you see the match count plus example descriptions that matched — and examples that*didn’t*— so a too-greedy keyword shows itself immediately. - For an\n**AI tag**, the question runs against a time-boxed sample (~5 seconds of the on-device model) and you get “≈N records would match” with positive and negative examples side by side.\n\nSeeing both sides matters. A false positive is obvious from the matched list; a false *negative* only shows up when the UI also shows you what fell outside the rule. Nothing is written until you save. The codegen model partners with you in managing the taxonomy — when a generated program had to classify something inline, it can propose the question back as a reusable AI tag, so the next time it’s a stored-tag filter instead of fresh inference.\n\n## The shape of it\n\nNone of this is exotic — it’s the same batch-vs-online split every data system eventually grows. The local twist is that the expensive resource isn’t a cloud bill, it’s *your* GPU and *your* foreground work, so the scheduler’s job is as much politeness as throughput. Compute judgments once, store them as tags, spend the AI budget only on the fuzzy tail, and always show the user what a rule would do before it does it.\n\n**Demo:**[demo.millfolio.app](https://demo.millfolio.app)** How it works:**[millfolio.app](https://millfolio.app)** Code:**[github.com/millfolio](https://github.com/millfolio)", "url": "https://wpnews.pro/news/managing-a-small-local-ai-budget-mac-m2-16gb", "canonical_source": "https://millfolio.com/blog/local-ai-infra-tags/", "published_at": "2026-07-11 04:17:19+00:00", "updated_at": "2026-07-11 04:35:11.727692+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-infrastructure"], "entities": ["millfolio", "Mac M2"], "alternates": {"html": "https://wpnews.pro/news/managing-a-small-local-ai-budget-mac-m2-16gb", "markdown": "https://wpnews.pro/news/managing-a-small-local-ai-budget-mac-m2-16gb.md", "text": "https://wpnews.pro/news/managing-a-small-local-ai-budget-mac-m2-16gb.txt", "jsonld": "https://wpnews.pro/news/managing-a-small-local-ai-budget-mac-m2-16gb.jsonld"}}