{"slug": "i-built-an-autonomous-accounting-tool-to-let-ai-do-my-taxes", "title": "I built an autonomous accounting tool to let AI do my taxes", "summary": "A developer released Autonomous Accounting, an open-source tool that uses a user-selected vision LLM to convert receipts, invoices and bank statements into a reconciled, categorized ledger plus a downloadable audit binder. The tool runs entirely on one machine, requires Python 3.11+, Node 20+ and PostgreSQL 15+, and supports any OpenAI-compatible vision endpoint including vLLM, llama.cpp, Ollama's /v1 and LM Studio; it hosts no model of its own. Reconciliation uses four matching passes, uncertain items are flagged for human review rather than guessed, and documents are sent only to the endpoint the user configures via LLM_BASE_URL.", "body_md": "[Watch the demo](#see-it-work)  ·  [Quickstart](#quickstart)  ·  [How it works](#how-it-works)  ·  [Your data](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/data-schema.md)  ·  [Known issues](#go-deeper)\n\n**Autonomous Accounting turns a pile of receipts, invoices and bank statements into a\nreconciled, categorized ledger** — plus an audit binder with every source document\nattached. A vision LLM you choose reads the documents, a four-pass matcher pairs them with\nyour bank lines, and anything it is not sure of is flagged for you instead of guessed.\nEverything runs on one machine you own: no account to create, no vendor to pay, no copy of\nyour books anywhere but your own disk.\n\nTwo minutes twenty: one invented company's January, from eight documents and a bank CSV to a downloadable audit binder.\n\n[Watch the full demo (2:20, captioned, no sound)](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/media/demo.mp4)\n\n<sub>Everything shown is synthetic. `python scripts/gen_demo_data.py` writes the same month on\nyour machine — see [Try the demo month](#try-the-demo-month).</sub>\n\n| **A review queue, not a black box.** Exact pairs approve themselves; date gaps and cross-currency pairs wait for you. | **A ledger with CRA/GIFI categories.** Monthly and annual, with a GST/HST/PST summary and QuickBooks- and Xero-style CSVs. | **An audit binder in one ZIP.** The XLSX workbook, a self-contained HTML report, and the document behind every line. | \n\n- **Extraction.** Receipts, invoices and PDF bank statements are rasterised and read by a\nvision LLM into structured JSON (vendor, date, amount, tax, currency), with totals\ncross-checked against line items. Uploads queue and answer`202` ; a background worker\ndrains the queue, so dropping half a year of documents at once cannot time out, and a\nrestart re-queues whatever was mid-flight.\n- **Reconciliation.** Four passes over the unmatched set: strict same-currency, FX-relaxed\ninside a configurable band, an aggressive pass for splits and installments, and an LLM\npass that links across accounts.\n- **Categorization.** A deterministic regex pre-pass (`config/category_rules.yaml` ) locks\nwhat it recognises, vendor clustering normalises descriptors, and only the remainder\nreaches the model, under a per-run call budget.\n- **Reports.** Monthly and annual ledgers, a GST/HST/PST tax summary, a PDF report,\nQuickBooks- and Xero-style CSVs, a shareable read-only link, the ZIP audit binder — and an\noptional \"reasonableness\" pass that compares your expense ratios against published\nCanadian industry data.\n\n- **It flags; it does not guess.** A wrong number in a tax filing is worse than a blank.\nBank fees, interest and internal transfers are excluded rather than force-matched.\n- **It never sums across currencies.** A USD invoice paid in CAD is an FX match for review,\nnever an \"exact\" one.\n- **Your documents go only where you point them.** Set`LLM_BASE_URL` to a server on your own\nmachine or LAN and nothing leaves. An empty provider key means*off* ; nothing silently\nfalls back to a paid provider, and`GET /health` shows where an upload would go. If the\nendpoint is unreachable, jobs park in`waiting_for_model` and resume by themselves.\n- **No telemetry, no analytics, no webfonts.** Loading a page fetches nothing from anyone\nelse. The log files on your disk are the only record.\n- **Row-level security on every table** , keyed on the signed-in user and enforced from the\ncatalogue by a test.\n\nThe long form — backups, encryption at rest, what reaches a model — is in\n[Honesty about data](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/data-handling.md).\n\nYou need **Python 3.11+**, **Node 20+**, **PostgreSQL 15+** on loopback, and any\nOpenAI-compatible, vision-capable endpoint — vLLM, llama.cpp, Ollama's `/v1`, LM Studio.\nThis project hosts no model of its own.\n\n```\npython -m venv .venv && . .venv/bin/activate    # Windows: .venv\\Scripts\\activate\npip install -r requirements.txt\ncp .env.example .env                            # then fill in the three values it names\n\ncreatedb autonomous_accounting                  # the order below is not interchangeable\npsql -d autonomous_accounting -f db/local_auth_schema.sql\npsql -d autonomous_accounting -f db/schema_pg.sql\npython -m db.migrate\npsql -d autonomous_accounting -f db/local_grants.sql\n\nnpm --prefix web ci && npm --prefix web run build\npython -m uvicorn server.app:app --port 8080    # then open http://localhost:8080\n```\n\nRegistration is closed by default. Set `AUTH_ALLOW_SIGNUP=1`, restart, sign up, then set it\nback to `0`. `GET /health` reports database, storage and LLM reachability.\n\n## **The three `.env` values, and how to generate the secrets**\n\n`.env` values, and how to generate the secrets\nGenerate each secret exactly as `.env.example` says:\n\n``` python\npython -c \"import secrets;print(secrets.token_urlsafe(48))\"   # AUTH_JWT_SECRET\npython -c \"import secrets;print(secrets.token_urlsafe(48))\"   # STORAGE_URL_SECRET\n```\n\nand set `DATABASE_URL` to the database you are about to create. Every other variable, with\nits default, is in `.env.example`. `AUTH_AUTOCONFIRM=1` (the default) means a new account\nworks immediately without any mail transport.\n\n## **Why the schema files run in that order**\n\n`local_auth_schema.sql` must run first: it creates the `auth` schema, `auth.uid()` and the\n`authenticated` role, all of which `schema_pg.sql` needs. `schema_pg.sql` is the whole\nschema — every table, index, trigger and row-level-security policy — as a single baseline;\n`db/migrations/` ships empty and `python -m db.migrate` therefore has nothing to do on a\nfresh install, but run it anyway so anything added after the baseline is picked up.\n`local_grants.sql` must run last: it grants on the tables the steps above created, and\nwithout it every query fails with *permission denied*. Each step is idempotent, so\nre-running the sequence on an existing database changes nothing. (Nothing in `db/` needs a\nserver feature newer than PostgreSQL 13; 15 is just the oldest release still maintained\nupstream.)\n\n## **Developing the frontend with the Vite dev server**\n\nFastAPI serves the built bundle from the same origin. For frontend work, run the Vite dev\nserver instead, which proxies `/api` to the backend:\n\n```\ncd web\nnpm ci\nnpm run dev            # http://localhost:5174\n```\n\n## **Peer-benchmark data (optional, one command)**\n\nThe reasonableness engine needs a benchmark table, which is **not** committed:\n\n```\npython scripts/ingest_ised_benchmarks.py --year 2024 --out config/ised_benchmarks.json\n```\n\nThat downloads the Financial Performance Data CSVs published by Innovation, Science and\nEconomic Development Canada on the Government of Canada Open Government Portal, normalises\nthem into per-industry cohort cells, validates them fail-loud, and writes a large JSON file\nthat `.gitignore` excludes. The data is licensed under the **Open Government Licence –\nCanada**; the generated file and every report built from it carry the attribution the\nlicence requires. No Open Government Licence data file is distributed with this repository\n— you build the table yourself. See `NOTICE`.\n\nEvery run validates structure and plausibility. `--anchor anchor.json` adds an optional\ncheck that one named cell still carries the figures you expect, so a refresh that quietly\nchanges what a cell means aborts instead of overwriting your table; with no anchor\nsupplied, no cell-specific figures are demanded.\n\n`--seed-only` writes a small table of invented example figures instead, with no download —\nenough for the engine and its tests to run, and labelled in its own `meta` as not being\nindustry data. Skip the step entirely and the feature stays off; nothing else is affected.\n\nThe files under `tests/fixtures/` are per-parser unit fixtures; uploaded through the UI they\nmatch nothing, because no receipt among them belongs to any bank line. To see the whole\npipeline work, generate a coherent synthetic month instead:\n\n```\npython scripts/gen_demo_data.py          # writes data/demo/, which is gitignored\n```\n\nThat writes one invented company's January 2026 — a 12-row CAD statement in the built-in\nCSV layout, seven PDFs and one PNG — plus a `README.txt` saying what each file is there to\nshow. Then, in the app:\n\n1. Set `OWN_COMPANY_PATTERNS=example corp` in`.env` and restart, so the invoice*issued\nby* the demo company is read as income and can match the incoming wire.\n2. Sign up, choose **Manitoba** as the province (the demo documents charge GST and RST),\nand upload`demo_bank_cad.csv` as the statement.\n3. Upload all eight documents at once. They queue; each takes roughly a minute on a 27B-class local vision model, less on a hosted one.\n4. Press **Match Receipts** . Expect exact same-day pairs to be auto-approved without a\nmodel call, a six-day date gap and a USD invoice paid in CAD to land in**Review** , the\nmonthly fee, interest and inter-account transfer to be marked as needing no receipt,\ntwo bank lines left asking for a document, and a cash receipt left unmatched.\n5. Approve the pending pairs in **Review** , then download the audit binder from**Reports → January 2026** and open`proof_of_transaction/` and`index.html` .\n\nWhat that run looks like on one model is written up under *Verification gaps* in\n`docs/backlog.md`, next to the defects it found.\n\n- **Not a hosted service.** Nothing to sign up for. You run it or it does not run.\n- **Not something you buy.** The software takes no payment. Every signed-in account has\nfull access to every feature.\n- **Not supported.** No warranty, no SLA, no security response commitment, no promise that\nan upgrade will not want a manual migration. See the licence.\n- **Not a filing service.** It produces a ledger and reports for a Canadian small\ncorporation, and models GST/HST/PST only. It files nothing with the CRA and it is not\naccounting advice. Check the numbers before you use them.\n\n| Layer | What runs | \n|---|---|\n| API + web app | FastAPI on uvicorn, port `8080` by default; serves`/api/*` and the built SPA from one origin | \n| Database | PostgreSQL with row-level security keyed on the signed-in user | \n| File storage | A directory on local disk, served only through short-lived HMAC-signed URLs | \n| Frontend | React 19 + Vite + TypeScript, Tailwind, TanStack Query, Zustand | \n| Extraction | Any **OpenAI-compatible, vision-capable** endpoint you point`LLM_BASE_URL` at. Gemini and OpenAI are optional fallbacks that stay inert while their keys are empty | \n| Auth | Local email + password ( `server/local_auth.py` ): HS256 access tokens, rotating refresh tokens, PBKDF2-HMAC-SHA256 hashes. Sign in with Google is optional | \n\n| [**Bring your own data**](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/data-schema.md) | The six YAML tables under `config/` you are expected to edit, every record shape and status vocabulary, the built-in bank parsers (BMO, Wise, PayPal, Amazon; everything else goes through the LLM statement parser), and the minimum CSV a hand-made statement needs | \n| [**Honesty about data**](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/data-handling.md) | Where everything sits, what is and is not encrypted, backups, what reaches a model | \n| [**Known issues**](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/backlog.md) | Open engineering items. Read *Security hardening* before exposing an instance beyond loopback: the defaults — one owner, loopback only, registration closed — are what this was reviewed for | \n| [**Testing**](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/docs/testing.md) | `./scripts/gates.sh` runs ruff, pytest, typecheck and build — there is no CI. How`pytest` isolates its database, and what the synthetic fixtures do not prove | \n| [**Working on it with a coding agent**](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/.claude/README.md) | `CLAUDE.md` and`AGENTS.md` are the entry points;`.claude/` is an optional Claude Code harness that denies agent reads of`.env` ,`data/` and`logs/` . Nothing in the application depends on it | \n\nOne thing a stranger will meet early: **Gmail's OAuth redirect has to match what you\nregistered.** The code builds `<BASE_URL>/api/onboarding/gmail/callback`, and `BASE_URL`\ndefaults to `http://localhost:8080`. If you serve the app anywhere else, set `BASE_URL` (or\n`GMAIL_REDIRECT_URI`) before connecting Gmail, and register the same value with Google.\nGmail, PayPal and Wise gather integrations stay hidden until you configure credentials.\n\nSee [CONTRIBUTING.md](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/CONTRIBUTING.md): synthetic fixtures only, `ruff` and `tsc` clean, one\ntopic per pull request, sign off your commits (`git commit -s`). The most useful\ncontribution is a statement parser or a category rule for a layout the engine does not read\nyet — built from invented data, never real books.\n\nIf this saved you a bookkeeping afternoon, a star helps the next small-business owner find it.\n\nAutonomous Accounting — self-hosted bookkeeping with LLM document extraction and bank reconciliation. Copyright (C) 2026 Autonomous Accounting contributors\n\nThis program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.\n\nThis program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.\n\nYou should have received a copy of the GNU Affero General Public License along with this\nprogram. If not, see [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).\n\nFull text in [LICENSE](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/LICENSE). Because this is the AGPL, running a modified version as\na network service obliges you to offer its source to its users. Third-party attributions\n— including PyMuPDF, itself AGPL — are in [NOTICE](https://github.com/CodeGameDev29/KFAutonomousAccounting/blob/main/NOTICE).", "url": "https://wpnews.pro/news/i-built-an-autonomous-accounting-tool-to-let-ai-do-my-taxes", "canonical_source": "https://github.com/CodeGameDev29/KFAutonomousAccounting", "published_at": "2026-09-22 01:45:53+00:00", "updated_at": "2026-09-22 02:23:41.812013+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-products", "developer-tools"], "entities": ["Autonomous Accounting", "Python 3.11", "Node 20", "PostgreSQL 15", "vLLM", "llama.cpp", "Ollama", "LM Studio"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-autonomous-accounting-tool-to-let-ai-do-my-taxes", "markdown": "https://wpnews.pro/news/i-built-an-autonomous-accounting-tool-to-let-ai-do-my-taxes.md", "text": "https://wpnews.pro/news/i-built-an-autonomous-accounting-tool-to-let-ai-do-my-taxes.txt", "jsonld": "https://wpnews.pro/news/i-built-an-autonomous-accounting-tool-to-let-ai-do-my-taxes.jsonld"}}