{"slug": "tinytot-tree-of-thoughts-inference-server", "title": "TinyToT – Tree of Thoughts Inference Server", "summary": "TinyToT, a lightweight inference server compatible with Ollama, achieves 97% accuracy on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering without requiring model weights, training, or a GPU. It separates fact storage into plain markdown files and uses a TF-IDF cosine similarity index for retrieval, with only a 20-50M parameter composition step, enabling CPU-based operation at under 1ms per query.", "body_md": "A lightweight, Ollama-compatible inference server that answers questions through\nknowledge retrieval, structured reasoning chains, and MCP tool calling —\nwith **no model weights, no training, and no GPU required.**\n\nIt scores 97% on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering. It runs on a laptop CPU in under 1ms per query.\n\nEvery large language model above 1B parameters is doing two completely different things and charging you for both:\n\n**Fact storage**— memorising billions of facts in FFN weight matrices at roughly 1–4 bits per parameter** Composition**— combining retrieved facts into coherent answers\n\nTinyToT separates these. Facts live in plain markdown files where they belong.\nThe retrieval engine is a TF-IDF cosine similarity index — zero learnable parameters,\nperfect recall, instant updates. The only thing that would need parameters is the\ncomposition step, and as the Phi-1 and DistilBERT research lines show, that requires\nroughly **30–50M parameters** — not 7B, not 70B.\n\nThe practical upshot: a 7B parameter model is using an estimated ~6.5B of those parameters as an expensive, lossy, non-updatable database. TinyToT externalises that database as text files. What remains is a ~20-50M parameter composition problem.\n\n| Benchmark category | Score | Comparable SoTA target |\n|---|---|---|\n| Graduate-level science | 5/5 | Humanity's Last Exam |\n| Clinical medicine | 5/5 | HealthBench Professional |\n| Law and legal reasoning | 5/5 | LegalBench |\n| Finance and economics | 5/5 | Hebbia Finance Benchmark |\n| Software engineering | 5/5 | SWE-bench domain knowledge |\n| Arithmetic and algebra | 5/5 | GSM8K / MATH |\n| Logic and deduction | 5/5 | BIG-bench logical reasoning |\n| Letter counting / language | 5/5 | BIG-bench word tasks |\nOverall |\n34/35 = 97% |\n\nGSM8K full test set (1,319 problems): **98.2%** at **126 queries/second** on CPU.\n\nTinyToT operates in four modes, chosen automatically from the prompt:\n\n| Mode | Trigger | Response |\n|---|---|---|\nCompute |\nArithmetic, algebra, geometry, logic | Evaluated directly — no retrieval |\nDirect answer |\n\"What is X?\", factual lookups | Key fact from knowledge base |\nJSON scoring |\n`\"score\":` , `\"rationale\":` , \"Reply with JSON\" |\n`{\"score\": float, \"rationale\": str}` |\nReasoning trace |\nEverything else | Full Tree of Thoughts trace grounded in knowledge |\n\nThe knowledge base is plain `.md`\n\nfiles — drop one in `data/knowledge/`\n\nand restart.\nNo training, no configuration, no GPU.\n\n```\n# Install (requires Python 3.8+, pipenv)\nmake install\n\n# Start the server on port 11434 (Ollama-compatible)\nmake run\n\n# Shut down cleanly\nmake stop\ngit clone <repository-url>\ncd TinyToT\nmake install   # creates pipenv venv and installs all deps including dev\n```\n\nSee `make help`\n\nfor all available targets.\n\nTinyToT is a drop-in replacement for Ollama. Point any Ollama-compatible client\nat `http://localhost:11434`\n\nwith model name `tinytot`\n\n.\n\n```\n# Direct factual answer — graduate-level knowledge\ncurl http://localhost:11434/api/chat \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"tinytot\",\"messages\":[{\"role\":\"user\",\"content\":\"What is the Heisenberg uncertainty principle?\"}],\"stream\":false}'\n# → \"The Heisenberg uncertainty principle states that position and momentum\n#    cannot both be known exactly simultaneously. Δx·Δp ≥ ℏ/2 ...\"\n\n# Compute engine — unseen arithmetic, no retrieval needed\ncurl http://localhost:11434/api/generate \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"tinytot\",\"prompt\":\"If 3x + 7 = 22, what is x?\",\"stream\":false}'\n# → \"5\"\n\n# Letter counting — a notorious LLM failure mode\ncurl http://localhost:11434/api/generate \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"tinytot\",\"prompt\":\"How many r'\\''s in strawberry?\",\"stream\":false}'\n# → \"3\"\n\n# Logical deduction — modus ponens\ncurl http://localhost:11434/api/generate \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"tinytot\",\"prompt\":\"No reptiles are warm-blooded. A snake is a reptile. Is a snake warm-blooded?\",\"stream\":false}'\n# → \"No. A snake is not warm-blooded because no reptiles are warm-blooded.\"\n\n# JSON scoring (for eval harnesses)\ncurl http://localhost:11434/api/generate \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"tinytot\",\"prompt\":\"Reply with JSON. Response: Paris is the capital of France.\",\"stream\":false}'\n# → {\"score\": 0.97, \"rationale\": \"Response aligns with known content: ...\"}\n```\n\nAdd knowledge by creating `.md`\n\nfiles in `data/knowledge/`\n\n:\n\n```\n## My Domain\n\nThe key fact about this topic is X. Additional context goes here.\n\nA second paragraph covers a related subtopic and becomes its own passage.\n```\n\nEach paragraph is a separate searchable passage. TinyToT uses TF-IDF cosine similarity to retrieve the best match. Confidence ≥ 0.50 returns the passage directly; confidence ≥ 0.20 grounds the reasoning chain.\n\n**The Hermes bridge:** TinyToT automatically reads Hermes Learning Journal files\n(`yyyy-mm-dd.md`\n\nwith `> source: ... · hash: ...`\n\nprovenance lines). Drop any Hermes\njournal file into `data/knowledge/`\n\nand the learnings become immediately retrievable\n— no retraining, no configuration.\n\nTinyToT solves a wide class of problems through direct computation rather than retrieval — this is how it handles out-of-distribution arithmetic that no LLM can do reliably without a code interpreter:\n\n| Capability | Example | Answer |\n|---|---|---|\n| Arithmetic | `347 * 18` |\n`6246` |\n| Algebra | `If 3x + 7 = 22, x = ?` |\n`5` |\n| Geometry | `Volume of sphere radius 4` |\n`268` |\n| Percentages | `30% profit on $50 cost` |\n`65` |\n| Unit conversion | `32°F to Celsius` |\n`0°C` |\n| Letter counting | `How many r's in strawberry?` |\n`3` |\n| Work rate | `3 workers 6h → 9 workers` |\n`2h` |\n| Multi-leg distance | `60km/h 2h then 80km/h 1h` |\n`200` |\n| Logic deduction | `All mammals warm. Dolphin mammal?` |\n`Yes` |\n| Date reasoning | `Days between 2024-01-01 and 2024-03-15` |\n`74` |\n\nThe compute engine uses Python's `ast`\n\nmodule for safe expression evaluation.\nIt never calls `eval()`\n\nor `exec()`\n\n.\n\nReasoning chains live in `data/categories/*.md`\n\n. Each file defines a domain with\nnamed chains and step-by-step thoughts. The TF-IDF index is built automatically\nfrom chain content — no keyword lists to maintain.\n\n```\n---\ncategory: my_domain\n---\n\n## Chain 1: My Reasoning Approach\n<!-- Handles: keyword1, keyword2 -->\nThought 1: First reasoning step.\nThought 2: Second reasoning step.\nThought 3: Conclusion.\n```\n\nTool patterns live in `data/schema/information_patterns.md`\n\n. When a prompt matches\na tool pattern, TinyToT returns an Ollama-format `tool_calls`\n\nresponse for mcphost\nto execute.\n\n```\nmcphost -m \"ollama:tinytot\" -p \"Search for the latest AI developments\"\n```\n\n| Endpoint | Method | Description |\n|---|---|---|\n`/api/generate` |\nPOST | Ollama-compatible text generation |\n`/api/chat` |\nPOST | Ollama-compatible chat with tool support |\n`/api/tags` |\nGET | List available models |\n`/api/show` |\nPOST | Model details |\n`/api/pull` |\nPOST | No-op (compatibility) |\n`/api/quit` |\nPOST/GET | Graceful server shutdown |\n\n```\nmake tests        # run pytest with branch coverage (gate: ≥80%)\nmake unit-tests   # unit tests only\nmake lint         # ruff check\nmake format       # ruff format\nmake precommit    # run all pre-commit hooks (pipenv run pre-commit run --all-files)\nmake docs         # build Sphinx HTML docs from docstrings\nmake docs-serve   # serve docs on localhost\nmake benchmark    # run routing + retrieval + GSM8K benchmarks\nmake ingest       # ingest external trace corpora (GSM8K, ToT traces)\nmake build        # build wheel for distribution\ntinytot/\n  _secrets_shim.py  # secrets module shim for Python 3.14+ compatibility\n  content.py        # I/O: chain loading, category discovery, knowledge loading\n  retrieval.py      # TF-IDF index, cosine similarity, ranking, knowledge lookup\n  compute.py        # Safe AST arithmetic, algebra, geometry, logic, letter counting\n  tools.py          # Tool detection, parameter extraction, MCP tool matching\n  inference.py      # Response mode dispatch, ToT orchestration, answer shaping\n  server.py         # FastAPI app, Ollama-compatible endpoints, /api/quit\n  ingest.py         # Convert external corpora (GSM8K, ToT traces) to knowledge\n  benchmark.py      # Routing accuracy, retrieval precision, GSM8K evaluation\n  cli/\n    generate_api_docs.py   # Auto-generates Markdown API docs from __all__\n  tests/\n    conftest.py            # Shared fixtures (in-memory category/knowledge dirs)\n    unit/                  # Per-module unit tests\n\ndata/\n  categories/     # Domain reasoning chains (*.md)\n  knowledge/      # Knowledge base passages (*.md) — 43 files, 23,881 passages\n  schema/         # Tool detection patterns\n\ndocs/\n  USER_GUIDE.md   # Architecture deep-dive and parameter analysis\n  source/         # Sphinx source (conf.py, index.md, api/)\n```\n\nMIT — see [LICENSE.md](/guilt/TinyToT/blob/main/LICENSE.md).", "url": "https://wpnews.pro/news/tinytot-tree-of-thoughts-inference-server", "canonical_source": "https://github.com/guilt/TinyToT", "published_at": "2026-07-10 05:53:50+00:00", "updated_at": "2026-07-10 06:06:14.016878+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["TinyToT", "Ollama", "Phi-1", "DistilBERT", "GSM8K", "MATH", "BIG-bench", "SWE-bench"], "alternates": {"html": "https://wpnews.pro/news/tinytot-tree-of-thoughts-inference-server", "markdown": "https://wpnews.pro/news/tinytot-tree-of-thoughts-inference-server.md", "text": "https://wpnews.pro/news/tinytot-tree-of-thoughts-inference-server.txt", "jsonld": "https://wpnews.pro/news/tinytot-tree-of-thoughts-inference-server.jsonld"}}