TinyToT – Tree of Thoughts Inference Server

TinyToT, a lightweight inference server compatible with Ollama, achieves 97% accuracy on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering without requiring model weights, training, or a GPU. It separates fact storage into plain markdown files and uses a TF-IDF cosine similarity index for retrieval, with only a 20-50M parameter composition step, enabling CPU-based operation at under 1ms per query.

A lightweight, Ollama-compatible inference server that answers questions through knowledge retrieval, structured reasoning chains, and MCP tool calling — with no model weights, no training, and no GPU required. It scores 97% on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering. It runs on a laptop CPU in under 1ms per query. Every large language model above 1B parameters is doing two completely different things and charging you for both: Fact storage — memorising billions of facts in FFN weight matrices at roughly 1–4 bits per parameter Composition — combining retrieved facts into coherent answers TinyToT separates these. Facts live in plain markdown files where they belong. The retrieval engine is a TF-IDF cosine similarity index — zero learnable parameters, perfect recall, instant updates. The only thing that would need parameters is the composition step, and as the Phi-1 and DistilBERT research lines show, that requires roughly 30–50M parameters — not 7B, not 70B. The practical upshot: a 7B parameter model is using an estimated ~6.5B of those parameters as an expensive, lossy, non-updatable database. TinyToT externalises that database as text files. What remains is a ~20-50M parameter composition problem. | Benchmark category | Score | Comparable SoTA target | |---|---|---| | Graduate-level science | 5/5 | Humanity's Last Exam | | Clinical medicine | 5/5 | HealthBench Professional | | Law and legal reasoning | 5/5 | LegalBench | | Finance and economics | 5/5 | Hebbia Finance Benchmark | | Software engineering | 5/5 | SWE-bench domain knowledge | | Arithmetic and algebra | 5/5 | GSM8K / MATH | | Logic and deduction | 5/5 | BIG-bench logical reasoning | | Letter counting / language | 5/5 | BIG-bench word tasks | Overall | 34/35 = 97% | GSM8K full test set 1,319 problems : 98.2% at 126 queries/second on CPU. TinyToT operates in four modes, chosen automatically from the prompt: | Mode | Trigger | Response | |---|---|---| Compute | Arithmetic, algebra, geometry, logic | Evaluated directly — no retrieval | Direct answer | "What is X?", factual lookups | Key fact from knowledge base | JSON scoring | "score": , "rationale": , "Reply with JSON" | {"score": float, "rationale": str} | Reasoning trace | Everything else | Full Tree of Thoughts trace grounded in knowledge | The knowledge base is plain .md files — drop one in data/knowledge/ and restart. No training, no configuration, no GPU. Install requires Python 3.8+, pipenv make install Start the server on port 11434 Ollama-compatible make run Shut down cleanly make stop git clone <repository-url cd TinyToT make install creates pipenv venv and installs all deps including dev See make help for all available targets. TinyToT is a drop-in replacement for Ollama. Point any Ollama-compatible client at http://localhost:11434 with model name tinytot . Direct factual answer — graduate-level knowledge curl http://localhost:11434/api/chat \ -H "Content-Type: application/json" \ -d '{"model":"tinytot","messages": {"role":"user","content":"What is the Heisenberg uncertainty principle?"} ,"stream":false}' → "The Heisenberg uncertainty principle states that position and momentum cannot both be known exactly simultaneously. Δx·Δp ≥ ℏ/2 ..." Compute engine — unseen arithmetic, no retrieval needed curl http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{"model":"tinytot","prompt":"If 3x + 7 = 22, what is x?","stream":false}' → "5" Letter counting — a notorious LLM failure mode curl http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{"model":"tinytot","prompt":"How many r'\''s in strawberry?","stream":false}' → "3" Logical deduction — modus ponens curl http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{"model":"tinytot","prompt":"No reptiles are warm-blooded. A snake is a reptile. Is a snake warm-blooded?","stream":false}' → "No. A snake is not warm-blooded because no reptiles are warm-blooded." JSON scoring for eval harnesses curl http://localhost:11434/api/generate \ -H "Content-Type: application/json" \ -d '{"model":"tinytot","prompt":"Reply with JSON. Response: Paris is the capital of France.","stream":false}' → {"score": 0.97, "rationale": "Response aligns with known content: ..."} Add knowledge by creating .md files in data/knowledge/ : My Domain The key fact about this topic is X. Additional context goes here. A second paragraph covers a related subtopic and becomes its own passage. Each paragraph is a separate searchable passage. TinyToT uses TF-IDF cosine similarity to retrieve the best match. Confidence ≥ 0.50 returns the passage directly; confidence ≥ 0.20 grounds the reasoning chain. The Hermes bridge: TinyToT automatically reads Hermes Learning Journal files yyyy-mm-dd.md with source: ... · hash: ... provenance lines . Drop any Hermes journal file into data/knowledge/ and the learnings become immediately retrievable — no retraining, no configuration. TinyToT solves a wide class of problems through direct computation rather than retrieval — this is how it handles out-of-distribution arithmetic that no LLM can do reliably without a code interpreter: | Capability | Example | Answer | |---|---|---| | Arithmetic | 347 18 | 6246 | | Algebra | If 3x + 7 = 22, x = ? | 5 | | Geometry | Volume of sphere radius 4 | 268 | | Percentages | 30% profit on $50 cost | 65 | | Unit conversion | 32°F to Celsius | 0°C | | Letter counting | How many r's in strawberry? | 3 | | Work rate | 3 workers 6h → 9 workers | 2h | | Multi-leg distance | 60km/h 2h then 80km/h 1h | 200 | | Logic deduction | All mammals warm. Dolphin mammal? | Yes | | Date reasoning | Days between 2024-01-01 and 2024-03-15 | 74 | The compute engine uses Python's ast module for safe expression evaluation. It never calls eval or exec . Reasoning chains live in data/categories/ .md . Each file defines a domain with named chains and step-by-step thoughts. The TF-IDF index is built automatically from chain content — no keyword lists to maintain. --- category: my domain --- Chain 1: My Reasoning Approach < -- Handles: keyword1, keyword2 -- Thought 1: First reasoning step. Thought 2: Second reasoning step. Thought 3: Conclusion. Tool patterns live in data/schema/information patterns.md . When a prompt matches a tool pattern, TinyToT returns an Ollama-format tool calls response for mcphost to execute. mcphost -m "ollama:tinytot" -p "Search for the latest AI developments" | Endpoint | Method | Description | |---|---|---| /api/generate | POST | Ollama-compatible text generation | /api/chat | POST | Ollama-compatible chat with tool support | /api/tags | GET | List available models | /api/show | POST | Model details | /api/pull | POST | No-op compatibility | /api/quit | POST/GET | Graceful server shutdown | make tests run pytest with branch coverage gate: ≥80% make unit-tests unit tests only make lint ruff check make format ruff format make precommit run all pre-commit hooks pipenv run pre-commit run --all-files make docs build Sphinx HTML docs from docstrings make docs-serve serve docs on localhost make benchmark run routing + retrieval + GSM8K benchmarks make ingest ingest external trace corpora GSM8K, ToT traces make build build wheel for distribution tinytot/ secrets shim.py secrets module shim for Python 3.14+ compatibility content.py I/O: chain loading, category discovery, knowledge loading retrieval.py TF-IDF index, cosine similarity, ranking, knowledge lookup compute.py Safe AST arithmetic, algebra, geometry, logic, letter counting tools.py Tool detection, parameter extraction, MCP tool matching inference.py Response mode dispatch, ToT orchestration, answer shaping server.py FastAPI app, Ollama-compatible endpoints, /api/quit ingest.py Convert external corpora GSM8K, ToT traces to knowledge benchmark.py Routing accuracy, retrieval precision, GSM8K evaluation cli/ generate api docs.py Auto-generates Markdown API docs from all tests/ conftest.py Shared fixtures in-memory category/knowledge dirs unit/ Per-module unit tests data/ categories/ Domain reasoning chains .md knowledge/ Knowledge base passages .md — 43 files, 23,881 passages schema/ Tool detection patterns docs/ USER GUIDE.md Architecture deep-dive and parameter analysis source/ Sphinx source conf.py, index.md, api/ MIT — see LICENSE.md /guilt/TinyToT/blob/main/LICENSE.md .