TinyToT – Tree of Thoughts Inference Server

A lightweight, Ollama-compatible inference server that answers questions through knowledge retrieval, structured reasoning chains, and MCP tool calling — with no model weights, no training, and no GPU required.

It scores 97% on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering. It runs on a laptop CPU in under 1ms per query.

Every large language model above 1B parameters is doing two completely different things and charging you for both:

Fact storage— memorising billions of facts in FFN weight matrices at roughly 1–4 bits per parameter** Composition**— combining retrieved facts into coherent answers

TinyToT separates these. Facts live in plain markdown files where they belong. The retrieval engine is a TF-IDF cosine similarity index — zero learnable parameters, perfect recall, instant updates. The only thing that would need parameters is the composition step, and as the Phi-1 and DistilBERT research lines show, that requires roughly 30–50M parameters — not 7B, not 70B.

The practical upshot: a 7B parameter model is using an estimated ~6.5B of those parameters as an expensive, lossy, non-updatable database. TinyToT externalises that database as text files. What remains is a ~20-50M parameter composition problem.

GSM8K full test set (1,319 problems): 98.2% at 126 queries/second on CPU.

TinyToT operates in four modes, chosen automatically from the prompt:

The knowledge base is plain .md

files — drop one in data/knowledge/

and restart. No training, no configuration, no GPU.

make install

make run

make stop
git clone <repository-url>
cd TinyToT
make install   # creates pipenv venv and installs all deps including dev

See make help

for all available targets.

TinyToT is a drop-in replacement for Ollama. Point any Ollama-compatible client at http://localhost:11434

with model name tinytot

.

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","messages":[{"role":"user","content":"What is the Heisenberg uncertainty principle?"}],"stream":false}'

curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"If 3x + 7 = 22, what is x?","stream":false}'

curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"How many r'\''s in strawberry?","stream":false}'

curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"No reptiles are warm-blooded. A snake is a reptile. Is a snake warm-blooded?","stream":false}'

curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"Reply with JSON. Response: Paris is the capital of France.","stream":false}'

Add knowledge by creating .md

files in data/knowledge/

:

## My Domain

The key fact about this topic is X. Additional context goes here.

A second paragraph covers a related subtopic and becomes its own passage.

Each paragraph is a separate searchable passage. TinyToT uses TF-IDF cosine similarity to retrieve the best match. Confidence ≥ 0.50 returns the passage directly; confidence ≥ 0.20 grounds the reasoning chain.

The Hermes bridge: TinyToT automatically reads Hermes Learning Journal files (yyyy-mm-dd.md

with > source: ... · hash: ...

provenance lines). Drop any Hermes journal file into data/knowledge/

and the learnings become immediately retrievable — no retraining, no configuration.

TinyToT solves a wide class of problems through direct computation rather than retrieval — this is how it handles out-of-distribution arithmetic that no LLM can do reliably without a code interpreter:

The compute engine uses Python's ast

module for safe expression evaluation. It never calls eval()

or exec()

.

Reasoning chains live in data/categories/*.md

. Each file defines a domain with named chains and step-by-step thoughts. The TF-IDF index is built automatically from chain content — no keyword lists to maintain.

---
category: my_domain
---

## Chain 1: My Reasoning Approach
<!-- Handles: keyword1, keyword2 -->
Thought 1: First reasoning step.
Thought 2: Second reasoning step.
Thought 3: Conclusion.

Tool patterns live in data/schema/information_patterns.md

. When a prompt matches a tool pattern, TinyToT returns an Ollama-format tool_calls

response for mcphost to execute.

mcphost -m "ollama:tinytot" -p "Search for the latest AI developments"

make tests        # run pytest with branch coverage (gate: ≥80%)
make unit-tests   # unit tests only
make lint         # ruff check
make format       # ruff format
make precommit    # run all pre-commit hooks (pipenv run pre-commit run --all-files)
make docs         # build Sphinx HTML docs from docstrings
make docs-serve   # serve docs on localhost
make benchmark    # run routing + retrieval + GSM8K benchmarks
make ingest       # ingest external trace corpora (GSM8K, ToT traces)
make build        # build wheel for distribution
tinytot/
  _secrets_shim.py  # secrets module shim for Python 3.14+ compatibility
  content.py        # I/O: chain , category discovery, knowledge 
  retrieval.py      # TF-IDF index, cosine similarity, ranking, knowledge lookup
  compute.py        # Safe AST arithmetic, algebra, geometry, logic, letter counting
  tools.py          # Tool detection, parameter extraction, MCP tool matching
  inference.py      # Response mode dispatch, ToT orchestration, answer shaping
  server.py         # FastAPI app, Ollama-compatible endpoints, /api/quit
  ingest.py         # Convert external corpora (GSM8K, ToT traces) to knowledge
  benchmark.py      # Routing accuracy, retrieval precision, GSM8K evaluation
  cli/
    generate_api_docs.py   # Auto-generates Markdown API docs from __all__
  tests/
    conftest.py            # Shared fixtures (in-memory category/knowledge dirs)
    unit/                  # Per-module unit tests

data/
  categories/     # Domain reasoning chains (*.md)
  knowledge/      # Knowledge base passages (*.md) — 43 files, 23,881 passages
  schema/         # Tool detection patterns

docs/
  USER_GUIDE.md   # Architecture deep-dive and parameter analysis
  source/         # Sphinx source (conf.py, index.md, api/)

MIT — see LICENSE.md.