AI Disclosure:
I love C++ and Python but AI helps me get more out and try things that I probably would not other wise do to time. local models were used to help make and test inside my program.
Toaster Loop — This is my first-ever public post about the project
This is the first time I’m showing this to anyone and is still a work in progress, so this first post is the “what/why/how”. I will try to keep this short as my readme has gotten to long and covers most of everything.
The problem I wanted gone
Every “local AI” stack I tried was a pile of half-connected parts: llama.cpp here, an agent framework there, fine-tuning scripts somewhere else entirely, and none of it remembered anything. Meanwhile the cloud stuff remembers everything but belongs to someone else and they are happy to sell it. Wendell hit a nerve: I wanted the curmudgeonly-home lab PC to run its own AI — one that gets better at my tasks over time without phoning home.
The goal
build Jarvis not Skynet, make it new user friendly and still have all the dials and knobs for the power users.
( here is a screenshot of me talking to an agent that made it’s own dataset about Toaster Loop and trained its self on it. then I asked it questions about the program with out letting it look at any files)
What Toaster Loop is
A single Python tool (toaster-loop) that runs the whole loop on one machine, over loopback only. Zero API keys, zero cloud calls. and sorry I know the frontend needs help
┌───────────────────────────── the loop ─────────────────────────────┐
run tasks ─▶ trace buffer ─▶ curate ─▶ train (LoRA SFT/DPO/GRPO) ─▶ export GGUF ─▶ validate
▲ │ pass
│ ▼
└───── serve --auto / hot-swap deploy ── active model ── promote ──────┘
(fail → previous model stays live; models rollback anytime) First time using
download a models gguf and safetensor version (these get tied together for running and training)
set up a serve instance with the model then setup an agent to tie to the instance, all settings are opt in but if you are new and don’t know all the options there is an enable all recommendations at the top of the options page to get a good base without setting everything,
there is more then one way to train the easiest is easy train, you can select an agent and it will set training base lines and when it has 50 good traces it will train and hot swap the model.
Home lab angle (localist) This is the “everyday useful thing” category in spirit: your box, your models, your data, your dashboards. Loopback-only by construction (bind address hardcoded to 127.0.0.1), token-auth on every API route with CSRF/DNS-rebinding defenses, and it’s Prometheus-scrapable so toaster-loop shows up next to Uptime Kuma and Home Assistant in your Grafana.
The side quest: Toaster Engine (C++, not public yet) llama.cpp is great, but I want things it fundamentally can’t do: run MOE models larger than VRAM+RAM by streaming experts at expert granularity from NVMe, locally upcycle dense models to MOE, and do it on AMD/Intel/NVIDIA with a Vulkan-only backend (no CUDA, no ROCm — cross-vendor or bust). So I’m writing the engine from scratch: one C++ shared library behind a stable C ABI, descriptor-driven model support (“support = data, not code”), CPU-tier computing cold experts concurrently with the GPU.
Status, honestly: design phase complete, and just started coding**.** Ten spec documents (reference-landscape autopsy of DeepSpeed/Muon/KTransformers/etc., Vulkan feasibility verdict, 22 MOE families matrix, memory-hierarchy math, upcycling staged plan, training stack, foundation/runtime/API architecture, roadmap with spikes S1–S8 and a risk register). Devember is where the first spike — measuring real per-expert sizes to settle the slot-sizing math — actually runs. I’ll post the specs’ honest performance projections (including the deliberately negative rows, like “Kimi-K3 on a 4070: correctness gate only, never a throughput gate”) as the build log starts.
Links
Feedback wanted any and all, especially: What home lab integration would make this actually earn its electricity in your lab?