What Is LLM Actually Doing? A Fellow Engineer's Take on Vectors, Next-Token Prediction, and Fail-back Routing

A developer detailed how large language models function as probability engines rather than calculators, explaining vector embeddings and next-token prediction, and shared a Go-based fail-back routing pattern used in NEXT4I's AI layer to improve reliability by trying multiple models in sequence.

Why treating an LLM as a probability engine, not a brain, changes how you architect around it. The reasoning behind NEXT4I's AI layer. LLM BuildinPublic SystemArchitecture AI Model AI AI Router AI Stable I used to wonder why the model confidently gives you a wrong number. It's not a bug in the traditional sense, it's the model doing exactly what it's built to do: predicting the next token from probability, with zero actual arithmetic happening underneath. TLDR; An LLM large language model converts text into vectors numeric coordinates in a high-dimensional meaning-space and generates output via next-token prediction, sampled with parameters like top-k and temperature. Because it's fundamentally a probability engine and not a calculator or a database, I designed NEXT4I with automatic model fail-back routing and task-based model selection instead of trusting any single model as a source of truth. Every token gets embedded into a vector, often with hundreds or thousands of dimensions. Semantically similar tokens end up close together in that space. That's why semantic search vector-based retrieval can match "large flying animal consumes insects" to "big bird eats worms" even with zero shared keywords, unlike old-school lexical search TF-IDF/BM25 which needs literal term overlap. GPUs handle this well because they're already wired for massive parallel floating-point math the same math used to shade millions of pixels per frame , so throwing billions of similarly-directed vectors at a GPU is a natural fit, not a coincidence. Given a prompt, the model doesn't look up an answer, it samples one token at a time from a probability distribution. Two knobs matter in practice: top k: 2 only sample from the top-2 most likely next tokens temperature: 0.2 low = deterministic/precise, high = creative/varied Low temperature + low top k gives you consistent, "boring" output, good for structured extraction. High temperature gives you variety, good for brainstorming, bad for anything requiring precision. There's no calculator inside the model. It doesn't evaluate x y , it predicts digits that are statistically plausible given the prompt, one token at a time. It gets 2 2 right because that pattern is everywhere in training data. It confidently botches large multiplication because it's still just sampling digits, not computing. This is exactly why production systems now delegate real math to a tool call a Python sandbox, a calculator function instead of trusting raw model output. Here's the simple pattern, stripped of any specific business logic, a reusable fail-back wrapper for any set of same-tier model clients: package modelrouter import "context" "errors" "fmt" // ModelClient is any backend that can answer a prompt. type ModelClient interface { Name string Complete ctx context.Context, prompt string string, error } // TieredRouter tries each client in a tier in order until one succeeds. type TieredRouter struct { tier ModelClient } func NewTieredRouter clients ...ModelClient TieredRouter { return &TieredRouter{tier: clients} } // Complete attempts each model in the tier, fail-back on error. func r TieredRouter Complete ctx context.Context, prompt string string, error { var errs error for , client := range r.tier { resp, err := client.Complete ctx, prompt if err == nil { return resp, nil } errs = append errs, fmt.Errorf "%s: %w", client.Name , err } return "", errors.Join errs... } This is intentionally boring: try the next model in the same tier on failure, return the first success. No retries with backoff yet, no circuit breaker, just the core fail-back idea. In NEXT4I's actual implementation, tiers are populated dynamically and health state feeds back into ordering, but that logic is abstracted here on purpose, the generic version above is what's actually useful to share. A minimal router that inspects task complexity before picking a tier: package modelrouter type Complexity int const Simple Complexity = iota Complex func ClassifyAndRoute task string, simpleTier, complexTier TieredRouter TieredRouter { if estimateComplexity task == Simple { return simpleTier } return complexTier } func estimateComplexity task string Complexity { if len task < 100 { return Simple } return Complex } estimateComplexity can be as crude as a length/keyword heuristic or as sophisticated as a small classifier model, the point is the routing decision happens before the expensive call, not after. If there's one thing worth taking away: treat the model as a probability engine you can't fully trust, and let the system around it, fail-back, routing, tool calls for math, carry the reliability burden instead. Thanks for reading all the way to the end, I'll keep working on more articles like this. Explore the NEXT4I journey and read the original article at: https://go.next4i.com/next4i/journey/en https://go.next4i.com/next4i/journey/en