Speculative tool execution guesses the next tool while the model is still reasoning, runs it in parallel, and hides the latency — discarding the guess when it's wrong.
TL;DR: An agent's think→act→observe loop is serial, so it burns a huge share of wall-clock sitting idle while a tool runs. Speculative tool execution borrows branch prediction from CPUs: a cheap predictor guesses the next tool call and runs it during the model's reasoning. If the guess matches, the result is already there and the wait disappears; if not, it's discarded and the real call runs — never a wrong answer. A confidence gate avoids wasting compute on low-odds guesses. In a runnable Go demo, a 58% hit rate cut wall-clock by 1.3× with zero correctness impact. Four 2026 papers (Speculative Actions, SPORK, PASTE, Speculate While You Reason) report 20–48% real speedups.
Mental model: a chef who starts searing the steak they're pretty sure you'll order while you're still reading the menu. If you order it, dinner is early. If you don't, they toss it and cook what you asked — you never get served the wrong dish, they just did some work in the background.
Agents run a strictly serial loop: reason, emit a tool call, wait for the result, reason again. That wait — a web request, a database query, a code execution — is dead time. The GPU (or your API budget) idles while the tool runs. Prior work measures this idle at 16–37% of wall-clock in typical workloads and up to 61% in tool-heavy ones. For a chatty, multi-step agent, that's the single biggest lever on latency, and no amount of prompt tuning touches it.
The catch is that the loop is only logically serial. The model doesn't strictly need to finish reasoning before the tool runs — it just needs the tool's result to be there when it's done.
CPUs solved this with branch prediction: guess which way a branch goes and execute ahead speculatively, rolling back if wrong. LLM inference reuses the idea in speculative decoding. Applied to agents it becomes speculative tool execution:
The correctness guarantee is the whole point: the agent's actual output is always the ground truth. Speculation can only ever remove waiting, never change the answer.
A learned predictor over recurring tool patterns:
func (p *predictor) predict(last string) (guess string, confidence float64) {
row := p.counts[last]
best, bestN, total := "", 0, 0
for tool, n := range row {
total += n
if n > bestN { best, bestN = tool, n }
}
return best, float64(bestN) / float64(total)
}
And the speculate-while-thinking core — the guessed tool runs in a goroutine, overlapping the model's reasoning; a confidence gate keeps us from speculating when the odds are poor:
guess, conf := pred.predict(last)
if guess != "" && conf >= gate {
specCh = make(chan string, 1)
go func(t string) { specCh <- runTool(t) }(guess) // runs in parallel with think
}
time.Sleep(thinkMS * time.Millisecond) // the model reasons
if specCh != nil && guess == actual {
<-specCh // correct: result already ready — latency hidden
hits++
} else {
runTool(actual) // miss: discard the speculation, run the real tool
}
Speculative Tool Execution — guess the next tool and run it while the model thinks
24-step agent loop; think=40ms, tool=60ms per step.
serial agent loop 2.443s (think, then wait for the tool, every step)
speculative loop 1.869s (14 hits / 4 misses / 1 gated out)
next-tool hit rate 58%
wall-clock speedup 1.31x (tool latency hidden behind reasoning)
wasted speculations 4 (discarded on a miss — never a wrong answer)
The four misses cost some throwaway background work; the one gated-out step shows the confidence gate declining to gamble. The 14 hits each hid a tool call behind reasoning that was going to happen anyway. Correctness is untouched — every actual tool call still ran (or was proven already run).
Reality check: the wall-clock here comes from fixed sleeps (and drifts a little run-to-run), so read the hit rate and the mechanism, not the exact milliseconds. The direction is real: the 2026 papers report 20–48% speedups on live agents — SPORK predicts the next tool at 74–99% accuracy, PASTE cuts task-completion time ~48% — and your gain scales with how predictable your tool sequences are and how much of each step is actually tool latency.
This jumped from theory to a small research wave in 2026, and the designs converge on the same shape. SPORK forks a probe off the agent's own KV cache to predict the next tool with 74–99% accuracy and gates dispatch on confidence — exactly the gate-and-discard structure above. PASTE mines recurring tool-call patterns (the assumption this demo's predictor encodes) and reports up to 48% lower task completion time. Speculative Actions frames it generally: a fast model stages likely actions so that validation, not waiting, is the critical path. And Speculate While You Reason shows the agent is its own best predictor.
The transferable engineering idea needs no training: any time your agent has predictable structure and idle wait, you can overlap them. Recurring tool sequences, known data-fetch fan-outs, and retry-then-reopen patterns are all speculatable today with a controller sitting over a normal completion API.
It models control flow and timing, not reasoning: "thinking" and "tool latency" are fixed sleeps and the predictor is a first-order Markov count over tool names — real speculators also predict tool arguments (much harder) and pay for a draft model. Two caveats the papers stress: speculation only helps when tool latency is a real slice of step time, and mispredicted side effects must be safe to discard (a read-only call is free to speculate; a charge_customer
is not). Start by speculating idempotent, read-only tools on your most predictable loops.
charge_customer
or any write you can't cleanly roll back.
go run . # standard library only
Papers (2026)
Background