What 586 Real Traces Taught Us About LLM Agent Determinism

A developer ran 586 traces of a LangGraph folder-research agent against AgentPulse, a metadata-only observability platform, using qwen2.5:7b on an RTX 4060 and a Pixel 8 Pro running a 3B model via llama.cpp in Termux. At temperature 0, trace shapes were nearly identical across 20 runs per question (standard deviation 0.0 on span counts for nine of ten questions), while higher temperatures selectively degraded multi-file synthesis questions before simple lookups. The team found mean-based regression detection produced false positives on bimodal traces and built a median/IQR-based quality-warning evidence adapter, published on GitHub.

How temperature selectively degrades agent reasoning — and what that means for observability. On November 3, 2026, we ran 586 traces of a LangGraph folder-research agent against AgentPulse, our metadata-only observability platform. The agent used Ollama with qwen2.5:7b on an RTX 4060 Legion , plus 6 traces from a Pixel 8 Pro running qwen3b via llama.cpp in Termux. The experiment: same 10 questions, same corpus, four temperatures 0, 0.3, 0.7, 1.0 . We measured trace shapes — span counts, durations, tool calls — to understand how sampling temperature affects agent behavior. Why this matters: If you're building observability for AI agents, you need to know what "normal" looks like. Without a determinism baseline, every alert is a guess. At temp=0, qwen2.5:7b produced identical trace shapes across 20 runs per question. Nine out of ten questions had standard deviation 0.0 on span counts. The tenth Q1 had sd 0.4 — still remarkably tight. | Question | Spans temp=0 | SD | |---|---|---| | Q2-Q9 except Q1 | 6-6, 8-8, 2-2, 11-11 | 0.0 | | Q1 | 8-10 | 0.4 | This is the baseline. When temp=0, trace-shape changes are genuine regression signal , not noise. Your alert thresholds can be tight. Here's what surprised us. Temperature doesn't add uniform noise. It selectively degrades complex reasoning while simple lookups survive. At temp=0.3: At temp=1.0: The pattern: Multi-file synthesis questions degrade at 0.3. Simple lookups survive to 1.0. The breakdown is question-dependent , not just temperature-dependent. Implication: Your eval thresholds can't just be "wider at higher temp." They need to be per-question-type. A simple lookup at temp=0.7 should still be deterministic. A complex reasoning task at temp=0.3 is already noisy. Our first eval prototype used mean comparison. It got 7/10 "correct" — but the 3 false positives were instructive. At temp=0.7, traces become bimodal : sometimes the agent does full work 8 spans , sometimes it produces a degenerate 2-span trace. The mean shifts, triggering a "regression" alert. But the median and IQR tell the real story: high variance with overlapping IQR = noise , not regression. The rule: We validated this on 500 traces. Mean-based detection fails. Median/IQR works. We validated AgentPulse across three very different machines: The phone was the hard one. We fought through: But it works. The phone agent runs, generates traces, and uploads them to the same workspace. Metadata-only observability works on mobile. From these findings, we built the quality-warning evidence adapter — the statistical engine that connects real trace data to AgentPulse's alerting system. Core logic: The adapter is live on GitHub codex/quality-warning-evidence, commit 3df33f0 . The API integration follows. If you're building agent observability: Temperature doesn't just make agents "more random." It selectively kills complex reasoning first. Your monitoring needs to know the difference. 586 traces. 3 devices. 1 night. The data is real, the findings are publishable, and the code is on GitHub.