# What 586 Real Traces Taught Us About LLM Agent Determinism

> Source: <https://dev.to/vishwajitsinh_mahida_2547/what-586-real-traces-taught-us-about-llm-agent-determinism-30cj>
> Published: 2026-11-04 03:41:00+00:00

*How temperature selectively degrades agent reasoning — and what that means for observability.*

On November 3, 2026, we ran 586 traces of a LangGraph folder-research agent against AgentPulse, our metadata-only observability platform. The agent used Ollama with qwen2.5:7b on an RTX 4060 (Legion), plus 6 traces from a Pixel 8 Pro running qwen3b via llama.cpp in Termux.

The experiment: same 10 questions, same corpus, four temperatures (0, 0.3, 0.7, 1.0). We measured trace shapes — span counts, durations, tool calls — to understand how sampling temperature affects agent behavior.

**Why this matters:** If you're building observability for AI agents, you need to know what "normal" looks like. Without a determinism baseline, every alert is a guess.

At temp=0, qwen2.5:7b produced **identical trace shapes** across 20 runs per question. Nine out of ten questions had standard deviation 0.0 on span counts. The tenth (Q1) had sd 0.4 — still remarkably tight.

| Question | Spans (temp=0) | SD | 
|---|---|---|
| Q2-Q9 (except Q1) | 6-6, 8-8, 2-2, 11-11 | 0.0 | 
| Q1 | 8-10 | 0.4 | 

This is the baseline. When temp=0, trace-shape changes are **genuine regression signal**, not noise. Your alert thresholds can be tight.

Here's what surprised us. Temperature doesn't add uniform noise. It **selectively degrades** complex reasoning while simple lookups survive.

At temp=0.3:

At temp=1.0:

**The pattern:** Multi-file synthesis questions degrade at 0.3. Simple lookups survive to 1.0. The breakdown is **question-dependent**, not just temperature-dependent.

**Implication:** Your eval thresholds can't just be "wider at higher temp." They need to be per-question-type. A simple lookup at temp=0.7 should still be deterministic. A complex reasoning task at temp=0.3 is already noisy.

Our first eval prototype used mean comparison. It got 7/10 "correct" — but the 3 false positives were instructive.

At temp=0.7, traces become **bimodal**: sometimes the agent does full work (8 spans), sometimes it produces a degenerate 2-span trace. The mean shifts, triggering a "regression" alert. But the median and IQR tell the real story: high variance with overlapping IQR = **noise**, not regression.

**The rule:**

We validated this on 500 traces. Mean-based detection fails. Median/IQR works.

We validated AgentPulse across three very different machines:

The phone was the hard one. We fought through:

But it works. The phone agent runs, generates traces, and uploads them to the same workspace. **Metadata-only observability works on mobile.**

From these findings, we built the **quality-warning evidence adapter** — the statistical engine that connects real trace data to AgentPulse's alerting system.

**Core logic:**

The adapter is live on GitHub (codex/quality-warning-evidence, commit 3df33f0). The API integration follows.

If you're building agent observability:

Temperature doesn't just make agents "more random." It selectively kills complex reasoning first. Your monitoring needs to know the difference.

*586 traces. 3 devices. 1 night. The data is real, the findings are publishable, and the code is on GitHub.*