# Hermes Agent Needs a Flight Recorder - So I Built One

> Source: <https://dev.to/ale007xd/hermes-agent-needs-a-flight-recorder-so-i-built-one-3gea>
> Published: 2026-05-29 11:01:26+00:00

*This is a submission for the Hermes Agent Challenge*

Autonomous agents can now write code, call tools, browse the web, mutate files, and delegate to subagents. But when they fail, they fail invisibly.

"An agent ran overnight, caught an unhandled exception loop, and burned $50 in tokens while corrupting our staging database."

If you've spent more than a week building production systems with autonomous agents, you've lived some version of this nightmare.

Most agent runtimes don't crash cleanly. They slide into retry storms, silently ignore failed tool calls, or recurse through delegation loops until budgets evaporate.

Airplanes have flight recorders. Distributed systems have OpenTelemetry. **Autonomous agents need TraceGuard.**

**TraceGuard** is a lightweight Python library and CLI that acts as an isolated, non-invasive execution flight recorder for autonomous agent runtimes.

It consumes append-only JSONL execution traces and detects the three silent killers of agentic workflows:

```
traceguard traces/my_agent_run.jsonl --strict
# exit 0 = clean · exit 1 = WARN · exit 2 = CRITICAL
```

Instead of scraping human-readable terminal logs, TraceGuard turns runtime execution into a structured, replayable execution event contract.

**GitHub:** [https://github.com/Ale007XD/traceguard](https://github.com/Ale007XD/traceguard)

Modern agent frameworks can browse the web, write files, execute shell commands, and coordinate sub-agents. But when something goes wrong, you're usually left with a giant wall of terminal output and one impossible question:

**What actually happened?**

Not what the LLM said. Not the final output. The actual execution state:

Distributed systems engineers solved these problems decades ago using structured traces, append-only logs, and replayable execution histories. Agent runtimes are now complex enough to require the same discipline.

Autonomous agents are stochastic distributed runtimes.

| Distributed System Failure | Agent Equivalent | Observability Primitive |
|---|---|---|
| Retry storm | Same tool called repeatedly without progress | Sliding window counter over event stream |
| Silent failure | Tool fails, agent continues anyway | Error propagation trace |
| Circular dependency | Agent A delegates to B which delegates back to A | Delegation cycle detection |
| State divergence | Agent acts on corrupted or stale state | Replayable transition history |

```
Agent Runtime
      │
      ▼
Append-Only Event Stream
      │
      ▼
  TraceGuard
      │
  ┌───┴───────┬──────────────┐
  ▼           ▼              ▼
Retry      Silent       Recursive
Storms    Failures     Delegation
```

Every execution step becomes a formal state transition. The runtime stops being an opaque, ephemeral process and becomes a replayable execution artifact.

Hermes Agent currently exposes beautiful terminal output optimized for humans. Production observability requires something fundamentally different: machine-readable execution semantics.

Example event:

```
{
  "event_id": "3f8a1c2d-...",
  "session_id": "hermes-session-001",
  "timestamp": "2026-05-29T10:00:00.050Z",
  "schema_version": "1.0",
  "type": "tool_call",
  "tool_name": "bash",
  "tool_args": {
    "command": "git status --porcelain"
  }
}
```

Each event is:

The missing primitive is not another dashboard. It is a structured execution event stream.

Detects identical tool invocations repeating without successful progress.

**Example:** `bash → fail`

→ `bash → fail`

→ `bash → fail`

(retry storm)

Detects agents continuing execution after failed or empty tool outputs.

**Example:** `read_file → empty`

→ `continue execution`

(silent corruption)

Detects sub-agent delegation cycles and self-recursion.

**Example:** `planner → coder → coder → planner`

(recursive loop)

Each detector operates independently over the same append-only event stream. Multiple detectors can fire simultaneously on the same execution trace.

TraceGuard is intentionally designed as an external execution observer.

```
LLM proposes
      │
      ▼
Runtime executes
      │
      ▼
TraceGuard observes
      │
      ▼
Governance layer enforces invariants
```

This is the critical distinction. Prompt engineering cannot reliably solve retry storms, hidden execution corruption, or delegation cycles. Prompt-layer control is insufficient. **Execution-layer governance is required.**

`schema.py`

)`recorder.py`

)`detectors.py`

)`guard.py`

)The core invariant is simple: **Record every transition. Analyze the record.**

Once execution becomes replayable, agent runtimes stop behaving like black boxes.

Hermes Agent currently produces terminal output optimized for human inspection. TraceGuard proposes a complementary execution event contract — a machine-readable stream of typed, versioned, append-only events emitted alongside the human-readable output.

This aligns with the discussion in [issue #169](https://github.com/NousResearch/hermes-agent/issues/169) on structured execution semantics.

The integration path is additive: TraceGuard requires no changes to Hermes internals. Emit events to a JSONL file; TraceGuard reads them externally.

``` bash
$ traceguard traces/retry_storm.jsonl
[WARN] RetryStormDetector: tool 'bash' called 4 times without success (threshold=3)
[WARN] SilentFailureDetector: step 2 failed, execution continued without error handling
[WARN] SilentFailureDetector: step 4 failed, execution continued without error handling
[WARN] SilentFailureDetector: step 6 failed, execution continued without error handling
[WARN] SilentFailureDetector: step 7 failed, execution continued without error handling

$ traceguard traces/recursive_delegation.jsonl
[CRITICAL] RecursiveDelegationDetector: delegation cycle detected — planner → coder → planner

$ traceguard traces/clean.jsonl
✓ No anomalies detected.

$ traceguard traces/retry_storm.jsonl --strict; echo "exit: $?"
exit: 1
python
from traceguard import TraceGuard

guard = TraceGuard()
report = guard.analyze("traces/my_agent_run.jsonl")

for anomaly in report.anomalies:
    print(f"[{anomaly.severity}] {anomaly.detector}: {anomaly.message}")

if report.is_clean:
    print("✓ No anomalies detected.")
```

`frozen=True`

event modelsNo external runtime dependencies. No framework lock-in.

TraceGuard was developed and iterated with Hermes Agent as the primary development environment — reading files, applying patches, running tests, and diagnosing failures through FSM-structured execution loops.

The irony is deliberate: a tool for governing agent execution traces was built by an agent whose execution was governed by the same FSM principles.

Hermes drove: reading source files → generating S&R patches → applying changes → running pytest → diagnosing failures → iterating.

Most failures in autonomous systems are not model failures. They are execution failures:

The model is usually doing exactly what it was asked to do. The runtime simply lacks governance.

"LLMs propose. Runtimes govern."

TraceGuard is to autonomous agents what OpenTelemetry became for distributed systems.

Built for the **Hermes Agent Challenge 2026**.

**Repository:** [https://github.com/Ale007XD/traceguard](https://github.com/Ale007XD/traceguard)

Built on [llm-nano-vm](https://github.com/Ale007XD/nano_vm) — deterministic FSM execution infrastructure.