# I Was Running 3 AI Coding Agents Locally and Had No Idea What They Were Breaking > Source: > Published: 2026-09-10 13:51:50+00:00 Like many developers over the past few months, my daily workflow quietly shifted from typing code manually to orchestrating AI coding agents directly from the command line. On any given Tuesday, I might have **Claude Code** refactoring an API endpoint in one terminal pane, **Kimi Code** writing benchmark scripts in another, and **Google Antigravity (Agy CLI)** running multi-agent tasks in an isolated git worktree. They were getting work done. But after a couple of weeks, I had that familiar uneasy feeling every engineer gets when running services without an APM: `1` and rewrite the code 4 times before I noticed? None of these CLIs talk to each other. None of them give you an aggregated dashboard. And worst of all, checking external cloud provider billing pages only shows aggregate monthly dollar amounts with a 6-hour delay—useless for real-time debugging. So I dug into what was actually happening on my machine, and what I found led me to build **[GroundControlAI](https://github.com/varunrai/GroundControlAI)**: a zero-overhead, real-time observability daemon and cyber-cockpit for local AI agents. Here is what I learned from reverse-engineering the local log streams, and how you can track your own agents. The first surprise was that these agent tools are already recording rich telemetry locally. They just dump it into scattered, format-incompatible log files: `~/.claude/projects/**/*.jsonl`): Logs exact prompt tokens, completion tokens, cache-creation tokens, cache-read tokens, reasoning tokens, and tool invocations (`Bash`, `Edit`, `Write`).`~/.kimi-code/sessions/**/wire.jsonl`): Records precise Time-to-First-Token (TTFT) latency in milliseconds, stream decode durations, background task exit codes, and process IDs.`~/.gemini/antigravity-cli/brain/**/*.jsonl`): Records multi-agent hierarchy graphs, planner deliberation thinking blocks, and subagent delegation handoffs. The data was already there. It just needed an ingestion engine that didn't murder my CPU with heavy polling loops. Once I started aggregating this stream into a central SQLite database, three patterns immediately jumped out. Modern models (like Claude 3.5 Sonnet) offer huge discounts for cached prompt tokens—down from $3.00/1M to $0.30/1M (a **90% discount**). When an agent stays within the cache lifetime window, multi-turn coding is remarkably cheap. But the moment you trigger a cache miss (e.g., passing a slightly modified system prompt or letting the 5-minute TTL expire), that 180,000-token repo context is re-billed at full price. Seeing a live **Cache Savings %** counter on the screen completely changed how I interact with agents. I stopped killing and restarting sessions unnecessarily because I could literally see the cache hits saving $15–$25 per working session. Have you ever seen an agent show `Thinking...` or `Deliberating...` for two minutes straight? Often, it's not thinking. It ran a bash command like `pytest tests/`, hit an `ImportError` (exit code `1`), tried to fix it with an invalid argument, hit exit code `2`, and is quietly spiraling in a retry loop. By capturing `tool_executions` with their raw commands and exit codes, GroundControlAI exposes a real-time SRE stream. If you see a cluster of red `[EXIT 1]` pills, you know immediately that the agent is stuck in a circular loop and you can intervene before it burns 200,000 tokens on hallucinated fixes. ``` 14:32:01 ▶ TOOL: [Bash] `pytest tests/test_normalizer.py` [STATUS: EXIT 1] 14:32:05 ▶ TOOL: [Edit] `normalizer.py` (Line 14-38) [STATUS: SUCCESS] 14:32:08 ▶ TOOL: [Bash] `pytest tests/test_normalizer.py` [STATUS: EXIT 0] ``` When human developers edit code, they touch a file 1 or 2 times per task. When an agent edits the same file 12 times in 10 minutes, that file is a **hotspot** indicating brittle logic, failing test assertions, or conflicting instructions. Tracking file operations (`read`, `edit`, `write`) per repository reveals instant code smells. A telemetry tool that burns 15% CPU polling log files defeats the purpose of running lightweight local tools. GroundControlAI uses the Linux kernel's `inotify` subsystem via Python's `watchdog`. Instead of scanning files or parsing entire 50MB `.jsonl` transcripts on every disk write, the collector daemon maintains an in-memory dictionary of byte offsets (`f.tell()`): ``` # collector.py (simplified concept) file_offsets = {} def process_file_incrementally(file_path): last_offset = file_offsets.get(file_path, 0) with open(file_path, "r", encoding="utf-8") as f: f.seek(last_offset) new_lines = f.readlines() file_offsets[file_path] = f.tell() # Save offset for next turn for line in new_lines: parse_and_insert_turn(line) ``` Because it only seeks and reads the newly appended bytes, turn ingestion takes **under 1 millisecond** and idles at **0.00% CPU**. The data lands in an SQLite database running in **WAL (Write-Ahead Logging)** mode, allowing the collector daemon to write continuously while the FastAPI web server reads concurrently without lock contention. ``` [Agent Log Files] (Claude, Kimi, Agy) │ (inotify kernel events) ▼ [Collector Daemon] ──(f.tell() incremental bytes) │ ├──> [Path Normalizer] (auto-resolves worktrees & mono-repos) ├──> [Pricing Engine] (calculates tokens & 90% cache discounts) │ ▼ [SQLite DB (WAL Mode)] │ (live concurrent read) ▼ [FastAPI Cyber-Cockpit] ──> [http://localhost:8080] ``` *(You can explore the full interactive architecture diagram online at **[varunrai.github.io/GroundControlAI](https://varunrai.github.io/GroundControlAI/)**).* One subtle issue with modern agents (especially Google Antigravity / Agy CLI) is that they spawn isolated git worktrees for tasks (`~/.ao/data/worktrees/my-repo/subagent-1`). If your dashboard keys projects by raw working directory, your single repository ends up split into 15 disjointed "projects" in the UI. To solve this without hardcoded directory names, GroundControlAI includes a dynamic path normalizer. It inspects the directory structure: `.git` is a worktree file, it reads the `gitdir: ` pointer and resolves the main parent repository directly.`apps/web`, `infrastructure/terraform`), it rolls it up to the canonical project root. The result is a clean, unified view per repository regardless of how many worktrees or subagents were spawned. GroundControlAI is fully open-source (MIT License) and packaged into a Docker Compose stack that mounts your local agent directories read-only: ``` git clone https://github.com/varunrai/GroundControlAI.git cd GroundControlAI docker compose up -d ``` Navigate to **[http://localhost:8080](http://localhost:8080)** in your browser. The collector immediately scans existing logs to hydrate your historical stats, then transitions into live kernel-watching mode for any new agent turns. AI coding assistants are no longer just fancy autocomplete—they are autonomous junior engineers running terminal commands, editing files, and making API calls. They deserve the same observability standards we apply to production backends. If you're running Claude Code, Kimi Code, or Agy CLI, give it a spin: PRs and new agent log parser contributions are very welcome!