{"slug": "i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-tool", "title": "I got tired of not knowing what my AI agents were doing, so I built a tiny observability tool", "summary": "A developer built Otterscope, a lightweight observability tool for AI agents that runs as a single Go binary with SQLite, avoiding the complexity of hosted or multi-service solutions. The tool captures OpenTelemetry traces, displays agent runs with LLM and tool call details, token counts, and cost estimates, and integrates evaluations directly into production traces.", "body_md": "I build small LLM agents. Not the impressive kind you see in demos, just\n\npractical little things that answer support questions or dig through some\n\ndocs and come back with an answer. And for months the most annoying part of\n\nthat wasn't the model or the prompt. It was that I had no idea what the thing\n\nwas actually doing.\n\nYou know the feeling. A run takes nine seconds and you don't know why. It\n\ncalled a tool, then called it again, then a third time, and somewhere in\n\nthere it spent forty cents. The logs are a wall of JSON. You tweak a prompt,\n\nship it, and you genuinely can't tell if you made it better or worse until a\n\nuser complains. I'd been squinting at `print()`\n\noutput like it was 2015.\n\nSo I went looking for something to just show me my agent runs. And the tools\n\nexist — they're fine, some are really good — but they all wanted something I\n\ndidn't want to give.\n\nThe hosted ones want your traces on their servers. Which means your prompts,\n\nwhich for me means actual customer messages, sitting in someone else's\n\ndatabase so I can look at a dashboard. No thanks.\n\nThe self-hostable ones are built for companies, and it shows. The one\n\neverybody recommends needs Postgres *and* ClickHouse *and* Redis *and* an S3\n\nbucket to run. Four stateful services. I'm logging maybe a few thousand model\n\ncalls a day on a side project. Standing up a ClickHouse cluster to look at\n\nthat is like renting a forklift to carry a bag of groceries.\n\nI sat with that for a bit and realized the mismatch was the whole problem.\n\nThis isn't company-scale data. It's my-laptop-scale data. And the moment you\n\naccept that, the design gets really boring in a good way: it's just a web app\n\nwith a SQLite file. That's it. One process.\n\nSo I built that. It's called Otterscope.\n\nIt's one Go binary. You run it, it opens a SQLite file, and it listens for\n\nOpenTelemetry traces on the standard port. You point your agent at it with\n\none environment variable — the same one every OpenTelemetry setup already\n\nuses — and your runs start showing up. No account, no config file, no other\n\nservices to babysit.\n\nThe thing I cared most about getting right was the *shape* of the data.\n\nEveryone else shows you spans, which is the raw OpenTelemetry unit, and it\n\nreads like a stack trace. I wanted runs. One agent execution is one run, and\n\ninside it you see the steps: this LLM call, that tool call, the loop where it\n\ncalled the same tool three times. Click into a call and you read the real\n\nmessages that went in and came back out, with the token counts and the cost\n\nsitting right there. When something goes sideways, you can actually see where.\n\nThere's a cost table that knows the current prices for the major providers,\n\nso a run tells you what it cost instead of just how many tokens it burned. If\n\nit hits a model I don't have a price for, it shows you the tokens and stays\n\nhonest about not knowing the dollar amount, which matters more than it\n\nsounds — I'd rather see a blank than a made-up number.\n\nAnd then there's the part I'm most attached to: the evals live in the same\n\nplace as the traces. You write a check (\"the answer should never contain 'I\n\ncan't help with that'\") or point an LLM at the run as a judge, and the result\n\ngets stamped onto the actual production run. Most tools make evals a separate\n\nproduct you run against a separate dataset. Here it's just a column on the\n\nruns you already have. There's a compare view too, so \"did my prompt change\n\nmake last week worse than the week before\" is a question you can actually\n\nanswer by looking, instead of by vibes.\n\nIf you build one of these, here's the thing nobody warns you about: there is\n\nno single format for AI traces. There are at least three, all live in the\n\nwild right now, and they disagree.\n\nOpenTelemetry has a \"GenAI\" convention, except it's mid-migration, so there's\n\nan old attribute layout and a new one and both are being emitted by real\n\ntools today. Then Arize's OpenInference is its own thing, and it's what the\n\nOpenAI Agents SDK and a bunch of the LangChain world speak. Then the Vercel\n\nAI SDK does its own `ai.*`\n\nattributes, and — this is the fun one — its spans\n\nmix the old and new OpenTelemetry styles together in the same span, so if you\n\nguess based on one attribute you get it wrong.\n\nSo a big chunk of Otterscope is just a translation layer that eats all of\n\nthese and turns them into one clean model. It's the least glamorous code in\n\nthe project and probably the most valuable, because it means you don't have\n\nto care which dialect your framework speaks. And because I keep the raw\n\npayload of everything that comes in, when I improve that translation later,\n\nyou can replay your old data through the new version and it just gets better\n\nretroactively. That felt worth doing.\n\nIt's single-user right now. There's no login, no teams, no roles. That's on\n\npurpose — I built the thing I needed, and I'm one person. It binds to\n\nlocalhost by default so you're not accidentally exposing it, and if you want\n\na team version with real auth, that's a \"when someone actually asks\" feature,\n\nnot a \"build it on spec because a spreadsheet said B2B\" feature.\n\nIt's also young. It'll have rough edges. But it does the thing I wanted, which\n\nwas to stop flying blind.\n\nThe fastest way is Docker:\n\n```\ndocker run -p 8317:8317 -p 4318:4318 -v otterscope:/data ghcr.io/otterscope/otterscope\n```\n\nThen point any OpenTelemetry-instrumented agent at `http://localhost:4318`\n\nand open `http://localhost:8317`\n\n. There's a `sample`\n\ncommand that fills it\n\nwith fake runs if you just want to click around first. The repo has setup\n\nsnippets for Pydantic AI, the OpenAI Agents SDK, LangGraph, and the Vercel AI\n\nSDK.\n\nRepo's here: [https://github.com/otterscope](https://github.com/otterscope)\n\nIf you'd rather run it on a server than on your laptop, I wrote up the whole\n\nthing on a small VPS separately — creating the box, Docker vs a systemd\n\nservice, and how to keep it private with an SSH tunnel or lock it down with a\n\nfirewall: ** Self-hosting observability for your AI agents on a $6**.\n\nBecause the tool's whole personality is \"small and predictable,\" and that's\n\nwhat I want from the box it runs on too.\n\nA $6 Droplet runs this comfortably. There's no managed database bill on top,\n\nbecause the database is a file — I back it up by copying the file or\n\nsnapshotting the Droplet, and I'm done. The pricing is flat and I can do the\n\nmath in my head, which matters when the whole point of the project is *not*\n\nhaving a surprise infrastructure bill for a side project. I don't want to\n\nfind out at the end of the month that some egress line item ate the savings.\n\nI've also just had a good time on it. The docs are readable, the Droplet\n\ncomes up in under a minute, and the Marketplace Docker image means \"spin up a\n\nbox that can run a container\" is a checkbox rather than an afternoon. For\n\nsomething like this — one binary, one file, one small server sitting next to\n\nyour agents — it fits the shape of the thing. That's really the whole reason.\n\nAnyway. If you've been squinting at agent logs, go self-host something and\n\nstop. It doesn't have to be mine, but it turns out it really doesn't need to\n\nbe complicated.", "url": "https://wpnews.pro/news/i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-tool", "canonical_source": "https://dev.to/remdore/i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-observability-tool-3p67", "published_at": "2026-07-16 16:51:03+00:00", "updated_at": "2026-07-16 17:05:41.524111+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure", "mlops", "large-language-models"], "entities": ["Otterscope", "OpenTelemetry", "SQLite", "Go", "Arize", "OpenInference", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-tool", "markdown": "https://wpnews.pro/news/i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-tool.md", "text": "https://wpnews.pro/news/i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-tool.txt", "jsonld": "https://wpnews.pro/news/i-got-tired-of-not-knowing-what-my-ai-agents-were-doing-so-i-built-a-tiny-tool.jsonld"}}