{"slug": "i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map", "title": "I pentested my own AI hub and shipped the method, not the map", "summary": "A developer who runs an observability hub for AI-assisted coding conducted a read-only penetration test of the stack, revealing that nearly all serious security defects were in recently written controls. The audit found that scope attributes bypassed allow-lists in both the OpenTelemetry Collector and Loki, leaking sensitive data as index labels, and that a trace ID sanitization statement silently failed on every record due to an invalid empty string. The developer shipped the method, not the map, emphasizing the need to verify controls dynamically.", "body_md": "I ran a penetration test on my own infrastructure last week. No Burp Suite, no exploit fired at production, no CVE popped. The whole engagement came down to one habit: refusing to believe a control was working until I had watched it work.\n\nThe target is a small observability hub I built for my own AI-assisted coding. Six services in one compose file: a tunnel, an OpenTelemetry Collector taking metrics and logs from Claude Code, Prometheus, Grafana, Loki, and a status API. The public surface is three aggregate numbers. Everything else stays private. That boundary, three numbers out and nothing else, was the whole thing I was testing.\n\nThe word \"pentest\" carries a picture that does not match, so: no attack traffic at the live system. The platform bills by usage and there is a WAF in front, so a flood of probes would have cost money and poisoned its own results. What I did was a read-only audit of the code and config, plus a dynamic run against the whole stack brought up locally in Docker.\n\nI expected the findings to cluster around the parts nobody had looked at. They did the opposite. Nearly every serious defect sat inside a control written days or hours earlier, usually by me, usually with a comment beside it naming what it protected against. Old code has been observed: it has run against real traffic and somebody has been surprised by it. A defence written yesterday has only been reasoned about, which feels like the same thing and is not.\n\nThe privacy boundary is an allow-list rather than a deny-list, and that part was right. Claude Code was measured sending five identity attributes, `user.email`\n\namong them carrying a real address, and no flag turns them off. A `delete_key`\n\nfor each works until the client adds a sixth, and this telemetry is beta: its attribute set is not a contract.\n\n```\n- context: resource\n  statements:\n    - keep_keys(resource.attributes, [\"service.name\"])\n    - set(resource.attributes[\"service.name\"], \"claude-code\")\n```\n\nThe second line is not redundant: `keep_keys`\n\nfilters keys, not values, and `service.name`\n\nis the one attribute that becomes an index label in Loki. On 2026-08-20 a sender holding the ingest token wrote it as `claude-code-…victim@example.com`\n\n, and the address arrived as an index label.\n\nWhat was wrong was a sentence in my design note calling the two barriers on that boundary independent. One is in the Collector, one in Loki, which re-filters whatever reaches it. `keep_keys(log.attributes, …)`\n\ngoverns record attributes, and Loki's `otlp_config`\n\nhas three sections, all three of attributes. Scope attributes crossed both untouched: remove Loki's list to test the isolation and a planted `scope.secret`\n\nwas suddenly queryable, while identity and content stayed out. The repair reads like a no-op and is the whole fix:\n\n```\n- context: scope\n  statements:\n    - keep_keys(scope.attributes, [])\n```\n\nWhat should have warned me is that the same hole existed twice. Two days later the metrics path turned out to be leaking scope attributes as `otel_scope_*`\n\nlabels past any allow-list, while the comment beside that exporter declared the boundary closed.\n\nA line in the log path was meant to zero the trace ID on every record before storage. Written correctly, in the right place, covered by a proof that ran it.\n\n```\n# looks right, fails on every record\n- set(log.trace_id.string, \"\")\n```\n\n`ParseTraceID`\n\nwants 32 hex characters and the empty string is not one, so the statement failed on every record, the Collector logged `failed to execute statement`\n\nand carried on, and the field reached Loki untouched. Measured on 2026-08-21 against the first real traffic: two warnings per record, around 180 per session, and a barrier that was declared and absent. The proof could not see it, because the payload never carried a trace ID. Green, and blind.\n\nIts twin is worse, because there the failure was conditional. OTTL documents that `set`\n\ndoes nothing at all if the value resolves to nil, so a line collapsing the log body to the event name did nothing on any record without an `event.name`\n\n, and a body containing a prompt and an address arrived in Loki verbatim. Neither proof could have caught it: the client always sends `event.name`\n\n, and the synthetic payload had to include it to satisfy a different assertion. The defence was a no-op in exactly the case it existed for. Both close the same way, by making the payload carry the thing.\n\nThose shell proofs are the instrument this project uses to not have silent failures. During the audit I found a silent failure inside the instrument, six hours old and mine.\n\nTwo of them pinned the Collector image literally, `0.158.0`\n\n, under a comment claiming it was the same digest as production. A dependency PR had moved compose and the Railway Dockerfile to `0.159.0`\n\n, Dependabot does not read shell, and the proofs went on pulling the old image and passing. So the sentence I had written to verify that upgrade, \"contract proof green on the new image\", was false. The pin is no longer copied: it is read out of `docker-compose.yml`\n\n, and each proof prints the image it is running on.\n\nThen I wrote a CI gate so it could not happen again, and an adversarial review found the gate born broken. It counted how many proofs derive their image by searching the whole file for the string `docker-compose.yml`\n\n, comments included, so the comment describing the derivation survived the derivation: delete the real line and the count stayed at three and the gate stayed green. Fifteen lines below a comment forbidding exactly that pattern, in a file where the same mistake had already been made three times.\n\nThe shape has siblings. A blocking image scan went green because uninstalling pip is not removing it: `ensurepip/_bundled/`\n\nkeeps a second copy as a wheel, and the scanner does not read inside an archive. Each of those looks like a pass. None is evidence of the property you wanted.\n\nI brought the metrics path up locally with fake secrets and pushed one metric carrying identity with a valid token: an email in `user.email`\n\n, an id in `organization.id`\n\n, and a hostile value inside `service.name`\n\nitself.\n\n``` php\n# auth first, before anything else gets a vote\nno auth      -> 401\nwrong token  -> 401\nvalid token  -> 200   # accepted, now let's see what survives it\n```\n\nThen read the exporter. Here is the single series it exposes, in full:\n\n```\nclaude_code_token_usage{job=\"claude-code\",model=\"claude-opus-5\",session_id=\"sess-canary\",type=\"input\"} 4242\n```\n\nThe email is gone, the organization id is gone, the hostile `service.name`\n\nwas pinned instead of becoming a label. What stayed are the three keys I allowed.\n\nAnd `4242`\n\nwent all the way to the public numbers, which is the honest half. An allow-list of names does not constrain values: whoever holds the ingest token can write `claude_code.token.usage`\n\nwith any number in it, and the public queries read that counter with `max_over_time(…[25h])`\n\n, so an injected spike stays stuck for twenty-five hours. Measured on a test stack: `1e12`\n\ntokens. It does not close here, because the token identifies the trusted producer and these numbers have no second source. What does close is the number arriving from somebody else: the three public queries now carry `{job=\"otel-collector\"}`\n\n.\n\nThe payload is also synthetic, so it proves the allow-list discards what I hand it, not that the client only sends that. On this same project a synthetic payload has already confirmed a query and then lied to me.\n\nThere is a version of this post that lists every residual weakness in the running system by name, with the exact route and the exact window. I wrote that report. It stays in the drawer.\n\nThe obvious objection is that the repository is public, so what am I withholding. The answer is the aggregation. Every defect above is closed in the open, with the measurement that found it sitting in the commit that fixed it. A list of what is still open, in one place, with the routes and the timings next to each other, is a different object. It is not a disclosure, it is directions.\n\nIf you send telemetry from an AI coding client, read one raw record before you read your config. Identity ships by default in this class of product, and every deny-list you write today is a list of the fields that existed this morning.\n\nIf you lean on two barriers, the useful question is not whether both are configured. It is which set of data only one of them is actually seeing. Break each on purpose and query the other back.\n\nAnd if you have proofs, ask what your last three green runs actually ran against. Mine were pulling an image production had already left behind, and they told me so in the friendliest way available: by passing.", "url": "https://wpnews.pro/news/i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map", "canonical_source": "https://dev.to/mk023/i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map-4k3n", "published_at": "2026-08-22 03:32:31+00:00", "updated_at": "2026-08-22 03:43:09.255799+00:00", "lang": "en", "topics": ["ai-safety", "ai-infrastructure", "developer-tools"], "entities": ["Claude Code", "OpenTelemetry Collector", "Prometheus", "Grafana", "Loki", "Docker", "OTTL"], "alternates": {"html": "https://wpnews.pro/news/i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map", "markdown": "https://wpnews.pro/news/i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map.md", "text": "https://wpnews.pro/news/i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map.txt", "jsonld": "https://wpnews.pro/news/i-pentested-my-own-ai-hub-and-shipped-the-method-not-the-map.jsonld"}}