{"slug": "scanning-agent-transcripts-for-secrets-without-sending-them-anywhere", "title": "Scanning agent transcripts for secrets, without sending them anywhere", "summary": "Oye Collective has built a local PII detection feature for coding agent transcripts that scans for secrets without sending data to the cloud. The system combines regex for structured tokens like API keys with a quantized ONNX model for context-sensitive items like names and addresses, running entirely on-device to avoid the security risk of uploading transcripts to a third party. The model weights (1.6 GB) are downloaded only on demand and the inference session is disposed after five minutes of inactivity to minimize resource usage.", "body_md": "*Originally published on the Oye Collective blog.*\n\nYou paste a `.env`\n\ninto a Claude Code session to debug a failing deploy. The agent reads it, fixes the config, and moves on. Twenty minutes later the task is done and you close the terminal.\n\nThe secret is still there. Not in the terminal, in the transcript. Every agent session writes a JSONL file to disk, and that file now holds your production database URL in clear text, sitting in a directory you will never think about again.\n\nMultiply that by a few hundred sessions. That is the actual state of most machines running coding agents today.\n\nThe obvious fix is to scan those transcripts and mask what you find. The obvious way to scan them is to send them to something that is good at finding secrets, which in 2026 means a cloud deployed model.\n\nYeah that's right. To find out whether your transcripts contain credentials, you would upload all of your transcripts, including the ones that contain credentials, to a third party. **The scan itself becomes the leak.** We were not comfortable with this trade, so the whole PII feature had to run locally or not exist at all.\n\nLocally means a model small enough to ship, fast enough to feel instant, and accurate enough to be worth turning on. Those three pull against each other, and most of what follows describes the places where they pull the hardest.\n\nOur first instinct was regex, and it works better than people expect. An AWS access key has a shape. So does a JWT, an IBAN, a card number. A regex with a checksum validator behind it will find those with precision a neural network struggles to match, and it costs microseconds.\n\nThen you hit the things that have no shape at all.\n\n```\nmy colleague Sarah handles the billing, she's at 14 Rue de Rivoli\nthe staging password is the same as my dog's name plus 1999\n```\n\nNo pattern matches those. A name is only a name in context. An address is only an address because of the words around it. And \"the staging password is\" is a phrase, not a format.\n\nSo we run both. **The two approaches fail in opposite directions, which means the union of them is strictly better than either one.** The model finds names, addresses, and password-phrased secrets that no expression will ever catch. The regex finds bare structured tokens that the model walks straight past.\n\nA composite detector runs every ready backend and merges the results. Regex is cheap and always ready, so masking works from the first launch.\n\nWe integrated OpenAI's privacy filter, quantized to int8 and exported to ONNX, which is run through ONNX Runtime. Weights plus tokenizer come to a little over 1.6 GB on disk.\n\nThat number is a problem for an app whose entire promise is that it sits quietly in your menu bar and does not bother your machine. A resident 1.6 GB is not quiet.\n\nWe made two decisions to handle this.\n\nFirst, nobody downloads it unless they ask for it. The feature is off by default and the weights arrive only when you turn it on, so a user who never enables PII detection carries none of it. Each artifact is pinned by SHA-256 against a pinned model revision, and a hash mismatch refuses the file rather than loading it. Hashing 1.6 GB takes a few seconds, which is long enough that the UI needs a distinct `verifying`\n\nstate or it looks frozen at 100%.\n\nSecond, the inference session has to be disposable.\n\n``` js\ntimer.setEventHandler { [weak self] in\n    guard let self else { return }\n    self.lock.lock(); defer { self.lock.unlock() }\n    if self.session != nil, Date().timeIntervalSince(self.lastUsed) >= self.idleTimeout {\n        self.session = nil  // release ~1.6 GB; ORTEnv stays (cheap)\n    }\n}\n```\n\nAfter five minutes with no scans, the ORT session gets torn down, and the next scan spends about a second rebuilding it. That fits how the feature actually gets used. You scan a transcript, read through what it flagged, and then don't touch it again for an hour. Paying a one second rebuild every now and then beats keeping 1.6 GB pinned in memory all day for an app that lives in your menu bar.\n\nOnce the model is running, the remaining problems are all about its output. It emits 33 classes per token in a BIOES scheme. In plain terms, for every token the model answers one question. Is this the start of a piece of PII, the middle of one, the end of one, a complete one on its own, or not part of one at all?\n\nTake the argmax at each position independently and you get sequences that cannot mean anything. A `begin-email`\n\nfollowed directly by an `outside`\n\n. An `inside-address`\n\nwith no `begin`\n\nbefore it. The labels are individually most likely and collectively nonsense, and every one of those is a span you cannot mask because you do not know where it ends.\n\nSo the decode is a constrained Viterbi pass. Illegal transitions are not penalized, they are unrepresentable.\n\n```\ncase (.begin, .inside), (.inside, .inside):\n    return from.category == to.category ? biases.insideToContinue : nil\ncase (.begin, .end), (.inside, .end):\n    return from.category == to.category ? biases.insideToEnd : nil\ndefault:\n    return nil  // any other transition is illegal\n}\n```\n\nReturning `nil`\n\nremoves the edge from the lattice entirely. The best path is then the best path through legal sequences only, and the decoded output is always a set of complete, well formed spans. The decoder is pure and deterministic, which means it unit tests from synthetic logits with no model loaded, and that has caught more bugs than any other single decision in this pipeline.\n\nThose spans have one problem left. They live in token space, and masking happens on the original string. To mask a secret you need its exact character range in that string. Off by one and you either leak a character of the credential or mask a character of the surrounding text.\n\nThe tokenizer we use returns token ids and token surfaces. It does not return offsets back into the source. That is a real gap, and the usual answer is to build a lookup table mapping tokens to byte positions, which is fiddly and easy to get subtly wrong around multibyte characters.\n\nThere is a much better answer hiding in how byte-level BPE works. The surface form of a byte-level token is drawn from a 256 symbol alphabet where each symbol stands for exactly one source byte. So a token's length in source bytes is just the number of unicode scalars in its surface. No table, no mapping, no special cases.\n\n``` js\nvar pos = 0\nfor surface in surfaces {\n    let n = surface.unicodeScalars.count  // == source UTF-8 byte count\n    ranges.append(pos..<(pos + n))\n    pos += n\n}\n```\n\nCumulate those and you have exact byte ranges for every token, which convert to `String.Index`\n\nranges by snapping to scalar boundaries. We verified it byte for byte against the reference tokenizer across ASCII, accented Latin, CJK, emoji, and code samples, because a claim like this can hold for English and quietly break on a Japanese variable name.\n\nThe model has a window. Real transcripts are much longer than it, so long inputs are processed as overlapping windows, 4096 tokens with 256 tokens of overlap on each side.\n\nOverlapping raises a new question. If a credit card number happens to straddle the boundary between two windows, which window owns it?\n\nThe tempting answer is to only trust spans fully inside a window's core region. It is also the wrong one, because a span straddling a seam is then partly outside both cores, gets rejected twice, and renders in clear text. The bug is invisible in testing unless a secret lands exactly on a boundary.\n\nWe assign each span to exactly one window, by its starting token. The trailing overlap is still there and still gives the model context to read the entity correctly, but ownership is unambiguous and the trusted ranges partition the sequence cleanly. A span that begins in a window belongs to that window, whole, however far past the boundary it runs.\n\nTwo spans overlap. Maybe the regex and the model both flagged an email with slightly different boundaries. You have to produce one span, so you pick one and drop the other.\n\nThat is wrong, and it is wrong in this specific way, If the span you dropped extended past the one you kept, the tail you discarded renders in clear text. You have not resolved a conflict, you have created a leak out of two correct detections.\n\nSo overlaps are always unioned, never dropped. The merged span covers everything either detector flagged and carries the label of whichever was more confident.\n\nWe ended up leaning on this rule in three different places in the pipeline, and it holds because the two mistakes are nowhere near the same size. Mask two extra characters of a sentence and a careful reader might notice and shrug. Leave four characters of an API key on screen and we've failed at the reason this feature exists. Every time we had to pick between the two, that comparison made the choice for us.\n\nMost privacy engineering comes down to noticing which of your errors are recoverable.\n\nAll of this ships in [Actvt](https://actvt.io), a macOS menu bar app that watches your machine and the coding agents on it. The PII detection is one part of it, off by default, and everything above happens on your Mac with no network call in the path.\n\nWe wrote it this way because the alternative was asking people to upload their agent transcripts to find out whether their agent transcripts contained anything they would not want uploaded. Some problems answer themselves.\n\nIf you are running agents at any scale and have thought about this differently, we would genuinely like to hear it. The failure modes here are subtle, and we would rather find ours in a conversation than in an incident.\n\n*Oye Collective builds production AI agents inside real businesses, and Actvt gives you observability into your own. Reach us at oyecollective.com.*", "url": "https://wpnews.pro/news/scanning-agent-transcripts-for-secrets-without-sending-them-anywhere", "canonical_source": "https://dev.to/2nji/scanning-agent-transcripts-for-secrets-without-sending-them-anywhere-k0k", "published_at": "2026-07-29 10:52:37+00:00", "updated_at": "2026-07-29 11:03:42.664937+00:00", "lang": "en", "topics": ["ai-safety", "developer-tools", "machine-learning", "natural-language-processing"], "entities": ["Oye Collective", "Claude Code", "OpenAI", "ONNX Runtime"], "alternates": {"html": "https://wpnews.pro/news/scanning-agent-transcripts-for-secrets-without-sending-them-anywhere", "markdown": "https://wpnews.pro/news/scanning-agent-transcripts-for-secrets-without-sending-them-anywhere.md", "text": "https://wpnews.pro/news/scanning-agent-transcripts-for-secrets-without-sending-them-anywhere.txt", "jsonld": "https://wpnews.pro/news/scanning-agent-transcripts-for-secrets-without-sending-them-anywhere.jsonld"}}