Logging AI tool usage without logging sensitive data

A developer outlined a logging pattern for AI features that records metadata by default and content only by exception, capturing call IDs, prompt versions, token counts, latency and status while omitting raw prompt and response text. The approach uses an HMAC-keyed fingerprint to group identical inputs without enabling guess-based reversal, and a redaction pass that strips secrets, emails, card numbers and phone numbers from opt-in debug samples.

When a team adds an AI feature, the first debugging instinct is to log everything: the full prompt, the full response, the user's message. It works. It also means your logs now hold a copy of every customer email, contract and support ticket that went through the feature, sitting in a system with weaker access controls than your database and a much longer memory. You need logs. You need them to debug bad answers, answer audit questions and keep costs under control. You don't need them to be a second copy of your customers' data. Here's a practical middle path. Before writing any code, list the questions your logs must answer. For most small teams it's these: Notice that none of these needs the full text of the prompt or the response. That's the key idea: log metadata by default, and content only by exception. This is the record to write for every call: python import hmac import json import logging import os import time import uuid log = logging.getLogger "ai usage" FP KEY = os.environ "AI LOG FP KEY" .encode random secret, not in code def fingerprint text: str - str: """Keyed, non-reversible fingerprint so identical inputs can be grouped.""" return hmac.new FP KEY, text.encode "utf-8" , "sha256" .hexdigest :16 def log ai call , feature, prompt name, prompt version, model, actor id, record id, prompt text, response text, usage, status, started : started: value of time.monotonic taken before the call usage = usage or {} failed calls often have no usage data call id = str uuid.uuid4 log.info json.dumps { "event": "ai call", "call id": call id, "feature": feature, e.g. "ticket summary" "prompt": prompt name, "prompt version": prompt version, "model": model, "actor id": actor id, internal user or service id, not an email "record id": record id, e.g. ticket id "input chars": len prompt text , "input fp": fingerprint prompt text , "output chars": len response text or "" , "input tokens": usage.get "input tokens" , "output tokens": usage.get "output tokens" , "status": status, "ok", "error", "blocked" "latency ms": int time.monotonic - started 1000 , } return call id With this, you can trace any answer to a prompt version and a record, add up costs by feature, and see failure rates, all without storing a word of the customer's text. A note on the fingerprint: a plain hash of short or predictable text an email address, a yes/no answer can be reversed by guessing, so the fingerprint is keyed with a secret AI LOG FP KEY , so it can't be checked by guessing without the key. The fingerprint is for grouping long, varied inputs "were these two calls the same request?" , not for hiding identifiers. Identifiers like emails shouldn't go into it in the first place. Sometimes you need to see what was sent, for example while fixing a specific bug. When that happens, log a redacted sample , not the raw text, and only when a flag is switched on. A simple redaction pass catches the most common sensitive patterns: python import re PATTERNS = re.compile r"\b ?:sk|pk|api|key|token|ghp|xox bp - A-Za-z0-9 - {12,}\b", re.I , " SECRET " , re.compile r" \w.+- +@ \w- +\. \w.- +" , " EMAIL " , re.compile r"\b ?:\d - ? {13,19}\b" , " CARD " , re.compile r" ?< \w. \+?\d ?: \s - {0,2}\d {9,14} ? \w. " , " PHONE " , def redact text: str, limit: int = 500 - str: for pattern, label in PATTERNS: text = pattern.sub label, text return text :limit Order matters: run the most specific patterns first. Secrets go before emails, cards and phone numbers, or a phone pattern can eat the digits in the middle of a key and leave the rest in the log. python import os DEBUG CONTENT = os.getenv "AI LOG CONTENT" == "1" call id = log ai call ... same call as above if DEBUG CONTENT: log.info json.dumps { "event": "ai call content", "call id": call id, "prompt sample": redact prompt text , "response sample": redact response text or "" , } Be honest about the limits. Regular expressions catch emails, phone numbers, card-like numbers and obvious keys. They won't catch names, many provider key formats cloud access keys, bearer tokens , addresses, medical details or anything written in free text. That's why content logging stays off by default, is switched on for a short time for a specific problem, and is truncated. If your feature handles health, financial or legal information, don't log content at all; reproduce the problem with test data instead. Metadata and content have different risks, so treat them differently: | | Metadata log | Content sample log | |---|---|---| | Default | Always on | Off; switched on per incident | | Who can read it | Engineering and whoever handles costs | A named few engineers | | How long it's kept | As long as your audit needs agree this with whoever owns compliance | Days, not months | | Where | Your normal log system | A separate log stream or index with its own access rule and expiry | Most log platforms let you route by a field here, event and set a separate retention period per stream. Use that, so content samples expire automatically and nobody has to remember to delete them. Your own logs are only one place prompts end up: type exc . name and status code, not str exc . call id instead, and look the details up where access is controlled. If you work with larger clients, sooner or later someone will ask how you control AI use. The metadata log answers most of it: feature , model actor id , record id prompt version , plus your repository history Being able to answer those from a dashboard, rather than by searching raw text, is a good sign you've got the balance right. Good AI logs tell you what happened without telling everyone what your customers said. Ahmed Vaseer is Founder & CEO - Ingenious Concepts, an AI-first technology and digital growth agency. 80+ projects, ISO 27001 certified via Global CB.