{"slug": "logging-ai-tool-usage-without-logging-sensitive-data", "title": "Logging AI tool usage without logging sensitive data", "summary": "A developer outlined a logging pattern for AI features that records metadata by default and content only by exception, capturing call IDs, prompt versions, token counts, latency and status while omitting raw prompt and response text. The approach uses an HMAC-keyed fingerprint to group identical inputs without enabling guess-based reversal, and a redaction pass that strips secrets, emails, card numbers and phone numbers from opt-in debug samples.", "body_md": "When a team adds an AI feature, the first debugging instinct is to log everything: the full prompt, the full response, the user's message. It works. It also means your logs now hold a copy of every customer email, contract and support ticket that went through the feature, sitting in a system with weaker access controls than your database and a much longer memory.\n\nYou need logs. You need them to debug bad answers, answer audit questions and keep costs under control. You don't need them to be a second copy of your customers' data.\n\nHere's a practical middle path.\n\nBefore writing any code, list the questions your logs must answer. For most small teams it's these:\n\nNotice that none of these needs the full text of the prompt or the response. That's the key idea: **log metadata by default, and content only by exception.**\n\nThis is the record to write for every call:\n\n``` python\nimport hmac\nimport json\nimport logging\nimport os\nimport time\nimport uuid\n\nlog = logging.getLogger(\"ai_usage\")\n\nFP_KEY = os.environ[\"AI_LOG_FP_KEY\"].encode()   # random secret, not in code\n\ndef fingerprint(text: str) -> str:\n    \"\"\"Keyed, non-reversible fingerprint so identical inputs can be grouped.\"\"\"\n    return hmac.new(FP_KEY, text.encode(\"utf-8\"), \"sha256\").hexdigest()[:16]\n\ndef log_ai_call(*, feature, prompt_name, prompt_version, model, actor_id,\n                record_id, prompt_text, response_text, usage, status, started):\n    # started: value of time.monotonic() taken before the call\n    usage = usage or {}                      # failed calls often have no usage data\n    call_id = str(uuid.uuid4())\n    log.info(json.dumps({\n        \"event\": \"ai_call\",\n        \"call_id\": call_id,\n        \"feature\": feature,                  # e.g. \"ticket_summary\"\n        \"prompt\": prompt_name,\n        \"prompt_version\": prompt_version,\n        \"model\": model,\n        \"actor_id\": actor_id,                # internal user or service id, not an email\n        \"record_id\": record_id,              # e.g. ticket id\n        \"input_chars\": len(prompt_text),\n        \"input_fp\": fingerprint(prompt_text),\n        \"output_chars\": len(response_text or \"\"),\n        \"input_tokens\": usage.get(\"input_tokens\"),\n        \"output_tokens\": usage.get(\"output_tokens\"),\n        \"status\": status,                    # \"ok\", \"error\", \"blocked\"\n        \"latency_ms\": int((time.monotonic() - started) * 1000),\n    }))\n    return call_id\n```\n\nWith this, you can trace any answer to a prompt version and a record, add up costs by feature, and see failure rates, all without storing a word of the customer's text.\n\nA note on the fingerprint: a plain hash of short or predictable text (an email address, a yes/no answer) can be reversed by guessing, so the fingerprint is keyed with a secret (`AI_LOG_FP_KEY`), so it can't be checked by guessing without the key. The fingerprint is for grouping long, varied inputs (\"were these two calls the same request?\"), not for hiding identifiers. Identifiers like emails shouldn't go into it in the first place.\n\nSometimes you need to see what was sent, for example while fixing a specific bug. When that happens, log a **redacted sample**, not the raw text, and only when a flag is switched on.\n\nA simple redaction pass catches the most common sensitive patterns:\n\n``` python\nimport re\n\nPATTERNS = [\n    (re.compile(r\"\\b(?:sk|pk|api|key|token|ghp|xox[bp])[-_][A-Za-z0-9_-]{12,}\\b\", re.I), \"[SECRET]\"),\n    (re.compile(r\"[\\w.+-]+@[\\w-]+\\.[\\w.-]+\"), \"[EMAIL]\"),\n    (re.compile(r\"\\b(?:\\d[ -]?){13,19}\\b\"), \"[CARD]\"),\n    (re.compile(r\"(?<![\\w.])\\+?\\d(?:[\\s()-]{0,2}\\d){9,14}(?![\\w.])\"), \"[PHONE]\"),\n]\n\ndef redact(text: str, limit: int = 500) -> str:\n    for pattern, label in PATTERNS:\n        text = pattern.sub(label, text)\n    return text[:limit]\n```\n\nOrder matters: run the most specific patterns first. Secrets go before emails, cards and phone numbers, or a phone pattern can eat the digits in the middle of a key and leave the rest in the log.\n\n``` python\nimport os\n\nDEBUG_CONTENT = os.getenv(\"AI_LOG_CONTENT\") == \"1\"\n\ncall_id = log_ai_call(...)  # same call as above\n\nif DEBUG_CONTENT:\n    log.info(json.dumps({\n        \"event\": \"ai_call_content\",\n        \"call_id\": call_id,\n        \"prompt_sample\": redact(prompt_text),\n        \"response_sample\": redact(response_text or \"\"),\n    }))\n```\n\nBe honest about the limits. Regular expressions catch emails, phone numbers, card-like numbers and obvious keys. They **won't** catch names, many provider key formats (cloud access keys, bearer tokens), addresses, medical details or anything written in free text. That's why content logging stays off by default, is switched on for a short time for a specific problem, and is truncated. If your feature handles health, financial or legal information, don't log content at all; reproduce the problem with test data instead.\n\nMetadata and content have different risks, so treat them differently:\n\n|  | Metadata log | Content sample log | \n|---|---|---|\n| **Default** | Always on | Off; switched on per incident | \n| **Who can read it** | Engineering and whoever handles costs | A named few engineers | \n| **How long it's kept** | As long as your audit needs (agree this with whoever owns compliance) | Days, not months | \n| **Where** | Your normal log system | A separate log stream or index with its own access rule and expiry | \n\nMost log platforms let you route by a field (here, `event`) and set a separate retention period per stream. Use that, so content samples expire automatically and nobody has to remember to delete them.\n\nYour own logs are only one place prompts end up:\n\n`type(exc).__name__`) and status code, not `str(exc)`.` call_id` instead, and look the details up where access is controlled.\nIf you work with larger clients, sooner or later someone will ask how you control AI use. The metadata log answers most of it:\n\n`feature`, `model`)` actor_id`, `record_id`)` prompt_version`, plus your repository history)\nBeing able to answer those from a dashboard, rather than by searching raw text, is a good sign you've got the balance right.\n\nGood AI logs tell you what happened without telling everyone what your customers said.\n\n*Ahmed Vaseer is Founder & CEO - Ingenious Concepts, an AI-first technology and digital growth agency. 80+ projects, ISO 27001 certified via Global CB.*", "url": "https://wpnews.pro/news/logging-ai-tool-usage-without-logging-sensitive-data", "canonical_source": "https://dev.to/ahmedvaseer/logging-ai-tool-usage-without-logging-sensitive-data-171k", "published_at": "2026-10-30 06:11:20+00:00", "updated_at": "2026-10-30 06:19:23.405278+00:00", "lang": "en", "topics": ["ai-tools", "mlops", "developer-tools", "ai-agents"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/logging-ai-tool-usage-without-logging-sensitive-data", "markdown": "https://wpnews.pro/news/logging-ai-tool-usage-without-logging-sensitive-data.md", "text": "https://wpnews.pro/news/logging-ai-tool-usage-without-logging-sensitive-data.txt", "jsonld": "https://wpnews.pro/news/logging-ai-tool-usage-without-logging-sensitive-data.jsonld"}}