Logging AI tool usage without logging sensitive data

When a team adds an AI feature, the first debugging instinct is to log everything: the full prompt, the full response, the user's message. It works. It also means your logs now hold a copy of every customer email, contract and support ticket that went through the feature, sitting in a system with weaker access controls than your database and a much longer memory.

You need logs. You need them to debug bad answers, answer audit questions and keep costs under control. You don't need them to be a second copy of your customers' data.

Here's a practical middle path.

Before writing any code, list the questions your logs must answer. For most small teams it's these:

Notice that none of these needs the full text of the prompt or the response. That's the key idea: log metadata by default, and content only by exception.

This is the record to write for every call:

import hmac
import json
import logging
import os
import time
import uuid

log = logging.getLogger("ai_usage")

FP_KEY = os.environ["AI_LOG_FP_KEY"].encode()   # random secret, not in code

def fingerprint(text: str) -> str:
    """Keyed, non-reversible fingerprint so identical inputs can be grouped."""
    return hmac.new(FP_KEY, text.encode("utf-8"), "sha256").hexdigest()[:16]

def log_ai_call(*, feature, prompt_name, prompt_version, model, actor_id,
                record_id, prompt_text, response_text, usage, status, started):
    usage = usage or {}                      # failed calls often have no usage data
    call_id = str(uuid.uuid4())
    log.info(json.dumps({
        "event": "ai_call",
        "call_id": call_id,
        "feature": feature,                  # e.g. "ticket_summary"
        "prompt": prompt_name,
        "prompt_version": prompt_version,
        "model": model,
        "actor_id": actor_id,                # internal user or service id, not an email
        "record_id": record_id,              # e.g. ticket id
        "input_chars": len(prompt_text),
        "input_fp": fingerprint(prompt_text),
        "output_chars": len(response_text or ""),
        "input_tokens": usage.get("input_tokens"),
        "output_tokens": usage.get("output_tokens"),
        "status": status,                    # "ok", "error", "blocked"
        "latency_ms": int((time.monotonic() - started) * 1000),
    }))
    return call_id

With this, you can trace any answer to a prompt version and a record, add up costs by feature, and see failure rates, all without storing a word of the customer's text.

A note on the fingerprint: a plain hash of short or predictable text (an email address, a yes/no answer) can be reversed by guessing, so the fingerprint is keyed with a secret (AI_LOG_FP_KEY), so it can't be checked by guessing without the key. The fingerprint is for grouping long, varied inputs ("were these two calls the same request?"), not for hiding identifiers. Identifiers like emails shouldn't go into it in the first place.

Sometimes you need to see what was sent, for example while fixing a specific bug. When that happens, log a redacted sample, not the raw text, and only when a flag is switched on.

A simple redaction pass catches the most common sensitive patterns:

import re

PATTERNS = [
    (re.compile(r"\b(?:sk|pk|api|key|token|ghp|xox[bp])[-_][A-Za-z0-9_-]{12,}\b", re.I), "[SECRET]"),
    (re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"), "[EMAIL]"),
    (re.compile(r"\b(?:\d[ -]?){13,19}\b"), "[CARD]"),
    (re.compile(r"(?<![\w.])\+?\d(?:[\s()-]{0,2}\d){9,14}(?![\w.])"), "[PHONE]"),
]

def redact(text: str, limit: int = 500) -> str:
    for pattern, label in PATTERNS:
        text = pattern.sub(label, text)
    return text[:limit]

Order matters: run the most specific patterns first. Secrets go before emails, cards and phone numbers, or a phone pattern can eat the digits in the middle of a key and leave the rest in the log.

import os

DEBUG_CONTENT = os.getenv("AI_LOG_CONTENT") == "1"

call_id = log_ai_call(...)  # same call as above

if DEBUG_CONTENT:
    log.info(json.dumps({
        "event": "ai_call_content",
        "call_id": call_id,
        "prompt_sample": redact(prompt_text),
        "response_sample": redact(response_text or ""),
    }))

Be honest about the limits. Regular expressions catch emails, phone numbers, card-like numbers and obvious keys. They won't catch names, many provider key formats (cloud access keys, bearer tokens), addresses, medical details or anything written in free text. That's why content logging stays off by default, is switched on for a short time for a specific problem, and is truncated. If your feature handles health, financial or legal information, don't log content at all; reproduce the problem with test data instead.

Metadata and content have different risks, so treat them differently:

Most log platforms let you route by a field (here, event) and set a separate retention period per stream. Use that, so content samples expire automatically and nobody has to remember to delete them.

Your own logs are only one place prompts end up:

type(exc).__name__) and status code, not str(exc). call_id instead, and look the details up where access is controlled. If you work with larger clients, sooner or later someone will ask how you control AI use. The metadata log answers most of it:

feature, model) actor_id, record_id) prompt_version, plus your repository history) Being able to answer those from a dashboard, rather than by searching raw text, is a good sign you've got the balance right.

Good AI logs tell you what happened without telling everyone what your customers said.

Ahmed Vaseer is Founder & CEO - Ingenious Concepts, an AI-first technology and digital growth agency. 80+ projects, ISO 27001 certified via Global CB.