PII Meets GenAI: What Actually Happens When Sensitive Data Enters Your Prompt?

A developer explores what happens to sensitive data when it enters generative AI prompts, arguing that the risk extends beyond the model provider to application logs, traces, and other storage. The post advises minimizing data sent to models and considering the entire data flow, including responses, to avoid accidental exposure.

"Don’t put sensitive data into an AI prompt." I hear this advice a lot, and while it makes sense, I think it oversimplifies the problem. As more applications start using GenAI to summarize documents, answer questions, support employees, or process customer information, sensitive data is going to show up in some of these interactions. So instead of only asking whether sensitive data should enter a prompt, I think there is another useful question to ask: What actually happens to that data once it does? Let’s take a simple example. A Simple Customer Support Request Say I have a customer support application that uses an LLM to summarize complaints. A request might look like this: Summarize this customer complaint: Customer: Sarah Williams Email: sarah.williams@example.com Account: 839274 Complaint: I was charged twice for my subscription. There is obviously sensitive PII here. My first instinct might be to look at the model provider. Does the provider retain the prompt? Is the data used for training? How long is it stored? These are important questions, but they are not the only ones. The model is just one part of the journey, and that makes me think about the entire data flow. The request would look something like this: But real applications rarely stop there. There may be application logs, API logs, traces, monitoring systems, conversation history, or other storage around that request. That is why I find it more useful to ask something like: Where can this data exist throughout the entire request? Once I start looking at it that way, some risks that have nothing to do with the model itself become much easier to spot. Let’s look at this basic debugging statement: logger.info f"Sending prompt: {prompt}" I've now potentially copied the customer's name, email, account number, and complaint into my logging platform. The AI provider could have strong protections around my API data, but that doesn't really help with the extra copy I just created myself. In many cases, I probably don't need the entire prompt to troubleshoot the request. I could log something like: logger.info "Sending AI request", extra={ "request id": request id, "model": model name } I'm not saying prompts should never be logged. There may be perfectly valid reasons to capture them. I just don't want it to happen accidentally because somebody added a debug statement six months ago and nobody thought about it again. Before figuring out how to protect all of this data, I would ask an even simpler question: Does the model actually need it? Going back to our original example: Customer: Sarah Williams Email: sarah.williams@example.com Account: 839274 Complaint: I was charged twice for my subscription. If all I want is a summary of the complaint, maybe I only need: The customer says they were charged twice for their subscription. Same task, but now we're sending much less data. Of course, this isn't going to work for every use case, as sometimes the model genuinely needs sensitive information to do what I'm asking it to do. Sometimes removing unnecessary data is easier than figuring out how to protect it everywhere else. It is also easy to focus so much on the prompt that I forget about what comes back. Suppose the model responds with: Sarah Williams reported that account 839274 was charged twice. The sensitive information is back again, and here I am wondering, now what? Maybe I log the response or store it in conversation history. Maybe I send it to another service that consumes it, or maybe it gets displayed somewhere it shouldn't. The data doesn't suddenly stop being sensitive because the model returned it. So I need to think about both sides of the interaction: What am I sending in, and where is the result going afterward? Before putting an application like this into production, I would start with five fairly simple questions: This isn't meant to replace a proper privacy or security review. For me, it is just a practical way to catch some obvious problems earlier, when they are usually much easier to fix. When we talk about sensitive data and GenAI, I think we sometimes put too much attention on the model itself. The model matters, of course, but it is only one part of a much bigger journey. The same piece of information can move through the application, logs, APIs, monitoring systems, the model provider, and eventually show up again in the response. That is why I think a better starting point is to simply follow the data from beginning to end. What am I sending? Where does it go? Where could another copy be created? And most importantly, do I actually need to send all of it in the first place? To me, that is much more useful than simply saying, " Don’t put PII into AI. " There will be cases where sensitive data is necessary for the application to do its job. When that happens, what matters is knowing where that data is going and making sure its journey through the application is intentional.