While working on Agentic AI for production support, one question came to my mind: Do we really know what data we are sending to the LLM?
Let's take a simple production incident.
Host: ip-10–0–21–145
Memory: 1024 MB
Contact: [user@example.com](mailto:user@example.com)
Authorization: Bearer abc.def.ghi
ERROR: Service failed due to disk space issue
For RCA, the LLM mainly needs to understand: "Service failed because of a disk space issue."
It doesn't really need the actual hostname, email, AWS resource details, file paths, capacity values or authorization token.
This is where I started looking at the LLM as a Data Egress Boundary.
Instead of:
Production Data → RAG → LLM
I am looking at:
Production Data → Clean/Sanitization Layer → RAG → LLM
Before anything goes to an LLM or embedding model, sensitive information should be cleaned or redacted.
But there is another important point.
RAG itself is not the security layer.
We normally think that our data is safe because we are using RAG. But before storing a document in a vector database, we generate embeddings.
So the actual flow can be:
Raw Data → Embedding Model → Vector DB → Retrieval → LLM
This means sanitization should happen before embedding as well, not only before the final LLM call. The same principle should apply to retrieval queries and AI observability logs.
For high-risk information like API keys, passwords, JWTs or bearer tokens, I prefer a simple rule:
If sensitive data is still detected after sanitization → don't call the model.
In my view, AI governance should not be only a policy document saying:
**"Don't send sensitive data to AI." **
Raw Context → Sanitize → Validate → RAG / LLM
The objective is not to remove useful context.
The objective is: Give AI enough context to solve the problem, but don't give it information it doesn't need.
#AgenticAI #AIGovernance #LLM #RAG #GenAI #AIArchitecture #ProductionSupport