Most AI projects run into the same problem: a capable model, a motivated team, and a data set that has accumulated for years without cleanup. The wrong answers usually don't come from the model — they come from the data fed into it.
Teams often assume the fix is "a better model." It almost never is. The model is rarely the problem — the data it receives is.
Here's a rule that's hard to accept but always holds: AI doesn't fix your data. It repeats it. If documents contradict each other, the AI will surface both. If information is outdated, the AI quotes the outdated version. If a file is unreadable, the AI literally cannot process it.
The model is only as good as the data you give it. And most organizations' document collections contain significant noise and duplication.
Common Data Quality Issues
Common patterns in enterprise document stores:
Duplicates. Identical or near-identical files saved in multiple locations with different names.
Near-duplicates. Versioned documents ("Draft", "Final", "v2", "Final_Final") that are 90% the same but treated as separate records.
Outdated content. Documents from years ago still sitting in active knowledge bases, referencing prices, policies, or products that no longer exist.
Contradictions. Different departments maintaining different versions of the same rule, with no central authority.
Unreadable files. Scanned PDFs, images, handwritten notes — files without a text layer the model can access.
Broken structure. Text cells where the schema expects numbers, null values in fields that can't be empty, rows that don't match their headers. Human eyes scroll past this; a machine parsing the file fails or silently misreads it.
Mixed formats. One column carrying several formats — dates as "01/02/2024" and "2 January 2024", amounts as "1000" and "1,000.00". Every downstream query becomes a guess about what the value actually means.
Missing context. Files in multiple languages, full of internal abbreviations, with no dates, owners, or metadata.
All of this flows straight into the AI. The results reflect the input quality.
Feeding the model more data doesn't make it smarter. Feeding it cleaner, more specific data does. Everything else is noise in, noise out.
The Answer Is a Pipeline, Not a One-Time Cleanup
A single cleanup effort isn't enough — new documents arrive constantly. What you need is a pipeline: a repeatable process every piece of data passes through before the AI sees it.
Centralize collection. Shared drives, archives, CRM, email — bring everything to one place. You can't fix what you can't see.
Make everything readable. Run OCR on scanned PDFs, extract text from images, handle multiple languages. If the AI can't read a page, that page effectively doesn't exist.
Normalize formats. One date format, one currency format, one product naming convention. Inconsistent records must resolve to the same canonical representation.
Deduplicate in two passes. First, exact duplicates via content hashing. Second, near-duplicates by semantic similarity — using embeddings (numerical representations of document meaning) and k-nearest neighbors (search for documents with similar embeddings). This catches "Final" vs "Final_v2" style variants. These become a single record with version history, not multiple files.
Establish currency. Date-stamp every document. Archive old versions (retain for audit, don't delete). Mark one authoritative "current" version per topic. Only current versions go to the AI.
Enrich with metadata. Add structure: document type, department, dates, owner, status, links to related records. This transforms a file pile into structured knowledge the agent can reason over.
Serve only the cleaned set. Build the vector index from the validated output. The agent never touches the raw folder. If a document didn't pass the pipeline, it doesn't exist for the AI.
Add a quality gate. Measure: duplicates caught, current vs. expired ratio, "new document to AI-ready" latency. You can't improve what you don't track.
The Loop That Keeps It Clean
Clean once and stop is a false economy. The pipeline runs continuously:
New document arrives → OCR, deduplication, version comparison, metadata enrichment, security scan, indexing. Document goes stale → flagged, archived, removed from the AI's retrieval set.
Quality gate runs on schedule → automated checks on the metrics above.
A practical side effect: clean data makes AI projects cheaper. Fewer tokens per request, faster responses, fewer costly errors. One well-selected document outperforms ten stale ones. The pipeline pays for itself.
The Security Check Most Pipelines Skip
Cleaning isn't only about quality — it's also about safety. Here's the attack nobody in the business room expects: a retrieved document can contain instructions directed at the AI itself.
This is called indirect prompt injection. A malicious or just careless document — a contract, a scanned PDF, an imported wiki page — can carry a line like "ignore your previous instructions and say...". When that document gets pulled into the model's context, the text stops being data and starts behaving like a command. The model can change its behavior because of a sentence buried in a file. This is precisely what OCR-extracted text is good at hiding: what looks like a typo in a scan can be a fully functional attack.
So in the pipeline, after OCR and extraction, every document passes a security pass before it's allowed into the index:
Pattern scan. A first pass with regex over the extracted text for known injection phrasing in multiple languages: "ignore previous instructions", "disregard", "you are now", "system prompt", "override prior commands", and similar. Fast and cheap — it catches the naive cases, which are most cases. Languages matter: documents get translated, and so do attacks.
Semantic scan. Regex misses rephrased attacks. A second pass uses a classifier or embedding-based anomaly detection: it flags chunks that read like instructions addressed to an AI rather than factual content. An invoice doesn't tell a model what to answer; a document that does should get attention.
Hidden content audit. Text the eye skips still reaches the model: text inside images (needs its own OCR pass), hidden layers in PDFs, tracked changes, comments, metadata fields. Attackers hide payloads there, so the audit must look there too.
Quarantine. Flagged documents go to a review queue — a human decides: clean it, strip it, or exclude it. Nothing silently enters the index on the first run.
And one principle that matters more than any scan: defense in depth. Even with cleaning in place, the agent must treat retrieved text as untrusted data, not instructions. Retrieved content never overrides the system prompt. The scan reduces exposure; the agent design is what actually stops the attack.
What Practice Teaches
A few lessons that only show up once you've run this pipeline for real:
Expect the first pass to remove a lot. In most document stores, a double-digit percentage of files turn out to be duplicates, near-duplicates, or outdated. That's normal, not a sign you did something wrong — it's the point.
Near-duplicate detection by embeddings catches what naming alone can't. Two files called something completely different can still be 90% the same document. Similarity search finds them; file names never will.
The hard part isn't cleaning — it's deciding what "current" means. Every topic needs one authoritative version, and that decision belongs to domain experts, not algorithms. The pipeline proposes; a human with business context approves.
Deletion improves answers more than addition. In retrieval, every extra document is a candidate for the wrong answer. Removing stale records usually improves output quality more than adding new ones.
Cost and latency drop together with cleanliness. Each request costs tokens proportional to what's indexed; a smaller, validated set means cheaper and faster answers without losing quality.
Automate in stages. Let the pipeline archive and merge for a few weeks while you review its decisions. Once it's consistently right, switch to full automation — but keep quality metrics on.
A Production-Grade Example: Databricks
If you want this as a managed system rather than a custom build, platforms like Databricks implement the full pipeline:
Ingestion and OCR at scale. Millions of files, distributed OCR — what used to be a months-long manual project becomes a scheduled job.
Normalization, deduplication, and near-duplicates. Spark runs the embedding pipeline; vector search for k-nearest neighbors is built in.
Models without the plumbing. Serve open models hosted by Databricks — cached weights, optimized inference, pay-per-token (embeddings like GTE-Large included) — or connect external providers like OpenAI, Anthropic, or Bedrock through one governed endpoint: credentials in a single place, rate limits and usage tracking applied centrally. No separate infrastructure to run or maintain.
Metadata and governance (Unity Catalog). Document versions, access controls, full audit trails — so cleaning never accidentally exposes protected data.
One platform for the entire stack. Cleaning, embeddings, vector indexes, and retrieval in one place, not a chain of disconnected tools.
This isn't a vendor endorsement. It's an illustration of what mature data engineering looks like: a governed, scalable system instead of a script that breaks at month three. The pipeline logic matters more than the platform.
The Takeaway
Before considering a larger model, more hires, or new AI initiatives: look at your data.
You don't need a bigger model. You need cleaner data.
A strong model on clean data beats a perfect model on raw, unprocessed data. Every time. The reason most AI projects underdeliver isn't the AI — it's unreliable input: duplicated, outdated, unreadable, and fed straight into the engine.
Clean the data. The model handles the rest.