When building an AI product, it's tempting to start with the fashionable pieces.
Vector database.
RAG.
Agents.
Multimodal models.
Then connect everything to an LLM and hope the final prompt makes sense of it.
While building the automotive AI pipeline behind Inspecly, we ended up taking almost the opposite approach.
The first question wasn't:
Which LLM should we use?
It was:
What information do we actually have, how reliable is it, and which system should process it?
That distinction changed the architecture.
The input is messy by default
A driver rarely describes a vehicle problem like a mechanic.
They might say:
"My car makes a strange noise when I start it."
But a request can also contain:
These inputs do not have the same reliability.
An OBD code is structured information.
A photo is visual evidence.
A voice message represents what the driver observed.
Vehicle metadata may require exact lookup.
Treating all of them as equivalent pieces of text would be a mistake.
So before asking an LLM to reason about the problem, we normalize the available evidence.
Conceptually:
{ "vehicle": { "make": "...", "model": "...", "vin": "optional" }, "description": "The engine loses power when accelerating.", "voice_transcription": null, "obd_codes": ["..."], "images": [...] }
This object doesn't contain a diagnosis.
It describes what we actually know.
Why not send everything to one multimodal LLM?
You absolutely could.
Description + images + OBD + vehicle information → one model → final answer.
It is very attractive for a prototype.
It's also difficult to control.
Consider these inputs:
OBD code
→ deterministic lookup
Photo
→ visual analysis
Voice
→ transcription
Vehicle information
→ exact lookup/API
Unknown technical information
→ tool-based retrieval
Safety constraint
→ explicit business rule
These are fundamentally different operations.
Putting everything into one giant prompt hides those differences.
It also becomes much harder to answer:
Where did this conclusion come from?
That's a serious problem once the system moves beyond a demo.
Structured data comes first
One design decision became particularly important for us:
If reliable structured information already exists, use it directly.
For example, when an OBD diagnostic trouble code is available, we first query our curated OBD database.
A record can contain information such as:
{ "code": "...", "explanation": "...", "possible_causes": [], "possible_actions": [], "validation_status": "reviewed" }
Why retrieve semantically similar paragraphs from documents when the system can perform an exact lookup against validated fields?
This is an important distinction.
Structured data and RAG solve different problems.
Structured data is excellent for:
exact identifiers,
validated fields,
controlled records,
deterministic queries.
RAG becomes useful when knowledge primarily lives inside documents.
What if the structured knowledge is missing?
That's where agents become useful.
Our internal database cannot contain every code, every manufacturer-specific interpretation and every vehicle configuration.
When structured knowledge is missing, a tool-using agent can search for additional information.
But there is an important rule:
Retrieved information should not silently become equivalent to validated information.
Instead, preserve provenance.
For example:
{ "code": "...", "source_type": "tool_agent", "sources": [], "validation_status": "unverified" }
The final system should know whether a piece of information came from:
a reviewed internal database,
an external technical source,
a tool-using agent,
image analysis,
or the driver themselves.
The model doesn't only need context.
It needs context with provenance.
Images are evidence, not diagnosis
Vision models are another useful component.
A photo might reveal:
The output should therefore look closer to:
{ "observation": "Possible fluid trace", "confidence": "medium", "limitations": [ "The source is not visible" ], "requires_physical_inspection": true }
rather than:
Your vehicle has an oil leak.
That difference matters.
AI systems often sound more certain than the evidence actually allows.
Voice is just another source of context
Voice is valuable because describing a mechanical problem through a form can be difficult.
The message gets transcribed, then becomes another input to the evidence layer.
But again:
is not necessarily a confirmed technical fact.
It's something reported by the driver.
That distinction should survive the entire pipeline.
The next step: a structured evidence layer
Today, multiple processing paths can eventually contribute information to the generation context.
As the number of sources increases, simple concatenation becomes harder to control.
You eventually need something closer to:
{
"reported_symptoms": [],
"obd_findings": [],
"visual_findings": [],
"retrieved_information": [],
"missing_information": [],
"conflicts": [],
"safety_flags": []
}
The final LLM can then generate from this normalized evidence instead of receiving an unstructured wall of text.
This makes several things easier:
It also makes the architecture much easier to evolve.
So where is RAG?
We aren't starting with a large automotive RAG pipeline.
And that's deliberate.
Adding PDFs to a vector database isn't the difficult part.
The difficult part is knowing whether a retrieved procedure applies to the correct:
A perfectly retrieved technical procedure for the wrong engine generation can still be completely wrong for the vehicle in front of you.
So our current priority is:
Validated structured data
↓
Explicit rules / APIs
↓
Tool-based retrieval when needed
↓
LLM generation
RAG becomes much more valuable when we have a controlled corpus of manufacturer manuals and validated technical documentation.
Eventually, the source router could look roughly like:
This isn't a universal hierarchy.
The broader point is more important:
A generated or retrieved paragraph should not silently override a reviewed fact.
Two audiences, same evidence
Another interesting problem is that the same evidence needs different outputs.
A driver needs:
A garage needs:
We don't need two independent AI analyses.
We need two representations of the same evidence.
That distinction has become an important part of the product architecture.
What I'm learning from building this
The quality of an AI response doesn't start with the final prompt.
It starts much earlier:
Input collection
↓
Normalization
↓
Source routing
↓
Evidence + provenance
↓
Conflict / uncertainty handling
↓
Generation
A better model can improve generation.
It cannot fix an architecture where every source is mixed into one untraceable context.
My current takeaway is simple:
Use deterministic systems where knowledge is deterministic.
Use agents where flexibility is useful.
Use RAG when document retrieval is actually the problem.
And preserve uncertainty instead of asking the LLM to hide it behind a confident answer.
I wrote a more detailed version of this architecture — including the current pipeline, future evidence layer and planned RAG architecture — here:
Original deep dive:
I'm also building AI DevList, where I curate useful resources about agents, LLM engineering, MCP, RAG, evals and production AI — with a short explanation of why each resource matters.
I'm curious how other teams handle this:
Do you start from the LLM and build outward, or from the evidence and build inward?