Building Production-Grade LLM Applications
A short note, just sharing some experience that might be useful for those who are starting to explore building LLM applications. A few things I've come to understand that have become important notes for me.
1. LLMs are stateless.
Don't assume that once you've successfully integrated with an LLM vendor's API, you automatically get the same experience as interacting with ChatGPT directly. ChatGPT is a product; what we call through the API is a model — and the model has no memory at all.
You say "my name is Iwan" in turn 1? Don't expect the LLM to remember it in turn 2. What actually happens: the entire conversation history is re-sent with every request. So "memory management" isn't the model's job — it's ours :D
And this brings its own challenges, because it isn't as simple as big vs. small memory; it's about keeping accuracy, cost, and performance in balance.
2. RAG is not part of the LLM.
RAG is an architecture around the model, not a built-in capability. We first retrieve the relevant documents, then inject them into the context. The model itself knows nothing about our database.
A simple example: say we have an FAQ document we want to use as a reference for our app. We chunk that document, turn it into embeddings, and store it in a vector database. Later, when a user asks a question, we turn the question into an embedding too, use it to find the most relevant FAQ chunks — and only then inject the results into the context before sending it to the model.
3. The LLM is an orchestrator, not a calculator.
This is the one I emphasize most. Never hand calculations over to an LLM, especially anything financial.
Not because the AI isn't smart enough, but because structurally it predicts the next token probabilistically — it doesn't execute arithmetic. When we ask it to add two numbers, it isn't calculating. It's guessing the most likely-looking answer.
The solution: function calling. The model's job is to understand the user's intent and choose which function to call. The computation is handed off to a deterministic backend — one that can be tested and audited.
It's not just about accuracy, it's also about auditability. If someone asks, "why is the number this?", the answer has to be traceable to a function whose code we can actually show. Not to a model that "thinks it's probably that."
There's still a lot I haven't figured out on my end, especially around evaluation and cost. If anything here is off, or if you've had a different experience, I'd be glad to hear your input.