{"slug": "aie-2-0-inside-the-llm-api-call", "title": "aie_2.0: inside the LLM API call", "summary": "A developer's technical lesson explains that large language models are stateless prediction machines with no built-in memory, and that the appearance of conversational recall comes from the messages array, which resends the entire conversation history on every API call. The writeup details how each message is labeled with a role (such as user, assistant, or system) so the model can distinguish speakers, and describes the controls and response structure involved in an LLM API call.", "body_md": "In the [last lesson](https://heymeraki.substack.com/p/aie_10-building-it?r=8ei6qe&utm_campaign=post-expanded-share&utm_medium=web), we established that an LLM (Large Language Model) is a prediction machine. It does not know things, it simply predicts the most likely next word based on all that it was trained on. We also established that it is stateless, no memory, every time you send a message, it starts fresh with no memory of the previous messages.\n\nThat raised a question for us. *If it has no memory, how is it that a conversation with Claude or ChatGPT feels like a real conversation with callouts and references to previous messages?* *How is it that it seems to remember what you said five messages ago?*\n\nThe answer is what this lesson is about. It has to do with how the API call is structured. Think of it like a phone call where the person on the other end has no memory. Every time you call, they pick up with no idea who you are or what you have discussed before. The only way to have a conversation is to read them the full transcript from the beginning, every single time. That is what happens when you make an LLM API call, you are sending the entire conversation history, fresh, on every single call.\n\nIn the build that followed aie_1.0, you made your first call to an LLM API. You sent a message and were able to get a response back, and you saw the token count. But the code was handed to you in steps without the full picture. *What were you actually sending? What controls what comes back? And what is inside that response beyond just the text?*\n\nWe fill that in this lesson. By the end, you will understand what the model receives when you make a call, the controls you have over what it produces, and what the response gives you back.\n\n#### *The messages array*\n\nBack to our phone call analogy. When you make a call, there are two people involved in it. There is you and whoever you are calling. Each person has a role. You know who is speaking at any given moment because you recognise their voice, or because they say their name, or because the context makes it obvious.\n\nThe LLM does not have any of that. It cannot hear tone. It does not know who is who. When it receives a call, all it has is a list of messages and a label on each one telling it who sent it.\n\nThat list is the ***messages array*** and we will define it properly here:\n\n**Messages array*** is the list of all that you send to the model on a single API call. It is structured as a sequence of messages, each one labelled with who sent it. The model reads this list from top to bottom and uses it to understand the full context before it responds. Everything the model knows about the conversation is in this list. If something is not there, the model does not know it exists.*\n\nIn the build from aie_1.0, your messages array looked like this:\n\n```\nmessages=[\n    {\"role\": \"user\", \"content\": \"What is a large language model?\"}\n]\n```\n\nIt only had one item, so one role and one piece of content. This is the simplest version of the messages array. It is just you asking a question.\n\nIn a real conversation however, that list grows. Every time that you send a message, it gets added to the list and every time the model responds, that also gets added. By the time you are five messages in, the model is receiving all of those five messages on every single call. Not just the most recent one, it is receiving all of them.\n\nThis is why it seems that the model remembers things. It does not have memory, you are handing it the full conversation history when you hit send and it reads through all of it before it replies.\n\n#### *The three roles*\n\nYou must have noticed the word `role` in that code. In any conversation, different people have different relationships to what is being said. You, the person you are speaking to, and in some situations a third party who has set the rules for the conversation before you started it. You can think of a moderator, or a host for this. They are not in the conversation but they shaped it.\n\nThe messages array has a role for each of these. We’ll go over each one.\n\n- `user` is you. Or more accurately, in terms of a product being built, it is whoever is on the other end of your product. In our build from aie_1.0, it was you typing a question. In a product, it would be your customer, the person using the feature you built.\n- `assistant` is the model. When you are building a back-and-forth conversation, one where the model needs to*remember* what it said earlier, you include its previous responses in the messages array labelled as`assistant` . The model reads the entries and understands what it said before, so it does not contradict itself or get repetitive.\n- `system` is a role that does not belong to the user or the model. It belong to the engineer. The system message sits at the top of the messages array before any conversation begins. It is where you tell the model who it is, what it should do, never do, and you define its behaviour. It is not seen by the user, it serves as your direct line to the model before anything else is done.\n\nA conversation with all three roles could look like this:\n\n```\nmessages=[\n    {\"role\": \"system\", \"content\": \"You are a customer support assistant for a travel company. Only answer questions about bookings, itineraries, and travel policies. Do not discuss anything else.\"},\n    {\"role\": \"user\", \"content\": \"Can I change my flight date?\"},\n    {\"role\": \"assistant\", \"content\": \"Yes, you can change your flight date up to 24 hours before departure. Would you like me to walk you through the process?\"},\n    {\"role\": \"user\", \"content\": \"Yes please\"}\n]\n```\n\nThe model receives all four of those messages at once. It reads the system instruction first, which tells it what kind of assistant it is. Then it reads the conversation so far. Then it generates what comes next.\n\nThat is the messages array. It is a complete picture of the conversation, including instructions, history, and the latest question. They are assembled and sent fresh on every single call.\n\n### *Parameters (model, max_tokens, temperature)*\n\nNow that you know what you are sending, what about what comes back?\n\nGoing back to the phone call analogy, you now know that you are sending a full transcript every time you call. But you also have some control over how the person on the other end responds. You can tell them to keep it brief. You can tell them to be more creative or more predictable. You can even specify which version of them you want to speak to.\n\nIn the API, these controls are called ***parameters***. You pass them alongside your messages on every call.\n\n**Parameters** are settings you include with every API call that shape how the model responds. They sit outside the messages array. They do not change what the model knows about the conversation, they change how it behaves when generating a response.\n\nThree of them matter most for you right now.\n\n- `model` tells the API which version of Claude to use. In the build from aie_1.0 you used`claude-haiku-4-5` . Haiku is lighter, faster, and cheaper. There are more capable models in the Claude family that handle more complex reasoning, but they take longer and cost more. You choose based on what the task actually needs.\n- `max_tokens` . To understand what this does, you need to understand how the model measures length. It does not measure in words or sentences. It measures in***tokens*** .*A **token** is the smallest unit the model works with. Think of it as a chunk of text. \"Hello\" is one token. \"Unbelievable\" might be three. A rough rule of thumb is that one token is about three to four characters of English text. So a short paragraph might be 80 tokens. A full page might be 400.*`max_tokens` sets a hard ceiling on how many of those chunks the model can produce in its response. Set it to 512 and the model cannot write more than 512 tokens back, no matter what. If its response would naturally run longer, it gets cut off, even mid-sentence. This is important to know because longer responses cost more and take more time to generate. You set it to something that makes sense for what you are asking the model to produce.\n- `temperature` is an interesting one. When the model picks its next word, it is working from a range of possibilities ranked by how likely each one is. The most probable word sits at the top. Temperature controls how tightly the model sticks to that ranking.***Temperature** is a number between 0 and 1 that controls how predictable or varied the model's output is. At 0, the model always picks the most probable next word. The output is consistent. If you send the same input twice and you get a very similar response both times. At 1, it samples more broadly from the range of possibilities. The output is more varied and sometimes more creative, but it is less predictable.*Think of it in terms of the person on the other end of that phone call. Temperature 0 is someone who always gives the most expected, safe answer. Temperature 1 is someone who might surprise you but not necessarily in a meaningful way. *For a feature that needs to produce the same reliable output every time temperature 0 is what you want. For something where variety is the point you might want something higher. It is an engineering choice that you make deliberately.*Here is what a full API call looks like with all three parameters set: \n\n```\nresponse = client.messages.create(\n    model=\"claude-haiku-4-5\",\n    max_tokens=256,\n    temperature=0,\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are a customer support assistant. Classify the message into one category only: Billing, Technical, Account, or Shipping.\"},\n        {\"role\": \"user\", \"content\": \"My invoice shows a charge I did not make.\"}\n    ]\n)\n```\n\n \n  - `model` picks Haiku. It is fast, cheap, and right for a classification task.\n  - `max_tokens` is set low because the answer should be one word.\n  - `temperature` is 0 because you want the same answer every time for the same input.\n  - The system message tells the model what its job is.\n  - The user message is the input it needs to classify.\n\n#### *What is inside the response?*\n\nWhen the API call comes back, you do not just receive the model’s text. You receive an object, like a container that holds several things at once. The text is in there, but so is information about the call. You can think of it like a delivery that comes with a receipt attached, where the receipt tells you what the package cost.\n\nYou saw this in the build from aie_1.0. After the API call, you wrote:\n\n```\ntext = response.content[0].text\ninput_tokens = response.usage.input_tokens\noutput_tokens = response.usage.output_tokens\n```\n\nLet us go through what each component is doing:\n\n- `response` is the full object the API sends back. Everything the call produces, the model's reply, the token counts, and the metadata about the request all lives inside it. You access different parts of it using dot notation, the same way you would access a property on an object in JavaScript.\n- `response.content` is a list of content blocks. This is where the model’s response to you is. In almost every case there is just one block, but the API is built to support multiple so it is always structured as a list. The`[0]` reaches in and takes the first item and`.text` pulls the words out of it as a plain string so you can use it in the rest of your code.\n- `response.usage` handles accounting. Every call keeps a record of how much it cost in tokens, and this is where you can find it.\n  - `response.usage.input_tokens` is how many tokens you sent to the model. This is not just your user message, it is everything in the messages array. The system message, any conversation history, the user’s question. All of it gets counted as input. Tokens get their own article later in the series.\n  - `response.usage.output_tokens` is how many tokens the model generated back. This is the response measured in the same unit.*Both of these numbers matter because you pay for them. It can feel like minute detail when it is just one call you are making in a terminal but it compounds in a real product where thousands of calls are being made a day for users.*\n\nYou now have the full picture of what is inside an API call. The messages array is the model’s entire world for that call, it carries the conversation history, your instructions, and the latest message. The parameters give you control over how the model responds. And the response object gives you back more than just text.\n\nThe next piece is the build. We take everything covered here and put it into practice by building a real multi-turn conversation in code. You’ll see the messages array grow with every exchange and the model responds as if it remembers. That’ll be aie_2.1.\n\nAfter the build, aie_2.2 goes deeper into what the API can do. Things about structured outputs, function calling, and streaming responses. They become essential when you start building features for users.", "url": "https://wpnews.pro/news/aie-2-0-inside-the-llm-api-call", "canonical_source": "https://heymeraki.substack.com/p/aie_20-inside-the-llm-api-call", "published_at": "2026-09-17 19:01:23+00:00", "updated_at": "2026-09-17 19:25:10.671974+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-tools", "developer-tools"], "entities": ["Claude", "ChatGPT"], "alternates": {"html": "https://wpnews.pro/news/aie-2-0-inside-the-llm-api-call", "markdown": "https://wpnews.pro/news/aie-2-0-inside-the-llm-api-call.md", "text": "https://wpnews.pro/news/aie-2-0-inside-the-llm-api-call.txt", "jsonld": "https://wpnews.pro/news/aie-2-0-inside-the-llm-api-call.jsonld"}}