RSS The flaws never left the model. We just got good at building around them.
It's been nearly four years since ChatGPT was released. For most of us that first encounter was GPT-3.5, and the friction was enough to convince a lot of developers that this thing was not built to handle code. The hallucination rate was high, and the complaints came down to four issues.
- AI was bad at math.
- AI was stuck in the past.
- AI had a short memory window.
- AI was toxic, reflecting the worst parts of the internet.
Take a look at what people were saying back then. So were all of these solved over the past few years? Is that why the technology is suddenly so useful?
Nope. All of these problems still exist, even in the newest models. Some have workarounds and others have ways to limit how often they show up, but every one of them is still sitting in the model.
Being a developer through this has felt strange. For a while the tools were hit or miss. Some days they saved me real time, other days they just made more work, and I never knew which I'd get. Then in January 2026, at the peak of Claude Code, half my feed was people realizing an agent could just write the code for them. The model underneath hadn't changed much, but the tooling around it had finally caught up.
That's worth slowing down for. Chasing each new tool means relearning something that may not matter a year from now. It's agentic swarms first, then loops, then graphs, and on and on. Meanwhile the four limits underneath don't move, and every trick worth knowing is a workaround for one of them. Learn the limits and the rest of the news cycle starts making sense on its own.
AI is bad at math #
First, AI is bad at math, and the solution is just a workaround.
So if you feed a batch of math equations directly into new frontier models like Fable or Sol through their API instead of their native apps, you'll still see them miss some of the equations you sent.
I sent GPT Sol High 200 simple addition questions and it got one wrong. Put that AI in charge of your company's accounting and it's fumbling a transaction every couple hundred entries.
Here is the one it missed.
52467998
52467898
So why did it get it wrong?
When you ask an AI a simple equation like 1 + 1
, it gets it right, because that answer is found in its training data. Give it a random, long pair like 35653046 + 16814852
, though, and there's a chance that exact equation has never shown up before.
With no memorized answer to lean on, the model does the only thing it knows how to. A language model only ever predicts the next likely token, so it never really calculates. It ends up imitating the standard algorithm, the long-addition method from grade school, working through it one digit at a time. But it's still picking each digit by what looks statistically likely instead of doing the actual math. In our case, that's where it slipped and wrote a single wrong digit.
That step-by-step working out is what makes it a reasoning model. Before answering, it reduces one hard problem into a chain of simpler ones it can handle, which is how the run above got 199 of 200 right. It's a real jump, but each step is still a guess, so it never quite reaches a guarantee.
Yet drop the same problem into ChatGPT or Claude and it gets it right. Something outside the model is stepping in.
The app gets around the model's weak math by handing the problem to an outside tool. When it notices a hard math problem, it sends the numbers to a server running Python and lets a small script do the calculation instead of the model.
That wrapper around the model is called the harness. A harness is just whatever code runs around the AI. It can be all sorts of things, but here it's the part that catches the math problem and calls the Python script.
Here's how it works.
To see this in action, watch how your ChatGPT or Claude app handles a math prompt. If the answer is simple enough to already be in its training data, it just responds directly. If the problem is a little harder, it first tries to work it out itself with the standard algorithm. And once the math gets complex enough that neither of those is reliable, it s, reaches for the Python tool, and computes the exact answer. The model is trained to judge which of these three routes fits the problem in front of it. When it decides it needs the Python tool, the harness is there to execute the tool call.
The whole thing feels like a bit of a hack. In the end, language models only output text. So to make tool calling work, developers train the model to spit out a specific JSON structure whenever it wants a tool. The harness watches the model's output, spots that JSON, runs the tool, and feeds the result back so the model can write the final answer. The illustration above walks through the raw events if you want to see exactly what that looks like.
Tool calling took off around 2023. Meta's Toolformer trained a model to reach for tools like a calculator on its own, and OpenAI soon made it reliable enough to ship in ChatGPT. You can see the tools any app uses by asking it to list all your internal tools
.
Early on, developers wired every tool into the harness by hand. If you wanted a model to read your GitHub issues or query your database, someone had to build that specific integration into that specific app.
Anthropic standardized this in late 2024 with the Model Context Protocol, or MCP. Instead of baking each tool into the harness, you run a small MCP server that advertises what it can do, and any MCP-aware app can plug into it. That Python calculator from earlier could sit behind an MCP server, and Claude, Codex, and Cursor could all reach for it without anyone writing the same integration three times.
Underneath it's the same tool-calling loop. MCP just agrees on a common shape for how a tool describes itself and hands back its results, so a tool gets built once and every app can use it.
AI is stuck in the past #
Every model is trained on data up to a certain date, and that date is its cutoff. After that the model is frozen and doesn't know about anything newer.
Take GPT Sol High, with a cutoff of February 2026. Ask its API who's in the 2026 World Cup final with web search off, and it can't give you the right answer.
Who won the World Cup Final 2026?
gpt-5.6-sol
answeredThe 2026 FIFA World Cup Final has not yet been played. It is scheduled for July 19, 2026, at MetLife Stadium in New Jersey.
So why did it get it wrong? The final is in July 2026, past its cutoff, so it never made it into the training data.
The way around this is RAG, short for Retrieval-Augmented Generation. That web search we just turned off is a version of it. With it on, the model can look up what's current and answer based on that. There are a few types of RAG. The web search one pulls from the open web. The basic version works off a document you give it instead. They both end up handing the model outside information, but they go about it pretty differently. Here's the basic version of the document one.
As context windows get bigger, do we still need RAG?
Yes. Even a 10 million token window is tiny next to a real codebase or document store, and stuffing everything in is slow, expensive, and something models handle unevenly, since they tend to lose track of the middle. Pulling in just the relevant piece stays the cheaper, sharper move, and that holds even as the shape of retrieval keeps changing.
Do apps like Codex and Claude Code use RAG?
For navigating code, mostly not the classic version. Instead of embedding your whole codebase into a vector database, they lean on agentic search, where the model runs grep, lists files, reads them, and follows imports the way a developer would. Anthropic pulled the vector index out of Claude Code in 2025, and Codex's CLI is told to prefer ripgrep for the same job. Vector RAG is still the norm for document question answering and large knowledge bases, just not for reading code. Boris from the Claude Code team explains the reasoning here.
AI has a short memory #
A regular program can pull millions of rows from a database and only load the one it needs. A model can't do that. Each request to a model carries the entire transcript so far, your messages and its own, all fed back in at once. What looks like one new message is the whole conversation, sent again.
It has to work this way because the model keeps no memory between turns, so every time it replies it re-reads the whole conversation from the top. There's a hard cap on how much of that text it can take in, and that cap is called the context window.
Why resend the whole transcript every time instead of just passing each message one by one?
Because the model is stateless. There is no storage inside the LLM, and its weights have stayed frozen since its cutoff date. So every request starts from a blank slate, and the only way it knows what came before is to see the whole conversation again. If the system sent only your latest line, the model would have no idea what you said a moment ago, so the transcript has to come along as the memory, and there is no memory unless the system resends it.
If the whole transcript has to be sent anyway, why send all of it on every single query? Why not send just the new message each time and let the model append it to what it already has?
It's technically possible, but it would be hugely wasteful.
At this point it stops being an LLM problem and becomes a hardware one. To keep the conversation on its side, the server would have to store the work the model did while reading the transcript. It keeps that work in a special cache called the KV cache. Without that cache the model would redo everything each step, so generating the 1,000th token would mean pushing all 999 before it back through from scratch. The cache holds the math already done, so each token gets computed only once.
The KV cache lives in VRAM, the fast memory attached to the GPU (think of it as RAM made for and soldered onto the GPU). VRAM is scarce and expensive, and a single model serves millions of people at once, so pinning every idle chat there while people read and type would eat up the most limited resource on the machine.
Your next message might not even hit the same machine. Requests get spread across a huge pool of GPUs, and the one holding your state may not be the one that takes your next turn. So resending the whole transcript ends up being the cheaper and more efficient way to do it.
There are really two ceilings here. One is the hardware, the KV cache and the VRAM it sits in, which caps what a single run can afford. The other is the context window, set in the model itself, the most text the model can work with in one turn, your message and its answer together.
The two rise together. A bigger context window needs a bigger KV cache to hold it, so it leans even harder on VRAM. But more VRAM alone won't buy you a bigger window.
Because the window is baked into the model, growing it also means training the model on longer stretches of text. A model only learns where each word sits in line for the lengths it was trained on, and once the text runs past that length it loses track of word order completely. That trained length is what sets the ceiling on the window, so the only way to lift the ceiling is to train on longer text from the start.
Two things have pushed context windows up fast, and both come down to spending. More money goes into VRAM, which buys room for a larger KV cache, and bigger training budgets let labs train models on much longer stretches of text. Back in the GPT-3.5 days you got about 4,000 tokens, a few thousand words before it started dropping the thread. These days 2 million is basically the floor for a serious model, and Google has pushed Gemini as far as 10 million in its own testing.
Still, it helps to keep the scale in perspective. 10 million tokens sounds like a lot, but it's actually pretty small. It works out to only about 40 MB of text. Plenty of the videos and images on your phone are already bigger than that, so feeding a model that kind of content fills the window fast.
AI is toxic #
RAG keeps a model's facts fresh, but it can't stop it from occasionally spitting out weird, unrelated, or toxic answers. That is because the model learned to talk from the internet, and the internet is itself weird and toxic, so those parts leak into what it says. When a model goes off the rails like that, it means the AI is unaligned.
The same next-word prediction that trips it up on math is what makes it toxic too. A cruel or false sentence can be the statistically likely one, and the model has no way to know, only that it fits. Alignment is the work of breaking that reflex, so the model learns to aim for a response that helps the person asking. Reinforcement learning is how you do it.
Why can't the model just learn this during pre-training?
Pre-training has one objective, predict the next word, and it runs over raw internet text that nobody sorted into helpful or harmful. You can't fold helpfulness into a process whose only yardstick is prediction. It needs feedback on whole answers, good or bad, and that is the job reinforcement learning does.
It is also far cheaper to do it this way. Pre-training is one enormous run over the whole internet that costs millions, so it happens in a single long session you don't get to redo easily. Alignment runs on a much smaller dataset, cheap enough that you can adjust the model, test how it behaves, and run it again, round after round, until it settles.
The RL process also works differently. Instead of copying a fixed dataset, the model is prompted with a question and gives an answer. That answer earns a reward based on how good it is, and the reward nudges its weights toward answering that way again. Those weights are the model's parameters, the millions or billions of numbers that decide how it responds, and their count is what people mean when they talk about a model's size. A 70-billion-parameter model is just 70 billion of these dials, each one nudged a little during training. It learns by trial and error.
The idea came from games and robotics, where the reward is easy to measure. You win the game or you don't. The score goes up or it doesn't. That gives the model a clean target to chase, a single number it can try to push higher.
Some things have no obvious score. You can't write a formula for a helpful answer, so a person has to judge.
OpenAI showed this back in 2017 by adding a human to the loop, rewarding a simulated agent based on whether it pulled off a backflip or not. There was no way to score a flip automatically, so every so often the person saw two short clips of the agent flailing and picked the one closer to a backflip, and that attempt got the positive reward. The nearer it came to a real flip, the more reward it earned, and about 900 of these judgments were enough to teach the whole move. You can watch it happen below.
The same trick grounds language models. People compare the model's answers, and it drifts toward the ones they prefer. This loop is called Reinforcement Learning from Human Feedback, or RLHF for short, and GPT-3.5 was the first big language model grounded this way.
To ground a base model like GPT-3.5, you start with a dataset of human-written questions and answers like this, usually thousands to millions of examples, nowhere near the petabytes used to pretrain a model from scratch.
OpenAI built their own dataset from scratch for InstructGPT, the paper behind what became ChatGPT and GPT-3.5. They hired about 40 contractors through Upwork and Scale AI to write and rate the examples.
From there, the full RLHF pipeline runs in a handful of stages. Here it is, step by step. RLHF got GPT-3.5 to behave most of the time, but it was leaky. Within weeks of ChatGPT's launch people were jailbreaking it, wrapping a request in a role-play or a fake developer mode until the model talked itself past its own training. The best known was DAN, short for Do Anything Now, a prompt that told the model to act as if it had no rules at all.
Even after RLHF, the model was still partly toxic. So what did newer models do to improve it?
Some of the progress came from more RLHF, retrained on the exact jailbreaks people kept finding. But two other approaches mattered more, and they push alignment beyond RLHF alone.
1. Using synthetic data instead of human-written data
RLHF is still needed, but synthetic data is how you scale and improve it fast. You start from the questions and answers humans already wrote, then use a model to generate many more in the same vein. You can also train a separate model to rate the answers, so one AI grades what another AI wrote and the human drops out of the loop.
Anthropic's Constitutional AI is the clearest version. You give the model a constitution, a short list of written principles, and it critiques and revises its own answers against them. It even labels which answers are more harmful on its own, so no human has to rate that part. The constitution is where its safety values come from, though helpfulness still leans on human feedback.
By now this reaches past alignment. Anthropic's transparency page lists newer models like Claude Fable 5 as trained on a mix of internet text, public and private datasets, and synthetic data generated by other models. It has become a normal ingredient across the whole pipeline, in both the pretraining mix and the post-training steps above.
It has also changed what a small model can do. Microsoft's Phi models are the clearest case, trained largely on synthetic textbook-style data and matching models several times their size on math and reasoning. The catch is quality. Well-curated synthetic data is what lifts a small model, while low-quality generated text can just as easily stall one.
2. Adding guardrails around the model
Guardrails are a layer that sits outside the model instead of inside its weights. Separate classifiers read what goes in and what comes out and block the clearly unsafe stuff, and a
system prompt People keep finding new jailbreaks, and each model closes the ones that came before it. That back and forth isn't going away. RLHF is still the heart of this, but it has help now, with synthetic data and guardrails working alongside it.
Where AI progress goes next #
Every weakness already has a practical workaround. The model hands complex arithmetic to an external tool, a code sandbox or calculator that returns the exact answer. For facts past the training cutoff, retrieval augmented generation pulls fresh documents into the prompt. And when safety or helpfulness slips, fine-tuning and reinforcement learning pull the output back into line. These three challenges are largely handled.
The frontier that's still wide open is time, what happens when you ask a model to keep working on one problem for a long stretch. That's where the next wave of progress is aimed.
Every hard problem that follows traces back to wanting one thing.
Long-running agents
Picture asking an agent to build an online store for your business, the whole thing, from the first line of code to a checkout that takes real orders. It designs the site, writes it, tests it, and puts it live, then stays on after launch. It watches how customers move through the store and ships the changes it thinks will sell more. It runs the support chatbot and the CRM behind it, remembering who bought what and following up. And as your business grows, it grows with it, learning from your own sales data what to build next. That's what everyone is building toward, an agent you can point at a real goal and trust to carry it on its own. People are already wiring models into harnesses built for exactly this. But the moment you run one for real, a few hard problems surface, and that's where the research is heading.
An agent that runs for a long time has to keep a lot in mind, so the obvious move is to grow the context window, and every provider has done it. The catch is that a bigger window doesn't mean the model uses all of it. Test after test finds the same thing. Models hold on to only part of a long window, the start and the end, and lose the thread in the middle.
Put the answer in the middle of the window and that's right where the model is worst at finding it.
Models can hold the whole document now. What they still can't do reliably is reason across all of it and lean on the middle as hard as they lean on the edges.
So people patch around it by hand. That's why the standard advice is to open a fresh chat whenever you switch topics, since a thread that's wandered through three unrelated problems is the kind of bloated context the model handles worst. The better fixes compress the context, or write the important state to a Markdown file so a new thread can read it and pick up where you left off instead of rereading everything. None of that fixes the weakness. It just keeps the window short enough that the weakness rarely bites.
Those tricks are enough for a single chat. For an agent meant to run for days, the same handoff becomes the whole design. It can't hold the store's full history in one window, so the harness cuts the job into short sessions and has the agent build one feature at a time. At the end of each session it commits its work and writes down what it finished and what's still open in a running log, so the next session can start cold, read that log, check the site still runs, and pick up where the last one stopped. The context stays small because the memory now lives outside it, on disk and in git, in files the agent keeps current.
Memory outside the window keeps the agent from getting lost, but it says nothing about how much work to take on at once. Left to itself an agent drifts one of two ways. It either swallows the whole project in one go and runs out of window partway through a feature, or it does a slice and calls the job finished before it is. The harness answers both by handing the agent an explicit list of what counts as done, and by making it test each feature the way a customer would before checking it off. On a run this long, the agent's own word that something works doesn't count for much.
All of that works toward one thing, keeping the model accurate across a long window, and accuracy is only half the problem. The window is expensive in plain hardware terms too. Every token the model is holding has to sit in fast memory as a KV cache, and that memory is the scarce, costly part of running one of these systems. Double the context and you roughly double what has to stay live on the GPU, so longer runs cost more to produce. That VRAM is scarce enough that demand for it has become a real driver of the rising memory prices of the last few years.
Both problems trace back to the same habit of stuffing everything into the window. The deeper fix is to stop leaning on it so hard. Right now a model's weights are frozen after training and reinforcement post-training, so everything it learns mid-task lives only in the context and disappears when the window clears.
The ambition is to change that, to let the model fold what the user teaches it back into its weights and learn from the context instead of fighting to hold all of it in view. That's continuous learning, and it's the real prize behind long-running agents. We're not there yet. Until we are, the whole thing runs on harnesses and context-management tricks that hold the accuracy loss and the cost at bay.
If every lab is converging on the same benchmarks and context length, does it still matter which one you build on?
It does. Three things still separate them:
RL tuning— how each lab shapes behavior and draws its safety lines. Anthropic trains against a published constitution; Grok bets the other way, treating heavy tuning as a tax on usefulness.Harness design— how the model uses tools, search, and its own context once a task runs long. On a single-turn question it barely matters; stretch the task out and what counts is whether the harness catches a bad tool call and recovers or just builds on it.Context management— they all summarize under the hood rather than drop old turns. What differs is the trigger point, what survives a compaction, and how much of the advertised window is usable after the harness's overhead.
Only the first of these is public. A lab publishes its values in a constitution or a Model Spec, but harness design and context management never show up in a document like that. The only way to know how a model behaves on your workload is to run your own eval against your own task.
The model is still bad at math, stuck at its training cutoff, short on memory, and prone to the occasional ugly answer. None of that got fixed in the model. What changed is the engineering around it, the harness, retrieval, the KV cache, the guardrails. That's the whole reason these tools went from something you couldn't trust with code to something I open every morning.
Long-running agents are the next round of this. Every new pattern people try, the agent graphs, the loops, the context tricks, routes around one of these same limits. Once you can name the limit, the trick makes sense, and you can read the latest work and try it yourself while it's still new instead of chasing the news cycle.
More from the blog #
- 3 min read
Intro: I still write in the Age of AI and you should too
A personal counter-movement against passive consumption to save what’s left of my attention span.
- 3 min read
How to Think Through Hard Questions
Why constraints are the answer, and writing is the tool
- 24 min read
Addis Ababa’s Water Progress Trap
A look at Addis Ababa's water shortage, the stalled Stage III plan, and the choices the city still has before the 2050 crisis arrives.
Have a product to build or fix? #
I take on a small number of contract and consulting projects: new builds, MVP-to-production rebuilds, and platform work. Tell me what you're building and I'll reply within a day.
For inquiries, errors, or statements that feel misleading or wrong, email me at eyosias.dev@gmail.com. You can also join the discussion in the @yosi_koda Telegram channel.