A friend of mine spent three weeks “training” his AI assistant. That’s the word he used, training, even though nobody had actually trained anything. He’d fed it his writing voice, the shorthand names for his projects, the way he liked his emails structured. By week two it felt like the thing finally got him.
Then one morning, nothing. It greeted him like a stranger. No writing voice, no project names, no shorthand. He was annoyed, and honestly, a little hurt. He told me, “I thought it remembered me.”
It never did. What he had was context. Nobody had told him the difference, and that gap between what people think AI does and what it’s actually doing is quietly costing the industry a lot more than a few disappointed users.
Context is everything currently sitting in the conversation window.
Every message you’ve typed, every response the model gave you, every document you pasted in, all of it lives in one active window that the model reads fresh each time you send something. It’s less like a memory and more like a very attentive read of the entire chat so far, every single time.
Close the tab, hit a length limit, or start a new session, and that window is gone. Not archived. Not filed away somewhere for later. Gone, the way a thought disappears the moment you stop holding it in your head.
This is closer to human working memory than anything else.
Working memory is what lets you hold a phone number in your head just long enough to dial it, or follow a sentence from its subject to its verb without losing the thread. It’s fast, it’s vivid, and it’s temporary by design. Nobody expects working memory to survive a good night’s sleep, yet somehow we expect context windows to.
Memory is different in almost every way that matters. It’s selective. It survives across sessions. And critically, it isn’t replayed, it’s retrieved.
Your brain doesn’t hand you a transcript of your last conversation with a close friend, it hands you the two or three things worth carrying forward: they just started a new job, they don’t eat meat, don’t bring up their dad right now.
Real memory in an AI system works the same way. Something gets written down after a conversation ends, and later, when it’s actually relevant, it gets pulled back out and placed in front of the model, not dumped wholesale but chosen for the moment. That act of choosing, deciding what’s worth keeping and what isn’t, is the entire job. It’s also the part almost nobody is doing well.
Context windows are easy to market. A number that goes from 8,000 tokens to 200,000 to a million reads like obvious progress, and in a narrow sense it is.
But the headlines rarely say “can process more text in a single sitting.” They imply something closer to “remembers more about you,” and that’s the sleight of hand.
A bigger window means the model can hold more of the current conversation before it starts dropping the earliest parts.
It says nothing about what happens the moment that conversation ends. You can have a context window large enough to hold an entire novel and still have an AI product with the memory of a goldfish, because size and persistence are not the same axis.
The illusion is strongest in demos, which is exactly why it spreads so easily. Inside one long session, an AI with a huge context window can feel eerily attentive, referencing something you said forty messages ago, adjusting its tone to match yours. It feels like being known. Then the session resets and the trick is exposed. What looked like memory was just a very long sentence that hadn’t ended yet.
I’ve sat in enough product conversations where the fix for “the AI doesn’t remember our users” was proposed as “let’s just stuff more into the context on every call.” It works, for a while, and then the bill arrives. Every extra token in context is a token you pay for and wait on, on every single request, whether it’s relevant or not. Latency creeps up. Costs creep up faster. And the model, buried under a pile of loosely relevant history, often gets worse at the actual task in front of it, not better.
Treating context expansion as a substitute for real memory is a bit like solving “I can’t find anything in my house” by building a bigger house and keeping every item you’ve ever owned in it. Technically everything’s still there. Good luck finding the car keys.
Memory done properly is a design problem, not an engineering flex. It means deciding, on purpose, what deserves to persist and what should be allowed to fade. Building the memory layer for an AI product taught me that the hard way. The instinct at first is to save everything, on the theory that more information can only help. It doesn’t. A model handed twenty loosely related facts about a user often performs worse than one handed the two facts that actually matter for the task at hand. Curation beats volume, almost every time.
A few things tend to separate memory that works from memory that just looks impressive in a pitch deck.
It’s selective. Not every message deserves to be remembered forever, and a system that treats a passing comment with the same weight as a stated preference will eventually drown in noise.
It’s retrieved, not recited. The right memory shows up at the right moment because something in the current conversation triggered it, not because the system unloaded everything it knows the second you said hello.
It’s visible and controllable. People deserve to know what’s being remembered about them and to be able to correct or delete it, the same way you’d want to correct a friend who misremembered something you said. A system that quietly remembers things you never agreed to have remembered isn’t building trust, it’s spending it without asking.
And it earns its keep over time rather than all at once. The best memory systems I’ve seen behave a little like a good colleague: they don’t need you to re-explain your job every morning, but they also don’t bring up something you mentioned once, offhand, three months ago, at an awkward moment.
Next time a product claims its AI “remembers you,” ask a different question than “how big is the context window.” Ask what it remembers, why it chose to remember that, and what happens the day after your conversation ends. That’s where the real answer lives, not in the token count on a spec sheet.
Context is what an AI is reading right now.
Memory is what it chose to keep.
The first one is easy to make bigger. The second one is what actually makes an AI worth coming back to.
Thank you!!
You can find more of my thoughts and work credibility on LinkedIn, Instagram, My portfolio .
If you would like to connect directly, feel free to reach out on WhatsApp at +91 955990461 or email me at nishkarshgupta381@gmail.com.
I am always open for my readers :)
Before you go
Context vs Memory: The AI Industry’s Most Expensive Confusion was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.