365 Days of AI · Day 16 · ~7 min read
In Day 7, I built a chatbot that remembers everything.
Tell it your name in message 1 — it still knows it in message 20. The trick was simple: every time you send a message, the app packs the entire conversation history into the prompt and sends it to the model. The model reads it all and replies as if it was paying attention the whole time.
If you haven’t read Day 7, the one line summary is: AI has no memory. The app fakes it by showing the model a full transcript every single time.
But here’s the problem I never addressed.
What happens at message 200? Or message 500?
The transcript keeps growing. Every message adds more tokens. And eventually — you hit the context window limit. The model starts forgetting the oldest messages. The “memory” breaks.
Today I fixed that. Three ways.
Every time you send a message to an AI model, it doesn’t just see your latest message. It sees the entire conversation — your system prompt, every message you’ve sent, every reply it gave, and your new message. All of it, every single time.
The context window is simply how much it can see at once. Think of it as the model’s attention span for a single request.
Everything has to fit inside it:
🔹 System Prompt
🔹 Chat History
🔹 Current User Message
🔹 Retrieved Chunks (if you’re using RAG)
Different models have different limits:
🔹 GPT-3.5 → 16K tokens
🔹 GPT-4o & GPT-4.5 → 128K tokens
🔹 Claude Sonnet 4.6 → 200K tokens
Sounds huge. But a long conversation adds up faster than you think. 1 token ≈ 4 characters. A chatty 50-message conversation? Easily 3,000–5,000 tokens gone.
What happens when it gets too full?
The oldest information starts getting dropped — silently. The model doesn’t tell you it forgot something. It simply no longer has access to that context. That’s why long chats sometimes feel like the AI suddenly forgot what you discussed earlier.
And this is different from hallucination — an important distinction:
🔹 Context window issue = the model lost information because it no longer fits
🔹 Hallucination = the model makes up information even when the context is available
In production, this is usually handled using strategies like:
🔹 Sliding Window
🔹 Conversation Summarisation
🔹 RAG (Retrieval-Augmented Generation)
🔹 A hybrid approach combining all of the above
Today I implemented all three in my chatbot. Let me show you how.
The simplest fix is to just keep the last N messages:
const MAX_MESSAGES = 10function trimToLastN(history) { return history.slice(-MAX_MESSAGES)}
One line. Keep the last 10 messages, drop everything older.
The upside — dead simple. Easy to understand. Works.
The downside — it’s blunt. Message 1 could be 5 words. Message 2 could be 500 words. You have no idea how many tokens you’re actually sending. You’re just counting messages, not actual size.
Good enough for a quick fix. But we can do better.
Instead of counting messages, count tokens. Keep as many recent messages as fit within a token budget.
I added two functions to chatbot.js:
const MAX_TOKENS = 2048const CHARS_PER_TOKEN = 4function estimateTokens(text) { return Math.ceil(text.length / CHARS_PER_TOKEN)}function trimToContextWindow(history) { const systemTokens = estimateTokens(SYSTEM_PROMPT) let availableTokens = MAX_TOKENS - systemTokens const trimmed = [] // Walk from newest to oldest, keep what fits for (let i = history.length - 1; i >= 0; i--) { const msgTokens = estimateTokens(history[i].content) if (availableTokens - msgTokens < 0) break trimmed.unshift(history[i]) availableTokens -= msgTokens } const dropped = history.length - trimmed.length if (dropped > 0) { console.log(`context window: trimmed ${dropped} old message(s)`) } return trimmed}
Walk the history backwards from the newest message. Keep adding messages as long as there’s token budget left. The moment you run out — stop.
Much smarter than counting messages.
Trimming still has one problem — you’re dropping messages forever.
What if message 3 was “my name is Priyanka” and that gets trimmed? The model forgets your name. Not great for a chatbot that’s supposed to remember you.
The smartest fix: instead of dropping old messages, summarise them.
When the history crosses 10 messages, the app now asks Ollama to summarise the older half of the conversation into 3–5 sentences. That summary gets placed at the top of the context. The recent messages follow it.
const SUMMARY_THRESHOLD = 10async function summariseHistory(oldMessages) { const conversation = oldMessages .map(m => `${m.role === 'user' ? 'User' : 'Assistant'}: ${m.content}`) .join('\n') const prompt = `Summarise the following conversation in 3-5 sentences. Focus on key facts the user shared (name, preferences, important context). Be concise. Do not add commentary.Conversation:${conversation}Summary:` const response = await fetch('http://localhost:11434/api/generate', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ model: 'llama3.1', prompt, stream: false }) }) const result = await response.json() return result.response.trim()}
The final buildContextHistory() function ties it all together:
async function buildContextHistory(history) { // Short enough — send everything as is if (history.length <= SUMMARY_THRESHOLD) return history const splitPoint = Math.floor(history.length / 2) const summary = await summariseHistory(history.slice(0, splitPoint)) if (summary) { // Prepend summary, keep recent messages return [ { role: 'assistant', content: `[Summary: ${summary}]` }, ...history.slice(splitPoint) ] } // If summarisation fails - fall back to token trimming return trimToContextWindow(history)}
Three levels:
Nothing ever breaks. The model always gets the best possible context it can.
The trimming and summarisation only happens before sending to the model. The full history is always saved to disk.
// Model gets smart trimmed contextconst contextHistory = await buildContextHistory(data.history)// But disk gets everything - nothing lostdata.history.push({ role: 'assistant', content: assistantReply })saveHistory(data)
Your conversation file is a record. You might need it later for debugging, analytics, or future features. So the model gets a trimmed window — the file stays complete.
Before:
message received: hey what's my name?history loaded — 14 messages so farsending to Ollama...
After:
message received: hey what's my name?history loaded — 14 messages so farcontext window: history has 14 messages, summarising old ones...summariser: summary generated — The user's name is Priyanka...context window: using summary + 7 recent messagesestimated prompt size: ~1203 tokenssending to Ollama...
That one extra block of logs tells me exactly what’s happening under the hood every single time.
This felt like a small code change. It was actually a big mental shift.
Context window isn’t just a spec sheet number you read in documentation and forget. It’s an active constraint you design around. The apps that feel like they “just work” over long conversations? Someone built exactly this logic behind the scenes. It doesn’t happen by default.
And the models that charge more per token? Part of what you’re paying for is a bigger attention span. That 1M token Gemini model isn’t just impressive — it’s solving a real engineering problem.
Full updated code on GitHub: chatbot-app
I’m Priyanka — Senior Software Engineer, AI learner, and someone who just made her chatbot smarter without touching the model. I write about AI every day on this series. Follow along for Day 17.
Your AI Doesn’t Forget. It Just Runs Out of Space. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.