Good morning, AI enthusiasts!
AI systems accumulate a lot over time: conversation history, retrieved facts, user preferences, tool results, training feedback, and instructions.
Keeping all of it is expensive or noisy, but removing information is not as simple as deleting the oldest thing. An agent may need an old fact again several steps later. A model trained to forget something can lose useful reasoning ability in the process. Even a failed tool call can contain information the agent needs to fix its next attempt.
Several of this week’s articles look at different versions of that problem, such as deciding what information should persist, when it should return, and what happens when we remove too much.
You’ll learn:
We also have a closer look at bias in LLM judges and a simple way to make large prompts easier to maintain across different formats and writers.
Let’s get into it!
This week in What’s AI, I’m breaking down how we built ToneBench and, more importantly, how we tested whether the LLM judges behind it could be trusted.
Every model writes five drafts for each of ten tasks because the same model can produce very different results from the same prompt. Four judges then score every draft across tone, substance, structure, hooks, slop, and other dimensions. We also measured the judges themselves and found clear lab preferences: on the October 26 board, GPT-6.1 Sol gave OpenAI drafts 4.1 more points than the other LLM judges’ scores would predict, and Claude Opus 5.5 gave Anthropic drafts a smaller boost.
The article goes through the full benchmark setup, why we generate multiple drafts, how one script gets scored, how we calibrated the judges against human ratings, and how I would use the results for writing today. One practical idea I’m testing is generating several cheap drafts with a smaller model, then using a stronger judge to select and merge the best parts instead of paying for one expensive generation.
Read the full article here or watch the video on YouTube. When we built Brown, the writing agent in our Agent Engineering course, we started with one prompt that contained all its writing rules.
That was manageable for one format. Once Brown had to write articles, video scripts, and social posts for different writers, every edit carried more risk. A rule added for scripts could affect articles, and changing one writer’s voice meant touching instructions that had nothing to do with voice.
We split the prompt by how often each part needed to change.
Four profile files stay fixed across tasks: mechanics, structure, terms, and tone. Two are loaded dynamically: the format profile and the writer’s voice.
Adding a video format now means changing one file. Adding a new writer means adding one voice profile. The shared writing rules stay untouched.
If a prompt is becoming hard to maintain, map its instructions into stable and variable groups first. Pull out one variable group, run the same evaluation set before and after, and keep the split only if quality holds while the system becomes easier to change. — Louis-François Bouchard, Towards AI Co-founder & Head of Community
This issue is brought to you thanks to MongoDB*:*
Persistent memory only helps if the agent can decide what should stay.
A user preference may need to last across sessions. A temporary plan or intermediate result may only matter until the task is finished. Keeping everything adds noise. Deleting too much makes the agent forget useful context.
MongoDB’s new Vercel AI SDK integration adds multi-tiered memory in Atlas, so each memory type can follow its own retention policy. It can also automatically create the vector indexes needed for retrieval. So instead of treating memory as one permanent store, you can separate short-lived context from facts and workflows the agent may need later.
The integration is open source, supports AI SDK v6 and v7, and is available on npm.
__pop1 built AI-Atlas, a project that evaluates public GitHub repositories to show what developers have actually built across areas such as machine learning, computer vision, NLP, LLMs, and AI agents. Each project is scored using a seven-part rubric covering technical depth, originality, completeness, reproducibility, and other engineering qualities, with the evaluation linked back to the source code. Developers can also claim their profiles and get a step-by-step roadmap for improving each project. The tool has already mapped 218 developers and analyzed 2,216 open-source AI repositories. You can map your own GitHub profile for free without creating an account. Check it out here, and if you work in ML research, open-source AI, or developer tooling, share your feedback with the creator in the thread!
The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or want a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!
-
Mtnmcg is working on an alternative machine learning architecture that outperforms transformers and is currently looking for people who can review it, share feedback, and help with the project. If you are interested, contact him in the thread!
-
Bee_protego is building a tool that helps you understand which words in your AI prompts actually matter and is currently looking for help from people with an AI background to test and improve it. If you think you can help, reach out to them in the thread!
-
Carlos1256 is studying towards becoming an ML engineer and currently focusing on software engineering fundamentals. If you are working towards the same goal and need a study partner, connect with them in the thread!
Meme shared by jacekde
Tidal Attention: A Moon-Cycle-Inspired Mechanism for Long-Context Memory by Hayanan
Tidal Attention proposes a memory-selection mechanism that combines recency, semantic relevance, and recurring patterns so older information can return when it becomes useful again. The article connects the idea to RetNet, Mamba, LongNet, Infini-attention, and Titans, then tests it with a simple cache simulation. The tidal policy outperformed LRU but performed worse than frequency-based caching. It also lays out an evaluation plan with ablations to test whether the recurrence mechanism actually improves long-context memory.
How you connect an agent to its tools determines which decisions the model makes and which the application controls. This article shows four patterns with a money-transfer example: individual tool calls, parallel execution with asyncio, sequential chains with LangChain’s LCEL, and LangGraph workflows with branching, merging, and human approval. The code examples show when each topology is useful as the workflow grows more complex.
Post-training depends first on having feedback that reliably separates better behavior from worse behavior. This article maps SFT, DPO, KTO, PPO, and GRPO to the types of training data and feedback they require, then explains the differences between RLHF, RLAIF, and RLVR. It also covers rubric-based rewards, feedback collected from production traces, dataset versioning, and a formula for comparing training cost with expected gains. It also includes held-out evaluations, regression slices, and matched baselines that check whether optimization actually transferred to real workloads.
An agent can often recover from a failed tool call if the runtime gives it enough information to correct the mistake. In the article’s ticket example, the OpenAI Agents SDK returned an error without telling the model which priority codes the server accepted, turning a recoverable failure into a dead end. The author tests eight frameworks and six models, then implements six recovery patterns covering useful error messages, sanitized exceptions, stored-value validation, turn and budget limits, repeated-call detection, and pinned defaults. Adding the approved error information allowed both hosted models to correct every failed ticket call, showing how strongly recovery depends on the information the harness returns to the model.
A model can stop producing memorized information without actually removing it, which makes machine unlearning difficult to evaluate. This article reviews approaches including TOFU, selective disentanglement, and R-TOFU, along with research showing that some forgetting methods can also damage a model’s ability to reason from newly provided evidence. A separate 240-case policy-retrieval experiment found that removing outdated records improved decisions, while removing useful history made them worse. The article uses these results to separate forgetting across three layers: stored evidence, model behavior, and downstream decisions, and explains why each needs to be tested independently.
If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards. LAI #151: Can AI Forget Without Getting Worse? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.