LAI #148: What Happens Between Prompt and Output?

Towards AI co-founder Louis-François Bouchard outlined a change to the Nova research agent that ranks sources on novelty as well as relevance, after its searches kept surfacing low-quality pages and multiple sources repeating the same point. Nova now keeps sources that pass its filters as short summaries and reads only five in full, partly based on whether each adds information missing from what the agent already collected. Bouchard also described testing 78 models across 127 configurations with 6,350 generated scripts and 25,400 judge assessments in ToneBench, which found the judges themselves have measurable preferences even when model identities are hidden.

Good morning, AI enthusiasts A lot of the frustrating bugs in AI systems now happen after the model has already done its part. The policy agent gets the right answer, but drops an important condition when it hands the task off. The batch job finishes, but your app never hears about it. Your latency dashboard looks fine, but the user is still waiting too long for the first token. Or the coding agent completes the command and leaves the credential sitting in its transcript. This week’s issue has a lot of useful examples of how to deal with those problems. You’ll learn: Let’s get into it This week in What’s AI, I’m sharing the setup I now use to switch between Claude Code and Codex without rebuilding context every time. The main change was moving my notes, skills, scripts, meeting recaps, and recurring instructions into an Obsidian vault that both agents can read. An index tells them what to load for each task, so I can give an agent the context it needs without passing my entire archive into every prompt. I also keep feedback with each skill, so repeated corrections can become part of the default workflow. I still use Codex, but I often let Claude Code plan the work and call Codex through MCP to execute it. I also move non-urgent jobs, such as daily news recaps, into scheduled runs so they do not compete with my daytime usage. The idea is to keep useful context and skills outside any one model or coding tool, so changing agents doesn’t mean starting over. The full article goes through the vault structure, the role of skills and feedback files, how I split planning from execution, and how the setup follows me across devices. Read it here https://louisbouchard.substack.com/p/why-i-stopped-letting-codex-lead or watch the video on YouTube https://www.youtube.com/watch?v=KeyqBiQYMok . When developers first give an agent search access, relevance is often the main filter: find pages related to the query, then let the model read them. That becomes expensive as the system scales. More searches bring in more pages, and relevant pages often overlap. You waste context, add latency, and spend money processing the same information multiple times. We ran into this while scaling Nova, the research agent we built in Agent Engineering https://towardsai.com/academy/agent-engineering/?utm source=TAI&utm medium=email&utm id=AItips . Its searches kept surfacing shady blogs, sales pitches, low-quality pages, and several sources making essentially the same point. So we stopped ranking sources on relevance alone. Nova now checks three things: Sources that pass the first filters stay in the research file as short summaries. Nova then chooses only five to read in full, partly based on whether they add something missing from the material already collected. If your agent already ranks search results by relevance, add novelty before loading the full page. Compare each source with what the agent already has. If it mostly repeats the same information, keep the summary or skip it. — Louis-François Bouchard, Towards AI Co-founder & Head of Community My AI Engineer talk on why AI writing still reads like slop is now live on YouTube I walk through ToneBench, where we tested 78 models across 127 configurations using 6,350 generated scripts and 25,400 judge assessments. I cover how we measure tone, substance, structure, hooks, slop, and consistency, and what the results show about model choice, reasoning effort, and cost. I also get into one of the harder parts of benchmarking writing: the judges themselves have measurable preferences, even when model identities are hidden. Panelreader. https://discord.com/channels/702624558536065165/983037843532308500/1561416545601720480 built a neural network from scratch in Python and NumPy to understand what frameworks like PyTorch and TensorFlow handle behind the scenes. The project implements a 2 → 3 → 1 XOR network with forward propagation, backpropagation, and gradient descent written manually. After 5,000 training epochs, the network correctly predicted all four XOR cases with a final MSE of 0.004715. The repo also includes mathematical explanations, training visualizations, and 12 editable diagrams that walk through how the network works. Check it out on GitHub https://discord.com/channels/702624558536065165/983037843532308500/1561416545601720480 and support a fellow community member. If you have feedback on the code or documentation, share it in the thread https://discord.com/channels/702624558536065165/983037843532308500/1561416545601720480 The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel https://discord.gg/rj6m9AF7eC Keep an eye on this section, too — we share cool opportunities every week 1. Hendrickop https://discord.com/channels/702624558536065165/1561629695571927090/1561629695571927090 is looking for an AI/ML learning partner. He is a beginner but wants to learn quickly and work on projects. If you are following the same path, connect with him in the thread https://discord.com/channels/702624558536065165/1561629695571927090/1561629695571927090 2. Martocierny https://discord.com/channels/702624558536065165/998978160605540454/1558427732512079933 is putting together a team to build a 2D multiplayer sandbox in the browser and is looking for coders, artists, testers, and even complete beginners. This is a hobby project, but if it sounds fun, reach out to him in the thread https://discord.com/channels/702624558536065165/998978160605540454/1558427732512079933 3. Gian 16992 https://discord.com/channels/702624558536065165/784477688551178240/1559773423767392267 is moving from backend orientation to AI engineering and is looking for people to discuss ideas, share resources, make projects, etc. If this sounds like a good next step for you, reach out to them in the thread https://discord.com/channels/702624558536065165/784477688551178240/1559773423767392267 Meme shared by bin4ry d3struct0r https://discord.com/channels/702624558536065165/830572933197201459/1562232543535829082 Dropout Finally Explained: Why Breaking Half Your Neural Networks Makes It Smarter https://pub.towardsai.net/dropout-finally-explained-why-breaking-half-your-neural-networks-makes-it-smarter-cb17ea5d1557?sk=f0f3dbbab2dc6dfdb6ab1ae66d17a94d by Simple ML https://medium.com/@simpleml123?source=post page---byline--cb17ea5d1557----------------------------------------- Neural networks can fit their training data closely but still perform much worse on new examples. Dropout addresses this by randomly removing units during training, so the network repeatedly trains different weight-sharing subnetworks instead of allowing the same neurons to depend on one another. This article works through the original dropout paper, explains co-adaptation and the weight-scaling rule, and connects dropout to regularization in linear models. It also reviews results across vision, speech, text, and genetics, along with practical guidance on keep probabilities, network size, max-norm constraints, and the additional training time dropout requires. 1. LLM Observability in Production: Metrics, Logs, Traces https://pub.towardsai.net/llm-observability-in-production-metrics-logs-traces-f1d37298d3a9?sk=dab9f192c13cc009e6e56e79d0849e70 by Ray Hu https://medium.com/@coolercoder?source=post page---byline--f1d37298d3a9----------------------------------------- Traditional API monitoring misses several problems specific to LLM applications. A streaming request can have acceptable total latency while still making the user wait too long for the first token, and token usage adds a cost dimension that normal request metrics don’t track. This article shows how to measure time to first token, time per output token, token usage, and cost with Prometheus, separate high-volume metadata from full prompt logs, and connect requests across OpenTelemetry traces with a shared trace id. It also covers redaction, sampling, and alerting so teams can trace a slow or expensive request back to the component that caused it. 2. Multi-Agent Trajectory Evaluation, Explained Simply https://pub.towardsai.net/multi-agent-trajectory-evaluation-explained-simply-2ce6e465d744?sk=413617c1dbb577f45f77435394547501 by Satsawat Natakarnkitkul https://medium.com/@net satsawat?source=post page---byline--2ce6e465d744----------------------------------------- A multi-agent system can return the correct final message even when one agent makes a serious mistake along the way. In the article’s example, a policy tool correctly returned that a damaged item had to be sent back before a refund, but the policy agent omitted that condition when handing the case to the orchestrator, so billing issued the refund immediately. The article shows how to evaluate each agent’s tool use, the information passed between agents, and the final outcome separately. It also runs cases multiple times to detect inconsistent behavior and uses step-level failure attribution to identify where the error entered the trajectory. 3. LLM Batch Webhooks: 5 Mechanisms, Zero Lost Jobs https://pub.towardsai.net/llm-batch-webhooks-5-mechanisms-zero-lost-jobs-573949dc28f2?sk=899fa5926ff5a4335910631bc6883652 by Ray Hu https://medium.com/@coolercoder?source=post page---byline--573949dc28f2----------------------------------------- Polling a batch API creates a failure case when the batch finishes, but the process checking its status stops running. This article replaces polling with a TypeScript webhook flow designed for duplicate deliveries, temporary failures, and unavailable downstream services. It implements signature verification, atomic deduplication, idempotent side effects, exponential backoff, a dead-letter queue for failed events, and a circuit breaker for unhealthy endpoints. It also compares how providers handle batch completion and retries, and shows how to replay failed deliveries without processing the same job twice. 4. The Key Is in the Clipboard https://pub.towardsai.net/the-key-is-in-the-clipboard-f374fd0be86e?sk=d94b8610ec76f3e7b1bac661d5b5c82b by Enzo Lombardi https://enzolombardi.net/?source=post page---byline--f374fd0be86e----------------------------------------- Coding agents often need credentials to run commands, but pasting a password or API key into chat leaves it in the model context and can also store it in local session transcripts. After examining 661 Claude Code transcripts, this article builds lsv, a Rust vault that lets an agent pass a secret into a command without providing a normal command for reading the value back. Secrets enter through the clipboard, appear as redacted references during use, and require Touch ID approval before access. If you are interested in publishing with Towards AI, check our guidelines and sign up https://contribute.towardsai.net/ . We will publish your work to our network if it meets our editorial policies and standards. LAI 148: What Happens Between Prompt and Output? https://pub.towardsai.net/lai-148-what-happens-between-prompt-and-output-e6de86481ee7 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.