LAI #141: The Questions AI Can’t Answer

Towards AI's newsletter #141 highlights the importance of session state in multi-turn AI conversations, noting that retrieval systems often fail when they only use the latest user message. The issue also features a community-built C implementation of Qwen 3.5 inference, which exposes lower-level operations for educational purposes. Louis-François Bouchard, Co-founder of Towards AI, emphasizes the need for context in retrieval and introduces the Towards AI Mentorship program offering access to 15+ senior AI engineers.

Good morning, AI enthusiasts A lot of AI engineering comes down to decisions that do not have one clean answer. Which trade-off matters more here? Is this system actually failing because of the model, the context, the retrieval layer, or the infrastructure around it? When should you optimize, and when should you leave something alone? This week’s issue is full of those kinds of decisions. You’ll learn: There’s also a community-built Qwen 3.5 implementation in C, a new poll on how people are working around coding-agent limits, and a few collaboration opportunities from the community. Let’s get into it I create a lot of resources on becoming an AI engineer. We have courses, tutorials, videos, a book, and even a GitHub roadmap that aims to organize the entire learning path. All of that is useful. But when I think about how I actually learned most of what I know, a huge part of it came from asking people questions. A lot of questions. For me, the problem was always access. I had to find someone who knew more than I did, hope they had time to answer, wait for a response, and often realize that the answer created three more questions. That is really the idea behind Towards AI Mentorship . We wanted to create a place where you can ask the questions that become too specific for a course, and where an AI can give you an answer but perhaps shouldn’t be the only opinion you trust. Instead of having one person you can occasionally ask, you have access to 15+ senior AI engineers and mentors. You can ask questions asynchronously, join two live sessions each week, bring in your architecture or project for feedback, and get help on the career side too, including resumes, portfolios, and projects. You can read more about why we built it in my blog post https://www.louisbouchard.ai/towards-ai-mentorship-for-ai-engineers/?utm source=chatgpt.com , or see everything included on the mentorship page https://towardsai.com/academy/mentorship/ . In a multi-turn conversation, the user’s latest message is written for the conversation, not for your retrieval system. It often depends on details mentioned several turns earlier. A simple example we use in the Context Engineering lesson of our Agent Engineering course https://towardsai.com/academy/agent-engineering/?utm source=newsletter&utm medium=email&utm id=AItips shows why it is important to know this. A user reports a mild headache and wants to avoid medication. A few turns later, they ask: “Could stress be causing this?” If you retrieve using only that question, the system no longer knows what “this” refers to. It also reduces the severity of the headache and hides the user’s preference for avoiding medication. Instead, maintain a small session state with the important facts from the conversation. Keep the last two or three raw messages as well, then build the retrieval query from that state plus the newest message. To check whether this actually improves retrieval, test it on 20 follow-up questions. Run each one twice: once with only the latest message, and once with the session state. Track: This gives retrieval the context it needs without sending the entire conversation back to the model every time. — Louis-François Bouchard, Towards AI Co-founder & Head of Community Exquisite peacock 20933 https://discord.com/channels/702624558536065165/983037843532308500/1543687057929535548 shared qwen35-in-c, a from-scratch C implementation of Qwen 3.5 inference. Rather than relying on PyTorch or a full serving framework, the project exposes the lower-level operations involved in loading the model, running its attention and linear-attention layers, maintaining state, and generating tokens. It is especially useful if you want to understand what an LLM inference runtime is actually doing beneath libraries such as Transformers or vLLM. The implementation is small enough to inspect rather than another abstraction to learn around. Check out the repo https://github.com/promptsmith1990/qwen35-in-c and support a fellow community member. If you have questions or feedback, share them in the thread https://discord.com/channels/702624558536065165/983037843532308500/1543687057929535548 . What stands out to me is that 47% of you hit your subscription limit at least once a day, while another 29% rarely or never do. That is quite a wide gap for the same generation of coding tools. I don’t think this necessarily measures how much someone codes with AI. It may measure how they have designed their workflow around AI. Long-running sessions, large contexts, higher reasoning effort, parallel agents, and more autonomous tasks can consume dramatically more capacity than short, bounded tasks. Both Claude Code and Codex explicitly meter usage differently depending on context, task complexity, model choice, and agentic work. So I’m curious whether heavy AI users eventually split into two camps: those who keep buying more capacity and those who redesign their workflows to make each token go further. For those hitting the daily limit, what have you optimized first: model routing, shorter contexts, splitting work across agents/providers, API fallbacks, or simply paying for more capacity? And for the heavy users who rarely hit a limit, I’m especially interested in your setup. That’s the more useful workflow to learn from. Share it in the thread https://discord.com/channels/702624558536065165/833660976196354079/1544094652716683284 The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel https://discord.gg/rj6m9AF7eC Keep an eye on this section, too — we share cool opportunities every week 1. Siammzm https://discord.com/channels/702624558536065165/1543076528693190818/1543076528693190818 is looking for a learning partner to build AI skills through studying, experimenting, discussing ideas, and building things. If you are also working towards the same goal, connect with them in the thread https://discord.com/channels/702624558536065165/1543076528693190818/1543076528693190818 2. Likelytobelaura. https://discord.com/channels/702624558536065165/998978160605540454/1544752703308042301 is building ConspiracyBench, a benchmark to evaluate whether models believe various conspiracy theories, and is looking for someone interested in working on the benchmark and eventually fine-tuning the model. If this sounds interesting, reach out to them in the thread https://discord.com/channels/702624558536065165/998978160605540454/1544752703308042301 3. Prafullmishra. https://discord.com/channels/702624558536065165/784477688551178240/1543227268875419778 is starting a group to build ML projects and share resources. If you are pursuing a similar track, contact him in the thread https://discord.com/channels/702624558536065165/784477688551178240/1543227268875419778 Meme shared by bin4ry d3struct0r https://discord.com/channels/702624558536065165/830572933197201459/1543812796024750141 LLM Continuous Batching Explained: The Secret Behind Fast LLMs https://pub.towardsai.net/llm-continuous-batching-explained-the-secret-behind-fast-llms-d4d8cbe083b6?sk=87af2a0f5be3be53f6cfcfe172b3f89b by Divy Yadav https://yadavdivy296.medium.com/?source=post page---byline--d4d8cbe083b6--------------------------------------- Continuous batching is why LLM responses stay fast even under heavy concurrent load. This article builds on that idea from single-request serving and static batching, then explains how prefill, decode, PagedAttention, and chunked prefill fit into the same system. It also separates continuous batching from dynamic batching and speculative decoding, with benchmarks demonstrating that scheduling alone can yield substantial throughput gains. 1. Tuning vLLM: What Every Setting Does to the Arithmetic https://pub.towardsai.net/tuning-vllm-what-every-setting-does-to-the-arithmetic-28d5f65e231b?sharedUserId=tai-tech by Satsawat Natakarnkitkul Net https://medium.com/@net satsawat?source=post page---byline--28d5f65e231b--------------------------------------- vLLM has dozens of settings, but only a handful usually deserve attention, and each controls a different bottleneck. This article maps six important knobs to the resource they actually change: concurrency, KV-cache capacity, work per scheduling step, prefix reuse, KV-cache precision, and eager execution. It also explains why vLLM’s unified token-budget scheduler allows chunked prefill, prefix caching, and speculative decoding to work together rather than as separate optimizations. Most importantly, it shows how to tune against your own traffic using TTFT, TPOT, throughput, and goodput instead of chasing a single benchmark number. 2. The Ultimate Guide to LLM Inference Optimization https://pub.towardsai.net/the-ultimate-guide-to-llm-inference-optimization-part-1-71ec501c189d?sk=1eb0c40d4774a401a014c7f8f8966e83 by Ashish Abraham https://medium.com/@ashishabraham02?source=post page---byline--71ec501c189d--------------------------------------- A large model can fit on paper and still be impractical to serve once weights, KV cache, latency, and throughput are factored in. This guide works through the model-level techniques that make inference cheaper: quantization, distillation, and speculative decoding. It shows the arithmetic behind mapping FP32 weights to lower-precision formats, explains when post-training quantization differs from quantization-aware training, and compares knowledge transfer through logits with Orca-style reasoning traces. It then shows how approaches such as EAGLE and Medusa use draft predictions to accelerate training of larger models without changing their final output distributions. 3. Polynomial Degree And Turning Points, Made Simple, Really Simple https://pub.towardsai.net/function-zoo-polynomial-rational-piecewise-inverse-a48c442aad56?sk=e1ad30c7ae3d40b3e2cfa2e2a0c42dc2 by Kamrun Nahar https://iknahar.medium.com/?source=post page---byline--a48c442aad56--------------------------------------- You can often predict what a function will look like before plotting a single point. This article builds that intuition through real examples instead of starting with definitions: UK stopping distances reveal how linear and quadratic terms combine, the MPG illusion shows what happens when you invert a ratio, and Runge’s classic experiment explains why higher-degree polynomials can behave badly at the edges. Tax brackets, skid marks, and Anscombe’s quartet extend the same reasoning to other function families. 4. Your Second GPU Is Bought for the Cache, Not the Model https://pub.towardsai.net/your-second-gpu-is-bought-for-the-cache-not-the-model-2d022eee49a7?sharedUserId=tai-tech by Satsawat Natakarnkitkul Net https://medium.com/@net satsawat?source=post page---byline--2d022eee49a7--------------------------------------- Adding GPUs does not solve one scaling problem; it can solve several completely different ones. This article separates the reasons to scale into model capacity, throughput, and KV-cache capacity, then maps each to data, tensor, pipeline, or expert parallelism. It also covers the problem that another GPU alone does not fix: prefill and decode competing for the same hardware, which systems such as DistServe, Mooncake, and NVIDIA Dynamo address by separating them into different pools and moving KV cache between them. If you are interested in publishing with Towards AI, check our guidelines and sign up https://contribute.towardsai.net/ . We will publish your work to our network if it meets our editorial policies and standards. LAI 141: The Questions AI Can’t Answer https://pub.towardsai.net/lai-141-the-questions-ai-cant-answer-9b506f5d2921 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.