LAI #141: The Questions AI Can’t Answer

Good morning, AI enthusiasts!

A lot of AI engineering comes down to decisions that do not have one clean answer. Which trade-off matters more here? Is this system actually failing because of the model, the context, the retrieval layer, or the infrastructure around it? When should you optimize, and when should you leave something alone?

This week’s issue is full of those kinds of decisions.

You’ll learn:

There’s also a community-built Qwen 3.5 implementation in C, a new poll on how people are working around coding-agent limits, and a few collaboration opportunities from the community.

Let’s get into it!

I create a lot of resources on becoming an AI engineer. We have courses, tutorials, videos, a book, and even a GitHub roadmap that aims to organize the entire learning path.

All of that is useful. But when I think about how I actually learned most of what I know, a huge part of it came from asking people questions. A lot of questions.

For me, the problem was always access. I had to find someone who knew more than I did, hope they had time to answer, wait for a response, and often realize that the answer created three more questions. That is really the idea behind Towards AI Mentorship. We wanted to create a place where you can ask the questions that become too specific for a course, and where an AI can give you an answer but perhaps shouldn’t be the only opinion you trust.

Instead of having one person you can occasionally ask, you have access to 15+ senior AI engineers and mentors. You can ask questions asynchronously, join two live sessions each week, bring in your architecture or project for feedback, and get help on the career side too, including resumes, portfolios, and projects.

You can read more about why we built it in my blog post, or see everything included on the mentorship page.

In a multi-turn conversation, the user’s latest message is written for the conversation, not for your retrieval system. It often depends on details mentioned several turns earlier.

A simple example we use in the Context Engineering lesson of our Agent Engineering course shows why it is important to know this. A user reports a mild headache and wants to avoid medication. A few turns later, they ask: “Could stress be causing this?”

If you retrieve using only that question, the system no longer knows what “this” refers to. It also reduces the severity of the headache and hides the user’s preference for avoiding medication. Instead, maintain a small session state with the important facts from the conversation. Keep the last two or three raw messages as well, then build the retrieval query from that state plus the newest message.

To check whether this actually improves retrieval, test it on 20 follow-up questions. Run each one twice: once with only the latest message, and once with the session state. Track:

This gives retrieval the context it needs without sending the entire conversation back to the model every time.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Exquisite_peacock_20933 shared qwen35-in-c, a from-scratch C implementation of Qwen 3.5 inference. Rather than relying on PyTorch or a full serving framework, the project exposes the lower-level operations involved in the model, running its attention and linear-attention layers, maintaining state, and generating tokens. It is especially useful if you want to understand what an LLM inference runtime is actually doing beneath libraries such as Transformers or vLLM. The implementation is small enough to inspect rather than another abstraction to learn around. Check out the repo and support a fellow community member. If you have questions or feedback, share them in the thread.

What stands out to me is that 47% of you hit your subscription limit at least once a day, while another 29% rarely or never do. That is quite a wide gap for the same generation of coding tools.

I don’t think this necessarily measures how much someone codes with AI. It may measure how they have designed their workflow around AI. Long-running sessions, large contexts, higher reasoning effort, parallel agents, and more autonomous tasks can consume dramatically more capacity than short, bounded tasks. Both Claude Code and Codex explicitly meter usage differently depending on context, task complexity, model choice, and agentic work.

So I’m curious whether heavy AI users eventually split into two camps: those who keep buying more capacity and those who redesign their workflows to make each token go further.

For those hitting the daily limit, what have you optimized first: model routing, shorter contexts, splitting work across agents/providers, API fallbacks, or simply paying for more capacity? And for the heavy users who rarely hit a limit, I’m especially interested in your setup. That’s the more useful workflow to learn from. Share it in the thread! The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Siammzm is looking for a learning partner to build AI skills through studying, experimenting, discussing ideas, and building things. If you are also working towards the same goal, connect with them in the thread!

2. Likelytobelaura. is building ConspiracyBench, a benchmark to evaluate whether models believe various conspiracy theories, and is looking for someone interested in working on the benchmark and eventually fine-tuning the model. If this sounds interesting, reach out to them in the thread!

3. Prafullmishra. is starting a group to build ML projects and share resources. If you are pursuing a similar track, contact him in the thread!

Meme shared by bin4ry_d3struct0r

LLM Continuous Batching Explained: The Secret Behind Fast LLMs by Divy Yadav

Continuous batching is why LLM responses stay fast even under heavy concurrent load. This article builds on that idea from single-request serving and static batching, then explains how prefill, decode, PagedAttention, and chunked prefill fit into the same system. It also separates continuous batching from dynamic batching and speculative decoding, with benchmarks demonstrating that scheduling alone can yield substantial throughput gains.

1. Tuning vLLM: What Every Setting Does to the Arithmetic by Satsawat Natakarnkitkul (Net)

vLLM has dozens of settings, but only a handful usually deserve attention, and each controls a different bottleneck. This article maps six important knobs to the resource they actually change: concurrency, KV-cache capacity, work per scheduling step, prefix reuse, KV-cache precision, and eager execution. It also explains why vLLM’s unified token-budget scheduler allows chunked prefill, prefix caching, and speculative decoding to work together rather than as separate optimizations. Most importantly, it shows how to tune against your own traffic using TTFT, TPOT, throughput, and goodput instead of chasing a single benchmark number.

2. The Ultimate Guide to LLM Inference Optimization by Ashish Abraham

A large model can fit on paper and still be impractical to serve once weights, KV cache, latency, and throughput are factored in. This guide works through the model-level techniques that make inference cheaper: quantization, distillation, and speculative decoding. It shows the arithmetic behind mapping FP32 weights to lower-precision formats, explains when post-training quantization differs from quantization-aware training, and compares knowledge transfer through logits with Orca-style reasoning traces. It then shows how approaches such as EAGLE and Medusa use draft predictions to accelerate training of larger models without changing their final output distributions.

3. Polynomial Degree And Turning Points, Made Simple, Really Simple by Kamrun Nahar

You can often predict what a function will look like before plotting a single point. This article builds that intuition through real examples instead of starting with definitions: UK stopping distances reveal how linear and quadratic terms combine, the MPG illusion shows what happens when you invert a ratio, and Runge’s classic experiment explains why higher-degree polynomials can behave badly at the edges. Tax brackets, skid marks, and Anscombe’s quartet extend the same reasoning to other function families.

4. Your Second GPU Is Bought for the Cache, Not the Model by Satsawat Natakarnkitkul (Net)

Adding GPUs does not solve one scaling problem; it can solve several completely different ones. This article separates the reasons to scale into model capacity, throughput, and KV-cache capacity, then maps each to data, tensor, pipeline, or expert parallelism. It also covers the problem that another GPU alone does not fix: prefill and decode competing for the same hardware, which systems such as DistServe, Mooncake, and NVIDIA Dynamo address by separating them into different pools and moving KV cache between them.

If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards. LAI #141: The Questions AI Can’t Answer was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.