cd /news/ai-agents/reading-five-agent-studies-in-one-we… · home › topics › ai-agents › article
[ARTICLE · art-172092] src=pub.towardsai.net ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Reading Five Agent Studies in One Week: What Their Honesty Sections Taught Me

A review of five AI agent studies published in newsletters and Medium digests found that their limitations sections made the reported numbers more useful, not less, according to the author's reading notes. Hubert Nakielski's Towards AI study tested 12 memory configurations over a scripted 60-turn conversation with six checkpoints and three runs each, using graphiti-core 0.30.2 on FalkorDB with Haiku 5.5 at roughly $0.085 to $0.088 per ingest and 228 to 241 LLM calls; add-only memory served a stale fact in 3 of 3 runs versus 0 of 3 for update-in-place, and only 4 of the 12 setups ever served a stale value. Fareed Khan's Level Up Coding multi-agent analyst over a TPC-DS SF100 warehouse of about 959 million rows in a 28.6 GB DuckDB file on a 16-core laptop with no GPU reported daily accuracy rising from 66.7% to 90.9% and false success falling from 47.1% to 9.8%, but its live p95 latency was 24 seconds against a 15-second target.

recommendation

Prefer update-in-place memory over add-only when facts change, and report failed hypotheses and harness bugs alongside results so readers can judge which numbers generalize.

more in playbook: Designing agent systems
by read15 min views3 publishedNov 18, 2026

Every week I go through a pile of newsletters and Medium digests looking for AI engineering posts worth my time. This week that came to more than a hundred items. Most of them make claims with numbers in them. A few of those numbers will end up in my own articles, my own design docs, or a Slack message to a colleague that starts with “apparently”.

That last word is where the trouble begins. Once a number leaves its article it loses its context. “Recall dropped to 0.27” sounds the same whether it came from a production system or a script on someone’s laptop. “Matches Excel on 99.99% of cells” sounds the same whether those cells are varied or mostly one stress file.

So this week I tried something. Instead of reading the results sections first, I read the parts most people skip: the limitations, the caveats, the “where it’s worse” paragraphs, the small print about who the author is. Five pieces stood out because their honesty sections made their numbers more useful to me, not less. I also kept a list of the opposite: posts where something in the text told me nobody had really read it before it went out.

A note on how I’m handling numbers here. Every figure below is the source author’s own claim, taken from my reading notes. I have not rerun any of these experiments. Where I say “he measured”, read “he reports that he measured”.

I used to treat caveats as a legal disclaimer. Something the author adds so nobody can call them wrong later.

I don’t think that anymore. A good limitations section is a map of where the number stops being true. Without it, I have to guess the boundary myself, and I always guess too generously because I want the number to apply to my problem.

The five articles below each drew one boundary clearly. Different boundaries, different kinds of honesty. That’s what made them a useful set to read side by side.

Hubert Nakielski’s piece in Towards AI is about agent memory going stale when facts change. The setup, as he describes it: 12 memory configurations, a scripted 60-turn conversation, six checkpoints, three runs each. He used a real graph memory library (graphiti-core 0.30.2 on FalkorDB) with Haiku 5.5, and he reports per-ingest costs of roughly $0.085 to $0.088 and 228 to 241 LLM calls.

His headline, by his own numbers: add-only memory served a stale fact in 3 of 3 runs, update-in-place in 0 of 3. Only 4 of the 12 setups ever served a stale value at all.

But the thing I’ll remember is smaller. He went in expecting plain RAG to go stale, and it didn’t. It never served a stale value in his runs. He wrote that down instead of quietly dropping the RAG arm or reframing the article around a different question.

That matters to me for a practical reason. When someone reports a hypothesis that failed, I know the experiment was capable of surprising them. An experiment that only ever confirms its author’s view might be a good experiment, or it might be a demo dressed as one. A failed prediction is evidence the setup had teeth.

He also admits to harness bugs, and that grading was done with a keyword regex over a scripted conversation. Both lower my confidence in the exact numbers. Both raise my confidence in the author.

What I can reuse: the direction of the finding (update-in-place beat add-only in his setup) and the experimental shape. What I won’t reuse: the exact ratios as if they generalise beyond a scripted 60-turn conversation.

Fareed Khan’s Level Up Coding piece builds a multi-agent analyst over a TPC-DS SF100 warehouse, which he reports as about 959 million rows in a 28.6 GB DuckDB file, on a 16-core laptop with no GPU. He reports a 296-question exam split into 102 development and 194 held-out questions.

The numbers he’s proud of, by his account: daily accuracy going from 66.7% to 90.9%, false success going from 47.1% to 9.8%, and a live median answer time of 3.8 seconds.

Then he tells you the parts that didn’t work. He had a 15 second p95 latency target and the live p95 in his notebook was 24.6 seconds. He had a 5% false-success target and missed it. And the one I respect most: after fixing something, he reused the held-out test set.

That last admission is the kind most people never make, because most people don’t notice they did it. Once you tune against your held-out set even once, it isn’t held out anymore. The 90.9% is now partly a development-set number. He says so, which means I know exactly how much discount to apply instead of discovering it later when my own reproduction comes in lower.

His repository is public under MIT and, per my notes, its README numbers match the article. There’s also a model-call cache of about four thousand calls so the run can be replayed without paying for inference. That’s a disclosure of a different kind: not “here’s what I got wrong” but “here’s how you can check me”.

What I can reuse: the architecture and the evaluation split, with the reused test set in mind. What I won’t reuse: “90.9% accuracy” as a clean held-out figure.

Mark Shv’s post about SharpCell, a .NET Excel formula engine written by Claude Code, has the most quotable headline numbers of the week. By his report: 256 workbooks, 253 of the 254 that contain formulas match Excel on every cell, and 80,044 of 80,046 individual cells match.

If I’d stopped there I’d have written “99.99% cell-level agreement with Excel” in a note somewhere. And it would have been misleading, through no fault of his.

Because he warns, right next to the number, that about half of those cells come from a single ROMAN/ARABIC stress-test file. Roman numeral conversion is a narrow corner of a spreadsheet engine. If half the denominator lives there, the cell-level percentage tells me much less about, say, financial or date functions than it first appears to.

This is the disclosure I find rarest. Not “my result has limits” but “my metric is shaped in a way that flatters me, and here’s how”. He gives a second, more honest coverage figure too: 423 of 490 functions covered by at least one Excel-computed case, which is the number I’d actually use.

Two more things I noted. The expected values never come from the agent; they come from workbooks Excel itself calculated, many taken unchanged from IronCalc’s test files. That’s the right shape for trusting code you didn’t write: the oracle is outside the thing being tested. And there’s a section on where SharpCell is worse, 23 functions with documented deviations, plus a plain statement that there’s no performance benchmark. He also says Claude helped draft the article, which I’ll come back to.

What I can reuse: the verification method, and the per-function coverage number. What I won’t reuse: the cell percentage without its denominator warning attached.

Chew Loong Nian’s Towards AI piece responds to a conference talk about permission-filtered retrieval. The claim in the title is that search-then-filter dropped 7 of 10 results in his test.

What makes it trustworthy is that “his test” is fully described. A 63-line FAISS script. 100,000 synthetic 64-dimensional vectors. Seed 7. Pinned library versions. And the script’s output is shown.

By his numbers: with about 10.3% of documents accessible, post-filtering returns 2.6 of 10 results (recall around 0.27). Pulling 1,000 candidates first still only gets 7.6. FAISS’s built-in ID selector reaches recall 0.67 at 10 probes and 0.98 at 50.

None of this is a production measurement, and he doesn’t pretend it is. The corpus is synthetic, the permission layout is synthetic, and the big-system figures (a 120 ms P90, 400 million vectors) come from the talk, not from his script. He keeps those separate.

This is the most copyable kind of honesty. I don’t need to trust him at all. I can run the same 63 lines with the same seed and either get his numbers or not. Small, synthetic, and disclosed beats large, real, and vague, at least for the purpose of deciding whether an idea is worth testing on my own data.

One caution I added for myself: in my notes, one block of numbers in the post doesn’t appear in the printed script output. That doesn’t make it wrong. It means those particular figures sit one step further from evidence than the rest, and I treat them that way.

What I can reuse: the mechanism (filtering after search loses most results at low access ratios) and the script itself as a starting point. What I won’t reuse: “7 of 10” as a general rate. It’s an average at roughly 10% access in a synthetic setup.

Heeki Park’s post is about testing non-deterministic agents with Amazon Bedrock AgentCore Evaluations. It’s a first-hand integration story. By his account, batch evaluations initially failed because CloudWatch Logs Insights lagged direct log reads by two to three minutes, and he fixed it with a polling backoff (5 seconds, times 1.5, capped at 15 seconds, with a 5 minute ceiling). One test case scored 1.00, 1.00 and 0.83, averaging 0.94 against a 0.70 threshold.

The disclosure here isn’t about the data. It’s about the author. He says he works at AWS and that he’s the author of Loom, the open-source tool he uses in the post.

I want to be clear that this doesn’t make the article an ad. My notes call it a genuine integration write-up, and the CloudWatch lag story is exactly the kind of detail you only get from someone who hit it. But knowing his position changes how I read two things: which alternatives he didn’t consider, and how much friction he might be smoothing over. With the disclosure, I can adjust. Without it, I’d find out from a footer or a LinkedIn profile and feel slightly misled, even if the content were identical.

There is a limit worth noting: the post has screenshots but no code blocks, so reproducing his setup means retyping from images. Disclosure of who you are doesn’t replace disclosure of what you ran.

What I can reuse: the CloudWatch lag lesson and the backoff values as a starting point. What I’d double-check: whether the tool comparison would look the same from someone outside AWS.

Here’s the week compressed into one view. Every number in it is the source author’s claim.

+------------------+--------------------------------------+----------------------------------------+| Article          | Disclosure it made                   | What that lets me do with its numbers  |+------------------+--------------------------------------+----------------------------------------+| Nakielski        | His RAG hypothesis failed (RAG never | Trust that the setup could surprise    || (agent memory)   | went stale); harness bugs; regex     | him; reuse the direction, not the      ||                  | grading on a scripted conversation   | exact ratios                           |+------------------+--------------------------------------+----------------------------------------+| Khan             | Missed the 15 s p95 target (24.6 s); | Discount the 90.9% as partly tuned on  || (data warehouse) | missed the 5% false-success target;  | test data; reuse architecture and the  ||                  | reused held-out set after a fix      | eval split                             |+------------------+--------------------------------------+----------------------------------------+| SharpCell        | About half the 80,046 cells come     | Ignore the cell-level percentage; use  || (Excel engine)   | from one ROMAN/ARABIC stress file;   | the 423/490 function coverage and the  ||                  | 23 functions deviate; no perf bench  | Excel-as-oracle method                 |+------------------+--------------------------------------+----------------------------------------+| Chew Loong Nian  | Synthetic data; 100,000 vectors;     | Rerun it myself; treat "7 of 10" as a  || (filtered RAG)   | seed 7; 63-line script; pinned       | synthetic average at ~10% access, not  ||                  | versions; talk figures kept separate | a general rate                         |+------------------+--------------------------------------+----------------------------------------+| Heeki Park       | Works at AWS; authors Loom, the tool | Use the CloudWatch lag lesson; check   || (AgentCore eval) | used in the post                     | tool comparisons from a neutral source |+------------------+--------------------------------------+----------------------------------------+

I’m not going to name authors in this section. The patterns are what matter, and each of these could happen to anyone who drafts with an AI assistant, including me.

Leaked scaffolding. Several posts this week contained phrases addressed to someone who wasn’t the reader. “As you cite it.” “According to your brief.” “Since you asked for a position.” These are fragments of a conversation between an author and a model, left in the published text. In one case a raw image brief survived, something like “[image:3, Layer decomposition. What it explains: …]”, describing a figure that was never made.

The phrase itself isn’t the problem. The problem is what it tells me about the rest: if nobody noticed a sentence addressed to “you”, nobody checked the numbers either. When a post says “as you cite it” next to a statistic, I read that as “this number came from a prompt, not from a source”. It goes in my notes as unverified, no matter how specific it looks.

Unfinished placeholders. One otherwise careful article about agent rules files had a full line reading roughly “[AUTHOR RESULT, to be completed before publication]”, followed by a reminder to add a commit link, a date and a model. What’s frustrating is that the same piece labels its own comparison matrix as illustrative values rather than measurements, which is exactly the right instinct. The placeholder just slipped through.

I don’t feel superior about this one. My own drafts have carried placeholder markers for results I hadn’t run yet. The difference between a draft and a published post is that someone looked for those markers. Now I look for them with a script (below).

Unnamed anecdotes. A customer said to run about twice as fast. A realtor offered as a satisfied user. Neither has a name, a company or a link. These show up in posts that otherwise look technical. An unnamed customer can’t be checked, can’t be asked a follow-up question, and can’t disagree with how they were described. I don’t treat them as false. I treat them as zero evidence, which is different, and which means they can’t carry an argument on their own.

None of these three patterns makes a post worthless. Some of the leaky posts this week summarised real primary sources fairly well. But they change my default: I go to the primary source and stop reading the summary.

Since half of what I criticised above is just “nobody searched the text before publishing”, here’s what I run before I publish anything. It’s plain grep, works locally, no API.

#!/usr/bin/env bash# lint-draft.sh: flag leaked scaffolding, placeholders and vague attributions in a markdown draft# usage: ./lint-draft.sh my-article.mdset -euo pipefailfile="${1:?usage: lint-draft.sh <file.md>}"status=0check() {  local label="$1" pattern="$2"  if grep -n -i -E "$pattern" "$file"; then    echo "^^^ $label"    echo    status=1  fi}check "Placeholder left in text" '\[(AUTHOR RESULT|TODO|TBD|FILL|MY RUN|image:[0-9]+)[^]]*\]|\[\[[^]]+\]\]|to be completed before publication'check "Prompt scaffolding addressed to 'you'" 'as you (cite|describe|mention|asked)|according to your (brief|notes|source)|your (brief|supplied material|source material)|since you asked'check "Unnamed anecdote" '(one|a) (customer|client|team|company|user|realtor) (saw|told|reported|said)|some teams (report|say)'check "Unlabelled 'illustrative' content to double-check" 'illustrative|hypothetical example'check "Em-dash or en-dash (house style)" "$(printf '\xe2\x80\x94|\xe2\x80\x93')"if [ "$status" -eq 0 ]; then  echo "No flags in $file"fiexit "$status"

It catches the obvious things only. It won’t tell you whether a number is honest. But the obvious things are exactly what slipped through in the posts above, and a 20-line script is cheaper than a correction note.

If you want a second pass that reads for meaning, a local model is enough. With Ollama running, something like this asks a small model to list every number in a draft and whether a source or a measurement is attached to it:

ollama pull qwen2.5:7b{  echo "List every number in the article below. For each one, say whether the text attributes it to a named source, to the author's own measurement, or to nothing. Output one line per number."  echo  cat my-article.md} | ollama run qwen2.5:7b

I treat the model’s output as a to-do list for me, not as a verdict. It misses things and occasionally invents an attribution, so I check each line by hand.

This is what I now ask before a number from someone else’s post goes into anything I write. It’s short on purpose. If I can’t answer a question in about a minute, the answer is effectively “no”.

If a post passes all six, I’ll quote its numbers with attribution. Four or five, I’ll quote them with the caveat attached. Fewer than that, I use the post as a pointer to something I need to verify or run myself.

I don’t know how much of this scales. Five careful posts in a week is a good week. Most weeks, the honest posts are also the long, member-only ones that take twenty minutes each, and the leaky ones are short and everywhere. A checklist helps me decide faster, but it doesn’t make the careful posts more common.

I’m also aware that I’m rewarding authors for disclosing weaknesses, which could in theory reward performative humility. I haven’t seen that yet. The five disclosures above all cost their authors something: a cleaner headline, a better-looking percentage, an appearance of neutrality. That cost is, I think, what makes them believable.

And SharpCell’s author said Claude helped draft his article, the same way leaked phrases tell me an assistant was involved elsewhere. Using an assistant isn’t the dividing line. Whether a person read and owned the result is.

AI Engineering, AI Agents, LLM, Evaluation, Benchmarking, Technical Writing, Software Engineering, Research Methods, RAG, Critical Thinking

Reading Five Agent Studies in One Week: What Their Honesty Sections Taught Me was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

── more in #ai-agents 4 stories · sorted by recency
── more on @hubert nakielski 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/reading-five-agent-s…] indexed:0 read:15min 2026-11-18 · —