On October 22 we helped run All Day AI, a free online conference with more than a hundred sessions and one hard rule: no vendor pitches. Brian Turcotte wrote about why before the event. Here’s a summary on a couple of them that focus on building with AI agents.
The speakers below work at a stock exchange, a bank, a genomics company, a travel platform, a legal software company, a security vendor, a chip maker, Google, and a handful of startups and consultancies, including one that tried to let agents write all of its code. Despite the different industries and backgrounds, the same five practices came up across these talks.
1. Review is the bottleneck, so approve on purpose
Sachin Hiriyanna, a staff engineer at Navan, opened with public GitHub data: 43 million pull requests (PRs) merged in August 2025, up 23% on the year before, while comment activity on those PRs barely moved. His analysis is that agents now produce PRs faster than humans can read them, and that reading every line of a 2,000-line agent PR on a Tuesday afternoon is not a plan. Neither is clicking the green check.
The recipe for his middle path reads four sentences:
- Read what can hurt.
- Let the agent read the rest.
- Open a file only when a question earns it.
- Approve on purpose.
In practice that is a short checklist of things that can hurt (crashes, lost updates, silent fallbacks, a diff that does something the description does not mention, widened permissions, untested branches), files sorted by risk before he opens any, and three sub-agents that review the same PR: one wired to the checklist, one to the product requirements document, one to the design file. They return one deduplicated list of questions, and every question passes through him before it reaches the author. “The job is to approve on purpose,” he said, “to be able to say I looked at the parts that mattered and I am putting my name on this.”
Shon Urbas of Pandium gave the counterexample from the other side of the PR. His agent wrote tests that pass by construction, assert True among them, and a human reviewer caught them. His team keeps two human reviewers per PR, and the second one still finds one or two real problems each time.
The conclusion is that the cost of shipping moved from writing code to reviewing it. The job is deciding what can hurt and reading that, and letting the machine read the rest.
2. Write down what the agent needs to know, in the repo
Parthiban Rajasekaran, a principal engineer at Capital One, described the PR that passes build, tests, and lint and still breaks the system: two services that were designed to be decoupled now read each other’s data store, because the design decision lived in a meeting and the agent read the code.
He listed four drift patterns that pass review one at a time:
- Layer violations
- Parallel abstractions (the agent writes its own fetch-and-retry instead of using the platform HTTP client)
- Bypassed standards
- Inconsistent failure handling
No single PR is wrong. The sum is a different system.
His fix needs no budget. Write a one-page architectural intent file at the repo root with the boundaries and the reasons. Encode the invariants as checks that fail the build (dependency-cruiser for JavaScript and TypeScript, ArchUnit on the JVM, custom lint anywhere), with failure messages that name the decision so the agent can correct itself. Add one question to the PR template: which decision does this change touch? “The goal is not zero drift,” he said. “It is drift caught while it is still cheap.”
Max Artomenko at Illumina applied the same idea in a regulated setting. His requirements and test cases lived in Jira, Jama, and a quality management system with closed APIs, out of reach for his coding agents. He synced them into Markdown in the repository and pointed the agents at them from AGENTS.md. On one mid-sized application the agent’s own coverage check found seven gaps between requirements and tests and about 10 requirements the code only partly satisfied. He puts the documentation time saved at roughly 80%, his own estimate.
The conclusion is that if the agent cannot read it, it does not exist. Decisions, requirements, and invariants belong next to the code, in a form the agent loads on every run.
3. Keep the checks outside the agent’s reach
Mradul Shrivastava of NASDAQ summarized his architecture in one line: “the LLM drafts and everything else gets checked.” A large language model (LLM) is probabilistic by nature, so he makes everything around it deterministic. Acceptance criteria are written with exact numbers and locked before any code exists, “so that the AI cannot adjust the test to fit its own code.” Prompts, memory, and model versions are pinned in Git and change only by reviewed PR. The compiler, linter, security scanner, and acceptance tests are the judge, and only humans can edit them. After three failed attempts the pipeline escalates to an engineer. Approved output is saved as an artifact, and the continuous integration (CI) pipeline runs the saved copy from then on. A new prompt version that scored 49 out of 50 on his golden dataset stayed blocked, because the price calculation it touched requires 100%.
Viktoriia Voloshyna showed what happens without that. In a live demo with Claude Code, she had the agent train a sentiment classifier on correctly labeled movie reviews where the positive ones came mostly from drama and the negative ones from horror. The agent reported 93.7% accuracy. The real figure on the larger set was 0.684. It had learned to tell drama from horror, and that is what it was scoring. A pytest suite passed. A leakage check passed. An iteration loop that burned close to a million tokens made the number worse each round. “Claude did not question anything,” she said, “unless we explicitly told Claude to question the class imbalance.”
Daniel Scheufler of Improving got there by running parallel agents and going into a meeting. He came back to a pile of “agent needs your input” pings, including a security agent in a panic about an API it had checked in a development environment with no firewall. Every review had degraded to “looks good to me.” His rule now is verification, not vibes: define done, define the test, and make the agent pass it before it moves on. He writes tests with a separate agent from the one that writes the code, on a different model.
The conclusion is that a test the agent can edit is a suggestion. The checks that decide whether work ships have to sit outside the agent’s reach, and a passing test only proves the agent answered the question it was asked.
4. The harness is where the leverage is
Ashish Kattamuri, a staff engineer at Proofpoint, explained why sessions degrade. The model starts from zero on every call; the harness assembles the whole conversation on every call. The context window is already busy before you type: in his own session, about 120,000 tokens of a million-token window were used before he typed his first message, by the system prompt, tool descriptions, skills, and memory files. Output tokens cost about five times input because generation is sequential. And a million-token window holds far fewer usable tokens, because recall drops for material in the middle. Compaction, clearing, and sub-agents are answers to those specific problems, and they fail when applied as habits.
Madeline Webster, a development manager at Clio, described the practice side. She has barely opened an integrated development environment in six months and works from the command line, orchestrating agents. Her rules came from failures: unplanned agent code was unreviewable and got thrown out, so plan mode first. Before a release she pointed agents at 40 to 50 minor bugs overnight with guardrails stated up front:
- Draft PRs only
- No merging
- A maximum PR size
- Product questions marked blocked rather than guessed
The result was 12 to 15 PRs, most ready, one or two reworked, in about three hours for a couple hundred dollars. Her observation afterward: the agent is more often stalled waiting for her than unable to do the work.
Chris Lucian, director of software development at Hunter Industries, runs agents for hours rather than minutes, and his point is that what keeps them going is everything around the model. Unit tests, coverage, mutation testing, linting, and static analysis are the feedback loop, and an agent with those around it corrects itself while nobody watches the live feed. He has his agents write their own skills (“create any skills you wish you had before you started this task”) and review their own instructions file afterward. His rule of thumb: under about 20 minutes of agent runtime you pay a context-switch cost; past it, you can go do something else.
The conclusion is that the model is the smallest lever. Context, skills, plan mode, and the rules you set before the run decide more than the model picker does.
5. Humans stay at the gates, and cheap models do the deciding
Nikolay Rodionov, CEO of Alpic, spent six months trying to make agents write all of his company’s code, partly to test the public claims that others had. The fully autonomous version failed. What survived is a human-gated pipeline with agents at specific steps. His first learning was that product agents do not work; reviewers stopped reading AI-written requirements. Writing specs detailed enough for one-shot implementation tripled the time his team spent on specs, and “every hour spent here was saving several hours spent on steering your agents.” Human code review stayed, because someone has to own the code at 3 a.m. Agent routines that run weekly are capped at five PRs per run. The team ships at least twice as fast end to end, by his account, with no drop in quality or uptime.
Lenin Alevski Huerta Arias, a security engineer at Google, took the cost question into the harness itself. Coding agents send every message to one expensive model, even when the decision is a label, a score, or a yes or no. “Not all the tokens or not all the messages are the same.” He uses small, fast, schema-constrained decision models in front of the large one: to triage prompt injection and escalate the sophisticated cases, to classify a shell command as dangerous or read-only before it runs, to route work to a browser agent or a fast agent, and to decide when to compact a session. They sit beside the large model rather than replacing it, since they lack multi-step reasoning. “Speed is not quality,” he said, and “confidence is not correctness,” so thresholds have to be tuned.
Chris Clark of OpenRouter supplied the data. Open-weight models now carry about 64% of the tokens on his platform and 28% of the spending. The blended price per million tokens fell about 80% in a year, and by his analysis most of that drop came from prompt caching rather than from cheaper models. His cheapest saving is unglamorous: “going to older agents that are unnecessarily pinned to a very expensive model.”
The conclusion is that the factory is real, and it still has people at the gates. The human gates stayed. What changed is that the expensive model now does only the work that needs it.
Where Kilo fits #
A software factory is only as trustworthy as its parts, and the five points above are the parts. Kilo’s Code Reviewer and REVIEWS.md are built for the review problem in point one, where the agent reads everything and the human approves on purpose. Kilo Memory, rules files, and SKILLS.md support are the repo-side context in points two and four. Kilo Sandbox and Cloud Agents are the answer to point three: the agent runs with auto-approve on, and the operating system decides what it can reach. Enkrypt AI guardrails check each prompt, tool call, and tool result against a policy your security team sets. Auto Efficient routes each request to the cheapest model that has proven it can do that class of task. Alevski made the case for small decision models from the stage; we made ours in the Jev post, and Auto Efficient is the closest thing Kilo ships. And safety scores sit on every model card. All of this still assumes humans at the gates. The speakers were clear that they kept theirs, and so do we.
What I am taking from it #
While generating code has gotten easier, deciding the quality of code and when it is ready to ship still requires a human step. Niklas Grebe of HERO Software put it plainly: building stopped being the bottleneck. What the 14 talks had in common is where they put the developer instead: deciding what can hurt, writing down what the agent cannot know, keeping the checks out of its reach, investing in the harness, and routing cheap decisions to cheap models. Recordings are free to watch if you subscribed at www.alldayai.com. Start with Hiriyanna for the review workflow and Shrivastava for the architecture.