Understanding Is the Bottleneck: Why Agent-Generated Code Needs Structured Summaries

Geoffrey Litt's talk 'Understanding Is the New Bottleneck' argues that AI makes writing code cheap but understanding it remains costly, a problem amplified in agent-assisted development where code review often becomes rubber-stamping. Developer Adl Alternative built the open-source tool 'what-you-did' to address this by generating structured JSON summaries of code changes, enabling both human triage and machine processing for MR review.

Understanding Is the Bottleneck: Why Agent-Generated Code Needs Structured Summaries Geoffrey Litt recently gave a talk called Understanding Is the New Bottleneck . His core point: AI makes writing code extremely cheap, but understanding code costs the same — or even more. I deeply agree. And in the context of Agent-assisted development, this problem is far worse than he describes. The status quo Agent writes code for you. You run tests. Green. Open MR. Reviewer sees 47 files changed, 3000 lines. Spends 5 minutes skimming. Approve. Nobody actually understands what the code does. This isn’t hypothetical — it’s daily reality. Tests become the only quality gate, and code review degrades into rubber-stamping. Why this is dangerous Tests cover the scenarios you can think of. The problems Agent introduces are often where you can’t: unreasonable abstractions, hidden state coupling, performance regressions, missed edge cases. Only a human who understands the code logic can catch these. Put differently: tests tell you “it’s not broken now,” but not “will it break in the future.” Where Geoffrey and I diverge Geoffrey’s answer is educational : he builds /explain-diff — rich HTML explainer docs with background context, literate diffs, interactive figures, and quizzes. His goal is to help humans learn the code deeply enough to participate in the creative process. I love this direction. But I’m solving a slightly different problem: what happens before a human decides to read? When a reviewer opens an MR with 47 files, they need triage — which files matter, what’s the shape of this change, where should I focus? They need a map before they start exploring . Geoffrey’s explainer is the exploration. I’m building the map. Why unstructured explanations aren’t enough I tried having Agent explain its changes in natural language. Problems: Unreliable : AI might miss files, or over-explain trivial changes Unconsumable : Natural language can’t be processed by downstream tools MR templates, impact analysis, change classification Unverifiable : You can’t assert “the explanation covers all changed files” Fundamentally, an essay-style explanation is a destination format . What I need is an intermediate data format — something both humans and machines can consume. The structured output approach So I built /what-you-did https://github.com/adlternative/what-you-did . Core design: - Use git diff --name-status as ground truth — build a skeleton containing every changed file first - Fill summaries per-file — entries can only be filled, never added or removed - Output JSON with a file count field for completeness verification - Aggregate by directory — provide different granularity levels for understanding { "file count": 5, "summary": "Refactored cache module into L1/L2 dual-layer architecture", "directories": { "path": "src/cache", "file count": 3, "summary": "Cache core logic split" } , "files": { "path": "src/cache/l1.ts", "status": "A", "category": "feature", "summary": "In-memory cache implementation" }, { "path": "src/cache/l2.ts", "status": "A", "category": "feature", "summary": "Disk cache implementation" }, { "path": "src/cache/index.ts", "status": "M", "category": "refactor", "summary": "Unified entry, routes to L1/L2" } } This isn’t prose for humans to read — it’s structured data for toolchains to consume. Humans can scan the summary at a glance; tools can auto-generate MR descriptions, trigger extra checks for specific modules, or compute change statistics by category. What this enables: MR review integration Structured data can be rendered into any shape. Here’s a concept for MR review — hover any file or directory to see its AI summary: The key insight: this data is available at page load time . No extra API calls. The JSON was generated when the branch was pushed — rendering is just a frontend component away. Key design decisions Skeleton first : Get the complete file list from git before filling content. This guarantees no files are missed. “Let AI analyze then output” has no such guarantee — especially on large diffs where context windows get tight. JSON not Markdown : Markdown is a rendering format. JSON is a data format. One JSON file can become an MR comment, a Slack notification, a weekly report entry, a changelog — each consuming what they need. Directory-level aggregation : Nobody wants to review 47 files one by one. But “8 files changed under src/auth, overall adding OAuth2 support” gives instant judgment. The relationship to Geoffrey’s work I see our approaches as complementary layers: Structured summary what-you-did → triage, orient, decide where to focus Rich explanation explain-diff → deep understanding of the parts that matter Quiz/verification → confirm you actually understood The first layer is fast, mechanical, and verifiable. The second is rich, educational, and creative. You need both. Conclusion The cost of writing code is approaching zero, but the cost of understanding code hasn’t changed. We need tools at multiple levels to bridge this gap — structured data for the map, rich explanations for the territory. Project: what-you-did on GitHub Geoffrey Litt 最近做了一个演讲叫 Understanding Is the New Bottleneck ，核心观点是：AI 让写代码变得极其廉价，但理解代码的成本没有变——甚至更高了。 我深有同感，而且在 Agent 辅助开发的场景下，这个问题比他描述的更严重。 现状 Agent 帮你写完代码，你跑测试，绿了，提 MR。评审人点开 diff，看到 47 个文件变更，3000 行代码。他花 5 分钟扫了一眼，approve。 没人真的理解这些代码做了什么。 这不是假设，这是我观察到的日常。测试通过成了唯一的质量门禁，代码评审退化成了形式审批。 为什么危险 测试覆盖的是你能想到的场景。Agent 引入的问题往往在你想不到的地方：不合理的抽象、隐含的状态耦合、性能退化、边界条件遗漏。这些只有人类理解代码逻辑后才能发现。 换句话说：测试告诉你”现在没坏”，但不告诉你”未来会不会坏”。 我和 Geoffrey 的分歧 Geoffrey 的答案是 教育导向 的：他做了 /explain-diff ——生成包含背景知识、文学化 diff、交互图示和测验的 HTML 解释文档。目标是帮助人类 深入学习 代码，从而能参与创造性过程。 我很喜欢这个方向。但我在解决一个稍微不同的问题： 在人类决定深入阅读之前会发生什么？ 当评审者打开一个 47 文件的 MR，他首先需要分诊——哪些文件重要、变更的整体形状是什么、应该聚焦在哪里？他需要先有一张 地图 ，然后才能开始 探索 。 Geoffrey 的解释器是探索工具。我在做地图。 为什么非结构化的解释不够 我试过让 Agent 用自然语言解释它做了什么。问题是： 不可靠 ：AI 可能漏掉文件，也可能过度解释无关紧要的改动 不可消费 ：自然语言没法被下游工具处理（MR 模板、影响分析、变更分类） 不可验证 ：你没法断言”解释覆盖了所有变更文件” 本质上，散文式解释是 终态格式 。我需要的是 中间数据格式 ——人和机器都能消费的东西。 结构化输出的思路 所以我做了 /what-you-did https://github.com/adlternative/what-you-did 。核心设计： - 用 git diff --name-status 作为 ground truth，先建一个包含所有变更文件的骨架 - 逐文件填充摘要，不允许增删条目 - 输出 JSON，带 file count 字段用于校验完整性 - 按目录聚合，提供不同粒度的理解入口 { "file count": 5, "summary": "重构缓存模块为 L1/L2 双层架构", "directories": { "path": "src/cache", "file count": 3, "summary": "缓存核心逻辑拆分" } , "files": { "path": "src/cache/l1.ts", "status": "A", "category": "feature", "summary": "内存缓存实现" }, { "path": "src/cache/l2.ts", "status": "A", "category": "feature", "summary": "磁盘缓存实现" }, { "path": "src/cache/index.ts", "status": "M", "category": "refactor", "summary": "统一入口，路由到 L1/L2" } } 这不是给人读的散文，是给工具链消费的结构化数据。人可以快速扫一眼 summary，工具可以自动生成 MR 描述、触发特定模块的额外检查、或者按 category 做变更统计。 下游怎么用：MR 评审界面集成 结构化数据可以被渲染成任何形态。这是一个 MR 评审的概念——hover 文件或目录查看 AI 摘要： 关键点：这些数据在页面加载时就已经有了。零额外 API 请求。JSON 在分支 push 时就生成了——渲染只是一个前端组件的事情。 关键设计决策 骨架先行 ：先从 git 拿到完整文件列表，再填充内容。保证不会漏文件。传统的”让 AI 分析 diff 然后输出”没有这个保证——尤其是大 diff 接近 context window 极限时。 JSON 而非 Markdown ：Markdown 是渲染格式，JSON 是数据格式。一份 JSON 可以变成 MR 评论、Slack 通知、周报条目、变更日志——各取所需。 目录级聚合 ：47 个文件没人想逐个看。但”src/auth 下改了 8 个文件，整体是在做 OAuth2 支持”——立刻就有判断力。 和 Geoffrey 工作的关系 我认为我们的方案是互补的层次： 结构化摘要 （what-you-did）→ 分诊、定向、决定聚焦点 深度解释 （explain-diff）→ 对重要部分的深入理解 测验/验证 → 确认你真的理解了 第一层快速、机械、可验证。第二层丰富、教育性、有创造力。两者都需要。 总结 写代码的成本在趋近于零，但理解代码的成本没变。我们需要多层次的工具来弥合这个差距——结构化数据做地图，深度解释做探索。