Are we just Sharpening Pencils?

A developer with a team of 10 programmers observed at a programming offsite that engineers given advanced AI models and generous budgets gravitate toward building developer tooling for themselves rather than tackling unfamiliar domains. The developer argues that LLMs struggle with complex tasks when the user lacks deep domain expertise to ask well-formed questions or spot errors, citing failed attempts at ceramics glaze chemistry versus a friend's successful AI-assisted electronics projects. The developer concludes that as models grow more capable, individuals and organizations will increasingly build larger personal tools reflecting their own 'embeddings,' widening a translation problem between differing conceptual frameworks.

I spent some time with my team of 10 programmers this week, at a programming offsite. This is a group that has a lot of experience with very advanced programming tools, access to cutting edge models, generous budgets and no real restrictions on what they build. What magical new thing did they including me mostly gravitate to? Building ever better tools for building more software faster. Developer tooling. Every “power user” or technologist I know, when they’re not worrying about AI safety, seems to be building their own private empire of custom tools, workflows, widgets, etc. Understanding this is important and sheds some valuable light on a challenge in the “packet switched organization” idea. Here’s another recent pattern, before we dig in. I’ve been trying to design ceramics glazes with LLMs. I have a friend who is an actual expert in this chemistry, and things that seem plausible and useful to me are often so wrong as to make them angry. It’s useless, and I’ve largely given up on it. At the same time, that friend has designed, with AI, several successful complex electronics projects. Why does one work and the other not? I think the root of both of these things is that it’s very hard to get an LLM to perform a complex task if you yourself don’t understand the domain deeply enough to ask a well-formed question, or spot BS in the answer. I have that expertise in some domains, like software design, and writing, but not in others, like glaze chemistry, where the training data is poor enough that the expertise is necessary. So why are the developers I know building tools for themselves? Well, because that’s what they know well. Just as most customers aren’t great product designers, most engineers aren’t necessarily either, even though they spend their time building products. Product design is hard, and it’s a distinct skill from product building what we used to call programming . And here’s the challenge for the “packet switched” world: each organization or person is an expert in itself. So, naturally, it will build tools and processes that make sense in its own context. I think of this as an ‘embedding’. An embedding is a choice about how to, well, embed the world in a specific model. It privileges some things at the cost of others. It’s a valid thing to do - it can optimize, and it can also make it easier to think in certain common patterns. In fact, this is most often the reason different groups and people have different embeddings - it’s faster for them. The intuitive person might think in terms of a whiteboard or images, but that’s just annoying, slow, and error prone for the accountant - they need a spreadsheet that computes answers in the form they can use to do their job. At the offsite, we worked in pairs or threes. It’s incredibly hard to do that with this kind of agentic support. We had big ambitions and did, actually, write a lot of code , but the more the models can do for you, the bigger the embedding translation task becomes. I think this is going to be a fundamental pattern: as the tools get more powerful, we will naturally want to build things for ourselves that express our own patterns and intentions. Those tools will get bigger as the models are more capable, and the translation problem will grow with them. What if you encounter someone whose concept of “document” is completely alien to yours - like a wiki, or a flavor wheel instead of a linear flow? One person thinks of a document as a linear sequence of paragraphs, but the other organizes it as a network where there is no “right” order to read it. “Move this earlier” makes sense in the first model but in the second embedding it might not even be a coherent question. But in the second, an overview of concepts might be very easy to build. You might have to take a fairly long bit of attention to even understand what they’re saying, or how to say something back. This challenge actually inspired me to start working on yet another developer tool, that turns the flow “inside out” and has continually running agents whose charter is to understand and facilitate translation between embeddings, so the cognitive load on the humans is minimized. This feels like a fumbling first step in the direction of an actual “packet switched” tool. Of course, then we have the next problem: if you can’t spot BS in domains where you aren’t an expert, how do you know that what you get from someone else in this world is valid? There may not be a new answer here - we may place trust in relationships, credentials, and responsibility. But it might be the case that we can compute and infer trust more quickly and in a more fine-grained way with these tools. I don’t know yet. In the best case of the packet-switched world, work rapidly travels to wherever it can best be serviced. But the understanding of that work needs to travel with it or be there when it arrives. Tools that translate understanding, surface hidden disagreements, build common languages or otherwise minimize human cognitive load will be absolutely critical. But in the meantime, I am now watching this pattern, and asking myself: are we more productive, or just more busy? Are we just sharpening pencils? And are they weird pencils that only we know how to use? 1 footnote-1 1 footnote-anchor-1 As an amusement, I took some of the comments from earlier posts and asked an LLM to generate imaginary comments for this one. See if you recognize yourself Everything below this is AI generated. Everything above is me An AI-generated imaginary comments section. These commenters are fictional. Their objections are annoyingly reasonable. Conceptual Carl: “You’ve found the problem, but I think you’re still understating it. Your two document models can represent the same material differently. What happens when they disagree about what constitutes evidence, who has authority, or what success means? Some differences require negotiation, not translation. An agent that makes everyone’s language sound compatible could actually conceal the disagreement.” Captain Follow-Through: “You want agents to reduce the cognitive load of understanding someone else’s work. Fine. But what tells me I can rely on that person when things go wrong? Understanding their model isn’t knowing whether they’ll tell me bad news promptly, own a mistake, or follow through. You keep trying to transport understanding when some of what organizations accumulate is a history of dependable behavior.” Transaction-Cost Terry: “You’re describing a tradeoff between local efficiency and coordination costs. Shared tools impose inconveniences on individuals partly to reduce those costs. Everyone hates the standard company system, but everyone understands what its fields mean. If AI makes customization cheaper, that doesn’t establish that customization is cheaper overall . Your weird pencils may be an argument for standards.” Product-Manager Pat: “You gave ten programmers excellent programming tools and no particular external problem to solve. They built programming tools. Is this a revelation about the future of software, or a fairly predictable consequence of the assignment? “Also, your glaze story complicates your thesis. You say the information is bad, then use the failure to argue that the user needs more expertise. Which bottleneck are we discussing? An expert recognizing that an answer is rubbish isn’t the same as an expert being able to obtain a good answer.” Impatient Ian: “We’ve been to an offsite, a pottery studio, product management school, an accountant’s office, and a flavor wheel. I understand that people think differently. Please get to the weird pencils.” Practical Priya: “The document example finally lets me understand what you mean. But I’m not convinced my organization is ‘an expert in itself.’ Different departments already disagree about how it works. Who gets to choose the embedding that our agents will translate? The official process and what people actually do are often very different.” Unfortunately Correct Colin: “You say you can’t reliably use AI for complex work you don’t understand. Your proposed solution is to use AI to translate other people’s complex work into something you can understand. How will you spot the bullshit in that translation? Haven’t you moved the glaze problem into the middle of your organizational infrastructure?”