LLM: Learningful Links and Musings

Some brief meta:

• On AI, dispositionally, I’m somewhere in between “all of this is crap” and “the singularity is upon us”. In 2026, to borrow a friend’s metaphor, we get to work with fun aliens. They have structurally poor epistemics, poor taste, they’ll shit all over your codebase if you let them. But, if you give them a harness and the right context, they can be scarily good at knowledge work tasks. The aliens’ eagerness is a great match for my inertia to start (or resume) working on something, so on balance I feel more capable with them.
• I prefer to run my own copy of the alien on my own computer, but in various circumstances I’ll rent someone else’s bigger, smarter alien. A lot of my interest reflects this preference order.
• Given the topic, echoing Dynomight: I, a real human, wrote every word of this post. (Same for all posts on this blog.)
• Many of these links came from friends. I am grateful but did bad job keeping track of what came from whom. If you’d like ‘via’ credit, please ask and I’ll happily provide it.
• Websites break over time, so prepend https://web.archive.org/web/

to any URL that doesn’t load when you read this.

Benchmarks #

It’s difficult to systematically compare LLMs on how ‘smart’ or ‘capable’ they are. A lot of benchmarks test LLMs on a publicly-visible set of questions; these die of saturation when AI labs gobble up the questions as training data. It’s difficult also because LLM capability is highly dimensional, and subtle changes in the prompt or harness can cause large, unpredictable changes in output quality. It’s difficult also because AI labs spend a lot of resources to influence our perception of model capabilities.