Revisiting Joel's Test

Joel Spolsky's 2000 Joel Test for software teams has been updated by an unnamed author at exe.dev into the Shelley Test, adding nine agent-era questions such as agentic code review, LLM-supervised continuous deployment, and end-to-end integration tests. The new test, named after the coding agent Shelley, reflects how AI agents now write code and supervise deployments, making peer code review obsolete and requiring new practices like adversarial subagent reviews and fast merge queues.

Back in 2000, Joel Spolsky wrote an influential blog post, The Joel Test: 12 Steps to Better Code https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ , for quickly identifying whether a software team is high-functioning. These are the twelve questions he nailed to the Internet. - Do you use source control? - Can you make a build in one step? - Do you make daily builds? - Do you have a bug database? - Do you fix bugs before writing new code? - Do you have an up-to-date schedule? - Do you have a spec? - Do programmers have quiet working conditions? - Do you use the best tools money can buy? - Do you have testers? - Do new candidates write code during their interview? - Do you do hallway usability testing? Those questions are still relevant, but, with the advent of agents, there are more questions. I’m calling this the Shelley Test, after our coding agent, which is named for the Unix shell, Mary Shelley, and Percy Bysshe Shelley: - Do you use agentic code review? - Do you deploy continuously, supervised by LLM agents? - Do you have end-to-end integration tests? - Do you and your coding agents have easy access to observability tooling? - Do you have access to the latest models from the best providers? - Do you have a merge queue and does it complete in 3 minutes or less? - Is it easy for your team to stand up new tools and agents, for development and to help with everything around development? - Do your team members regularly discuss their tools, their workflows, and adjust when necessary? - Is your product legible to coding agents as users? 1. Do you use agentic code review? Peer-based code review is dead https://crawshaw.io/blog/agent-principal-agent . The LLM is writing the code, and the engineer is responsible for it. Adding a rubber stamp, however delayed, from another human, is done for. Even on the best teams, we all know that a small, targeted change might generate a long code review cycle and a bike-shedding session, whereas if you combine two weeks’ worth of changes and dump them on your neighbor, it’s LGTM. Instead, ask your agent to use subagents with different models to do an adversarial code review to check that what’s in the commit is what’s on the tin, and it will do wonders. Does your harness only support one family of models? Use a harness that doesn’t kneecap you. See also “Review the Reviews” https://blog.exe.dev/review-the-reviews and Roborev https://github.com/kenn-io/roborev . 2. Do you deploy continuously, supervised by LLM agents? We are not dogmatic about whether this means once an hour, on every commit, or once a day, but inventory in your repo will face reality, and it is paramount that it does so soon. Shorter cycles are better. Continuous deployment requires integration tests that you can trust, and that’s a good thing. It also requires feature flag infrastructure; also a good thing. Athena https://blog.exe.dev/athena-deploys-exe , our bot that supervises continuous deployments, is now indispensable. It reads logs, checks metrics, and writes down lessons for the next deploy. Athena chats on Slack about it, and has the power to pull the plug on a deploy. If you’ve been holding back on continuous deployment because you’re not confident that your deployment platform has all the right metrics gates, abandon that project immediately, and write yourself an Athena agentic loop. It’s deeply embedded in our deployment control center software, but write to us if you want us to extract the heart of it. And yeah, we give our bots names https://blog.exe.dev/botiquette so they’re easy to refer to . 3. Do you have end-to-end integration tests? When a bug inevitably sneaks through, you have found where your testing is insufficient. LLMs are great at tests. See, for example, this commentary on the Go cryptography standard testing https://bsky.app/profile/filippo.abyssdomain.expert/post/3msj4dfbihb2u . Having the integration test infrastructure in place allows you to feel confident that deploys won’t break core functionality. Incidentally, if you must have an external dependency, “ digital twins https://factory.strongdm.ai/techniques/dtu ” for that dreaded external API have never been easier to build. 4. Do you and your coding agents have easy access to observability tooling? In the LLM age, a monitoring stack must be computer-queryable, preferably with SQL, preferably with business data available to JOIN with. At exe.dev, we love Clickhouse https://clickhouse.com/clickstack . Use a bot to do initial triage and maintenance of alerts. Our bot is called Sisyphus. Instead of using the observability tool or its agent to look at metrics and diagnose tricky customer bugs, point your regular coding agent to the observability tool. Combining code and logs is fire. 5. Do you have access to the latest models from the best providers? This is a repeat of Joel’s “Do you use the best tools money can buy?” Joel’s question is being tested as CFOs find out that an extravagant IntelliJ or Tableau subscription is the least of their worries now. 6. Do you have a merge queue and does it complete in 3 minutes or less? Don’t use long-lived branches. Commit your changes to main, and keep main green. The way to do so is to run the tests before they are merged in. The faster your tests, the easier this is Your merge queue will get backed up or require cleverness if it’s slow. Engineering with LLMs is starved for human attention: the long latencies destroy that attention. We’ve abandoned GitHub Actions since https://sketch.dev/blog/lightweight-merge-queue https://sketch.dev/blog/lightweight-merge-queue was written, but the basics are still functioning. Agents are incredible at speeding up your CI if given the opportunity. The last build at time of writing took 2m30s or so, used many, many lanes of parallelism on one big honkin’ machine, and had acceptable CPU utilization, though the 60s tail leaves much room for improvement 7. Is it easy for your team to stand up new tools and agents, for development and to help with everything around development? The prior math on whether it’s worthwhile to build a tool https://xkcd.com/1205/ is now wrong, because coding agents build very passable tools in one-shot. It must be easy to host and iterate https://blog.exe.dev/devtools-must-be-open-source on these tools. 8. Do your team members regularly discuss their tools, their workflows, and adjust when necessary? We are in an age of exploration https://blog.exe.dev/bones-of-the-software-factory , and sharing what is working and not working with your peers is imperative. It’s always been a good idea, but now you’re missing out on the compounding effects if you don’t do it. We do this constantly: on Slack, on the telephone, and in our team meetings. 9. Is your product legible to coding agents as users? Your users are judging your software based on whether Claude Code can operate it. Can it? Do you have an llms.txt https://exe.dev/llms.txt as very obvious documentation? Does auth work? APIs are king again. Not to get all Clayton Christensen up in here, but a worse product operated by a mid-tier coding agent is destroying the better product unavailable to the coding agent.