Do Automated Evals Work?

Machine learning engineer Hamel Husain argues that automated evaluations (evals) for AI systems are flawed and often misleading, based on his 20+ years of experience and work at Airbnb and GitHub. He teaches a course on AI evals for engineers and PMs, with over 4,500 students from companies including OpenAI, Anthropic, and Google.

Hamel Husain’s Blog Notes on applied AI engineering, machine learning, and data science. I’m a machine learning engineer with 20+ years of experience. I’ve worked at Airbnb and GitHub, including early LLM research https://arxiv.org/abs/1909.09436 used by OpenAI for code understanding https://openai.com/index/introducing-text-and-code-embeddings . I’ve also led and contributed to popular open-source ./oss/opensource.html machine-learning tools. I’m currently working to bring data science back to AI: helping teams debug, analyze, and measure their systems. I call this “evals,” and it’s the focus of my teaching, consulting, and writing. 🎓 Learn With Me I co-teach AI Evals for Engineers and PMs , with over 4,500 students from 500+ companies including OpenAI, Anthropic, and Google . Check the page for upcoming cohorts. 📮 Blog I often share my experience building AI products. Below is a selection of my long-form writing. No matching items 📬 Follow Me The best way to follow my activity is X Twitter https://twitter.com/HamelHusain .