{"slug": "do-automated-evals-work", "title": "Do Automated Evals Work?", "summary": "Machine learning engineer Hamel Husain argues that automated evaluations (evals) for AI systems are flawed and often misleading, based on his 20+ years of experience and work at Airbnb and GitHub. He teaches a course on AI evals for engineers and PMs, with over 4,500 students from companies including OpenAI, Anthropic, and Google.", "body_md": "# Hamel Husain’s Blog\n\nNotes on applied AI engineering, machine learning, and data science.\n\nI’m a machine learning engineer with 20+ years of experience. I’ve worked at Airbnb and GitHub, including early LLM [research](https://arxiv.org/abs/1909.09436) used by OpenAI for [code understanding](https://openai.com/index/introducing-text-and-code-embeddings). I’ve also led and contributed to popular [open-source](./oss/opensource.html) machine-learning tools.\n\nI’m currently working to bring data science back to AI: helping teams debug, analyze, and measure their systems. I call this “evals,” and it’s the focus of my teaching, consulting, and writing.\n\n## 🎓 Learn With Me\n\nI co-teach ** AI Evals for Engineers and PMs**, with over 4,500 students from 500+ companies (including OpenAI, Anthropic, and Google). Check the page for upcoming cohorts.\n\n## 📮 Blog\n\nI often share my experience building AI products. Below is a selection of my long-form writing.\n\nNo matching items\n\n## 📬 Follow Me\n\nThe best way to follow my activity is [X (Twitter)](https://twitter.com/HamelHusain).", "url": "https://wpnews.pro/news/do-automated-evals-work", "canonical_source": "https://hamel.dev/", "published_at": "2026-07-11 07:00:00+00:00", "updated_at": "2026-07-11 21:39:54.062807+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-safety", "ai-ethics", "ai-research"], "entities": ["Hamel Husain", "Airbnb", "GitHub", "OpenAI", "Anthropic", "Google"], "alternates": {"html": "https://wpnews.pro/news/do-automated-evals-work", "markdown": "https://wpnews.pro/news/do-automated-evals-work.md", "text": "https://wpnews.pro/news/do-automated-evals-work.txt", "jsonld": "https://wpnews.pro/news/do-automated-evals-work.jsonld"}}