Show HN: I asked LLMs to choose between popular developer tools

Preseason, an open-source benchmark launched on Hacker News, measures which developer tools large language models recommend when asked to build real web apps, tracking recommendations across categories like databases, auth, hosting, and payments. The project runs fixed prompts against a fixed panel of models, parses answers for recommended tools, and publishes rankings and head-to-head comparisons to make AI-driven tool recommendations inspectable and reproducible. Preseason currently supports Vercel and Supabase Cloud, with a live demo at preseason.ai.

Measure which developer tools LLMs recommend when asked to build real web apps. Preseason is an open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps. We run a fixed set of web-app prompts against a fixed panel of models, parse each answer for recommended tools and services, then publish rankings, head-to-head comparisons, and methodology notes. The goal is to make AI-driven developer-tool recommendations inspectable, reproducible, and contestable, so you can see which tools AI coding assistants are most likely to put in front of developers. 🌐 Live demo: https://preseason.ai https://preseason.ai Preseason currently tracks recommendations across categories like: - databases - auth - hosting - analytics - payments - background jobs - UI/component libraries - observability - AI/model providers For each prompt × model run, we record whether the model recommended a known tool, no tool, or an invalid/unrecognized answer. - Which database does each model recommend most often for a new SaaS app? - Does GPT-4.1 prefer Supabase, Firebase, Neon, or plain Postgres? - Which tools win head-to-head when two options appear in similar prompts? - Are some models more likely to recommend "no tool" or hallucinate unknown tools? Recommendations from AI coding assistants shape developer tool adoption faster than blog posts or Twitter threads. If a foundation model quietly favors one database or hosting provider, that preference scales to every developer using it. We think the methodology behind that should be open, reproducible, and contestable, not a private dashboard. Preseason exists so anyone can: - See what today's LLMs recommend, with frozen prompts and model snapshots that are inspectable in this repo - Run their own benchmark on their own prompts or model panel - Submit issues when results look off and have an open paper trail - The benchmark measures recommendations, not whether a tool is objectively better. - Results depend on the frozen prompt set and model snapshots. - Tool-name parsing is intentionally strict; unknown names go to review instead of being guessed. - The project is early, so rankings should be treated as directional rather than definitive. pnpm run setup installs deps and starts local Supabase cp .env.example .env.local fill with supabase status + OpenRouter key pnpm run db:migrate pnpm run db:seed pnpm run db:seed-dev pnpm run dev App is at http://localhost:3000 http://localhost:3000 . Full setup details, including the env var table and troubleshooting, are in docs/SETUP.md /betocmn/preseason/blob/main/docs/SETUP.md . The supported launch path is Vercel + Supabase Cloud. Docker Compose and plain Postgres self-hosting are not supported yet because Preseason currently depends on Supabase Auth. See docs/SELF HOSTING.md /betocmn/preseason/blob/main/docs/SELF HOSTING.md . ┌────────────┐ ┌──────────────┐ ┌─────────────┐ │ Cron │───────▶│ OpenRouter │───────▶│ Response │ │ /api/cron │ prompt │ one model │ answer │ parser │ │ /benchmark│ └──────────────┘ └──────┬──────┘ └────────────┘ │ ▲ ▼ │ every 6 min ┌────────────────┐ │ │ Case decision │ │ │ tool / none / │ │ │ invalid │ │ └────────┬───────┘ │ │ │ ▼ ┌─────┴──────┐ QC pass ┌─────────────────────────────┐ │ Season │◀──────────────│ Rankings + head-to-head │ │ frozen │ │ matches public │ └────────────┘ └─────────────────────────────┘ Every active season freezes a set of prompt versions and model snapshots. The cron route at /api/cron/benchmark-run walks every prompt × model combination, requires the model to produce a strict machine-readable appendix, parses each response into a case decision tool / none / invalid , and publishes runs that pass QC. Public pages, including rankings, category indexes, and head-to-head matches, only read from published benchmark data. Unrecognized tool names are held in a candidate queue for admin review rather than guessed at. App Router, React Server Components Next.js 15 https://nextjs.org : typed API tRPC v11 https://trpc.io + Drizzle ORM https://orm.drizzle.team/ Postgres + email-OTP auth Supabase https://supabase.com : model gateway OpenRouter https://openrouter.ai + Tailwind CSS v4 https://tailwindcss.com shadcn/ui https://ui.shadcn.com + Testcontainers for an integration-tested Postgres Vitest https://vitest.dev for lint + format Biome https://biomejs.dev : local development environment docs/SETUP.md : supported deployment path docs/SELF HOSTING.md : every env var explained docs/CONFIGURATION.md : system overview docs/ARCHITECTURE.md : glossary of project terms docs/CONCEPTS.md : how rankings are produced docs/METHODOLOGY.md : what's planned next docs/ROADMAP.md How Benchmarks Work /betocmn/preseason/blob/main/docs/guides/how-benchmarks-work.md How Prompts Work /betocmn/preseason/blob/main/docs/guides/how-prompts-work.md How Rankings Work /betocmn/preseason/blob/main/docs/guides/how-rankings-work.md How Cron Benchmarks Work /betocmn/preseason/blob/main/docs/guides/how-cron-benchmarks-work.md How Matches Work /betocmn/preseason/blob/main/docs/guides/how-matches-work.md How LLM Service Works /betocmn/preseason/blob/main/docs/guides/how-llm-service-works.md How Evals Work /betocmn/preseason/blob/main/docs/guides/how-evals-work.md Recommendation Methodology /betocmn/preseason/blob/main/docs/guides/recommendation-methodology.md Pull requests are very welcome. See CONTRIBUTING.md /betocmn/preseason/blob/main/CONTRIBUTING.md for how to set up, what we look for in PRs, and our triage SLA. New to the project? Look for issues labelled . https://github.com/betocmn/preseason/labels/good%20first%20issue good first issue We follow the Contributor Covenant /betocmn/preseason/blob/main/CODE OF CONDUCT.md . Security reports go through SECURITY.md /betocmn/preseason/blob/main/SECURITY.md . MIT /betocmn/preseason/blob/main/LICENSE . See the LICENSE file. Third-party tool logos under public/logos/ are used under nominative fair use; see docs/LOGO POLICY.md /betocmn/preseason/blob/main/docs/LOGO POLICY.md .