{"slug": "show-hn-i-asked-llms-to-choose-between-popular-developer-tools", "title": "Show HN: I asked LLMs to choose between popular developer tools", "summary": "Preseason, an open-source benchmark launched on Hacker News, measures which developer tools large language models recommend when asked to build real web apps, tracking recommendations across categories like databases, auth, hosting, and payments. The project runs fixed prompts against a fixed panel of models, parses answers for recommended tools, and publishes rankings and head-to-head comparisons to make AI-driven tool recommendations inspectable and reproducible. Preseason currently supports Vercel and Supabase Cloud, with a live demo at preseason.ai.", "body_md": "**Measure which developer tools LLMs recommend when asked to build real web apps.**\n\nPreseason is an open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps.\n\nWe run a fixed set of web-app prompts against a fixed panel of models, parse each answer for recommended tools and services, then publish rankings, head-to-head comparisons, and methodology notes.\n\nThe goal is to make AI-driven developer-tool recommendations inspectable, reproducible, and contestable, so you can see which tools AI coding assistants are most likely to put in front of developers.\n\n🌐 **Live demo:** [https://preseason.ai](https://preseason.ai)\n\nPreseason currently tracks recommendations across categories like:\n\n- databases\n- auth\n- hosting\n- analytics\n- payments\n- background jobs\n- UI/component libraries\n- observability\n- AI/model providers\n\nFor each prompt × model run, we record whether the model recommended a known tool, no tool, or an invalid/unrecognized answer.\n\n- Which database does each model recommend most often for a new SaaS app?\n- Does GPT-4.1 prefer Supabase, Firebase, Neon, or plain Postgres?\n- Which tools win head-to-head when two options appear in similar prompts?\n- Are some models more likely to recommend \"no tool\" or hallucinate unknown tools?\n\nRecommendations from AI coding assistants shape developer tool adoption faster than blog posts or Twitter threads. If a foundation model quietly favors one database or hosting provider, that preference scales to every developer using it. We think the methodology behind that should be open, reproducible, and contestable, not a private dashboard.\n\nPreseason exists so anyone can:\n\n- See\n**what** today's LLMs recommend, with frozen prompts and model snapshots that are inspectable in this repo - Run\n**their own** benchmark on their own prompts or model panel - Submit\n**issues** when results look off and have an open paper trail\n\n- The benchmark measures recommendations, not whether a tool is objectively better.\n- Results depend on the frozen prompt set and model snapshots.\n- Tool-name parsing is intentionally strict; unknown names go to review instead of being guessed.\n- The project is early, so rankings should be treated as directional rather than definitive.\n\n```\npnpm run setup                  # installs deps and starts local Supabase\ncp .env.example .env.local      # fill with `supabase status` + OpenRouter key\npnpm run db:migrate\npnpm run db:seed\npnpm run db:seed-dev\npnpm run dev\n```\n\nApp is at [http://localhost:3000](http://localhost:3000). Full setup details, including the env\nvar table and troubleshooting, are in [ docs/SETUP.md](/betocmn/preseason/blob/main/docs/SETUP.md).\n\nThe supported launch path is Vercel + Supabase Cloud. Docker Compose and plain\nPostgres self-hosting are not supported yet because Preseason currently depends\non Supabase Auth. See [ docs/SELF_HOSTING.md](/betocmn/preseason/blob/main/docs/SELF_HOSTING.md).\n\n```\n   ┌────────────┐        ┌──────────────┐        ┌─────────────┐\n   │  Cron      │───────▶│  OpenRouter  │───────▶│  Response   │\n   │  /api/cron │ prompt │  (one model) │ answer │  parser     │\n   │  /benchmark│        └──────────────┘        └──────┬──────┘\n   └────────────┘                                        │\n         ▲                                               ▼\n         │ every 6 min                          ┌────────────────┐\n         │                                      │  Case decision │\n         │                                      │  tool / none / │\n         │                                      │  invalid       │\n         │                                      └────────┬───────┘\n         │                                               │\n         │                                               ▼\n   ┌─────┴──────┐    QC pass    ┌─────────────────────────────┐\n   │  Season    │◀──────────────│  Rankings + head-to-head    │\n   │  (frozen)  │               │  matches (public)           │\n   └────────────┘               └─────────────────────────────┘\n```\n\nEvery active **season** freezes a set of prompt versions and model\nsnapshots. The cron route at `/api/cron/benchmark-run`\n\nwalks every\nprompt × model combination, requires the model to produce a strict\nmachine-readable appendix, parses each response into a case decision\n(`tool`\n\n/ `none`\n\n/ `invalid`\n\n), and publishes runs that pass QC.\n\nPublic pages, including rankings, category indexes, and head-to-head matches, only read from published benchmark data. Unrecognized tool names are held in a candidate queue for admin review rather than guessed at.\n\n(App Router, React Server Components)[Next.js 15](https://nextjs.org): typed API[tRPC v11](https://trpc.io)+[Drizzle ORM](https://orm.drizzle.team/)(Postgres + email-OTP auth)[Supabase](https://supabase.com): model gateway[OpenRouter](https://openrouter.ai)+[Tailwind CSS v4](https://tailwindcss.com)[shadcn/ui](https://ui.shadcn.com)+ Testcontainers for an integration-tested Postgres[Vitest](https://vitest.dev)for lint + format[Biome](https://biomejs.dev)\n\n: local development environment`docs/SETUP.md`\n\n: supported deployment path`docs/SELF_HOSTING.md`\n\n: every env var explained`docs/CONFIGURATION.md`\n\n: system overview`docs/ARCHITECTURE.md`\n\n: glossary of project terms`docs/CONCEPTS.md`\n\n: how rankings are produced`docs/METHODOLOGY.md`\n\n: what's planned next`docs/ROADMAP.md`\n\n[How Benchmarks Work](/betocmn/preseason/blob/main/docs/guides/how-benchmarks-work.md)[How Prompts Work](/betocmn/preseason/blob/main/docs/guides/how-prompts-work.md)[How Rankings Work](/betocmn/preseason/blob/main/docs/guides/how-rankings-work.md)[How Cron Benchmarks Work](/betocmn/preseason/blob/main/docs/guides/how-cron-benchmarks-work.md)[How Matches Work](/betocmn/preseason/blob/main/docs/guides/how-matches-work.md)[How LLM Service Works](/betocmn/preseason/blob/main/docs/guides/how-llm-service-works.md)[How Evals Work](/betocmn/preseason/blob/main/docs/guides/how-evals-work.md)[Recommendation Methodology](/betocmn/preseason/blob/main/docs/guides/recommendation-methodology.md)\n\nPull requests are very welcome. See [ CONTRIBUTING.md](/betocmn/preseason/blob/main/CONTRIBUTING.md) for\nhow to set up, what we look for in PRs, and our triage SLA. New to the\nproject? Look for issues labelled\n\n[.](https://github.com/betocmn/preseason/labels/good%20first%20issue)\n\n`good first issue`\n\nWe follow the [Contributor Covenant](/betocmn/preseason/blob/main/CODE_OF_CONDUCT.md). Security reports\ngo through [ SECURITY.md](/betocmn/preseason/blob/main/SECURITY.md).\n\n[MIT](/betocmn/preseason/blob/main/LICENSE). See the `LICENSE`\n\nfile. Third-party tool logos under\n`public/logos/`\n\nare used under nominative fair use; see\n[ docs/LOGO_POLICY.md](/betocmn/preseason/blob/main/docs/LOGO_POLICY.md).", "url": "https://wpnews.pro/news/show-hn-i-asked-llms-to-choose-between-popular-developer-tools", "canonical_source": "https://github.com/betocmn/preseason", "published_at": "2026-08-31 02:29:25+00:00", "updated_at": "2026-08-31 02:52:52.935306+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-research"], "entities": ["Preseason", "OpenRouter", "Supabase", "Vercel", "GPT-4.1"], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-asked-llms-to-choose-between-popular-developer-tools", "markdown": "https://wpnews.pro/news/show-hn-i-asked-llms-to-choose-between-popular-developer-tools.md", "text": "https://wpnews.pro/news/show-hn-i-asked-llms-to-choose-between-popular-developer-tools.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-asked-llms-to-choose-between-popular-developer-tools.jsonld"}}