Show HN: I asked LLMs to choose between popular developer tools

Measure which developer tools LLMs recommend when asked to build real web apps.

Preseason is an open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps.

We run a fixed set of web-app prompts against a fixed panel of models, parse each answer for recommended tools and services, then publish rankings, head-to-head comparisons, and methodology notes.

The goal is to make AI-driven developer-tool recommendations inspectable, reproducible, and contestable, so you can see which tools AI coding assistants are most likely to put in front of developers.

🌐 Live demo: https://preseason.ai

Preseason currently tracks recommendations across categories like:

• databases
• auth
• hosting
• analytics
• payments
• background jobs
• UI/component libraries
• observability
• AI/model providers

For each prompt × model run, we record whether the model recommended a known tool, no tool, or an invalid/unrecognized answer.

• Which database does each model recommend most often for a new SaaS app?
• Does GPT-4.1 prefer Supabase, Firebase, Neon, or plain Postgres?
• Which tools win head-to-head when two options appear in similar prompts?
• Are some models more likely to recommend "no tool" or hallucinate unknown tools?

Recommendations from AI coding assistants shape developer tool adoption faster than blog posts or Twitter threads. If a foundation model quietly favors one database or hosting provider, that preference scales to every developer using it. We think the methodology behind that should be open, reproducible, and contestable, not a private dashboard.

Preseason exists so anyone can:

• See what today's LLMs recommend, with frozen prompts and model snapshots that are inspectable in this repo - Run their own benchmark on their own prompts or model panel - Submit issues when results look off and have an open paper trail

• The benchmark measures recommendations, not whether a tool is objectively better.

• Results depend on the frozen prompt set and model snapshots.

• Tool-name parsing is intentionally strict; unknown names go to review instead of being guessed.

• The project is early, so rankings should be treated as directional rather than definitive.

pnpm run setup                  # installs deps and starts local Supabase
cp .env.example .env.local      # fill with `supabase status` + OpenRouter key
pnpm run db:migrate
pnpm run db:seed
pnpm run db:seed-dev
pnpm run dev

App is at http://localhost:3000. Full setup details, including the env var table and troubleshooting, are in docs/SETUP.md.

The supported launch path is Vercel + Supabase Cloud. Docker Compose and plain Postgres self-hosting are not supported yet because Preseason currently depends on Supabase Auth. See docs/SELF_HOSTING.md.

   ┌────────────┐        ┌──────────────┐        ┌─────────────┐
   │  Cron      │───────▶│  OpenRouter  │───────▶│  Response   │
   │  /api/cron │ prompt │  (one model) │ answer │  parser     │
   │  /benchmark│        └──────────────┘        └──────┬──────┘
   └────────────┘                                        │
         ▲                                               ▼
         │ every 6 min                          ┌────────────────┐
         │                                      │  Case decision │
         │                                      │  tool / none / │
         │                                      │  invalid       │
         │                                      └────────┬───────┘
         │                                               │
         │                                               ▼
   ┌─────┴──────┐    QC pass    ┌─────────────────────────────┐
   │  Season    │◀──────────────│  Rankings + head-to-head    │
   │  (frozen)  │               │  matches (public)           │
   └────────────┘               └─────────────────────────────┘

Every active season freezes a set of prompt versions and model snapshots. The cron route at /api/cron/benchmark-run

walks every prompt × model combination, requires the model to produce a strict machine-readable appendix, parses each response into a case decision (tool

/ none

/ invalid

), and publishes runs that pass QC.

Public pages, including rankings, category indexes, and head-to-head matches, only read from published benchmark data. Unrecognized tool names are held in a candidate queue for admin review rather than guessed at.

(App Router, React Server Components)Next.js 15: typed APItRPC v11+Drizzle ORM(Postgres + email-OTP auth)Supabase: model gatewayOpenRouter+Tailwind CSS v4shadcn/ui+ Testcontainers for an integration-tested PostgresVitestfor lint + formatBiome

: local development environmentdocs/SETUP.md

: supported deployment pathdocs/SELF_HOSTING.md

: every env var explaineddocs/CONFIGURATION.md

: system overviewdocs/ARCHITECTURE.md

: glossary of project termsdocs/CONCEPTS.md

: how rankings are produceddocs/METHODOLOGY.md

: what's planned nextdocs/ROADMAP.md

How Benchmarks WorkHow Prompts WorkHow Rankings WorkHow Cron Benchmarks WorkHow Matches WorkHow LLM Service WorksHow Evals WorkRecommendation Methodology

Pull requests are very welcome. See CONTRIBUTING.md for how to set up, what we look for in PRs, and our triage SLA. New to the project? Look for issues labelled

.

good first issue

We follow the Contributor Covenant. Security reports go through SECURITY.md.

MIT. See the LICENSE

file. Third-party tool logos under public/logos/

are used under nominative fair use; see docs/LOGO_POLICY.md.