# Show HN: I asked LLMs to choose between popular developer tools

> Source: <https://github.com/betocmn/preseason>
> Published: 2026-08-31 02:29:25+00:00

**Measure which developer tools LLMs recommend when asked to build real web apps.**

Preseason is an open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps.

We run a fixed set of web-app prompts against a fixed panel of models, parse each answer for recommended tools and services, then publish rankings, head-to-head comparisons, and methodology notes.

The goal is to make AI-driven developer-tool recommendations inspectable, reproducible, and contestable, so you can see which tools AI coding assistants are most likely to put in front of developers.

🌐 **Live demo:** [https://preseason.ai](https://preseason.ai)

Preseason currently tracks recommendations across categories like:

- databases
- auth
- hosting
- analytics
- payments
- background jobs
- UI/component libraries
- observability
- AI/model providers

For each prompt × model run, we record whether the model recommended a known tool, no tool, or an invalid/unrecognized answer.

- Which database does each model recommend most often for a new SaaS app?
- Does GPT-4.1 prefer Supabase, Firebase, Neon, or plain Postgres?
- Which tools win head-to-head when two options appear in similar prompts?
- Are some models more likely to recommend "no tool" or hallucinate unknown tools?

Recommendations from AI coding assistants shape developer tool adoption faster than blog posts or Twitter threads. If a foundation model quietly favors one database or hosting provider, that preference scales to every developer using it. We think the methodology behind that should be open, reproducible, and contestable, not a private dashboard.

Preseason exists so anyone can:

- See
**what** today's LLMs recommend, with frozen prompts and model snapshots that are inspectable in this repo - Run
**their own** benchmark on their own prompts or model panel - Submit
**issues** when results look off and have an open paper trail

- The benchmark measures recommendations, not whether a tool is objectively better.
- Results depend on the frozen prompt set and model snapshots.
- Tool-name parsing is intentionally strict; unknown names go to review instead of being guessed.
- The project is early, so rankings should be treated as directional rather than definitive.

```
pnpm run setup                  # installs deps and starts local Supabase
cp .env.example .env.local      # fill with `supabase status` + OpenRouter key
pnpm run db:migrate
pnpm run db:seed
pnpm run db:seed-dev
pnpm run dev
```

App is at [http://localhost:3000](http://localhost:3000). Full setup details, including the env
var table and troubleshooting, are in [ docs/SETUP.md](/betocmn/preseason/blob/main/docs/SETUP.md).

The supported launch path is Vercel + Supabase Cloud. Docker Compose and plain
Postgres self-hosting are not supported yet because Preseason currently depends
on Supabase Auth. See [ docs/SELF_HOSTING.md](/betocmn/preseason/blob/main/docs/SELF_HOSTING.md).

```
   ┌────────────┐        ┌──────────────┐        ┌─────────────┐
   │  Cron      │───────▶│  OpenRouter  │───────▶│  Response   │
   │  /api/cron │ prompt │  (one model) │ answer │  parser     │
   │  /benchmark│        └──────────────┘        └──────┬──────┘
   └────────────┘                                        │
         ▲                                               ▼
         │ every 6 min                          ┌────────────────┐
         │                                      │  Case decision │
         │                                      │  tool / none / │
         │                                      │  invalid       │
         │                                      └────────┬───────┘
         │                                               │
         │                                               ▼
   ┌─────┴──────┐    QC pass    ┌─────────────────────────────┐
   │  Season    │◀──────────────│  Rankings + head-to-head    │
   │  (frozen)  │               │  matches (public)           │
   └────────────┘               └─────────────────────────────┘
```

Every active **season** freezes a set of prompt versions and model
snapshots. The cron route at `/api/cron/benchmark-run`

walks every
prompt × model combination, requires the model to produce a strict
machine-readable appendix, parses each response into a case decision
(`tool`

/ `none`

/ `invalid`

), and publishes runs that pass QC.

Public pages, including rankings, category indexes, and head-to-head matches, only read from published benchmark data. Unrecognized tool names are held in a candidate queue for admin review rather than guessed at.

(App Router, React Server Components)[Next.js 15](https://nextjs.org): typed API[tRPC v11](https://trpc.io)+[Drizzle ORM](https://orm.drizzle.team/)(Postgres + email-OTP auth)[Supabase](https://supabase.com): model gateway[OpenRouter](https://openrouter.ai)+[Tailwind CSS v4](https://tailwindcss.com)[shadcn/ui](https://ui.shadcn.com)+ Testcontainers for an integration-tested Postgres[Vitest](https://vitest.dev)for lint + format[Biome](https://biomejs.dev)

: local development environment`docs/SETUP.md`

: supported deployment path`docs/SELF_HOSTING.md`

: every env var explained`docs/CONFIGURATION.md`

: system overview`docs/ARCHITECTURE.md`

: glossary of project terms`docs/CONCEPTS.md`

: how rankings are produced`docs/METHODOLOGY.md`

: what's planned next`docs/ROADMAP.md`

[How Benchmarks Work](/betocmn/preseason/blob/main/docs/guides/how-benchmarks-work.md)[How Prompts Work](/betocmn/preseason/blob/main/docs/guides/how-prompts-work.md)[How Rankings Work](/betocmn/preseason/blob/main/docs/guides/how-rankings-work.md)[How Cron Benchmarks Work](/betocmn/preseason/blob/main/docs/guides/how-cron-benchmarks-work.md)[How Matches Work](/betocmn/preseason/blob/main/docs/guides/how-matches-work.md)[How LLM Service Works](/betocmn/preseason/blob/main/docs/guides/how-llm-service-works.md)[How Evals Work](/betocmn/preseason/blob/main/docs/guides/how-evals-work.md)[Recommendation Methodology](/betocmn/preseason/blob/main/docs/guides/recommendation-methodology.md)

Pull requests are very welcome. See [ CONTRIBUTING.md](/betocmn/preseason/blob/main/CONTRIBUTING.md) for
how to set up, what we look for in PRs, and our triage SLA. New to the
project? Look for issues labelled

[.](https://github.com/betocmn/preseason/labels/good%20first%20issue)

`good first issue`

We follow the [Contributor Covenant](/betocmn/preseason/blob/main/CODE_OF_CONDUCT.md). Security reports
go through [ SECURITY.md](/betocmn/preseason/blob/main/SECURITY.md).

[MIT](/betocmn/preseason/blob/main/LICENSE). See the `LICENSE`

file. Third-party tool logos under
`public/logos/`

are used under nominative fair use; see
[ docs/LOGO_POLICY.md](/betocmn/preseason/blob/main/docs/LOGO_POLICY.md).