Weighted Averages Lie About AI Readiness — The Case for Bottleneck Scoring

A developer argues that weighted-average scoring models for AI readiness assessments are structurally flawed because they allow catastrophic weaknesses to be masked by strengths in other areas. To address this, they designed a 'bottleneck scoring' system that caps the total score when fatal preconditions—such as missing version control or automated tests—are absent, reducing the mechanism to a simple Math.min() function. The approach is implemented in an open-source, client-side assessment tool built with TypeScript, React, and Vite.

Hook Most self-assessment tools score you the same way: answer questions, multiply by weights, sum it up. Higher total = more ready. That model has a structural flaw. A team that doesn't use Git — but aces documentation, AI policy, and project fit — can score high. Yet without version control, every large AI-generated change is an irreversible overwrite. No amount of strength elsewhere compensates. Call it the honor student problem: additive scoring rewards averages when what actually matters is the weakest link. The article walks through how I designed a scoring model that caps the total score when a fatal precondition is missing — and why the whole mechanism collapses into a single Math.min . Target Audience Developers and tech leads who: - Are evaluating or building readiness / maturity assessments for AI-assisted development - Design scoring, ranking, or evaluation logic and care about the failure modes of weighted averages - Enjoy "small design decision, big consequence" write-ups grounded in a real open-source implementation Key Sections 1. The Honor Student Problem - How typical readiness assessments work: weighted average across categories - Failure mode: catastrophic weakness in one area is diluted by strength in others - Concrete example: no Git + perfect everything else = high score under additive scoring - Why this matters more for AI-driven development: AI multiplies change volume, so missing safety rails are amplified, not averaged away 2. Bottleneck Scoring: Average for Progress, Cap for Preconditions - Keep the weighted average — it's good at expressing continuous improvement - Add a second layer: fatal preconditions that cap the total score when absent - The entire mechanism reduces to Math.min baseScore, cap - Analogy: Liebig's law of the minimum the barrel with one short stave 3. The Six Preconditions and Their Caps - No version control Git → capped at 49 - Almost no written specs → capped at 49 - No task/ticket management → capped at 59 - No human review / production approval → capped at 59 - No automated tests and no change checklist → capped at 59 - No rules on what must not be fed to AI → capped at 69 - When multiple fire, the lowest cap wins rate-limited by the worst bottleneck 4. Why 49, 59, 69 — Caps as Level Ceilings - 100-point scale, 5 axes: Documentation 25 / Process 25 / Quality Assurance 20 / AI Usage 15 / Project Fit 15 - 5 levels Lv1: 0–29 … Lv5: 85–100 - Caps sit deliberately just below level boundaries 50, 70 : a cap encodes "the highest level you can reach while carrying this gap" 5. Proving It Works: Test Cases as Design Documentation - All answers max → 100, Lv5 - No Git, everything else perfect → forced to 49 Lv2 - Multiple caps at once → the minimum 49 is applied - Deterministic scoring same input, same output guaranteed by Vitest unit tests 6. Design Decisions That Followed From the Same Principle - "I don't know" scores 0.2, not 0 — not knowing is different from not having - Solo developers: team-only questions e.g., PR review practice get 0.5× weight - Recommendations prioritized by impact axis points / questions per axis × unmet degree , organized into a now / 1-month / 3-month roadmap - Strength selection enforces axis diversity - Of 8 development phases evaluated for AI fitness, documentation is the only one that can never be "not recommended" 7. The Tool Itself Briefly - Fully client-side static app: TypeScript, React 19, Vite 8, Tailwind CSS 4, shadcn-ui; GitHub Pages - Zero external transmission — answers stay in LocalStorage/IndexedDB, verified automatically with Playwright E2E tests - Scoring logic versioned SCHEMA VERSION ; old results get an "outdated version" badge - 5 languages, MIT license, just shipped — no usage numbers to brag about, and don't pretend otherwise Estimated Length 2,000–2,400 words design deep-dive / engineering decision write-up Tone Notes - This is a "one design decision, examined honestly" piece — the star is the scoring model, not the tool. Keep the tool as the implementation vehicle, mentioned mostly at the end. - Lead with the honor student problem as a relatable trap; most readers have built or used an additive rubric. - Use a before/after code-style diagram additive score vs capped score for the no-Git team to make the flaw visceral. - Be candid that the implementation is almost embarrassingly simple Math.min — the value is in choosing the caps, not the code. - Do not overstate the project: it's a personal OSS tool published in July 2026 with no track record yet. "A project I just shipped" is the right register. - End with a question inviting readers to name the bottleneck their own team is averaging away. SEO / Discoverability - Primary keywords: "AI readiness assessment", "scoring system design", "weighted average problems" - Secondary: "maturity model scoring", "bottleneck scoring", "self-assessment tool design" - The generalizable lesson cap vs average in any evaluation system makes this shareable beyond the AI niche — frame the opening so rubric designers of any kind feel addressed