{"slug": "weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring", "title": "Weighted Averages Lie About AI Readiness — The Case for Bottleneck Scoring", "summary": "A developer argues that weighted-average scoring models for AI readiness assessments are structurally flawed because they allow catastrophic weaknesses to be masked by strengths in other areas. To address this, they designed a 'bottleneck scoring' system that caps the total score when fatal preconditions—such as missing version control or automated tests—are absent, reducing the mechanism to a simple Math.min() function. The approach is implemented in an open-source, client-side assessment tool built with TypeScript, React, and Vite.", "body_md": "##\nHook\n\nMost self-assessment tools score you the same way: answer questions, multiply by weights, sum it up. Higher total = more ready.\n\nThat model has a structural flaw. A team that doesn't use Git — but aces documentation, AI policy, and project fit — can score high. Yet without version control, every large AI-generated change is an irreversible overwrite. No amount of strength elsewhere compensates.\n\nCall it the honor student problem: additive scoring rewards averages when what actually matters is the weakest link. The article walks through how I designed a scoring model that caps the total score when a fatal precondition is missing — and why the whole mechanism collapses into a single `Math.min()`\n\n.\n\n##\nTarget Audience\n\nDevelopers and tech leads who:\n\n- Are evaluating (or building) readiness / maturity assessments for AI-assisted development\n- Design scoring, ranking, or evaluation logic and care about the failure modes of weighted averages\n- Enjoy \"small design decision, big consequence\" write-ups grounded in a real open-source implementation\n\n##\nKey Sections\n\n###\n1. The Honor Student Problem\n\n- How typical readiness assessments work: weighted average across categories\n- Failure mode: catastrophic weakness in one area is diluted by strength in others\n- Concrete example: no Git + perfect everything else = high score under additive scoring\n- Why this matters more for AI-driven development: AI multiplies change volume, so missing safety rails are amplified, not averaged away\n\n###\n2. Bottleneck Scoring: Average for Progress, Cap for Preconditions\n\n- Keep the weighted average — it's good at expressing continuous improvement\n- Add a second layer: fatal preconditions that cap the total score when absent\n- The entire mechanism reduces to\n`Math.min(baseScore, cap)`\n\n- Analogy: Liebig's law of the minimum (the barrel with one short stave)\n\n###\n3. The Six Preconditions and Their Caps\n\n- No version control (Git) → capped at 49\n- Almost no written specs → capped at 49\n- No task/ticket management → capped at 59\n- No human review / production approval → capped at 59\n- No automated tests and no change checklist → capped at 59\n- No rules on what must not be fed to AI → capped at 69\n- When multiple fire, the lowest cap wins (rate-limited by the worst bottleneck)\n\n###\n4. Why 49, 59, 69 — Caps as Level Ceilings\n\n- 100-point scale, 5 axes: Documentation 25 / Process 25 / Quality Assurance 20 / AI Usage 15 / Project Fit 15\n- 5 levels (Lv1: 0–29 … Lv5: 85–100)\n- Caps sit deliberately just below level boundaries (50, 70): a cap encodes \"the highest level you can reach while carrying this gap\"\n\n###\n5. Proving It Works: Test Cases as Design Documentation\n\n- All answers max → 100, Lv5\n- No Git, everything else perfect → forced to 49 (Lv2)\n- Multiple caps at once → the minimum (49) is applied\n- Deterministic scoring (same input, same output) guaranteed by Vitest unit tests\n\n###\n6. Design Decisions That Followed From the Same Principle\n\n- \"I don't know\" scores 0.2, not 0 — not knowing is different from not having\n- Solo developers: team-only questions (e.g., PR review practice) get 0.5× weight\n- Recommendations prioritized by impact (axis points / questions per axis × unmet degree), organized into a now / 1-month / 3-month roadmap\n- Strength selection enforces axis diversity\n- Of 8 development phases evaluated for AI fitness, documentation is the only one that can never be \"not recommended\"\n\n###\n7. The Tool Itself (Briefly)\n\n- Fully client-side static app: TypeScript, React 19, Vite 8, Tailwind CSS 4, shadcn-ui; GitHub Pages\n- Zero external transmission — answers stay in LocalStorage/IndexedDB, verified automatically with Playwright E2E tests\n- Scoring logic versioned (SCHEMA_VERSION); old results get an \"outdated version\" badge\n- 5 languages, MIT license, just shipped — no usage numbers to brag about, and don't pretend otherwise\n\n##\nEstimated Length\n\n2,000–2,400 words (design deep-dive / engineering decision write-up)\n\n##\nTone Notes\n\n- This is a \"one design decision, examined honestly\" piece — the star is the scoring model, not the tool. Keep the tool as the implementation vehicle, mentioned mostly at the end.\n- Lead with the honor student problem as a relatable trap; most readers have built or used an additive rubric.\n- Use a before/after code-style diagram (additive score vs capped score for the no-Git team) to make the flaw visceral.\n- Be candid that the implementation is almost embarrassingly simple (\n`Math.min`\n\n) — the value is in choosing the caps, not the code.\n- Do not overstate the project: it's a personal OSS tool published in July 2026 with no track record yet. \"A project I just shipped\" is the right register.\n- End with a question inviting readers to name the bottleneck their own team is averaging away.\n\n##\nSEO / Discoverability\n\n- Primary keywords: \"AI readiness assessment\", \"scoring system design\", \"weighted average problems\"\n- Secondary: \"maturity model scoring\", \"bottleneck scoring\", \"self-assessment tool design\"\n- The generalizable lesson (cap vs average in any evaluation system) makes this shareable beyond the AI niche — frame the opening so rubric designers of any kind feel addressed", "url": "https://wpnews.pro/news/weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring", "canonical_source": "https://dev.to/yun_bow/weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring-2595", "published_at": "2026-07-18 13:38:49+00:00", "updated_at": "2026-07-18 13:58:44.128779+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "ai-safety", "ai-agents"], "entities": ["Git", "Vitest", "TypeScript", "React", "Vite", "Tailwind CSS", "Playwright", "GitHub Pages"], "alternates": {"html": "https://wpnews.pro/news/weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring", "markdown": "https://wpnews.pro/news/weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring.md", "text": "https://wpnews.pro/news/weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring.txt", "jsonld": "https://wpnews.pro/news/weighted-averages-lie-about-ai-readiness-the-case-for-bottleneck-scoring.jsonld"}}