AI, Quiz Me!

A developer published a Claude skill file called "codebase-quiz" that quizzes programmers on their own repositories to surface where their mental model of the code is wrong, and reported it worked well enough to schedule a recurring Friday session. The skill stores a per-repository profile.md and log.md under ~/.claude/codebase-quiz/<repo>/, derives 8–12 code areas from the repo's top-level structure, counts the user's commits per area via git log, and runs a default 8-question calibration round before adapting difficulty. The author framed the tool as a response to losing touch with codebases as agent-driven, non-deterministic development replaces hand-written code.

I’ve noticed my desire to write about programming in this age of AI deteriorating lately. Not because interesting things aren’t happening, certainly not, but because it seems that the insights I have are being had simultaneously by everybody else. I’m going to ignore that sentiment and write something anyway. Partly because I want to be a write, not a write-not https://paulgraham.com/writes.html , and partly because my brain rewards me when I publish something, and I miss that feeling. So let me tell you about this Claude skill that I’ve been enjoying lately. Ever since my programming process shifted to writing primarily-English-occasionally-curly-brackets rather than the other way around, I’ve had this sinking feeling that I am losing touch with whatever codebase I am working on. I’ve experienced jumps in abstraction before. I wrote x86 assembly for a good while, then C++ and then the languages that carry so much machinery around with them that it would have made assembly-me squirm. Embracing a higher level of abstraction means giving up some control but in return opening up a world of new opportunities. I get that. But something feels different this time. I guess first and foremost because an agent is not deterministic the way a compiler is. But also because it feels like this time, I get to decide how much control I am willing to give up. The agent is perfectly happy to accept vague and poorly thought out instructions and quite often do a reasonable job all the same. On the other hand, retaining control is fatiguing Which means that I am now in yet another battle of will power, like I am in so many other aspects of life, like my sweet tooth. Dangerous I need make an effort, I think, to keep an understanding of how every codebase I work with works. So I tried to get Claude to quiz me. This has worked so well that I have put a notice in my calendar to run it every Friday, and I think it’s really valuable. Here is the skill file: --- name: codebase-quiz description: Quiz the user on the current repo to find where their mental model of the code is wrong. Use when the user asks to be quizzed or tested on the codebase ”quiz me”, “codebase quiz”, “Friday quiz” . --- Codebase quiz Goal: find where my mental model of this codebase is wrong, not teach me. Don’t explain anything before I answer. Storage ~/.claude/codebase-quiz/<repo / , where <repo is the repository name from git remote get-url origin not the local path, so multiple clones share one history . - profile.md — tenure, areas name, paths, my 1–3 rating, current difficulty, questions asked so far , session length, onboarding date, top-level structure snapshot. - log.md — one dated section per session. Setup don’t narrate it If profile.md is missing → onboarding 1. Survey the repo: top-level apps/packages, CLAUDE.md files, rules, docs. Propose 8–12 areas, each with the paths it covers. 2. For each area, count my commits: git log --author=”$ git config user.email ” --oneline -- <paths | wc -l . Find my first commit date as proposed tenure. 3. Show the table area, paths, my commits and ask me to: correct tenure, edit the area list, rate each area 1–3, set session length default 8 questions . 4. Write profile.md , including a snapshot of the top-level directory listing. 5. This session is a calibration round: one medium question per area. At the end, record each area’s starting difficulty from the results, and note where my self-rating and my result disagree. If profile.md exists 1. Read profile.md and log.md . Note days since the last session. 2. If the top-level structure differs from the snapshot, or onboarding is older than 6 months, propose area-list changes before starting. 3. Run git log --merges --oneline --since=<last session date . 4. Plan <session length questions: - re-test past misconceptions more of these the longer since the last session , - ~2 on recent merges, - the rest favouring areas with a low rating or few questions asked so far. Every session For each planned question, read the code that settles it before asking. Docs, CLAUDE.md and rules files are sources for questions, not answer keys. If they disagree with the code, the code is the answer, and note the doc as stale. Questions - Kinds: predict given this input/state, what happens , trace what calls what, from UI action to network request , locate where is this decided , why / what breaks , spot the bug show a small plausible diff; what does it break . - No trivia: names, spellings, line numbers. Ask about behaviour and boundaries. - Prefer cases where the plausible default assumption is wrong. - Free-text answers, no multiple choice. - One question at a time. Ask my confidence 1–3 with it. Wait for my answer. After each answer - Verdict: correct / partly / wrong. Then the actual behaviour in ≤5 lines with file:line refs. - If partly or wrong: state the wrong assumption in one sentence. - One follow-up only if my answer points to a deeper misunderstanding. - Difficulty per area: 2 correct in a row → harder edge cases, cross-package . Wrong → same topic, simpler. - “disagree”: re-check the code, then concede or show the evidence. - “skip” and “explain more” are allowed. “stop” ends the session early; still do the End steps. End 1. Table: , area, kind, confidence, result. 2. Misconceptions found, each as “I believed X; actually Y file ”. 3. Docs/rules that contradicted the code, with the specific claim. 4. Append all of the above, dated, to log.md . 5. Update profile.md : per-area difficulty and questions-asked counts. Obviously, this is Claude-specific but you should be able to have your agent of choice adjust it as necessary quite easily.