# A complete engineering log of building `findmypylibrary` with an AI pair-programmer (Claude Code) > Source: > Published: 2026-09-20 18:00:36+00:00 *A complete engineering log of building `findmypylibrary` with an AI pair-programmer (Claude Code): what broke, how we found out, what we tried, what we measured, and why each decision went the way it did. Nothing is left out, including the mistakes that were ours.* `findmypylibrary` answers one question: *"I need to do X in Python. Which package?"* ``` pip install findmypylibrary findmypylibrary refresh findmypylibrary "fuzzy string matching" 1. RapidFuzz (score 0.90) rapid fuzzy string matching downloads/30d: 163,835,611 last release: 2026-08-30 2. pfzy (score 0.81) Python port of the fzy fuzzy string matching algorithm downloads/30d: 27,024,226 last release: 2022-01-28 3. fuzzywuzzy (score 0.73) Fuzzy string matching in python downloads/30d: 14,789,165 last release: 2020-02-13 ``` Three constraints were fixed on day one and never changed, and they explain most of the decisions below: The project started as a 0.0.1 placeholder on PyPI: a name reservation with a README that described two commands that did not exist yet. This is the story of getting from there to something we were willing to release. The plan needed, for the most-used packages on PyPI: name, description, download count and last release date. **The first obstacle:** PyPI has no free bulk endpoint for "all packages with downloads". Download statistics live in a public BigQuery dataset, and BigQuery needs a Google Cloud account and key. That breaks constraint 3. **What we did instead:** two public sources, neither needing a key. `https://pypi.org/pypi//json`), one request per package, for the summary and release date. **A small first bug:** the dataset's documented URL returned an HTML page, not JSON. It was a `301` redirect to a new domain that our first `curl` did not follow. We switched to the raw GitHub URL. Lesson filed for later: never assume a `200`, and follow redirects (this came back to bite us in Step 12). **Decision: how many packages?** The question was 200, 2,000 or 8,000. The answer from the product owner was "15,000, or whatever covers all actively downloaded libraries". When we opened the dataset it contained exactly 15,000 rows, down to packages with about 68,000 downloads a month. So "top 15,000 by 30-day downloads" became the definition of "active", for free. **The cost of that decision:** 15,000 packages means 15,000 HTTP requests per snapshot. Sequentially that is over an hour. With `httpx.AsyncClient` and a semaphore of 25 concurrent requests it takes a few minutes. Hold that thought; it becomes the scaling problem in Step 5. Architecture, deliberately boring: ``` php fetch.py two sources -> list of package dicts (async, bounded concurrency, retries) cache.py sqlite file in ~/.cache/findmypylibrary/ rank.py query -> ranked packages cli.py click commands: refresh, search ``` Three details worth recording. **1. A bare query had to work.** The README promised `findmypylibrary "parse messy pdfs"`, with no `search` subcommand. Click groups reject unknown command names, so we subclassed the group: ``` python class DefaultGroup(click.Group): def resolve_command(self, ctx, args): try: return super().resolve_command(ctx, args) except click.UsageError: return super().resolve_command(ctx, ["search", *args]) ``` This looked finished. It was not (Steps 12 and 15 found three more cases). **2. An ordering trap in the async crawl.** We used `asyncio.as_completed` to drive a progress bar. It yields results in completion order, so you cannot `zip` them back onto the input list to attach download counts. We caught this at design time and made each task return its own complete record, download count included, instead of re-joining afterwards. **3. We did not hammer PyPI while developing.** Every test of the pipeline during development used `refresh --limit 40`. The full 15,000-package crawl was run only when we needed real data. Result of the first full run: 14,999 of 15,000 packages (the missing one was a genuine `404`, a delisted package). The first ranking was textbook: a pure-Python BM25 over name + summary + keywords, then a weighted blend. ``` score = 0.60 * relevance + 0.25 * popularity + 0.15 * recency (each min-max normalised) ``` It produced sensible results for the first few queries we tried. That was the problem: we only tried a few. `openpyxl` loses to a package nobody has heard of Query: `read and write excel spreadsheets`. ``` 1. numbers-parser Read and write Apple Numbers spreadsheets 924,605 downloads 2. tifffile Read and write TIFF files 27,744,478 downloads 3. openpyxl A Python library to read/write Excel ... 339,316,525 downloads ``` `openpyxl` is the right answer and it has 370 times the downloads of the winner. **Diagnosis (we looked at the numbers instead of guessing).** `openpyxl` matched three of the four query words, so it was not a recall problem. Two things combined: `numbers-parser`'s whole summary is six words, three of which are in the query. BM25's length normalisation rewards that density. **The fix: stop blending, start gating.** Relevance became a *gate* (keep anything within 50% of the best match), and the survivors were ranked mainly by popularity. This is how real package search engines work: text match for recall, authority for ordering. We also weighted matches in the package name above matches in the summary, and added a small synonym list (excel/xlsx/spreadsheet and a dozen more). This was done test-first. The regression test asserts the outcome a user sees, not an internal number: ``` python def test_popular_relevant_package_beats_short_keyword_dense_niche_one(): result = names("read and write excel spreadsheets", [openpyxl, numbers_parser, tifffile]) assert result[0] == "openpyxl" ``` We rejected one idea here on principle: a local embedding model. It would have pulled in a deep-learning runtime and a model download for a tool whose selling point is being light. The product owner uses a strict audit methodology on a large TypeScript / Postgres / Playwright application, and asked for the same gates here: Applied literally, half of that is meaningless for a Python CLI with no browser, no database server and no TypeScript. Copying the letter of a process into a different stack is how you end up with ceremonies nobody can explain. So we translated the intent: ``` Their gate What it became here ----------------------------- --------------------------------------------------- TDD pytest, regression test written before each fix >= 90% coverage pytest-cov, enforced in CI 0 TypeScript / lint errors ruff (lint + format) and mypy, zero findings Playwright end-to-end subprocess tests that run the real installed binary Leftover / twin host inventory grep every importer of anything we change or remove "Ledger-green, product-red" assert the outcome users see, never a proxy for it ``` That last line is the core lesson of their methodology: a test that checks "the file exists" or "the score changed" can be green while the product is broken. It turned out to apply to us more than once. To isolate test runs, a cleanup command removed `~/.cache/findmypylibrary` before running pytest. That directory was not a test artefact. It was the real 15,000-package snapshot that had taken a full crawl to build. **Fix:** rebuild it, then make the mistake impossible rather than merely avoided. Every test now runs with `XDG_CACHE_HOME` pointed at a temporary directory, through one `autouse` fixture: ``` python @pytest.fixture(autouse=True) def isolated_cache_dir(tmp_path, monkeypatch): monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "xdg-cache")) ``` **Lesson:** "be careful" is not a control. A fixture that every test gets automatically is. (We broke this rule again, through a different door, in Step 15.) With a working tool, the obvious flaw surfaced: `refresh` makes 15,000 requests to PyPI. One developer doing that monthly is fine. A thousand users doing it is not, and it would get everyone rate-limited. **Decision: build the snapshot once, centrally, and let users download it.** A scheduled GitHub Actions workflow builds the snapshot monthly and publishes it as a GitHub Release asset. `findmypylibrary refresh` downloads that file (one request, a couple of seconds). `refresh --build-locally` keeps the full crawl for anyone who wants a snapshot as of right now. This is the same pattern the upstream dataset uses for itself, which is a good sign it is the right shape. **A guardrail worth mentioning:** the AI assistant's permission system refused to create a *public* GitHub repository on its own. It stopped, explained, and asked. Creating a public surface is a human decision. We approved it explicitly, and only then did it create the repo and push. We verified the loop end to end, not just the workflow: trigger it manually, confirm the release exists, then run `refresh` on a clean machine profile and confirm it downloads and answers queries. Asked directly, the honest answer was a table of what had *not* been verified: ``` Gap Status after this step ---------------------------------- --------------------------------------------- No CI; gates only ran by hand CI on every push and pull request Only tested on macOS, Python 3.12 Matrix: Linux, macOS, Windows x 3 Pythons Reader during a refresh? Concurrency test added Real HTTP 429 from PyPI Still mock-only. We will not hammer a public service to force a rate limit. Said so plainly. ``` Saying "this one we cannot test ethically" is part of being ready. Hiding it is not. We ran 15 everyday queries against the real snapshot. About half were good. Five failed in a way that would embarrass the tool on day one: ``` Query Missing from the top results Why (confirmed in the data) ------------------------------ ----------------------------- --------------------------------------- dataframes pandas its metadata never says "dataframe" resize images Pillow summary says "Imaging", not "images" plot charts matplotlib summary says "plotting", not "plot" connect to postgres database psycopg2 (pymssql came first) "postgres" is not "PostgreSQL" unit testing pytest matched one word; the gate dropped it ``` Two root causes. Matching was **exact-token**, with no stemming. And the text was tiny: summaries average **8 words**, and 259 packages have none at all. A 13-entry synonym list cannot paper over that. The same review found seven robustness gaps. Each is a story of "it works on the happy path": `refresh` saved whatever it got. A network drop at 10% would replace 15,000 packages with 1,500, and in the workflow that would be published to everyone.`releases/latest/...`. The first time we published a We decided not to publish until the ranking and gaps 1 to 5 were fixed. A tool that cannot find pandas is not a release. **The design decision.** We needed stemming and more text per package. More text kills the pure-Python BM25: building an index over 15,000 documents of ~250 tokens *on every query* would take seconds. The index had to be built once and shipped. SQLite already has this. FTS5 is a full-text engine with a Porter stemmer and a BM25 ranking function, it is compiled into standard CPython on all three platforms (the CI matrix later proved it), and it adds zero dependencies: ``` CREATE VIRTUAL TABLE packages_fts USING fts5( name, summary, keywords, topics, description, content='', tokenize='porter unicode61' ); ``` `content=''` makes it *contentless*: README text is searchable but not stored, which keeps the file small. Result: 19.8 MB on disk, 10.9 MB gzipped, and queries in tens of milliseconds. **More text, carefully.** We added each package's topic classifiers and a cleaned excerpt of its README. "Cleaned" matters: in pandas' README the word "DataFrame" first appears about 3,000 characters in, behind badges, links and HTML. We strip images, links, URLs, tags and reST directives before indexing. **The golden queries came first.** Before touching a ranking constant we wrote 40 everyday queries, each with a list of acceptable right answers (`"resize images" -> pillow, opencv-python, ...`). Writing the expectations *before* seeing results is the only way to keep yourself honest. Baseline on the new index: 37 of 40. **The discovery that shaped the design: README noise.** For `unit testing`, the results were `boto3` and `tqdm`. Why? boto3's README has a section that says "run the unit tests". tqdm's talks about iterations-per-second "units". We had given the description column a weight of 1 against 8 for the name, and it made almost no difference. The reason is a property of BM25: term frequency *saturates*, so once a term appears at all, the column weight barely moves the score. boto3's README-only match scored 0.73 of the best real match. **Fix:** score the core fields (name, summary, keywords, topics) and the description in two separate queries, and add the description score at a heavy discount. A parameter sweep showed the pass rate was flat at 39/40 across a wide band of settings, which is what you want to see: a plateau, not a knife-edge. **A better metric.** Pass/fail at top 5 hides ordering. We added mean reciprocal rank (1 for a right answer in first place, 1/2 for second, and so on). It showed that scaling popularity by "log-downloads divided by the maximum" barely separated a package with a million downloads from one with a billion (the logs are 6 and 9). Min-max scaling over the surviving candidates lifted MRR from 0.69 to 0.88. **A test whose premise was wrong.** We asserted that tqdm must not appear for "unit testing". It kept appearing. Instead of tuning until it vanished, we looked: tqdm's own PyPI classifiers include `Education :: Testing`. That is the package describing itself, not README noise. The test was wrong, so we fixed the test, and kept the assertion for boto3, which was the genuine case. The concurrency test (searches running while a refresh rewrites the snapshot) was passing. Its output contained a *warning*: the writer thread had died with `sqlite3.OperationalError: disk I/O error`. The test passed because it only collected errors from the reader thread. It was asserting a proxy. We reproduced it in isolation and asked SQLite for the extended error: `SQLITE_IOERR_LOCK`. The cause was opening the snapshot through a read-only URI (`file:...?mode=ro`). On macOS, those connections made a concurrent writer's lock acquisition fail, three to five times per run. Plain connections never failed. **Fix:** open normally and make the connection read-only with `PRAGMA query_only = ON`. **And** fix the test so it records writer failures too. A bug fix without the test fix would have left the same blind spot for the next bug. Each with the reason, because the reason is what transfers to other projects. `releases/download/snapshot-latest/...`, a tag that only the snapshot workflow writes to. Code releases can never change what that URL points at.`snapshot-v2.sqlite.gz`) and in a `meta` table. Old installs keep downloading a layout they understand. A mismatch produces a clear "run refresh" message.`--build-locally`. The user decides whether to make 15,000 requests.` status` command, and a warning after 45 days. pandas still did not appear for `dataframes` (polars did, so the golden query passed, but the review had named pandas). Could we fix it? pandas matches `dataframes` *only* through its README. boto3 matches `unit testing` *only* through its README. We swept the two relevant parameters and printed both positions side by side: ``` gate readme weight pandas for "dataframes" boto3 for "unit testing" ----- ------------- ----------------------- ------------------------ 0.05 0.5 7th 6th 0.10 0.5 7th 6th 0.15 0.5 7th 6th 0.20 0.5 not shown not shown 0.25 0.5 not shown not shown ``` They move together, every time. To a lexical ranker the two cases are the same case. You cannot have one without the other. **Decision:** precision wins. No boto3 for "unit testing", and pandas is found by the words its own metadata uses (`data analysis` puts it first). We wrote this limit into the README with those exact examples, instead of tuning until a demo looked good. The same sweep gave a free improvement (a README weight of 0.5 took the golden set to 40/40 with boto3 still excluded), so we took it. We also tried weighting the coverage gate by term rarity. It removed one junk result and dropped Pillow and pytest. Rejected, with the numbers in the commit message. **Mistake 2.** A commit was gated like this: ``` pytest -q | tail -3 && git commit ... && git push ``` pytest failed. The commit and push happened anyway, because the exit status of a pipeline is the exit status of its *last* command, and `tail` always succeeds. Red code reached the main branch. **Fix:** never put a pipe between a gate and the thing it guards. Capture output to a file and test the real exit code: ``` fail=0 pytest -q > out.txt 2>&1 || fail=1 ruff check -q . || fail=1 mypy src > /dev/null || fail=1 if [ "$fail" -ne 0 ]; then echo "GATES RED - NOT COMMITTING"; exit 1; fi ``` **Mistake 3**, found while investigating mistake 2. The "failures" were not real. To prove a regression test had teeth, we had temporarily changed a constant in `rank.py` from `0.5` to `1.0`, confirmed the test failed, and restored it. Both versions of the file had the same size and were written within the same second. Python validates cached bytecode by modification time and size, so it considered the stale `.pyc` valid and kept loading the `1.0` version. The source said one thing and the interpreter ran another. **Fix:** after any mutation check, delete `__pycache__`. CI, which starts from a clean checkout, confirmed the pushed commit was green all along. Two lessons: check exit codes, not output; and when results contradict the source code, suspect the cache. Asked "is anything left?", re-running our own checklist would only confirm what we already believed. So we started a second AI reviewer with no edit rights and a specific brief: *here is what the author believes is handled; find what they missed.* In parallel we attacked the real CLI with odd inputs. Between the two: `ValueError: min() iterable argument is empty`. This sat behind 98% coverage.` refresh --build-locally --limit 0` wiped the snapshot.`0 < 0.95 * 0`, which is false, and saved an empty list. 14,999 packages became 0, exit code 0.`-n -1` printed 389 results`403` with an HTML body raised `JSONDecodeError` out of the whole run. It now costs one package. We also follow redirects now (remember Step 1).`PRAGMA quick_check` before swapping a download in.`résumé parser` was tokenised as `r`, `sum`, `parser`.` findmypylibrary status bar widget``-n 2 parse pdf`. And two findings about the *tests*, which mattered more than any single bug: `"Traceback" not in output`. Click's test runner keeps uncaught exceptions in `result.exception` and never writes them to `output`. Those assertions could not fail. That is exactly why the search crash survived. The test helper now fails on any uncaught exception: ``` crashed = result.exception is not None and not isinstance(result.exception, SystemExit) assert not crashed, f"CLI crashed with {result.exception!r}" ``` **Performance, measured rather than assumed.** Every search took 0.30 seconds, and 0.14 of that was importing `httpx`, which search never uses. Importing the fetch module lazily halved the start-up time to 0.15 seconds. We also measured FTS5 `optimize` plus `VACUUM`: 5% smaller, no speed-up. Not worth a line of code, so we did not add it. Piping results to `head` closed the pipe early. A new, broad `except OSError` handler caught the resulting error and told the user to "check permissions and free disk space". We fixed it by letting `BrokenPipeError` through. Local tests passed. CI failed on all four Windows jobs. On Windows a closed pipe does not raise `BrokenPipeError` (`EPIPE`). It raises `OSError` with `EINVAL`. The real problem was the design, not the missing case: a catch-all around the *whole command* could not tell "I cannot write the cache" from "nobody is reading my output". We removed it and did two precise things instead: ``` php def run() -> None: try: main() except OSError as exc: if exc.errno not in (errno.EPIPE, errno.EINVAL): raise os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno()) sys.exit(1) ``` No amount of local care would have found this. A cross-platform CI matrix is not a formality. The golden set said 100%. But we had tuned on it. So we wrote 30 *new* queries, fixed their expected answers before running anything, and ran them once. ``` Tuned set (40 queries): 40/40 in the top 5 MRR 0.89 Held-out set (30 queries): 25/30 in the top 5 MRR 0.72 ``` 83%, not 100%. That gap is the price of tuning on your test set, and it is the number users actually experience. The failures had one pattern. A package with a billion downloads that matched only the *commonest* word of the query floated to the top: ``` php gui application -> idna, platformdirs, filelock (all match only "application") geospatial data -> pandas, platformdirs, tzdata (all match only "data") discord bot -> python-telegram-bot first (matches only "bot") ``` **Experiment 1:** multiply the whole score by how much of the query's *rare* vocabulary a package covers. The junk vanished, but so did the good partial matches (Pillow, pytest). Total passes fell from 65 to 61. Rejected. **Experiment 2:** apply that factor to the **popularity term only**. A package keeps its relevance score, but it only earns popularity credit in proportion to the informative words it matches. ``` exponent tuned set held-out set junk entries in four probe queries -------- ---------- ------------ ---------------------------------- 0.0 40 (0.892) 25 (0.715) 10 0.25 39 (0.869) 25 (0.737) 5 0.5 39 (0.840) 25 (0.764) 2 1.0 38 (0.843) 25 (0.759) 1 ``` Better on the held-out set. But the held-out set had now influenced a decision, so it was no longer held out. We wrote a **third** set of 25 fresh queries to validate: ``` php Validation set (25 queries): MRR 0.755 -> 0.858 ldap authentication: google-auth, google-auth-oauthlib, ... -> django-auth-ldap, python-ldap, ldap3 currency conversion: pillow, pymupdf, forex-python -> forex-python, currency-symbols, ... bloom filter: soupsieve, eth-bloom, Markdown -> eth-bloom, bloom-filter2, pybloom-live ``` Adopted. The one query it lost, `graph algorithms`, exposed a flaw in our own synonym list: "graph" had been grouped with "chart" and "plot", so matplotlib beat networkx. We removed it. **One more rejected idea, because rejections are results too.** pytest dropped out of `unit testing` (its keywords say "unittest", one word). Adding `unit <-> unittest` as a synonym fixed it and immediately polluted `unit conversion` with testing tools. The "general" fix, treating every adjacent pair of query words as a possible compound, made things much worse: 84 of 95 instead of 89, because rare accidental compounds like "machinelearning" inflate the top relevance score and push good packages below the gate. What worked was a four-entry list of real compounds (`unit test`, `time zone`, `data frame`, `web socket`) that applies only when both words are in the query. **Final numbers.** All 95 queries are now the permanent regression suite. 90 pass. Of the 55 that were never used for tuning, 49 passed when first run (89%). The README reports the 89% and explains why the 95% is flattering. The rebuilt offline fixture (2,743 real packages) reproduces the full-snapshot result exactly: same 90, same five misses. A second independent review used a different lens: *check every claim in the docs against the code, be a first-time user, think about supply chain.* It found: `refresh --limit 5` (without `--limit` and downloaded the full snapshot. Now a usage error.`status --json` ran a search for the word "status". Now a usage error.`id`, `relevance`), used `download_count` where the JSON output said `downloads_30d`, did not validate `top_n`, and raised an exception it did not export. All settled `trusted_schema = OFF`. **And the reviewer made a mistake of its own.** Told in plain words never to touch the real cache, it ran `refresh --limit 5` expecting a usage error, without isolating the cache directory. The command ran and re-downloaded the snapshot over the real one. The content was identical and nothing was lost, and it reported the deviation itself, first, at the top of its report. But it is Mistake 1 again, through a different door. It also found a real bug in the process (`--limit` being ignored), which is a nice illustration that accidents are data. **Lesson:** an instruction is not a sandbox. If something must not be touched, make it unreachable (an environment variable set for the whole process, a read-only mount), do not just ask nicely. A workflow you edited and did not run is an untested program. Each time we changed the refresh workflow we triggered it and checked the result, four times in total, including the path that only runs the *second* time (release already exists, asset replaced in place, notes updated after the upload). The publish workflow refuses a tag that is not on the main branch, refuses a tag that does not match the package version, and refuses to publish if the snapshot that version downloads is not already on the release page, so a new install can never hit a 404. Finally, the whole first-run journey from a wheel installed into a clean environment: search with no snapshot (clear message), contradictory options (usage error), `refresh` (real download), `status`, searches, JSON piped to `head`, the Python API, the typing marker, and proof that a search does not import the HTTP stack. ``` Tests 135, coverage 97%, ruff and mypy clean CI 15 jobs: Linux, macOS, Windows x Python 3.10 to 3.14 Ranking 90 of 95 golden queries; 89% on queries never used for tuning Snapshot 14,999 packages, 10.8 MB download, rebuilt monthly behind two gates Search ~0.15 s per invocation, offline Source distribution 18 KB (was 780 KB before we excluded the test corpus) ``` Known limits, stated in the README rather than discovered by users: matching is lexical, so numpy does not appear for "linear algebra"; about one search in ten will not show a package you would call right; a real PyPI rate limit has only ever been simulated; GitHub pauses scheduled workflows after 60 days without commits, and the 45-day staleness warning is the safety net for that. **On testing** **On evaluation** **On robustness** `--limit 0`, an empty crawl, an empty survivor list. **On process** `__pycache__`, in our case). ``` Decision Alternatives considered Why ---------------------------------------- --------------------------------- ------------------------------------------ Two keyless public data sources BigQuery no account or key for users or maintainers Top 15,000 packages 200 / 2,000 / 8,000 matches the upstream dataset; covers the long tail down to ~68k downloads a month Central monthly snapshot + download every user crawls 15,000 requests per user does not scale Fixed release tag "snapshot-latest" releases/latest, dated tags code releases must not move the URL; re-runs in a month must not fail SQLite FTS5 (porter, contentless) pure-Python BM25; embeddings stemming and a shipped index with zero new dependencies; embeddings break "lightweight" Core fields and README scored apart one weighted index BM25 term-frequency saturation makes column weights nearly useless against README noise Gate, then rank linear blend short-document bias swamped popularity Popularity credit x informative coverage hard rarity gate; whole-score only variant that removed junk without penalty losing the good partial matches Curated compounds (4 entries) synonym; compound every pair synonym leaked; general rule scored 84/95 Precision over pandas-for-"dataframes" lower the gate inseparable from boto3-for-"unit testing" No silent crawl fallback automatic fallback the user must opt in to 15,000 requests Trusted Publishing on a version tag manual twine upload with a token no stored or pasted secrets Lazy import of the HTTP stack eager import measured: 0.30 s -> 0.15 s per search No FTS optimize / VACUUM add it measured: 5% smaller, no speed-up ``` If you take one thing from this log: most of the bugs above were invisible to the checks we already had. They were found by asking a different kind of question each time ("what happens on Windows?", "what if this is zero?", "what does it score on queries it has never seen?", "what would someone who wants me to be wrong find?"). Asking the same question twice, more carefully, would have found none of them.