A complete engineering log of building `findmypylibrary` with an AI pair-programmer (Claude Code)

A developer built findmypylibrary, a Python CLI tool that answers "I need to do X in Python, which package?" by ranking packages from a 15,000-row PyPI download dataset. The tool fetches package summaries and release dates asynchronously with bounded concurrency, caches results in SQLite, and supports bare queries without a search subcommand. The engineering log details bugs encountered, including a redirect that returned HTML instead of JSON and an ordering trap in the async crawl.

A complete engineering log of building findmypylibrary with an AI pair-programmer Claude Code : what broke, how we found out, what we tried, what we measured, and why each decision went the way it did. Nothing is left out, including the mistakes that were ours. findmypylibrary answers one question: "I need to do X in Python. Which package?" pip install findmypylibrary findmypylibrary refresh findmypylibrary "fuzzy string matching" 1. RapidFuzz score 0.90 rapid fuzzy string matching downloads/30d: 163,835,611 last release: 2026-08-30 2. pfzy score 0.81 Python port of the fzy fuzzy string matching algorithm downloads/30d: 27,024,226 last release: 2022-01-28 3. fuzzywuzzy score 0.73 Fuzzy string matching in python downloads/30d: 14,789,165 last release: 2020-02-13 Three constraints were fixed on day one and never changed, and they explain most of the decisions below: The project started as a 0.0.1 placeholder on PyPI: a name reservation with a README that described two commands that did not exist yet. This is the story of getting from there to something we were willing to release. The plan needed, for the most-used packages on PyPI: name, description, download count and last release date. The first obstacle: PyPI has no free bulk endpoint for "all packages with downloads". Download statistics live in a public BigQuery dataset, and BigQuery needs a Google Cloud account and key. That breaks constraint 3. What we did instead: two public sources, neither needing a key. https://pypi.org/pypi/<name /json , one request per package, for the summary and release date. A small first bug: the dataset's documented URL returned an HTML page, not JSON. It was a 301 redirect to a new domain that our first curl did not follow. We switched to the raw GitHub URL. Lesson filed for later: never assume a 200 , and follow redirects this came back to bite us in Step 12 . Decision: how many packages? The question was 200, 2,000 or 8,000. The answer from the product owner was "15,000, or whatever covers all actively downloaded libraries". When we opened the dataset it contained exactly 15,000 rows, down to packages with about 68,000 downloads a month. So "top 15,000 by 30-day downloads" became the definition of "active", for free. The cost of that decision: 15,000 packages means 15,000 HTTP requests per snapshot. Sequentially that is over an hour. With httpx.AsyncClient and a semaphore of 25 concurrent requests it takes a few minutes. Hold that thought; it becomes the scaling problem in Step 5. Architecture, deliberately boring: php fetch.py two sources - list of package dicts async, bounded concurrency, retries cache.py sqlite file in ~/.cache/findmypylibrary/ rank.py query - ranked packages cli.py click commands: refresh, search Three details worth recording. 1. A bare query had to work. The README promised findmypylibrary "parse messy pdfs" , with no search subcommand. Click groups reject unknown command names, so we subclassed the group: python class DefaultGroup click.Group : def resolve command self, ctx, args : try: return super .resolve command ctx, args except click.UsageError: return super .resolve command ctx, "search", args This looked finished. It was not Steps 12 and 15 found three more cases . 2. An ordering trap in the async crawl. We used asyncio.as completed to drive a progress bar. It yields results in completion order, so you cannot zip them back onto the input list to attach download counts. We caught this at design time and made each task return its own complete record, download count included, instead of re-joining afterwards. 3. We did not hammer PyPI while developing. Every test of the pipeline during development used refresh --limit 40 . The full 15,000-package crawl was run only when we needed real data. Result of the first full run: 14,999 of 15,000 packages the missing one was a genuine 404 , a delisted package . The first ranking was textbook: a pure-Python BM25 over name + summary + keywords, then a weighted blend. score = 0.60 relevance + 0.25 popularity + 0.15 recency each min-max normalised It produced sensible results for the first few queries we tried. That was the problem: we only tried a few. openpyxl loses to a package nobody has heard of Query: read and write excel spreadsheets . 1. numbers-parser Read and write Apple Numbers spreadsheets 924,605 downloads 2. tifffile Read and write TIFF files 27,744,478 downloads 3. openpyxl A Python library to read/write Excel ... 339,316,525 downloads openpyxl is the right answer and it has 370 times the downloads of the winner. Diagnosis we looked at the numbers instead of guessing . openpyxl matched three of the four query words, so it was not a recall problem. Two things combined: numbers-parser 's whole summary is six words, three of which are in the query. BM25's length normalisation rewards that density. The fix: stop blending, start gating. Relevance became a gate keep anything within 50% of the best match , and the survivors were ranked mainly by popularity. This is how real package search engines work: text match for recall, authority for ordering. We also weighted matches in the package name above matches in the summary, and added a small synonym list excel/xlsx/spreadsheet and a dozen more . This was done test-first. The regression test asserts the outcome a user sees, not an internal number: python def test popular relevant package beats short keyword dense niche one : result = names "read and write excel spreadsheets", openpyxl, numbers parser, tifffile assert result 0 == "openpyxl" We rejected one idea here on principle: a local embedding model. It would have pulled in a deep-learning runtime and a model download for a tool whose selling point is being light. The product owner uses a strict audit methodology on a large TypeScript / Postgres / Playwright application, and asked for the same gates here: Applied literally, half of that is meaningless for a Python CLI with no browser, no database server and no TypeScript. Copying the letter of a process into a different stack is how you end up with ceremonies nobody can explain. So we translated the intent: Their gate What it became here ----------------------------- --------------------------------------------------- TDD pytest, regression test written before each fix = 90% coverage pytest-cov, enforced in CI 0 TypeScript / lint errors ruff lint + format and mypy, zero findings Playwright end-to-end subprocess tests that run the real installed binary Leftover / twin host inventory grep every importer of anything we change or remove "Ledger-green, product-red" assert the outcome users see, never a proxy for it That last line is the core lesson of their methodology: a test that checks "the file exists" or "the score changed" can be green while the product is broken. It turned out to apply to us more than once. To isolate test runs, a cleanup command removed ~/.cache/findmypylibrary before running pytest. That directory was not a test artefact. It was the real 15,000-package snapshot that had taken a full crawl to build. Fix: rebuild it, then make the mistake impossible rather than merely avoided. Every test now runs with XDG CACHE HOME pointed at a temporary directory, through one autouse fixture: python @pytest.fixture autouse=True def isolated cache dir tmp path, monkeypatch : monkeypatch.setenv "XDG CACHE HOME", str tmp path / "xdg-cache" Lesson: "be careful" is not a control. A fixture that every test gets automatically is. We broke this rule again, through a different door, in Step 15. With a working tool, the obvious flaw surfaced: refresh makes 15,000 requests to PyPI. One developer doing that monthly is fine. A thousand users doing it is not, and it would get everyone rate-limited. Decision: build the snapshot once, centrally, and let users download it. A scheduled GitHub Actions workflow builds the snapshot monthly and publishes it as a GitHub Release asset. findmypylibrary refresh downloads that file one request, a couple of seconds . refresh --build-locally keeps the full crawl for anyone who wants a snapshot as of right now. This is the same pattern the upstream dataset uses for itself, which is a good sign it is the right shape. A guardrail worth mentioning: the AI assistant's permission system refused to create a public GitHub repository on its own. It stopped, explained, and asked. Creating a public surface is a human decision. We approved it explicitly, and only then did it create the repo and push. We verified the loop end to end, not just the workflow: trigger it manually, confirm the release exists, then run refresh on a clean machine profile and confirm it downloads and answers queries. Asked directly, the honest answer was a table of what had not been verified: Gap Status after this step ---------------------------------- --------------------------------------------- No CI; gates only ran by hand CI on every push and pull request Only tested on macOS, Python 3.12 Matrix: Linux, macOS, Windows x 3 Pythons Reader during a refresh? Concurrency test added Real HTTP 429 from PyPI Still mock-only. We will not hammer a public service to force a rate limit. Said so plainly. Saying "this one we cannot test ethically" is part of being ready. Hiding it is not. We ran 15 everyday queries against the real snapshot. About half were good. Five failed in a way that would embarrass the tool on day one: Query Missing from the top results Why confirmed in the data ------------------------------ ----------------------------- --------------------------------------- dataframes pandas its metadata never says "dataframe" resize images Pillow summary says "Imaging", not "images" plot charts matplotlib summary says "plotting", not "plot" connect to postgres database psycopg2 pymssql came first "postgres" is not "PostgreSQL" unit testing pytest matched one word; the gate dropped it Two root causes. Matching was exact-token , with no stemming. And the text was tiny: summaries average 8 words , and 259 packages have none at all. A 13-entry synonym list cannot paper over that. The same review found seven robustness gaps. Each is a story of "it works on the happy path": refresh saved whatever it got. A network drop at 10% would replace 15,000 packages with 1,500, and in the workflow that would be published to everyone. releases/latest/... . The first time we published a We decided not to publish until the ranking and gaps 1 to 5 were fixed. A tool that cannot find pandas is not a release. The design decision. We needed stemming and more text per package. More text kills the pure-Python BM25: building an index over 15,000 documents of ~250 tokens on every query would take seconds. The index had to be built once and shipped. SQLite already has this. FTS5 is a full-text engine with a Porter stemmer and a BM25 ranking function, it is compiled into standard CPython on all three platforms the CI matrix later proved it , and it adds zero dependencies: CREATE VIRTUAL TABLE packages fts USING fts5 name, summary, keywords, topics, description, content='', tokenize='porter unicode61' ; content='' makes it contentless : README text is searchable but not stored, which keeps the file small. Result: 19.8 MB on disk, 10.9 MB gzipped, and queries in tens of milliseconds. More text, carefully. We added each package's topic classifiers and a cleaned excerpt of its README. "Cleaned" matters: in pandas' README the word "DataFrame" first appears about 3,000 characters in, behind badges, links and HTML. We strip images, links, URLs, tags and reST directives before indexing. The golden queries came first. Before touching a ranking constant we wrote 40 everyday queries, each with a list of acceptable right answers "resize images" - pillow, opencv-python, ... . Writing the expectations before seeing results is the only way to keep yourself honest. Baseline on the new index: 37 of 40. The discovery that shaped the design: README noise. For unit testing , the results were boto3 and tqdm . Why? boto3's README has a section that says "run the unit tests". tqdm's talks about iterations-per-second "units". We had given the description column a weight of 1 against 8 for the name, and it made almost no difference. The reason is a property of BM25: term frequency saturates , so once a term appears at all, the column weight barely moves the score. boto3's README-only match scored 0.73 of the best real match. Fix: score the core fields name, summary, keywords, topics and the description in two separate queries, and add the description score at a heavy discount. A parameter sweep showed the pass rate was flat at 39/40 across a wide band of settings, which is what you want to see: a plateau, not a knife-edge. A better metric. Pass/fail at top 5 hides ordering. We added mean reciprocal rank 1 for a right answer in first place, 1/2 for second, and so on . It showed that scaling popularity by "log-downloads divided by the maximum" barely separated a package with a million downloads from one with a billion the logs are 6 and 9 . Min-max scaling over the surviving candidates lifted MRR from 0.69 to 0.88. A test whose premise was wrong. We asserted that tqdm must not appear for "unit testing". It kept appearing. Instead of tuning until it vanished, we looked: tqdm's own PyPI classifiers include Education :: Testing . That is the package describing itself, not README noise. The test was wrong, so we fixed the test, and kept the assertion for boto3, which was the genuine case. The concurrency test searches running while a refresh rewrites the snapshot was passing. Its output contained a warning : the writer thread had died with sqlite3.OperationalError: disk I/O error . The test passed because it only collected errors from the reader thread. It was asserting a proxy. We reproduced it in isolation and asked SQLite for the extended error: SQLITE IOERR LOCK . The cause was opening the snapshot through a read-only URI file:...?mode=ro . On macOS, those connections made a concurrent writer's lock acquisition fail, three to five times per run. Plain connections never failed. Fix: open normally and make the connection read-only with PRAGMA query only = ON . And fix the test so it records writer failures too. A bug fix without the test fix would have left the same blind spot for the next bug. Each with the reason, because the reason is what transfers to other projects. releases/download/snapshot-latest/... , a tag that only the snapshot workflow writes to. Code releases can never change what that URL points at. snapshot-v2.sqlite.gz and in a meta table. Old installs keep downloading a layout they understand. A mismatch produces a clear "run refresh" message. --build-locally . The user decides whether to make 15,000 requests. status command, and a warning after 45 days. pandas still did not appear for dataframes polars did, so the golden query passed, but the review had named pandas . Could we fix it? pandas matches dataframes only through its README. boto3 matches unit testing only through its README. We swept the two relevant parameters and printed both positions side by side: gate readme weight pandas for "dataframes" boto3 for "unit testing" ----- ------------- ----------------------- ------------------------ 0.05 0.5 7th 6th 0.10 0.5 7th 6th 0.15 0.5 7th 6th 0.20 0.5 not shown not shown 0.25 0.5 not shown not shown They move together, every time. To a lexical ranker the two cases are the same case. You cannot have one without the other. Decision: precision wins. No boto3 for "unit testing", and pandas is found by the words its own metadata uses data analysis puts it first . We wrote this limit into the README with those exact examples, instead of tuning until a demo looked good. The same sweep gave a free improvement a README weight of 0.5 took the golden set to 40/40 with boto3 still excluded , so we took it. We also tried weighting the coverage gate by term rarity. It removed one junk result and dropped Pillow and pytest. Rejected, with the numbers in the commit message. Mistake 2. A commit was gated like this: pytest -q | tail -3 && git commit ... && git push pytest failed. The commit and push happened anyway, because the exit status of a pipeline is the exit status of its last command, and tail always succeeds. Red code reached the main branch. Fix: never put a pipe between a gate and the thing it guards. Capture output to a file and test the real exit code: fail=0 pytest -q out.txt 2 &1 || fail=1 ruff check -q . || fail=1 mypy src /dev/null || fail=1 if "$fail" -ne 0 ; then echo "GATES RED - NOT COMMITTING"; exit 1; fi Mistake 3 , found while investigating mistake 2. The "failures" were not real. To prove a regression test had teeth, we had temporarily changed a constant in rank.py from 0.5 to 1.0 , confirmed the test failed, and restored it. Both versions of the file had the same size and were written within the same second. Python validates cached bytecode by modification time and size, so it considered the stale .pyc valid and kept loading the 1.0 version. The source said one thing and the interpreter ran another. Fix: after any mutation check, delete pycache . CI, which starts from a clean checkout, confirmed the pushed commit was green all along. Two lessons: check exit codes, not output; and when results contradict the source code, suspect the cache. Asked "is anything left?", re-running our own checklist would only confirm what we already believed. So we started a second AI reviewer with no edit rights and a specific brief: here is what the author believes is handled; find what they missed. In parallel we attacked the real CLI with odd inputs. Between the two: ValueError: min iterable argument is empty . This sat behind 98% coverage. refresh --build-locally --limit 0 wiped the snapshot. 0 < 0.95 0 , which is false, and saved an empty list. 14,999 packages became 0, exit code 0. -n -1 printed 389 results 403 with an HTML body raised JSONDecodeError out of the whole run. It now costs one package. We also follow redirects now remember Step 1 . PRAGMA quick check before swapping a download in. résumé parser was tokenised as r , sum , parser . findmypylibrary status bar widget -n 2 parse pdf . And two findings about the tests , which mattered more than any single bug: "Traceback" not in output . Click's test runner keeps uncaught exceptions in result.exception and never writes them to output . Those assertions could not fail. That is exactly why the search crash survived. The test helper now fails on any uncaught exception: crashed = result.exception is not None and not isinstance result.exception, SystemExit assert not crashed, f"CLI crashed with {result.exception r}" Performance, measured rather than assumed. Every search took 0.30 seconds, and 0.14 of that was importing httpx , which search never uses. Importing the fetch module lazily halved the start-up time to 0.15 seconds. We also measured FTS5 optimize plus VACUUM : 5% smaller, no speed-up. Not worth a line of code, so we did not add it. Piping results to head closed the pipe early. A new, broad except OSError handler caught the resulting error and told the user to "check permissions and free disk space". We fixed it by letting BrokenPipeError through. Local tests passed. CI failed on all four Windows jobs. On Windows a closed pipe does not raise BrokenPipeError EPIPE . It raises OSError with EINVAL . The real problem was the design, not the missing case: a catch-all around the whole command could not tell "I cannot write the cache" from "nobody is reading my output". We removed it and did two precise things instead: php def run - None: try: main except OSError as exc: if exc.errno not in errno.EPIPE, errno.EINVAL : raise os.dup2 os.open os.devnull, os.O WRONLY , sys.stdout.fileno sys.exit 1 No amount of local care would have found this. A cross-platform CI matrix is not a formality. The golden set said 100%. But we had tuned on it. So we wrote 30 new queries, fixed their expected answers before running anything, and ran them once. Tuned set 40 queries : 40/40 in the top 5 MRR 0.89 Held-out set 30 queries : 25/30 in the top 5 MRR 0.72 83%, not 100%. That gap is the price of tuning on your test set, and it is the number users actually experience. The failures had one pattern. A package with a billion downloads that matched only the commonest word of the query floated to the top: php gui application - idna, platformdirs, filelock all match only "application" geospatial data - pandas, platformdirs, tzdata all match only "data" discord bot - python-telegram-bot first matches only "bot" Experiment 1: multiply the whole score by how much of the query's rare vocabulary a package covers. The junk vanished, but so did the good partial matches Pillow, pytest . Total passes fell from 65 to 61. Rejected. Experiment 2: apply that factor to the popularity term only . A package keeps its relevance score, but it only earns popularity credit in proportion to the informative words it matches. exponent tuned set held-out set junk entries in four probe queries -------- ---------- ------------ ---------------------------------- 0.0 40 0.892 25 0.715 10 0.25 39 0.869 25 0.737 5 0.5 39 0.840 25 0.764 2 1.0 38 0.843 25 0.759 1 Better on the held-out set. But the held-out set had now influenced a decision, so it was no longer held out. We wrote a third set of 25 fresh queries to validate: php Validation set 25 queries : MRR 0.755 - 0.858 ldap authentication: google-auth, google-auth-oauthlib, ... - django-auth-ldap, python-ldap, ldap3 currency conversion: pillow, pymupdf, forex-python - forex-python, currency-symbols, ... bloom filter: soupsieve, eth-bloom, Markdown - eth-bloom, bloom-filter2, pybloom-live Adopted. The one query it lost, graph algorithms , exposed a flaw in our own synonym list: "graph" had been grouped with "chart" and "plot", so matplotlib beat networkx. We removed it. One more rejected idea, because rejections are results too. pytest dropped out of unit testing its keywords say "unittest", one word . Adding unit <- unittest as a synonym fixed it and immediately polluted unit conversion with testing tools. The "general" fix, treating every adjacent pair of query words as a possible compound, made things much worse: 84 of 95 instead of 89, because rare accidental compounds like "machinelearning" inflate the top relevance score and push good packages below the gate. What worked was a four-entry list of real compounds unit test , time zone , data frame , web socket that applies only when both words are in the query. Final numbers. All 95 queries are now the permanent regression suite. 90 pass. Of the 55 that were never used for tuning, 49 passed when first run 89% . The README reports the 89% and explains why the 95% is flattering. The rebuilt offline fixture 2,743 real packages reproduces the full-snapshot result exactly: same 90, same five misses. A second independent review used a different lens: check every claim in the docs against the code, be a first-time user, think about supply chain. It found: refresh --limit 5 without --limit and downloaded the full snapshot. Now a usage error. status --json ran a search for the word "status". Now a usage error. id , relevance , used download count where the JSON output said downloads 30d , did not validate top n , and raised an exception it did not export. All settled trusted schema = OFF . And the reviewer made a mistake of its own. Told in plain words never to touch the real cache, it ran refresh --limit 5 expecting a usage error, without isolating the cache directory. The command ran and re-downloaded the snapshot over the real one. The content was identical and nothing was lost, and it reported the deviation itself, first, at the top of its report. But it is Mistake 1 again, through a different door. It also found a real bug in the process --limit being ignored , which is a nice illustration that accidents are data. Lesson: an instruction is not a sandbox. If something must not be touched, make it unreachable an environment variable set for the whole process, a read-only mount , do not just ask nicely. A workflow you edited and did not run is an untested program. Each time we changed the refresh workflow we triggered it and checked the result, four times in total, including the path that only runs the second time release already exists, asset replaced in place, notes updated after the upload . The publish workflow refuses a tag that is not on the main branch, refuses a tag that does not match the package version, and refuses to publish if the snapshot that version downloads is not already on the release page, so a new install can never hit a 404. Finally, the whole first-run journey from a wheel installed into a clean environment: search with no snapshot clear message , contradictory options usage error , refresh real download , status , searches, JSON piped to head , the Python API, the typing marker, and proof that a search does not import the HTTP stack. Tests 135, coverage 97%, ruff and mypy clean CI 15 jobs: Linux, macOS, Windows x Python 3.10 to 3.14 Ranking 90 of 95 golden queries; 89% on queries never used for tuning Snapshot 14,999 packages, 10.8 MB download, rebuilt monthly behind two gates Search ~0.15 s per invocation, offline Source distribution 18 KB was 780 KB before we excluded the test corpus Known limits, stated in the README rather than discovered by users: matching is lexical, so numpy does not appear for "linear algebra"; about one search in ten will not show a package you would call right; a real PyPI rate limit has only ever been simulated; GitHub pauses scheduled workflows after 60 days without commits, and the 45-day staleness warning is the safety net for that. On testing On evaluation On robustness --limit 0 , an empty crawl, an empty survivor list. On process pycache , in our case . Decision Alternatives considered Why ---------------------------------------- --------------------------------- ------------------------------------------ Two keyless public data sources BigQuery no account or key for users or maintainers Top 15,000 packages 200 / 2,000 / 8,000 matches the upstream dataset; covers the long tail down to ~68k downloads a month Central monthly snapshot + download every user crawls 15,000 requests per user does not scale Fixed release tag "snapshot-latest" releases/latest, dated tags code releases must not move the URL; re-runs in a month must not fail SQLite FTS5 porter, contentless pure-Python BM25; embeddings stemming and a shipped index with zero new dependencies; embeddings break "lightweight" Core fields and README scored apart one weighted index BM25 term-frequency saturation makes column weights nearly useless against README noise Gate, then rank linear blend short-document bias swamped popularity Popularity credit x informative coverage hard rarity gate; whole-score only variant that removed junk without penalty losing the good partial matches Curated compounds 4 entries synonym; compound every pair synonym leaked; general rule scored 84/95 Precision over pandas-for-"dataframes" lower the gate inseparable from boto3-for-"unit testing" No silent crawl fallback automatic fallback the user must opt in to 15,000 requests Trusted Publishing on a version tag manual twine upload with a token no stored or pasted secrets Lazy import of the HTTP stack eager import measured: 0.30 s - 0.15 s per search No FTS optimize / VACUUM add it measured: 5% smaller, no speed-up If you take one thing from this log: most of the bugs above were invisible to the checks we already had. They were found by asking a different kind of question each time "what happens on Windows?", "what if this is zero?", "what does it score on queries it has never seen?", "what would someone who wants me to be wrong find?" . Asking the same question twice, more carefully, would have found none of them.