From a Test-Suite Trace to a Search Signal: the Bootstrap Pipeline Story

A developer built a four-step "bootstrap pipeline" that indexes unfamiliar codebases by linking tests to the exact functions they execute, using a custom sys.settrace plugin to trace 1,727 tests. Measurement showed name-based linking failed entirely (0 of 109 tests named their target function) and that a single test executes 10.1 source functions on average, so the Tarantula heuristic was rejected as a primary-target selector in favor of complete dynamic traces. The tracer generalized to external projects including gemma_agent, linking 97.3% of 2,805 passing tests at a +17.4% one-time execution overhead.

The problem to which everything traces back: RAG search across an unfamiliar codebase sees the code, but misses the intent. Embeddings find a chunk by keywords, PropertyGraph finds a symbol by name, but to the question "where is the real business logic here, and what can I safely throw away?" no model answers — because the answer simply doesn't exist in static code representation. Hence the bootstrap pipeline was born for indexing a new project across 4 steps: @mcp app.tool as the outer system boundary; Step 3 was the main battleground. The argument wasn't "are tests needed" , but how to link a test to the exact function it actually executes . Steps 1 and 2 relied on clean static analysis, whereas step 3 lacked an obvious static answer. Initial intuition: "A test has a name, a function has a name, let's link by name." Measurement killed it instantly: 0 out of 109 tests in the evaluation sample named the function they actually executed. Import-only file-level match gave 77.9% — but file level is coarse noise a single file contains dozens of functions . We executed the entire suite 1,727 tests using a custom sys.settrace plugin: dynamic trace total tests traced: 1727 dynamic trace tests executing =1 src function: 1551 89.8% dynamic trace unique src functions executed: 1212 dynamic trace avg src functions per linked test: 10.1 median 6, range 1-118 tests with exact-name target hit in dynamic set: 47 2.7% A/B same session: 174.8s vs 198.6s - overhead +13.6% Takeaway: Dynamic tracing is the only deterministic linker, at the cost of a +13.6% one-time execution overhead. And an uncomfortable truth surfaced immediately: a single test executes 10.1 functions on average. "1 test = 1 function" was a naive myth. Thus, edges required a ranker : which of the 10 is the primary target? We tested the Tarantula heuristic "a function called infrequently by many tests is the primary target" . Hypothesis: ≥60-70% of tests have an unambiguous candidate at rank≤3. Reality: safe mkdir / get data root autouse fixtures, 234 callers each and project hash 223 callers . Verdict: Tarantula is unsuitable for selecting the main target, but works well as a confidence annotation for ~16% of tests. TESTS edges are thus built from the complete trace without truncation — they are correct by construction and do not require lossy ranking. Could coverage run Python 3.14, sys.monitoring run faster than our custom sys.settrace plugin? Hypothesis: Overhead <5%. Measurement: php baseline: 184.88s | coverage: 221.78s - overhead +19.96% target <5% REFUTED coverage.py was ~1.5x slower than our lightweight plugin. sys.monitoring remains a validation oracle for spot-checks, while sys.settrace stays as the main execution driver. We evaluated a full static score AST L1 calls / L2 name tokens / L3 imports against dynamic trace as ground truth. | Signal | Hit | Recall | Precision | Mean Candidates | |---|---|---|---|---| | L1 direct calls from test body | 88.4% | 30.3% | 68.0% | 2.9 | | L2 name tokens | 17.7% | 3.8% | 12.1% | — | | L3 file imports | 91.6% | 72.0% | 21.8% | 41.4 | | Union L1 + L2 + L3 | 90.4% | 70.0% | 20.6% | — | The initial assumption recall ≤ 30% was wrong: union recall reached 70% . Static signals were stronger than anticipated, but as a precise anchor L1 is narrow precision 68%, 2.9 candidates , and as a wide net L3 is noisy 41.4 candidates . Verdict: Dynamic trace remains the edge driver; static analysis acts as a companion layer and candidate supplier for mock tests which execute 0 real target functions dynamically . Does this generalize beyond our own repo? We ran the tracer against external projects: | Project | Language | Tests | Linked % | Overhead | |---|---|---|---|---| | gemma agent | Python | 2882 2874 pass | 97.3% 2805 | +17.4% 71.4s vs 60.8s | | commit- | Python | 27 | 100% | N/A | | codebase-memory-mcp | Go | 27 test funcs | — | go test : 51.0% pkg / 22.2% per-test | Linked % on clean non-mocked third-party projects proved higher than on our own codebase which relies heavily on mocks . The limitation is honest: dynamic execution is currently Python-only ; Go and TS require per-test tooling e.g., go test -coverprofile across N executions . PropertyGraph itself is polyglot, but the edge builder remains Python-first. Everything prior built test ──TESTS── function edges inside PropertyGraph without leveraging them during search. E17 closed the loop: SymbolIndexAdapter.get tests for symbol incoming TESTS edges + Searcher. append tests signal : appends up to 3 tests per function capped at min len, 6 per query , graph score = 0.4 vs 1.0 for definitions, sentinel chunk index = - 20 000 000 + line avoiding collisions with code chunks in RRF ranking. MSCODEBASE TESTS SIGNAL , A/B evaluation on a live PropertyGraph 7 target functions, true answers from trace : hit@1: off=7/7, on=7/7 | hit@3: 7/7 | MRR function : off=1.000, on=1.000 TESTS-signal: 6/7 queries received relevant covering tests in the response graph stage avg dt: off=3.43ms, on=3.27ms within noise floor RETRACTION: 0 broken links all files verified on disk The core invariant holds — function definitions are never displaced by test results MRR = 1.0 in both arms : tests follow strictly as secondary context. To validate beyond the narrow 7-query panel, we ran a wide panel of 35 identifier queries functions with the most TESTS edges : hit@1: off=33/35 94.3% , on=33/35 94.3% hit@3: off=34/35 97.1% , on=34/35 97.1% MRR function : off=0.957, on=0.957 TESTS-signal: 34/35 queries received new covering tests 97.1% graph stage avg dt: off=6.52ms, on=7.53ms overhead +15.3% Critical finding: TESTS-signal does not improve hit@1 off=on . It only adds context tests to already-found results: 97.1% of queries received new covering tests. This means TESTS-signal is context for LLM , not a search improvement. If LLM doesn't use tests, the signal is useless. TESTS-signal works only for Python : For non-Python projects, TESTS-signal does not work at all . Dynamic trace pytest + sys.settrace collects edges only for Python. For Go/TS, a separate connector is needed go test -coverprofile, Jest coverage , but this is not done . We tested TESTS-signal against 5 attack vectors: Conclusion: TESTS-signal is resilient to concurrency, boundaries, abuse, TOCTOU, and dependency failures. A transparent list of risks and open validation items. Full numbers for the A/B and red-team results are in the E17 section above — this list stays to the takeaways and the fixes. Evaluation scope. While expanded to a 35-query panel, evaluation is still performed on a single primary codebase without deep reranker interaction. A/B did not improve hit@1. See "Wide Panel" above — TESTS-signal does not help find the function . It only adds context to already-found results. → Conclusion: TESTS-signal is context for LLM , not a search improvement. → Risk: if LLM doesn't use tests, the signal is useless. → Fix: verify on real LLM pipeline not in this experiment . Overhead +15.3% for wide panel 6.52ms → 7.53ms avg graph stage time . For bootstrap one-time run this is acceptable. For prod search — may be critical with many queries. → Risk: at 1000 queries/sec, overhead may be noticeable. → Fix: cache TESTS-signal not done . Language limitation. Dynamic trace is currently Python-only ~34% function coverage in Python, 0% in JS/TS/Go . graph score = 0.4 is an empirical constant. Chosen to stay strictly below function definitions, but unverified against BM25/reranker weight interactions. → Verify on full pipeline; constant may become a parameter. Pointer :0 . Test nodes from dynamic trace lack line numbers line=0 . Indexers must resolve test decorator line positions before enabling in prod. Hub-function noise. safe mkdir linked to 234 tests yields low-signal noise. The cap of 3 tests prevents payload flooding but doesn't solve irrelevance. Dependence on a green test suite. On broken test suites, trace degrades failing tests = missing edges . Bootstrap applies to stable branches only. Mock tests are blind. 10.2% of tests execute 0 src functions; static companions cover 88 of 176, but not all. Graph reindex drift. Rebuilding the PropertyGraph may alter node ordering slightly. CI scale limit. Running trace on 100k+ test suites may breach CI execution windows. Clean-state status. Verification was executed locally; clean CI state verification requires a PR merge. A huge thank you to everyone who engages with these posts in the comments. Your feedback, real-world observations, counter-examples, and benchmark numbers directly shape these experiments. This kind of open technical critique is what keeps engineering honest. Every experiment, bug, failure, and idea here is mine — I earned them the hard way, in production, in public. AI worked as my editor: it helped me structure thoughts and polish my English. It did not invent the facts, because it has none of its own. No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am. Disclaimer & Status: draft source-material for the article . This is not a "feature advertisement", but an honest engineering story: figures are reproducible, weak points are named, and unaddressed risks are listed in the "What Could Go Wrong" section.