The problem to which everything traces back: RAG search across an unfamiliar codebase sees the code, but misses the intent. Embeddings find a chunk by keywords, PropertyGraph finds a symbol by name, but to the question "where is the real business logic here, and what can I safely throw away?" no model answers — because the answer simply doesn't exist in static code representation.
Hence the bootstrap pipeline was born for indexing a new project across 4 steps:
@mcp_app.tool) as the outer system boundary;
Step 3 was the main battleground. The argument wasn't "are tests needed", but how to link a test to the exact function it actually executes. Steps 1 and 2 relied on clean static analysis, whereas step 3 lacked an obvious static answer.
Initial intuition: "A test has a name, a function has a name, let's link by name."
Measurement killed it instantly: 0 out of 109 tests in the evaluation sample named the function they actually executed. Import-only (file-level match) gave 77.9% — but file level is coarse noise (a single file contains dozens of functions).
We executed the entire suite (1,727 tests) using a custom sys.settrace plugin:
[dynamic_trace] total tests traced: 1727
[dynamic_trace] tests executing >=1 src function: 1551 (89.8%)
[dynamic_trace] unique src functions executed: 1212
[dynamic_trace] avg src functions per linked test: 10.1 (median 6, range 1-118)
tests with exact-name target hit in dynamic set: 47 (2.7%)
A/B same session: 174.8s vs 198.6s -> overhead +13.6%
Takeaway: Dynamic tracing is the only deterministic linker, at the cost of a +13.6% one-time execution overhead. And an uncomfortable truth surfaced immediately: a single test executes 10.1 functions on average. "1 test = 1 function" was a naive myth. Thus, edges required a ranker: which of the 10 is the primary target?
We tested the Tarantula heuristic ("a function called infrequently by many tests is the primary target").
Hypothesis: ≥60-70% of tests have an unambiguous candidate at rank≤3.
Reality:
safe_mkdir / get_data_root (autouse fixtures, 234 callers each) and project_hash (223 callers).
Verdict: Tarantula is unsuitable for selecting the main target, but works well as a confidence annotation for ~16% of tests. TESTS edges are thus built from the complete trace without truncation — they are correct by construction and do not require lossy ranking.
Could coverage run (Python 3.14, sys.monitoring) run faster than our custom sys.settrace plugin?
Hypothesis: Overhead <5%.
Measurement:
baseline: 184.88s | coverage: 221.78s -> overhead +19.96% (target <5% REFUTED)
coverage.py was ~1.5x slower than our lightweight plugin. sys.monitoring remains a validation oracle for spot-checks, while sys.settrace stays as the main execution driver.
We evaluated a full static score (AST L1 calls / L2 name tokens / L3 imports) against dynamic trace as ground truth.
| Signal | Hit | Recall | Precision | Mean Candidates |
|---|---|---|---|---|
| L1 (direct calls from test body) | 88.4% | 30.3% | 68.0% | 2.9 |
| L2 (name tokens) | 17.7% | 3.8% | 12.1% | — |
| L3 (file imports) | 91.6% | 72.0% | 21.8% | 41.4 |
| Union (L1 + L2 + L3) | 90.4% | 70.0% | 20.6% | — |
The initial assumption (recall ≤ 30%) was wrong: union recall reached 70%. Static signals were stronger than anticipated, but as a precise anchor L1 is narrow (precision 68%, 2.9 candidates), and as a wide net L3 is noisy (41.4 candidates).
Verdict: Dynamic trace remains the edge driver; static analysis acts as a companion layer and candidate supplier for mock tests (which execute 0 real target functions dynamically).
Does this generalize beyond our own repo? We ran the tracer against external projects:
| Project | Language | Tests | Linked % | Overhead |
|---|---|---|---|---|
| gemma_agent | Python | 2882 (2874 pass) | 97.3% (2805) | +17.4% (71.4s vs 60.8s) |
| commit- | Python | 27 | 100% | N/A |
| codebase-memory-mcp | Go | 27 test funcs | — | go test : 51.0% pkg /22.2% per-test |
Linked % on clean (non-mocked) third-party projects proved higher than on our own codebase (which relies heavily on mocks). The limitation is honest: dynamic execution is currently Python-only; Go and TS require per-test tooling (e.g., go test -coverprofile across N executions). PropertyGraph itself is polyglot, but the edge builder remains Python-first.
Everything prior built test ──TESTS──> function edges inside PropertyGraph without leveraging them during search. E17 closed the loop:
SymbolIndexAdapter.get_tests_for_symbol() (incoming TESTS edges) + Searcher._append_tests_signal(): appends up to 3 tests per function (capped at min(len, 6) per query), graph_score = 0.4 vs 1.0 for definitions, sentinel chunk_index = -(20_000_000 + line) avoiding collisions with code chunks in RRF ranking.MSCODEBASE_TESTS_SIGNAL, A/B evaluation on a live PropertyGraph (7 target functions, true answers from trace):
hit@1: off=7/7, on=7/7 | hit@3: 7/7 | MRR(function): off=1.000, on=1.000
TESTS-signal: 6/7 queries received relevant covering tests in the response
graph_stage avg dt: off=3.43ms, on=3.27ms (within noise floor)
RETRACTION: 0 broken links (all files verified on disk)
The core invariant holds — function definitions are never displaced by test results (MRR = 1.0 in both arms): tests follow strictly as secondary context.
To validate beyond the narrow 7-query panel, we ran a wide panel of 35 identifier queries (functions with the most TESTS edges):
hit@1: off=33/35 (94.3%), on=33/35 (94.3%)
hit@3: off=34/35 (97.1%), on=34/35 (97.1%)
MRR(function): off=0.957, on=0.957
TESTS-signal: 34/35 queries received new covering tests (97.1%)
graph_stage avg dt: off=6.52ms, on=7.53ms (overhead +15.3%)
Critical finding: TESTS-signal does not improve hit@1 (off=on). It only adds context (tests) to already-found results: 97.1% of queries received new covering tests. This means TESTS-signal is context for LLM, not a search improvement. If LLM doesn't use tests, the signal is useless.
TESTS-signal works only for Python:
For non-Python projects, TESTS-signal does not work at all. Dynamic trace (pytest + sys.settrace) collects edges only for Python. For Go/TS, a separate connector is needed (go test -coverprofile, Jest coverage), but this is not done.
We tested TESTS-signal against 5 attack vectors:
Conclusion: TESTS-signal is resilient to concurrency, boundaries, abuse, TOCTOU, and dependency failures.
A transparent list of risks and open validation items. (Full numbers for the A/B and red-team results are in the E17 section above — this list stays to the takeaways and the fixes.)
Evaluation scope. While expanded to a 35-query panel, evaluation is still performed on a single primary codebase without deep reranker interaction.
A/B did not improve hit@1. See "Wide Panel" above — TESTS-signal does not help find the function. It only adds context to already-found results.
→ Conclusion: TESTS-signal is context for LLM, not a search improvement.
→ Risk: if LLM doesn't use tests, the signal is useless.
→ Fix: verify on real LLM pipeline (not in this experiment).
Overhead +15.3% for wide panel (6.52ms → 7.53ms avg graph_stage time). For bootstrap (one-time run) this is acceptable. For prod search — may be critical with many queries.
→ Risk: at 1000 queries/sec, overhead may be noticeable.
→ Fix: cache TESTS-signal (not done).
Language limitation. Dynamic trace is currently Python-only (~34% function coverage in Python, 0% in JS/TS/Go).
graph_score = 0.4 is an empirical constant. Chosen to stay strictly below function definitions, but unverified against BM25/reranker weight interactions.
→ Verify on full pipeline; constant may become a parameter.
Pointer :0. Test nodes from dynamic trace lack line numbers ( line=0). Indexers must resolve test decorator line positions before enabling in prod.
Hub-function noise. safe_mkdir linked to 234 tests yields low-signal noise. The cap of 3 tests prevents payload flooding but doesn't solve irrelevance.
Dependence on a green test suite. On broken test suites, trace degrades (failing tests = missing edges). Bootstrap applies to stable branches only.
Mock tests are blind. 10.2% of tests execute 0 src functions; static companions cover 88 of 176, but not all.
Graph reindex drift. Rebuilding the PropertyGraph may alter node ordering slightly.
CI scale limit. Running trace on 100k+ test suites may breach CI execution windows.
Clean-state status. Verification was executed locally; clean CI state verification requires a PR merge.
A huge thank you to everyone who engages with these posts in the comments. Your feedback, real-world observations, counter-examples, and benchmark numbers directly shape these experiments. This kind of open technical critique is what keeps engineering honest.
Every experiment, bug, failure, and idea here is mine — I earned them the hard way, in production, in public. AI worked as my editor: it helped me structure thoughts and polish my English. It did not invent the facts, because it has none of its own.
No AI detectors were consulted in the making of this disclosure. They have enough trouble agreeing on what I am.
Disclaimer & Status: draft (source-material for the article). This is not a "feature advertisement", but an honest engineering story: figures are reproducible, weak points are named, and unaddressed risks are listed in the "What Could Go Wrong" section.