A weak point in AI-assisted coding is not always broken syntax or a failing test suite. The original concern behind this project was about tests too: if the AI-assisted workflow helps with testing, is that set of tests enough?
If the testing pattern is biased, a bug may sit where the tests do not look. The early idea was adversarial and game-theoretic, even equilibrium-like: treat bugs or failure conditions as if they were trying to stay hidden, then ask what testing or investigation policy would still hold up.
That is not what v0.1.0
proves. The current repository is not a formal game-theoretic debugger, and it does not claim real-world debugging accuracy.
That is the frame of bug-cause-inference-game, a small Python prototype for cost-aware bug-cause investigation. The version discussed here is tag
v0.1.0
, which points to commit 9e30c93f246602d840c875e975c362e6ab1e7747
.This version is a preparation step. P1a asks how a cost-aware policy chooses the next investigation action on synthetic cases. P1b compares metadata-derived evidence with execution-grounded evidence in a small injected-bug scaffold. P1c looks beyond averages to worst-case and difficult buckets.
The conclusion is narrow: the prototype shows the design value and limits of cost-aware investigation and worst-case reporting, not a finished debugger.
The repository is careful about what it does not claim. It is not a production fault-localization engine, an automated repair tool, an LLM debugging benchmark, a fuzzing framework, or a formal game-theoretic debugger.
That boundary is part of the design. The project is closest to probabilistic debugging, Bayesian fault diagnosis, and active diagnosis: rank likely causes from observations, then choose the next observation under cost.
For AI-assisted coding workflows, that modest scope is the point. If generated code can look plausible while missing the intended behavior, the next layer of tooling should help decide what evidence to collect, not just produce another confident explanation.
P1a is the smallest slice. It works on synthetic observed-bug cases, updates posterior probabilities over coarse cause categories, compares investigation policies, and emits a DecisionReport
whose main output is recommended_next_action
.
The synthetic dataset contains 50 cases, built from 5 cause categories and a fixed seed 20260627
.
The 5 cause categories are:
boundary_condition
missing_null_handling
configuration_environment
race_order_dependence
specification_mismatch
Each case has 8 possible investigation actions:
inspect_error_log
run_boundary_tests
compare_environment
inspect_recent_diff
run_reproduction_matrix
add_instrumentation
check_spec_acceptance
run_concurrency_stress
That action list is the first useful abstraction. Instead of asking an agent to "debug this," the prototype asks it to choose among named actions with explicit costs.
The implemented P1a policies include random
, fixed_checklist
, posterior_greedy
, cheapest_first
, information_gain
, information_gain_per_cost
, and static_posterior
. The main policy is information_gain_per_cost
.
The stopping settings are also explicit: top probability threshold 0.75
, margin threshold 0.15
, budget limit 10
, max steps 5
, minimum expected information gain per cost 0.03
, and failure cost 12
.
That makes the prototype less magical. It cannot hide behind "the model decided." The design keeps the current belief state, the available actions, the action costs, and the next recommended action explicit.
The current P1a dataset is not hard. Before any investigation action, the generated cases already have initial top-1 accuracy of 70%
and initial top-2 accuracy of 100%
, with 15 initially-wrong top-1 cases out of 50.
Even with that caveat, the policy comparison is useful. On the synthetic setup, information_gain_per_cost
has mean cost_to_true_cause_top1
of 1.12
, while fixed_checklist
has 1.56
. The recorded fixed-checklist cost reduction is about 28%
, and the primary success rate within budget is 94%
.
The same result also exposes a warning sign. The primary wrong-stop rate is about 13%
, where wrong-stop means stopping on a high-confidence but incorrect cause hypothesis.
The initially-wrong subset makes the tradeoff sharper. On those cases, information_gain_per_cost
has lower mean cost than fixed_checklist
(3.73
versus 5.2
), but fixed_checklist
has the higher success rate (0.87
versus 0.8
).
So the safe reading is narrow: cost-aware active investigation reduced average investigation cost in this synthetic setup, but it did not prove real-world debugging accuracy.
That is already a useful habit for AI-assisted coding tools. A good report should show where the policy saves cost, where it stops too early, and where a boring checklist still wins.
P1b adds a small injected-bug checkout/pricing benchmark scaffold. It evaluates budget-aware policies on 20 buggy variants and 5 clean variants.
The P1b evaluator looks at failure discovery, function-level location ranking, coarse cause inference, and fix-intent prediction. It still does not generate patches.
P1b uses two observation modes. metadata_synth
is the frozen Phase A/B baseline that synthesizes evidence from variant metadata. execution_grounded
builds observations from checkout test results, exceptions, traced checkout functions, coverage-spectrum suspicion, and Phase C real-diff artifacts.
The P1b action set is larger than P1a's and closer to a debugging workflow. It includes test actions such as run_smoke_tests
, run_boundary_tests
, run_null_missing_tests
, run_config_matrix_tests
, and run_state_sequence_tests
, plus inspection actions such as inspect_traceback
, inspect_coverage_spectrum
, inspect_recent_diff
, and inspect_spec_clause
.
The main P1b policy is expected_utility_per_cost
.
This is still a small scaffold, but it creates a useful pressure test: does a policy that looks good with metadata-derived evidence still look good when the observations are tied to execution?
If this article had to be reduced to one lesson, it would be the P1b comparison. The metadata_synth
baseline made the primary policy look better than execution_grounded
evidence on several metrics.
For expected_utility_per_cost
, the primary-policy comparison reports:
| Metric | metadata_synth |
execution_grounded |
|---|---|---|
| bug discovery within budget | 0.55 | 0.40 |
| cause top-1 accuracy | 0.80 | 0.55 |
| fix-intent top-1 accuracy | 0.75 | 0.40 |
| mean investigation cost | 2.80 | 4.64 |
metadata_synth
still has a role as a baseline. The point is that its optimism gap is visible.
For AI-assisted coding, this is the more general warning. Metadata-derived evidence can make a policy look better than it looks with execution-grounded evidence. Execution-grounded evidence is still small here, but the gap is visible.
In v0.1.0
, P1c is an analysis-only report over the existing 25-variant P1b scaffold.
The first P1c goal is to ask whether existing investigation policies remain useful when the evidence is deliberately difficult, ambiguous, expensive, or misleading inside a small reproducible benchmark.
The planning notes name several candidate adversarial models: worst-case variant selection, evidence-ambiguity buckets, observation-cost stress, and observation dropout or delay.
The implemented label buckets are boundary_precision
, missing_optional_input
, config_normalization
, state_sequence
, spec_semantics
, and clean_false_positive
.
The important design choice is that P1c does not collapse the result into one score. Discovery, first-failure cost, localization, cause, fix-intent, false-positive, and mean-cost metrics can point to different weaknesses.
That choice fits the original concern better than an average-only report. If a bug behaves like a difficult opponent, the average case is often the least interesting case.
For the current primary P1b/P1c policy, expected_utility_per_cost
, the aggregate reading is a useful baseline: discovery 0.40
, cause top-1 0.55
, fix-intent top-1 0.40
, and mean investigation cost 4.64
.
Those averages need the bucket view next to them.
| Bucket | What P1c exposed |
|---|---|
state_sequence |
|
The primary brittle bucket: discovery, location top-3, cause top-1, and fix-intent top-1 are all 0.0 ; first-failure cost is 14 . |
|
config_normalization |
|
The next broad weak bucket: those four metrics are all 0.25 ; first-failure cost is 10.75 . |
|
missing_optional_input |
|
Successful but expensive: the four success metrics are 1.0 and first-failure cost is 1 , but mean investigation cost is 9.75 . |
The current clean variants do not trigger false positives in the selected-bucket report or current stress profiles, but that does not show false-positive risk is solved outside this scaffold.
This is why the P1c layer matters. It turns "the average looks acceptable" into "which class of case breaks the policy, and how?"
The word "game" should be handled carefully here. The current repository does not claim a formal player model, payoff model, minimax-optimal policy, or game-theoretic guarantee.
The game-like idea comes from the original motivation: if bugs may hide where tests do not look, treat hard bugs or failure conditions as if they are selected by an opponent that wants to stay hidden, then ask which investigation policy still performs under that selection pressure.
In v0.1.0
, that idea is still bounded. P1c selects worst cases and stress conditions inside a small injected benchmark; it does not generate new bugs, repair code, or claim real-world debugging accuracy.
That boundary keeps the current result readable. Formal game-theoretic claims stay outside this version until the project has an explicit player model, action space, payoff, and evaluation result.
For an AI-assisted coding workflow, the design lesson is deliberately small: That last point is easy to skip. It is also the part that makes the experiment reusable. A small benchmark can still teach something if it is honest about what it cannot show.
Source version: guriguri215-lang/bug-cause-inference-game at v0.1.0. Disclosure: this article draft was prepared with AI assistance, and the project claims and numbers were checked against the linked repository version and DEV's AI-assisted article guidance.