{"slug": "i-ran-sol-6-vs-6-1-on-tasks-from-my-own-repo-it-s-cheaper-and-at-least-as-good", "title": "I ran Sol 6 vs. 6.1 on tasks from my own repo. It's cheaper and at least as good", "summary": "A developer's head-to-head test of Sol 6 versus Sol 6.1 on tasks from their own repository found the newer model cheaper and at least as capable. The comparison used the `stet runs repair-ai-coverage` command with `--model-key model-one`, and the accompanying code changes add `--parse-retries` handling for saved `code_review` prompts whose evaluator responses failed JSON/schema parsing, rewriting validation artifacts only after a successful repair. The patch also adds a `prepareCoverageRepairPrompts` step that requires the original task projection, matching artifact digests, persisted test commands, and complete repository context, leaving unavailable provenance unresolved.", "body_md": "…`stet runs repair-ai-coverage --out /path/to/h2h-output --model-key model-one --c`\n\n292292\n\n293293``Use `--parse-retries` when saved `code_review` prompts exist but evaluator``\n\n294294`responses failed JSON/schema parsing. The command rewrites validation artifacts`\n\n295−`only after a successful repair and regenerates summaries/task details.`\n\n295+`only after a successful repair and regenerates summaries/task details. Stored`\n\n296+`prompts retain their measurement mode and truncation. Missing-prompt repair`\n\n297+`requires the original task projection, matching artifact digests, persisted test`\n\n298+`commands, and complete repository context; unavailable provenance stays`\n\n299+`unresolved. Attempt counts and retry reasons distinguish recovery from an`\n\n300+`initial successful grade; assertions and unrecoverable runtime errors are not`\n\n301+`retried.`\n\n296302\n\n297303`Regrade additive custom or bundled graders from canonical task details:`\n\n298304\n\n…``const coverageValidationGolden = `{``\n\n843843 `},`\n\n844844 `\"equiv_rescue_policy\": \"on\",`\n\n845845 `\"equivalence\": {`\n\n846+ `\"attempt_count\": 1,`\n\n846847 `\"confidence\": 0.95,`\n\n847848 `\"evaluator_model\": \"judge-a\",`\n\n848849 `\"issue_type\": \"stylistic\",`\n\n…`func (p *repairAICoverageProducer) Produce(mctx *MutationContext) (*ProduceResul`\n\n274274 `var mu sync.Mutex`\n\n275275 `runWithConcurrency(jobs, min(len(jobs), coverageRepairWorkers), func(job repairJob) {`\n\n276276 `reporter.recordTaskStart(variant, job.TaskID, job.CurrentWork, time.Now().UTC())`\n\n277− `result, repairErr := validate.RepairCoverage(ctx, validate.CoverageRepairConfig{`\n\n277+ `repairCfg := validate.CoverageRepairConfig{`\n\n278278 `Existing:           job.Validation,`\n\n279279 `AI:                 ai,`\n\n280280 `AIModelID:          resolvedAIModelID,`\n\n281281 `RepairEquivalence:  repairEquivalence && needsRepairCoverage(job.Validation, TaskGraderEquivalence, resolvedAIModelID),`\n\n282282 `RepairCodeReview:   repairCodeReview && needsRepairCoverage(job.Validation, TaskGraderCodeReview, resolvedAIModelID),`\n\n283283 `ReviewParseRetries: cfg.ParseRetries,`\n\n284− `})`\n\n284+ `}`\n\n285+ `cleanup, reconstructionIssues := prepareCoverageRepairPrompts(&repairCfg, manifest, NewTaskArtifacts(outputRoot, variant, job.TaskID, \"\"))`\n\n286+ `if cleanup != nil {`\n\n287+ `defer cleanup()`\n\n288+ `}`\n\n289+ `result := validate.CoverageRepairResult{Validation: job.Validation}`\n\n290+ `var repairErr error`\n\n291+ `if repairCfg.RepairEquivalence || repairCfg.RepairCodeReview {`\n\n292+ `result, repairErr = validate.RepairCoverage(ctx, repairCfg)`\n\n293+ `}`\n\n294+ `result.Unresolved = append(result.Unresolved, reconstructionIssues...)`\n\n285295 `if repairErr != nil {`\n\n286296 `reporter.recordTaskEnd(variant, job.TaskID, job.CurrentWork, \"failed\", time.Now().UTC())`\n\n287297 `mu.Lock()`\n\n…`func RunAdaptedReferenceTest(ctx context.Context, ai AIClient, promptCtx PromptC`\n\n292292 `composedDiff, composeErr := composeAdaptedReferenceTestPatch(runCtx, inputs, referenceTestPatch, canonicalDiff)`\n\n293293 `if composeErr != nil {`\n\n294294 `result.Reason = \"patch_compose_failed\"`\n\n295− `result.Summary = fmt.Sprintf(\"compose canonical reference test patch and adapted_diff: %v\", composeErr)`\n\n295+ `result.Summary = fmt.Sprintf(\"adapted_diff could not be composed with the canonical reference test patch: %v\", composeErr)`\n\n296296 `return result`\n\n297297 `}`\n\n298298 `composedPayload := payload`\n\n…`import (`\n\n99 `\"strings\"`\n\n1010 `\"time\"`\n\n1111\n\n12+ `\"stet/internal/commandlist\"`\n\n1213 `\"stet/internal/filecopy\"`\n\n1314 `\"stet/internal/gitops\"`\n\n1415 `\"stet/internal/harness\"`\n\n…`func legacyCommandsThatNameAgentTests(commands, agentFiles []string) []string {`\n\n192193 `files := normalizeAgentTestFiles(agentFiles)`\n\n193194 `var out []string`\n\n194195 `for _, command := range normalizeCommands(commands) {`\n\n195− `if commandHasPotentialNonPathTestSelection(command) || receiptBackedCommandHasShellControl(command) {`\n\n196+ `if _, _, safe := commandlist.TestInvocation(command); !safe || commandHasPotentialNonPathTestSelection(command) || receiptBackedCommandHasShellControl(command) {`\n\n196197 `continue`\n\n197198 `}`\n\n198199 `for _, file := range files {`\n\n…`import (`\n\n66 `\"fmt\"`\n\n77 `\"os\"`\n\n88 `\"path/filepath\"`\n\n9+ `\"strconv\"`\n\n910 `\"strings\"`\n\n1011\n\n12+ `\"stet/internal/commandlist\"`\n\n1113 `\"stet/internal/shellcmd\"`\n\n1214`)`\n\n1315\n\n…`func replaceExactFinalTestPath(command, sourcePath, replacement string) (string,`\n\n198200`}`\n\n199201\n\n200202`func certifiedGoReplayCommand(command, sourcePath string) (string, bool) {`\n\n201− `if !strings.HasSuffix(strings.ToLower(sourcePath), \".go\") || !goCommandTargetsAgentTestFiles(command, []string{sourcePath}) {`\n\n203+ `_, fields, safe := commandlist.TestInvocation(command)`\n\n204+ `if !safe || len(fields) < 3 || fields[0].Value != \"go\" || fields[1].Value != \"test\" || !strings.HasSuffix(strings.ToLower(sourcePath), \".go\") {`\n\n202205 `return \"\", false`\n\n203206 `}`\n\n204− `replay, changed := removeGoRunSelector(command)`\n\n205− `if !changed || hasRemainingGoTestSelector(replay) {`\n\n207+ `// Require one explicit package. A recursive selector or additional package`\n\n208+ `// can credit tests outside the certified source package.`\n\n209+ `sourceDirectory := filepath.Dir(sourcePath)`\n\n210+ `allFields, _ := shellcmd.FieldSpans(command)`\n\n211+ `// Go also consumes selectors from GOFLAGS. Recorded overrides are opaque`\n\n212+ `// to this bounded rewrite, so they cannot certify an unfiltered replay.`\n\n213+ `for _, field := range allFields {`\n\n214+ `if field.Start >= fields[0].Start {`\n\n215+ `break`\n\n216+ `}`\n\n217+ `if strings.HasPrefix(field.Value, \"GOFLAGS=\") {`\n\n218+ `return \"\", false`\n\n219+ `}`\n\n220+ `}`\n\n221+ `if len(allFields) >= 3 && allFields[0].Value == \"cd\" {`\n\n222+ `relative, err := filepath.Rel(allFields[1].Value, sourceDirectory)`\n\n223+ `if err != nil || relative == \"..\" || strings.HasPrefix(relative, \"../\") {`\n\n224+ `return \"\", false`\n\n225+ `}`\n\n226+ `sourceDirectory = relative`\n\n227+ `}`\n\n228+ `packagePath := \"./\" + filepath.ToSlash(sourceDirectory)`\n\n229+ `if sourceDirectory == \".\" {`\n\n230+ `packagePath = \".\"`\n\n231+ `}`\n\n232+ `if strings.TrimSuffix(fields[2].Value, \"/\") != packagePath {`\n\n233+ `return \"\", false`\n\n234+ `}`\n\n235+ `start, end := -1, -1`\n\n236+ `for i := 3; i < len(fields); i++ {`\n\n237+ `value := fields[i].Value`\n\n238+ `if value == \"-run\" || strings.HasPrefix(value, \"-run=\") {`\n\n239+ `if start >= 0 {`\n\n240+ `return \"\", false`\n\n241+ `}`\n\n242+ `start, end = fields[i].Start, fields[i].End`\n\n243+ `if value == \"-run\" {`\n\n244+ `if i+1 >= len(fields) || strings.HasPrefix(fields[i+1].Value, \"-\") {`\n\n245+ `return \"\", false`\n\n246+ `}`\n\n247+ `i++`\n\n248+ `end = fields[i].End`\n\n249+ `}`\n\n250+ `continue`\n\n251+ `}`\n\n252+ `// Only flags with self-contained values may remain. Positional words,`\n\n253+ `// passthrough args, and further selectors cannot be proven safe.`\n\n254+ `name := strings.SplitN(value, \"=\", 2)[0]`\n\n255+ `if name == \"-count\" {`\n\n256+ `countText, hasValue := strings.CutPrefix(value, \"-count=\")`\n\n257+ `count, err := strconv.Atoi(countText)`\n\n258+ `if !hasValue || err != nil || count <= 0 {`\n\n259+ `return \"\", false`\n\n260+ `}`\n\n261+ `}`\n\n262+ `switch name {`\n\n263+ `case \"-v\", \"-race\", \"-short\", \"-count\", \"-timeout\", \"-parallel\", \"-shuffle\", \"-tags\", \"-mod\":`\n\n264+ `if !strings.Contains(value, \"=\") && name != \"-v\" && name != \"-race\" && name != \"-short\" {`\n\n265+ `return \"\", false`\n\n266+ `}`\n\n267+ `default:`\n\n268+ `return \"\", false`\n\n269+ `}`\n\n270+ `}`\n\n271+ `if start < 0 {`\n\n206272 `return \"\", false`\n\n207273 `}`\n\n208− `return replay, true`\n\n274+ `return strings.TrimSpace(strings.TrimRight(command[:start], \" \\t\") + command[end:]), true`\n\n209275`}`\n\n210276\n\n211277`func normalizeReplayRelativePath(path string) string {`\n\n…`func TestValidateAgentTestReplayPlanRejectsUnprovenCommand(t *testing.T) {`\n\n6161 `t.Fatalf(\"validation error = %v\", err)`\n\n6262 `}`\n\n6363`}`\n\n64+\n\n65+`func TestCertifiedGoReplayPreservesOnlyProvenInvocation(t *testing.T) {`\n\n66+ `command := \"cd module && env MODE='test value' timeout 30s go test ./pkg -run 'TestHidden|TestOther' -count=1\"`\n\n67+ `got, ok := certifiedGoReplayCommand(command, \"module/pkg/hidden_test.go\")`\n\n68+ `if !ok || got != \"cd module && env MODE='test value' timeout 30s go test ./pkg -count=1\" {`\n\n69+ `t.Fatalf(\"replay = %q, %v\", got, ok)`\n\n70+ `}`\n\n71+ `for _, unsafe := range []string{`\n\n72+ `\"echo go test ./pkg -run TestHidden\", \"go test ./pkg -run TestHidden; true\",`\n\n73+ `\"go test ./pkg -run TestHidden && go test ./other\", \"go test ./... -run TestHidden\",`\n\n74+ `\"go test ./pkg ./other -run TestHidden\", \"go test ./pkg -run TestHidden -skip TestOther\",`\n\n75+ `\"go test ./pkg -run TestHidden -args payload\", \"go test ./pkg -run TestHidden -run TestOther\",`\n\n76+ `\"go test ./pkg -run TestHidden -count=0\", \"go test ./pkg -run TestHidden -count=-1\",`\n\n77+ `\"go test ./pkg -run TestHidden -count=invalid\", \"go test ./pkg -run TestHidden -count 0\",`\n\n78+ `\"go test ./pkg -run TestHidden -count -1\",`\n\n79+ `\"GOFLAGS=-count=0 go test ./pkg -run TestHidden\",`\n\n80+ `\"GOFLAGS='-run=Nothing' go test ./pkg -run TestHidden\",`\n\n81+ `\"env GOFLAGS=-count=0 go test ./pkg -run TestHidden\",`\n\n82+ `\"cd module && env GOFLAGS='-run=Nothing' go test ./pkg -run TestHidden\",`\n\n83+ `} {`\n\n84+ `if replay, ok := certifiedGoReplayCommand(unsafe, \"pkg/hidden_test.go\"); ok {`\n\n85+ `t.Errorf(\"accepted %q as %q\", unsafe, replay)`\n\n86+ `}`\n\n87+ `}`\n\n88+`}`\n\n89+\n\n90+`func TestLegacyReplayRequiresAnActualTestRunner(t *testing.T) {`\n\n91+ `commands := legacyCommandsThatNameAgentTests([]string{`\n\n92+ `\"echo tests/agent_test.sh\", \"cat tests/agent_test.sh\", \"rm tests/agent_test.sh\",`\n\n93+ `\"sh -c tests/agent_test.sh\", \"'CI=1' sh tests/agent_test.sh\", \"sh tests/agent_test.sh\",`\n\n94+ `}, []string{\"tests/agent_test.sh\"})`\n\n95+ `if len(commands) != 1 || commands[0] != \"sh 'tests/agent_test.sh'\" {`\n\n96+ `t.Fatalf(\"legacy replay commands = %q\", commands)`\n\n97+ `}`\n\n98+`}`\n\n…`func buildEquivalenceRetryPrompt(originalPrompt, rawOutput string, parseErr erro`\n\n1818 `)`\n\n1919`}`\n\n2020\n\n21−`func RunEquivalence(ctx context.Context, ai AIClient, promptCtx PromptContext) EquivalenceResult {`\n\n21+`func RunEquivalence(ctx context.Context, ai AIClient, promptCtx PromptContext) (result EquivalenceResult) {`\n\n22+ `mode := EquivalencePromptModeStandard`\n\n23+ `if goldBlindEquivalenceEnabled() {`\n\n24+ `mode = EquivalencePromptModeGoldBlind`\n\n25+ `}`\n\n26+ `history := newGradeAttemptClient(ai)`\n\n27+ `ai = history`\n\n28+ `defer func() { result.PromptMode = mode; history.annotateEquivalence(&result) }()`\n\n29+\n\n2230 `if goldBlindEquivalenceEnabled() {`\n\n2331 `// No gold reaches the model, so gold truncation cannot have mutilated`\n\n2432 `// the graded evidence: clear it before the truncation annotation runs.`\n\n…\n\n11`package validate`\n\n22\n\n33`import (`\n\n4+ `\"fmt\"`\n\n45 `\"os\"`\n\n56 `\"strconv\"`\n\n67 `\"strings\"`\n\n…`notes: <short explanation tied to the task obligations>`\n\n127128`// mode. In gold-blind mode the gold patch is dropped from the template data as`\n\n128129`// well as from the template, so no gold content can reach the model.`\n\n129130`func renderEquivalencePrompt(promptCtx PromptContext) (prompt string, goldBlind bool, err error) {`\n\n130− `if !goldBlindEquivalenceEnabled() {`\n\n131− `rendered, renderErr := renderPrompt(equivalencePromptTemplate, promptCtx.asTemplateData())`\n\n132− `return rendered, false, renderErr`\n\n131+ `mode := EquivalencePromptModeStandard`\n\n132+ `if goldBlindEquivalenceEnabled() {`\n\n133+ `mode = EquivalencePromptModeGoldBlind`\n\n134+ `}`\n\n135+ `rendered, err := RenderEquivalencePromptForMode(promptCtx, mode)`\n\n136+ `return rendered, mode == EquivalencePromptModeGoldBlind, err`\n\n137+`}`\n\n138+\n\n139+`// RenderEquivalencePromptForMode uses recorded measurement authority, never the process environment.`\n\n140+`func RenderEquivalencePromptForMode(promptCtx PromptContext, mode EquivalencePromptMode) (string, error) {`\n\n141+ `switch mode {`\n\n142+ `case EquivalencePromptModeStandard:`\n\n143+ `return renderPrompt(equivalencePromptTemplate, promptCtx.asTemplateData())`\n\n144+ `case EquivalencePromptModeGoldBlind:`\n\n145+ `data := promptCtx.asTemplateData()`\n\n146+ `delete(data, \"GoldPatch\")`\n\n147+ `return renderPrompt(equivalenceGoldBlindPromptTemplate, data)`\n\n148+ `default:`\n\n149+ `return \"\", fmt.Errorf(\"unknown equivalence prompt mode %q\", mode)`\n\n133150 `}`\n\n134− `data := promptCtx.asTemplateData()`\n\n135− `delete(data, \"GoldPatch\")`\n\n136− `rendered, renderErr := renderPrompt(equivalenceGoldBlindPromptTemplate, data)`\n\n137− `return rendered, true, renderErr`\n\n138151`}`\n\n…`func runPerCommandTests(`\n\n22022202 `cmdCfg.Commands = []string{cmd}`\n\n22032203 `goldRun, err := RunGoldBaselineTests(infraCtx, inputs, cmdCfg)`\n\n22042204 `gr.EnvCache = goldRun.EnvCache`\n\n2205− `gr.Result = TestCommandResult{Command: cmd}`\n\n2205+ `gr.Result = TestCommandResult{Command: cmd, AttemptCount: 1}`\n\n22062206 `if err != nil {`\n\n22072207 `gr.Err = err`\n\n2208− `if mode := inferInfraFailureModeFromError(err); mode != \"\" {`\n\n2208+ `if mode := retryableVerifierFailureMode(goldRun, err); mode != \"\" {`\n\n22092209 `gr.Result.GoldOutcome = TestOutcomeFail`\n\n22102210 `gr.Result.GoldFailureMode = mode`\n\n22112211 `gr.IsInfra = true`\n\n…`func runPerCommandTests(`\n\n22182218 `gr.Result.GoldOutcome = TestOutcomeUnknown`\n\n22192219 `gr.Notes = append(gr.Notes, fmt.Sprintf(\"gold baseline failed for %q: %v\", cmd, err))`\n\n22202220 `}`\n\n2221− `return`\n\n2221+ `if !isRetryableGoldInfrastructureFailure(gr.Result.GoldFailureMode) {`\n\n2222+ `return`\n\n2223+ `}`\n\n22222224 `}`\n\n22232225 `gr.EnvCache = goldRun.EnvCache`\n\n22242226 `applyGoldRunResult(gr, goldRun)`\n\n2225− `if gr.Result.GoldOutcome == TestOutcomeFail && isRetryableGoldInfrastructureFailure(gr.Result.GoldFailureMode) && budgetCtx.Err() == nil {`\n\n2227+ `if isUnsetFailureMode(gr.Result.GoldFailureMode) {`\n\n2228+ `gr.Result.GoldFailureMode = retryableVerifierFailureMode(goldRun, err)`\n\n2229+ `}`\n\n2230+ `if gr.Result.GoldOutcome != TestOutcomePass && isRetryableGoldInfrastructureFailure(gr.Result.GoldFailureMode) && budgetCtx.Err() == nil {`\n\n22262231 `beginGoldRetry(&gr.Result, goldRun)`\n\n2232+ `recordTestRetry(&gr.Result, gr.Result.GoldFailureMode)`\n\n22272233 `retryCfg := goldRetryAttemptConfig(overrideCfg, gr.Index, 2)`\n\n22282234 `retryCfg.Commands = []string{cmd}`\n\n22292235 `retryRun, retryErr := RunGoldBaselineTests(infraCtx, inputs, retryCfg)`\n\n…`func runPerCommandTests(`\n\n22442250 `gr.Result.GoldRetried = true`\n\n22452251 `finishGoldRetry(&gr.Result, retryRun)`\n\n22462252 `if gr.Result.GoldOutcome == TestOutcomePass {`\n\n2253+ `gr.Result.RecoveredAfterRetry = true`\n\n2254+ `gr.Err = nil`\n\n2255+ `gr.IsInfra = false`\n\n22472256 `gr.Notes = append(gr.Notes, fmt.Sprintf(\"gold baseline for %q recovered on retry after transient dependency fetch\", cmd))`\n\n22482257 `} else {`\n\n22492258 `gr.Notes = append(gr.Notes, fmt.Sprintf(\"gold baseline retry for %q did not recover\", cmd))`\n\n…`func runPerCommandTests(`\n\n24842493 `cmdCfg := overrideCfg`\n\n24852494 `cmdCfg.Commands = []string{entry.cmd}`\n\n24862495 `or.Attempted = true`\n\n2496+ `or.Result.AttemptCount++`\n\n24872497 `overrideRun, err := RunOverrideTests(budgetCtx, inputs, cmdCfg)`\n\n24882498 `or.EnvCache = overrideRun.EnvCache`\n\n24892499 `if err != nil {`\n\n2490− `if mode := inferInfraFailureModeFromError(err); mode != \"\" {`\n\n2500+ `if mode := retryableVerifierFailureMode(overrideRun, err); mode != \"\" {`\n\n24912501 `or.Result.AgentFailureMode = mode`\n\n24922502 `if mode == \"timeout\" {`\n\n24932503 `or.Notes = append(or.Notes, fmt.Sprintf(\"agent retest timeout for %q\", entry.cmd))`\n\n…`func runPerCommandTests(`\n\n24962506 `or.Notes = append(or.Notes, fmt.Sprintf(\"agent retest failed for %q: %v\", entry.cmd, err))`\n\n24972507 `or.Result.AgentOutcome = TestOutcomeUnknown`\n\n24982508 `or.Err = err`\n\n2499− `return`\n\n2509+ `if !isRetryableGoldInfrastructureFailure(or.Result.AgentFailureMode) {`\n\n2510+ `return`\n\n2511+ `}`\n\n25002512 `}`\n\n25012513 `or.Result.AgentOutcome = testOutcomeFromTrial(overrideRun.Trial)`\n\n25022514 `or.Result.AgentTrialName = overrideRun.Trial.TrialName`\n\n2503− `or.Result.AgentFailureMode = overrideRun.Trial.FailureMode`\n\n2515+ `or.Result.AgentFailureMode = retryableVerifierFailureMode(overrideRun, err)`\n\n25042516\n\n2505− `// Retry once if agent failed but gold passed.`\n\n2506− `if or.Result.AgentOutcome == TestOutcomeFail && or.Result.GoldOutcome == TestOutcomePass && budgetCtx.Err() == nil {`\n\n2517+ `// Retry only a typed transient infrastructure failure; assertions are terminal.`\n\n2518+ `if or.Result.AgentOutcome != TestOutcomePass && isRetryableGoldInfrastructureFailure(or.Result.AgentFailureMode) && or.Result.GoldOutcome == TestOutcomePass && budgetCtx.Err() == nil {`\n\n25072519 `retryCfg := overrideCfg`\n\n25082520 `retryCfg.Commands = []string{entry.cmd}`\n\n2521+ `recordTestRetry(&or.Result, or.Result.AgentFailureMode)`\n\n2522+ `or.Result.AgentRetried = true`\n\n25092523 `retryRun, retryErr := RunOverrideTests(budgetCtx, inputs, retryCfg)`\n\n25102524 `or.RetryEnvCache = &retryRun.EnvCache`\n\n2525+ `or.Result.AgentOutcome = testOutcomeFromTrial(retryRun.Trial)`\n\n2526+ `or.Result.AgentTrialName = retryRun.Trial.TrialName`\n\n2527+ `or.Result.AgentFailureMode = retryableVerifierFailureMode(retryRun, retryErr)`\n\n2528+ `or.Err = retryErr`\n\n25112529 `if retryErr == nil {`\n\n25122530 `retryOutcome := testOutcomeFromTrial(retryRun.Trial)`\n\n25132531 `if retryOutcome == TestOutcomePass {`\n\n…`func runPerCommandTests(`\n\n25152533 `or.Result.AgentTrialName = retryRun.Trial.TrialName`\n\n25162534 `or.Result.AgentFailureMode = retryRun.Trial.FailureMode`\n\n25172535 `or.Result.AgentRetried = true`\n\n2536+ `or.Result.RecoveredAfterRetry = true`\n\n2537+ `or.Err = nil`\n\n25182538 `or.Notes = append(or.Notes, fmt.Sprintf(\"agent retest for %q succeeded on retry\", entry.cmd))`\n\n25192539 `}`\n\n25202540 `}`\n\n…`func runGoldBatchSlice(`\n\n347347 `cleanups := []func(){}`\n\n348348 `for _, idx := range indexes {`\n\n349349 `gr := &goldResults[idx]`\n\n350+ `priorResult := gr.Result`\n\n350351 `priorRetried, priorEvidence := gr.Result.GoldRetried, gr.Result.GoldRetryEvidence`\n\n351352 `gr.IsInfra = false`\n\n352353 `gr.IsGlobalInfra = false`\n\n353354 `gr.Err = nil`\n\n354355 `if ctx.Err() != nil {`\n\n355356 `gr.Result = TestCommandResult{`\n\n356− `Command:           gr.Command,`\n\n357− `GoldOutcome:       TestOutcomeUnknown,`\n\n358− `GoldRetried:       priorRetried,`\n\n359− `GoldRetryEvidence: priorEvidence,`\n\n357+ `Command:             gr.Command,`\n\n358+ `GoldOutcome:         TestOutcomeUnknown,`\n\n359+ `GoldRetried:         priorRetried,`\n\n360+ `GoldRetryEvidence:   priorEvidence,`\n\n361+ `AttemptCount:        priorResult.AttemptCount,`\n\n362+ `RetryCount:          priorResult.RetryCount,`\n\n363+ `RetryReasons:        priorResult.RetryReasons,`\n\n364+ `RecoveredAfterRetry: priorResult.RecoveredAfterRetry,`\n\n360365 `}`\n\n361366 `gr.Err = ctx.Err()`\n\n362367 `continue`\n\n…`func runGoldBatchSlice(`\n\n364369 `stageRoot, cacheResult, cleanup, err := stageGoldBaselineTask(ctx, inputs, []string{commands[idx]}, false, overrideCfg.OutputDir)`\n\n365370 `gr.EnvCache = cacheResult`\n\n366371 `gr.Result = TestCommandResult{`\n\n367− `Command:           gr.Command,`\n\n368− `GoldRetried:       priorRetried,`\n\n369− `GoldRetryEvidence: priorEvidence,`\n\n372+ `Command:             gr.Command,`\n\n373+ `GoldRetried:         priorRetried,`\n\n374+ `GoldRetryEvidence:   priorEvidence,`\n\n375+ `AttemptCount:        priorResult.AttemptCount,`\n\n376+ `RetryCount:          priorResult.RetryCount,`\n\n377+ `RetryReasons:        priorResult.RetryReasons,`\n\n378+ `RecoveredAfterRetry: priorResult.RecoveredAfterRetry,`\n\n370379 `}`\n\n371380 `if err != nil {`\n\n372381 `gr.Err = err`\n\n…`func runGoldBatchSlice(`\n\n407416 `gr.Result.GoldRetried = true`\n\n408417 `gr.Result.GoldRetryEvidence = &TestRetryEvidence{First: *gr.FirstRetryEvidence}`\n\n409418 `}`\n\n419+ `recordTestRetry(&gr.Result, FailureModeInfraDependencyFetch)`\n\n410420 `gr.RetryDispatched = true`\n\n411421 `}`\n\n412422 `}`\n\n413423\n\n424+ `for _, item := range items {`\n\n425+ `var idx int`\n\n426+ `fmt.Sscanf(item.ProjectionKey, \"gold-%d\", &idx)`\n\n427+ `if attempt == 1 {`\n\n428+ `goldResults[idx].Result.AttemptCount++`\n\n429+ `}`\n\n430+ `}`\n\n414431 `results, err := batchRunner.RunBatch(ctx, harness.BatchRunConfig{`\n\n415432 `OutputDir:           filepath.Join(filepath.Dir(items[0].SourceTaskDir), \".tb-runs\"),`\n\n416433 `RunID:               fmt.Sprintf(\"validate-gold-batch-%d\", time.Now().UnixNano()),`\n\n…`func runGoldBatchSlice(`\n\n452469 `evidence := retainBatchGoldRetryAttemptEvidence(projected, gr.Result.GoldFailureMode, overrideCfg, gr.Index, attempt)`\n\n453470 `gr.FirstRetryEvidence = &evidence`\n\n454471 `}`\n\n472+ `if attempt > 1 && gr.Result.GoldOutcome == TestOutcomePass {`\n\n473+ `gr.Result.RecoveredAfterRetry = true`\n\n474+ `}`\n\n455475 `if attempt > 1 && gr.Result.GoldRetryEvidence != nil {`\n\n456476 `evidence := retainBatchGoldRetryAttemptEvidence(projected, gr.Result.GoldFailureMode, overrideCfg, gr.Index, attempt)`\n\n457477 `gr.Result.GoldRetryEvidence.Retry = &evidence`\n\n…`func runOverrideBatchSlice(`\n\n564584 `return`\n\n565585 `}`\n\n566586\n\n587+ `for _, item := range items {`\n\n588+ `var idx int`\n\n589+ `fmt.Sscanf(item.ProjectionKey, \"override-%d\", &idx)`\n\n590+ `overrideResults[idx-offset].Result.AttemptCount++`\n\n591+ `}`\n\n592+\n\n567593 `results, err := batchRunner.RunBatch(ctx, harness.BatchRunConfig{`\n\n568594 `OutputDir:           filepath.Join(filepath.Dir(items[0].SourceTaskDir), \".tb-runs\"),`\n\n569595 `RunID:               fmt.Sprintf(\"validate-override-batch-%d\", time.Now().UnixNano()),`\n\n…`func runOverrideBatchSlice(`\n\n604630 `projected := results[fmt.Sprintf(\"override-%d\", offset+i)]`\n\n605631 `or.Result.AgentOutcome = testOutcomeFromTrial(projected.Trial)`\n\n606632 `or.Result.AgentTrialName = projected.Trial.TrialName`\n\n607− `or.Result.AgentFailureMode = projected.Trial.FailureMode`\n\n608− `if or.Result.AgentOutcome == TestOutcomeFail && or.Result.GoldOutcome == TestOutcomePass && ctx.Err() == nil {`\n\n633+ `or.Result.AgentFailureMode = classifyBatchGoldFailureMode(projected)`\n\n634+ `if or.Result.AgentOutcome != TestOutcomePass && isRetryableGoldInfrastructureFailure(or.Result.AgentFailureMode) && or.Result.GoldOutcome == TestOutcomePass && ctx.Err() == nil {`\n\n609635 `stageRoot, cacheResult, cleanup, err := stageOverrideTask(ctx, inputs, []string{or.Command}, false, overrideCfg.OutputDir)`\n\n610636 `if err == nil {`\n\n611637 `or.RetryEnvCache = &cacheResult`\n\n…`func runOverrideBatchSlice(`\n\n623649 `if len(retryItems) == 0 {`\n\n624650 `return`\n\n625651 `}`\n\n652+ `for _, idx := range retryIndexes {`\n\n653+ `or := &overrideResults[idx]`\n\n654+ `recordTestRetry(&or.Result, or.Result.AgentFailureMode)`\n\n655+ `or.Result.AgentRetried = true`\n\n656+ `}`\n\n626657 `retryResults, err := batchRunner.RunBatch(ctx, harness.BatchRunConfig{`\n\n627658 `OutputDir:           filepath.Join(filepath.Dir(retryItems[0].SourceTaskDir), \".tb-runs\"),`\n\n628659 `RunID:               fmt.Sprintf(\"validate-override-retry-batch-%d\", time.Now().UnixNano()),`\n\n…`func runOverrideBatchSlice(`\n\n634665 `HarnessCLICacheMode: overrideCfg.HarnessCLICacheMode,`\n\n635666 `Items:               retryItems,`\n\n636667 `})`\n\n637− `if err != nil {`\n\n638− `return`\n\n639− `}`\n\n640668 `for _, idx := range retryIndexes {`\n\n641669 `or := &overrideResults[idx]`\n\n642670 `projected := retryResults[fmt.Sprintf(\"retry-%d\", offset+idx)]`\n\n643− `if testOutcomeFromTrial(projected.Trial) == TestOutcomePass {`\n\n644− `or.Result.AgentOutcome = TestOutcomePass`\n\n645− `or.Result.AgentTrialName = projected.Trial.TrialName`\n\n646− `or.Result.AgentFailureMode = projected.Trial.FailureMode`\n\n647− `or.Result.AgentRetried = true`\n\n671+ `or.Result.AgentOutcome = testOutcomeFromTrial(projected.Trial)`\n\n672+ `or.Result.AgentTrialName = projected.Trial.TrialName`\n\n673+ `or.Result.AgentFailureMode = classifyBatchGoldFailureMode(projected)`\n\n674+ `or.Err = err`\n\n675+ `if err != nil {`\n\n676+ `if isUnsetFailureMode(or.Result.AgentFailureMode) {`\n\n677+ `or.Result.AgentFailureMode = inferInfraFailureModeFromError(err)`\n\n678+ `}`\n\n679+ `or.Notes = append(or.Notes, fmt.Sprintf(\"agent retest retry for %q failed: %v\", or.Command, err))`\n\n680+ `}`\n\n681+ `if err == nil && or.Result.AgentOutcome == TestOutcomePass {`\n\n682+ `or.Result.RecoveredAfterRetry = true`\n\n648683 `or.Notes = append(or.Notes, fmt.Sprintf(\"agent retest for %q succeeded on retry\", or.Command))`\n\n649684 `}`\n\n650685 `}`\n\n…`go: golang.org/x/sys@v0.42.0: Get \"https://proxy.golang.org/golang.org/x/sys/@v/`\n\n50215021 `t.Fatalf(\"command results = %d, want 1\", len(summary.CommandResults))`\n\n50225022 `}`\n\n50235023 `result := summary.CommandResults[0]`\n\n5024+ `if result.AttemptCount != 3 || result.RetryCount != 1 || !result.RecoveredAfterRetry || len(result.RetryReasons) != 1 || result.RetryReasons[0] != FailureModeInfraDependencyFetch {`\n\n5025+ `t.Fatalf(\"retry metadata = %+v\", result)`\n\n5026+ `}`\n\n50245027 `if !result.GoldRetried || result.GoldOutcome != TestOutcomePass || result.GoldTrialName != \"gold-retry\" {`\n\n50255028 `t.Fatalf(\"gold retry result = %+v\", result)`\n\n50265029 `}`\n\n…`go: golang.org/x/sys@v0.42.0: Get \"https://proxy.golang.org/golang.org/x/sys/@v/`\n\n51115114 `t.Fatalf(\"command results = %d, want 1\", len(summary.CommandResults))`\n\n51125115 `}`\n\n51135116 `result := summary.CommandResults[0]`\n\n5117+ `if result.AttemptCount != 3 || result.RetryCount != 1 || !result.RecoveredAfterRetry || len(result.RetryReasons) != 1 || result.RetryReasons[0] != FailureModeInfraDependencyFetch {`\n\n5118+ `t.Fatalf(\"retry metadata = %+v\", result)`\n\n5119+ `}`\n\n51145120 `if !result.GoldRetried || result.GoldOutcome != TestOutcomePass || result.GoldTrialName != \"gold-retry\" {`\n\n51155121 `t.Fatalf(\"gold retry result = %+v\", result)`\n\n51165122 `}`\n\n…`func TestRunPerCommandTestsBatchedGoldTimeoutClassifiedAsTimeBudgetExhausted(t *`\n\n53825388 `}`\n\n53835389`}`\n\n53845390\n\n5385−`func TestRunPerCommandTestsBatchedRetriesFailedOverrideAndMarksRetried(t *testing.T) {`\n\n5391+`func TestRunPerCommandTestsBatchedRetriesDependencyFetchOverrideAndMarksRetried(t *testing.T) {`\n\n53865392 `t.Parallel()`\n\n53875393 `repo, base, head := createRepoWithCommits(t)`\n\n53885394 `dataset := t.TempDir()`\n\n…`func TestRunPerCommandTestsBatchedRetriesFailedOverrideAndMarksRetried(t *testin`\n\n54185424 `runner := &batchValidateRunner{`\n\n54195425 `results: map[string]TBTrialResult{`\n\n54205426 `\"gold-0\":     {TaskID: taskID, IsResolved: &pass, TrialName: \"gold-a\"},`\n\n5421− `\"override-0\": {TaskID: taskID, IsResolved: &fail, TrialName: \"override-a\"},`\n\n5427+ `\"override-0\": {TaskID: taskID, IsResolved: &fail, TrialName: \"override-a\", FailureMode: FailureModeInfraDependencyFetch},`\n\n54225428 `\"retry-0\":    {TaskID: taskID, IsResolved: &pass, TrialName: \"retry-a\"},`\n\n54235429 `},`\n\n54245430 `}`\n\n…`func TestRunPerCommandTestsBatchedRetriesFailedOverrideAndMarksRetried(t *testin`\n\n54435449 `t.Fatalf(\"expected 1 command result, got %d\", len(result.Tests.CommandResults))`\n\n54445450 `}`\n\n54455451 `cmd := result.Tests.CommandResults[0]`\n\n5452+ `if cmd.AttemptCount != 3 || cmd.RetryCount != 1 || !cmd.RecoveredAfterRetry || len(cmd.RetryReasons) != 1 || cmd.RetryReasons[0] != FailureModeInfraDependencyFetch {`\n\n5453+ `t.Fatalf(\"retry metadata = %+v\", cmd)`\n\n5454+ `}`\n\n54465455 `if !cmd.AgentRetried {`\n\n54475456 `t.Fatal(\"expected AgentRetried=true\")`\n\n54485457 `}`\n\n…\n\n11`package validate`\n\n22\n\n3−`import \"fmt\"`\n\n3+`import (`\n\n4+ `\"fmt\"`\n\n5+ `\"path/filepath\"`\n\n6+ `\"strings\"`\n\n7+`)`\n\n48\n\n59`// OfflinePromptInputs describes everything needed to reassemble the grader`\n\n610`// PromptContext for a single already-evaluated cell without re-running the`\n\n…`type OfflinePromptInputs struct {`\n\n1822 `// TBResultsRaw is the raw TB results JSON for this cell, or \"\" when`\n\n1923 `// unavailable (e.g. no-test-signal corpora).`\n\n2024 `TBResultsRaw string`\n\n25+ `// EffectiveTestCommands overrides dataset commands when non-nil. An empty`\n\n26+ `// non-nil slice preserves a persisted grade with no verifier commands.`\n\n27+ `EffectiveTestCommands []string`\n\n2128 `// PromptPatchMaxBytes caps the gold/agent patch bytes embedded in the`\n\n2229 `// prompt. Values <= 0 fall back to DefaultPromptPatchMaxBytes, matching the`\n\n2330 `// orchestrator/config default.`\n\n…`type OfflinePromptInputs struct {`\n\n2532 `// RepoContextOptions controls repo tree/excerpt extraction. The zero value`\n\n2633 `// inherits BuildRepoContext defaults, matching the live default path.`\n\n2734 `RepoContextOptions RepoContextOptions`\n\n35+ `// RequireRepoContext makes reconstruction fail when the source repository`\n\n36+ `// cannot be extracted, instead of grading an unavailable-context marker.`\n\n37+ `RequireRepoContext bool`\n\n38+ `// ExpectedArtifacts binds reconstruction to the original graded files.`\n\n39+ `// When supplied, every source path and digest must be present and match.`\n\n40+ `ExpectedArtifacts *ArtifactManifest`\n\n2841`}`\n\n2942\n\n3043`// BuildOfflinePromptContext reassembles the grader PromptContext for one cell`\n\n…`type OfflinePromptInputs struct {`\n\n3447`// that removes the extracted repo temp dir; callers should defer it.`\n\n3548`//`\n\n3649`// The assembly intentionally mirrors internal/validate/orchestrator.go so the`\n\n37−`// resulting context is byte-for-byte identical to a live grade except for the`\n\n38−`// template version: it reuses ReadTaskInputs, truncatePatch, and`\n\n39−`// BuildRepoContext, and applies the same graceful repo-context fallback.`\n\n50+`// assembly reuses ReadTaskInputs, truncatePatch, and BuildRepoContext. Callers`\n\n51+`// can supply persisted effective commands and require usable repository context;`\n\n52+`// otherwise it applies the live path's graceful repo-context fallback.`\n\n4053`func BuildOfflinePromptContext(in OfflinePromptInputs) (PromptContext, func(), error) {`\n\n4154 `inputs, err := ReadTaskInputs(in.DatasetPath, in.TaskID, in.AgentPatchPath)`\n\n4255 `if err != nil {`\n\n…`func BuildOfflinePromptContext(in OfflinePromptInputs) (PromptContext, func(), e`\n\n4558 `if err := verifyRepoSourceForConsumption(inputs); err != nil {`\n\n4659 `return PromptContext{}, nil, err`\n\n4760 `}`\n\n61+ `if in.ExpectedArtifacts != nil {`\n\n62+ `current := BuildArtifactManifest(inputs)`\n\n63+ `for _, pair := range []struct {`\n\n64+ `name      string`\n\n65+ `want, got ArtifactFileManifest`\n\n66+ `}{`\n\n67+ `{\"agent patch\", in.ExpectedArtifacts.AgentPatch, current.AgentPatch},`\n\n68+ `{\"gold patch\", in.ExpectedArtifacts.GoldPatch, current.GoldPatch},`\n\n69+ `{\"repository\", in.ExpectedArtifacts.RepoTarball, current.RepoTarball},`\n\n70+ `} {`\n\n71+ `if pair.want.Path == \"\" || pair.want.SHA256 == \"\" {`\n\n72+ `return PromptContext{}, nil, fmt.Errorf(\"missing original %s path or digest\", pair.name)`\n\n73+ `}`\n\n74+ `if filepath.Clean(pair.want.Path) != filepath.Clean(pair.got.Path) || !strings.EqualFold(strings.TrimPrefix(pair.want.SHA256, \"sha256:\"), pair.got.SHA256) {`\n\n75+ `return PromptContext{}, nil, fmt.Errorf(\"original %s provenance mismatch\", pair.name)`\n\n76+ `}`\n\n77+ `}`\n\n78+ `}`\n\n4879\n\n4980 `maxBytes := in.PromptPatchMaxBytes`\n\n5081 `if maxBytes <= 0 {`\n\n…`func BuildOfflinePromptContext(in OfflinePromptInputs) (PromptContext, func(), e`\n\n5283 `}`\n\n5384 `goldPatchPrompt, goldPatchTruncation := truncatePatch(inputs.GoldPatch, maxBytes)`\n\n5485 `agentPatchPrompt, agentPatchTruncation := truncatePatch(inputs.AgentPatch, maxBytes)`\n\n86+ `if in.ExpectedArtifacts != nil && (goldPatchTruncation.Truncated || agentPatchTruncation.Truncated) {`\n\n87+ `return PromptContext{}, nil, fmt.Errorf(\"reconstructed prompt patches would be truncated; original context budgets are unavailable\")`\n\n88+ `}`\n\n89+ `testCommands := inputs.TestCommands`\n\n90+ `if in.EffectiveTestCommands != nil {`\n\n91+ `testCommands = append([]string{}, in.EffectiveTestCommands...)`\n\n92+ `}`\n\n5593\n\n5694 `// Mirror orchestrator.go: on repo-context failure, degrade RepoTree to an`\n\n5795 `// \"unavailable: <err>\" marker and continue rather than failing the build.`\n\n…`func BuildOfflinePromptContext(in OfflinePromptInputs) (PromptContext, func(), e`\n\n5997 `repoCtx := RepoContext{}`\n\n6098 `builtRepoCtx, repoCleanup, repoErr := BuildRepoContext(inputs.RepoTarballPath, in.RepoContextOptions)`\n\n6199 `if repoErr != nil {`\n\n100+ `if in.RequireRepoContext {`\n\n101+ `return PromptContext{}, nil, fmt.Errorf(\"build required repo context: %w\", repoErr)`\n\n102+ `}`\n\n62103 `builtRepoCtx = RepoContext{`\n\n63104 `RepoTree: fmt.Sprintf(\"unavailable: %v\", repoErr),`\n\n64105 `}`\n\n65106 `} else {`\n\n66107 `cleanup = repoCleanup`\n\n108+ `if in.ExpectedArtifacts != nil && (builtRepoCtx.RepoTreeTruncation.Truncated || builtRepoCtx.RepoExcerptsTruncation.Truncated) {`\n\n109+ `if cleanup != nil {`\n\n110+ `cleanup()`\n\n111+ `}`\n\n112+ `return PromptContext{}, nil, fmt.Errorf(\"reconstructed repository context would be truncated; original context budgets are unavailable\")`\n\n113+ `}`\n\n67114 `}`\n\n68115 `repoCtx = builtRepoCtx`\n\n69116\n\n…`func BuildOfflinePromptContext(in OfflinePromptInputs) (PromptContext, func(), e`\n\n71118 `TaskYAML:            inputs.TaskYAML,`\n\n72119 `GoldPatch:           goldPatchPrompt,`\n\n73120 `AgentPatch:          agentPatchPrompt,`\n\n74− `TestCommands:        inputs.TestCommands,`\n\n121+ `TestCommands:        testCommands,`\n\n75122 `TBResults:           in.TBResultsRaw,`\n\n76123 `RepoPath:            repoCtx.RepoPath,`\n\n77124 `RepoTree:            repoCtx.RepoTree,`\n\n…`func BuildOfflinePromptContext(in OfflinePromptInputs) (PromptContext, func(), e`\n\n80127 `GoldPatchTruncated:  goldPatchTruncation.Truncated,`\n\n81128 `}, cleanup, nil`\n\n82129`}`\n\n130+\n\n131+`// RenderReviewPrompt renders the built-in review template from supplied evidence.`\n\n132+`func RenderReviewPrompt(promptCtx PromptContext) (string, error) {`\n\n133+ `return renderPrompt(reviewPromptTemplate, promptCtx.asTemplateData())`\n\n134+`}`\n\n…`func TestBuildOfflinePromptContextDegradesWhenTarballUnextractable(t *testing.T)`\n\n164164 `t.Fatalf(\"expected agent patch to be populated, got %q\", promptCtx.AgentPatch)`\n\n165165 `}`\n\n166166`}`\n\n167+\n\n168+`func TestBuildOfflinePromptContextRequiresUsableRepositoryWhenRequested(t *testing.T) {`\n\n169+ `datasetPath, taskID, agentPatchPath := writeOfflineTaskDir(t, false)`\n\n170+ `_, cleanup, err := BuildOfflinePromptContext(OfflinePromptInputs{`\n\n171+ `DatasetPath: datasetPath, TaskID: taskID, AgentPatchPath: agentPatchPath,`\n\n172+ `RequireRepoContext: true,`\n\n173+ `})`\n\n174+ `if cleanup != nil {`\n\n175+ `defer cleanup()`\n\n176+ `}`\n\n177+ `if err == nil || !strings.Contains(err.Error(), \"build required repo context\") {`\n\n178+ `t.Fatalf(\"expected required repository extraction failure, got %v\", err)`\n\n179+ `}`\n\n180+`}`\n\n181+\n\n182+`func TestBuildOfflinePromptContextUsesEffectiveVerifierCommands(t *testing.T) {`\n\n183+ `datasetPath, taskID, agentPatchPath := writeOfflineTaskDir(t, true)`\n\n184+ `for _, commands := range [][]string{nil, {}, {\"effective verifier\"}} {`\n\n185+ `ctx, cleanup, err := BuildOfflinePromptContext(OfflinePromptInputs{`\n\n186+ `DatasetPath: datasetPath, TaskID: taskID, AgentPatchPath: agentPatchPath,`\n\n187+ `EffectiveTestCommands: commands,`\n\n188+ `})`\n\n189+ `if cleanup != nil {`\n\n190+ `defer cleanup()`\n\n191+ `}`\n\n192+ `if err != nil {`\n\n193+ `t.Fatal(err)`\n\n194+ `}`\n\n195+ `want := commands`\n\n196+ `if commands == nil {`\n\n197+ `want = []string{\"pytest -q\"}`\n\n198+ `}`\n\n199+ `if !reflect.DeepEqual(ctx.TestCommands, want) {`\n\n200+ `t.Fatalf(\"effective commands %v: got %v, want %v\", commands, ctx.TestCommands, want)`\n\n201+ `}`\n\n202+ `}`\n\n203+`}`\n\n204+\n\n205+`func TestBuildOfflinePromptContextRejectsMutatedOriginalArtifacts(t *testing.T) {`\n\n206+ `for _, artifact := range []string{\"agent patch\", \"gold patch\", \"repository\"} {`\n\n207+ `t.Run(artifact, func(t *testing.T) {`\n\n208+ `dataset, taskID, patch := writeOfflineTaskDir(t, true)`\n\n209+ `inputs, err := ReadTaskInputs(dataset, taskID, patch)`\n\n210+ `if err != nil {`\n\n211+ `t.Fatal(err)`\n\n212+ `}`\n\n213+ `original := BuildArtifactManifest(inputs)`\n\n214+ `path := patch`\n\n215+ `if artifact == \"gold patch\" {`\n\n216+ `path = inputs.GoldPatchPath`\n\n217+ `}`\n\n218+ `if artifact == \"repository\" {`\n\n219+ `path = inputs.RepoTarballPath`\n\n220+ `}`\n\n221+ `if err := os.WriteFile(path, []byte(\"mutated evidence\"), 0o644); err != nil {`\n\n222+ `t.Fatal(err)`\n\n223+ `}`\n\n224+ `_, cleanup, err := BuildOfflinePromptContext(OfflinePromptInputs{`\n\n225+ `DatasetPath: dataset, TaskID: taskID, AgentPatchPath: patch,`\n\n226+ `ExpectedArtifacts: &original, RequireRepoContext: true,`\n\n227+ `})`\n\n228+ `if cleanup != nil {`\n\n229+ `defer cleanup()`\n\n230+ `}`\n\n231+ `if err == nil || !strings.Contains(err.Error(), artifact+\" provenance mismatch\") {`\n\n232+ `t.Fatalf(\"expected original %s mismatch, got %v\", artifact, err)`\n\n233+ `}`\n\n234+ `})`\n\n235+ `}`\n\n236+`}`\n\n237+\n\n238+`func TestBuildOfflinePromptContextRejectsAbsentOriginalArtifactDigest(t *testing.T) {`\n\n239+ `dataset, taskID, patch := writeOfflineTaskDir(t, true)`\n\n240+ `inputs, err := ReadTaskInputs(dataset, taskID, patch)`\n\n241+ `if err != nil {`\n\n242+ `t.Fatal(err)`\n\n243+ `}`\n\n244+ `original := BuildArtifactManifest(inputs)`\n\n245+ `original.AgentPatch.SHA256 = \"\"`\n\n246+ `_, cleanup, err := BuildOfflinePromptContext(OfflinePromptInputs{`\n\n247+ `DatasetPath: dataset, TaskID: taskID, AgentPatchPath: patch, ExpectedArtifacts: &original,`\n\n248+ `})`\n\n249+ `if cleanup != nil {`\n\n250+ `defer cleanup()`\n\n251+ `}`\n\n252+ `if err == nil || !strings.Contains(err.Error(), \"missing original agent patch\") {`\n\n253+ `t.Fatalf(\"expected absent original provenance failure, got %v\", err)`\n\n254+ `}`\n\n255+`}`\n\n256+\n\n257+`func TestBuildOfflinePromptContextRejectsNewTruncationWithOriginalProvenance(t *testing.T) {`\n\n258+ `dataset, taskID, patch := writeOfflineTaskDir(t, true)`\n\n259+ `if err := os.WriteFile(patch, []byte(largePatch(4_000)), 0o644); err != nil {`\n\n260+ `t.Fatal(err)`\n\n261+ `}`\n\n262+ `inputs, err := ReadTaskInputs(dataset, taskID, patch)`\n\n263+ `if err != nil {`\n\n264+ `t.Fatal(err)`\n\n265+ `}`\n\n266+ `original := BuildArtifactManifest(inputs)`\n\n267+ `_, cleanup, err := BuildOfflinePromptContext(OfflinePromptInputs{`\n\n268+ `DatasetPath: dataset, TaskID: taskID, AgentPatchPath: patch, ExpectedArtifacts: &original,`\n\n269+ `PromptPatchMaxBytes: 600,`\n\n270+ `})`\n\n271+ `if cleanup != nil {`\n\n272+ `defer cleanup()`\n\n273+ `}`\n\n274+ `if err == nil || !strings.Contains(err.Error(), \"would be truncated\") {`\n\n275+ `t.Fatalf(\"expected new truncation refusal, got %v\", err)`\n\n276+ `}`\n\n277+`}`\n\n…`const (`\n\n2828`var coverageRepairRetryBackoffs = []time.Duration{time.Second, 2 * time.Second}`\n\n2929\n\n3030`type CoverageRepairConfig struct {`\n\n31+ `EquivalenceAI            AIClient`\n\n32+ `EquivalencePrompt        string`\n\n33+ `EquivalencePromptContext *PromptContext`\n\n34+ `EquivalencePromptMode    EquivalencePromptMode`\n\n35+ `ReviewAI                 AIClient`\n\n36+ `ReviewPrompt             string`\n\n37+ `ReviewPromptContext      *PromptContext`\n\n38+\n\n3139 `Existing           ValidationResult`\n\n3240 `AI                 AIClient`\n\n3341 `AIModelID          string`\n\n…`type CoverageRepairConfig struct {`\n\n3745`}`\n\n3846\n\n3947`type CoverageRepairIssue struct {`\n\n48+ `` AttemptCount int      `json:\"attempt_count,omitempty\"` ``\n\n49+ `` RetryCount   int      `json:\"retry_count,omitempty\"` ``\n\n50+ `` RetryReasons []string `json:\"retry_reasons,omitempty\"` ``\n\n51+\n\n4052 `` Dimension string `json:\"dimension\"` ``\n\n4153 `` Reason    string `json:\"reason\"` ``\n\n4254 `` Category  string `json:\"category,omitempty\"` ``\n\n…`func RepairCoverage(ctx context.Context, cfg CoverageRepairConfig) (CoverageRepa`\n\n6981 `updated := false`\n\n7082\n\n7183 `if cfg.RepairEquivalence {`\n\n72− `equivalence, repaired, issue := repairEquivalenceFromPrompt(ctx, ai, cfg.Existing.Equivalence, cfg.AIModelID, cfg.ReviewParseRetries)`\n\n84+ `source := cfg.Existing.Equivalence`\n\n85+ `if cfg.EquivalencePrompt != \"\" {`\n\n86+ `source.Prompt = cfg.EquivalencePrompt`\n\n87+ `}`\n\n88+ `mode := source.PromptMode`\n\n89+ `if cfg.EquivalencePromptMode != \"\" {`\n\n90+ `if mode != \"\" && mode != cfg.EquivalencePromptMode {`\n\n91+ `return result, errors.New(\"equivalence repair prompt mode differs from original measurement\")`\n\n92+ `}`\n\n93+ `mode = cfg.EquivalencePromptMode`\n\n94+ `}`\n\n95+ `if cfg.EquivalencePromptContext != nil {`\n\n96+ `rendered, err := RenderEquivalencePromptForMode(*cfg.EquivalencePromptContext, mode)`\n\n97+ `if err != nil {`\n\n98+ `return result, err`\n\n99+ `}`\n\n100+ `if cfg.EquivalencePrompt != \"\" && cfg.EquivalencePrompt != rendered {`\n\n101+ `return result, errors.New(\"equivalence repair prompt differs from supplied context\")`\n\n102+ `}`\n\n103+ `source.Prompt = rendered`\n\n104+ `}`\n\n105+ `source.PromptMode = mode`\n\n106+ `dimensionAI := cfg.EquivalenceAI`\n\n107+ `if dimensionAI == nil {`\n\n108+ `dimensionAI = ai`\n\n109+ `}`\n\n110+ `history := newGradeAttemptClient(dimensionAI)`\n\n111+ `equivalence, repaired, issue := repairEquivalenceFromPrompt(ctx, history, source, cfg.AIModelID, cfg.ReviewParseRetries)`\n\n112+ `if repaired {`\n\n113+ `equivalence.PromptMode = mode`\n\n114+ `if cfg.EquivalencePromptContext != nil {`\n\n115+ `pc := *cfg.EquivalencePromptContext`\n\n116+ `if mode == EquivalencePromptModeGoldBlind {`\n\n117+ `pc.GoldPatchTruncated = false`\n\n118+ `}`\n\n119+ `equivalence = annotateEquivalenceTruncation(equivalence, pc)`\n\n120+ `}`\n\n121+ `history.annotateEquivalence(&equivalence)`\n\n122+ `mergeEquivalenceRepairHistory(&equivalence, source)`\n\n123+ `}`\n\n73124 `if issue != nil {`\n\n125+ `issue.AttemptCount = history.attempts`\n\n126+ `issue.RetryCount = max(0, history.attempts-1)`\n\n127+ `issue.RetryReasons = append([]string(nil), history.reasons...)`\n\n74128 `result.Unresolved = append(result.Unresolved, *issue)`\n\n75129 `} else {`\n\n76130 `result.Validation.Equivalence = equivalence`\n\n…`func RepairCoverage(ctx context.Context, cfg CoverageRepairConfig) (CoverageRepa`\n\n84138 `if strings.TrimSpace(source.Prompt) == \"\" && cfg.Existing.Review != nil {`\n\n85139 `source = *cfg.Existing.Review`\n\n86140 `}`\n\n87− `review, repaired, issue := repairReviewFromPrompt(ctx, ai, source, cfg.AIModelID, cfg.ReviewParseRetries)`\n\n141+ `if cfg.ReviewPrompt != \"\" {`\n\n142+ `source.Prompt = cfg.ReviewPrompt`\n\n143+ `}`\n\n144+ `if cfg.ReviewPromptContext != nil {`\n\n145+ `rendered, err := renderPrompt(reviewPromptTemplate, cfg.ReviewPromptContext.asTemplateData())`\n\n146+ `if err != nil {`\n\n147+ `return result, err`\n\n148+ `}`\n\n149+ `if cfg.ReviewPrompt != \"\" && cfg.ReviewPrompt != rendered {`\n\n150+ `return result, errors.New(\"review repair prompt differs from supplied context\")`\n\n151+ `}`\n\n152+ `source.Prompt = rendered`\n\n153+ `}`\n\n154+ `if source.Prompt != resolvedRepairReview(cfg.Existing).Prompt {`\n\n155+ `source.Output = \"\"`\n\n156+ `}`\n\n157+ `dimensionAI := cfg.ReviewAI`\n\n158+ `if dimensionAI == nil {`\n\n159+ `dimensionAI = ai`\n\n160+ `}`\n\n161+ `history := newGradeAttemptClient(dimensionAI)`\n\n162+ `review, repaired, issue := repairReviewFromPromptContext(ctx, history, source, cfg.AIModelID, cfg.ReviewParseRetries, cfg.ReviewPromptContext)`\n\n163+ `if repaired && history.attempts > 0 {`\n\n164+ `history.annotateReview(&review)`\n\n165+ `mergeReviewRepairHistory(&review, source)`\n\n166+ `}`\n\n167+ `if repaired && cfg.ReviewPromptContext != nil {`\n\n168+ `review = annotateReviewTruncation(review, *cfg.ReviewPromptContext)`\n\n169+ `}`\n\n88170 `if issue != nil {`\n\n171+ `issue.AttemptCount = history.attempts`\n\n172+ `issue.RetryCount = max(0, history.attempts-1)`\n\n173+ `issue.RetryReasons = append([]string(nil), history.reasons...)`\n\n89174 `result.Unresolved = append(result.Unresolved, *issue)`\n\n90175 `} else {`\n\n91176 `result.Validation.CodeReview = review`\n\n…`func RepairCoverage(ctx context.Context, cfg CoverageRepairConfig) (CoverageRepa`\n\n104189`}`\n\n105190\n\n106191`func repairEquivalenceFromPrompt(ctx context.Context, ai AIClient, existing EquivalenceResult, overrideModelID string, parseRetries int) (EquivalenceResult, bool, *CoverageRepairIssue) {`\n\n107− `prompt := strings.TrimSpace(existing.Prompt)`\n\n108− `if prompt == \"\" {`\n\n192+ `prompt := existing.Prompt`\n\n193+ `if strings.TrimSpace(prompt) == \"\" {`\n\n109194 `return existing, false, &CoverageRepairIssue{`\n\n110195 `Dimension: coverageRepairDimensionEquivalence,`\n\n111196 `Reason:    \"missing_prompt\",`\n\n…`func repairEquivalenceFromPrompt(ctx context.Context, ai AIClient, existing Equi`\n\n145230\n\n146231 `parsed.Status = AIStatusUsed`\n\n147232 `parsed.EvaluatorModel = resolveRepairEvaluatorModel(existing.EvaluatorModel, overrideModelID)`\n\n233+ `parsed.PromptMode = existing.PromptMode`\n\n234+ `parsed.Truncated = existing.Truncated`\n\n235+ `parsed.TruncationReason = existing.TruncationReason`\n\n148236 `parsed.Prompt = prompt`\n\n149237 `parsed.Output = output`\n\n150238 `return parsed, true, nil`\n\n…`func reviewVerdictIsGuardSuppressible(result ReviewResult) bool {`\n\n159247`}`\n\n160248\n\n161249`func repairReviewFromPrompt(ctx context.Context, ai AIClient, existing ReviewResult, overrideModelID string, parseRetries int) (ReviewResult, bool, *CoverageRepairIssue) {`\n\n162− `prompt := strings.TrimSpace(existing.Prompt)`\n\n163− `if prompt == \"\" {`\n\n250+ `return repairReviewFromPromptContext(ctx, ai, existing, overrideModelID, parseRetries, nil)`\n\n251+`}`\n\n252+\n\n253+`func repairReviewFromPromptContext(ctx context.Context, ai AIClient, existing ReviewResult, overrideModelID string, parseRetries int, promptCtx *PromptContext) (ReviewResult, bool, *CoverageRepairIssue) {`\n\n254+ `prompt := existing.Prompt`\n\n255+ `if strings.TrimSpace(prompt) == \"\" {`\n\n164256 `return existing, false, &CoverageRepairIssue{`\n\n165257 `Dimension: coverageRepairDimensionCodeReview,`\n\n166258 `Reason:    \"missing_prompt\",`\n\n…`func repairReviewFromPrompt(ctx context.Context, ai AIClient, existing ReviewRes`\n\n174266 `// re-apply that guard, so such verdicts must be resampled, never recovered.`\n\n175267 `if existing.Status == AIStatusUnavailable && strings.TrimSpace(existing.Output) != \"\" {`\n\n176268 `if reparsed, reparseErr := parseReviewOutput(existing.Output); reparseErr == nil &&`\n\n177− `!reviewVerdictIsGuardSuppressible(reparsed) {`\n\n269+ `((promptCtx == nil && !reviewVerdictIsGuardSuppressible(reparsed)) || (promptCtx != nil && validateReviewAgainstPromptContext(reparsed, *promptCtx) == nil)) {`\n\n178270 `reparsed.Status = AIStatusUsed`\n\n179271 `reparsed.Prompt = prompt`\n\n180272 `reparsed.Output = existing.Output`\n\n…`func repairReviewFromPrompt(ctx context.Context, ai AIClient, existing ReviewRes`\n\n184276 `if reparsed.EvaluatorModel == \"\" {`\n\n185277 `reparsed.EvaluatorModel = strings.TrimSpace(overrideModelID)`\n\n186278 `}`\n\n279+ `reparsed.AttemptCount = existing.AttemptCount`\n\n280+ `reparsed.RetryCount = existing.RetryCount`\n\n281+ `reparsed.RetryReasons = append([]string(nil), existing.RetryReasons...)`\n\n282+ `reparsed.RecoveredAfterRetry = existing.RecoveredAfterRetry`\n\n187283 `reparsed.Truncated = existing.Truncated`\n\n188284 `reparsed.TruncationReason = existing.TruncationReason`\n\n189285 `return reparsed, true, nil`\n\n…`func repairReviewFromPrompt(ctx context.Context, ai AIClient, existing ReviewRes`\n\n196292 `}`\n\n197293\n\n198294 `parsed, parseErr := parseReviewOutput(output)`\n\n295+ `if parseErr == nil && promptCtx != nil {`\n\n296+ `parseErr = validateReviewAgainstPromptContext(parsed, *promptCtx)`\n\n297+ `}`\n\n199298 `for attempt := 0; parseErr != nil && attempt < normalizedReviewParseRetries(parseRetries); attempt++ {`\n\n200299 `retryPrompt := buildReviewRetryPrompt(prompt, output, parseErr)`\n\n201300 `retryOutput, retryErr := runAIWithRetries(ctx, ai, retryPrompt)`\n\n…`func repairReviewFromPrompt(ctx context.Context, ai AIClient, existing ReviewRes`\n\n203302 `return existing, false, newReviewRuntimeFailureIssue(retryErr)`\n\n204303 `}`\n\n205304 `parsed, parseErr = parseReviewOutput(retryOutput)`\n\n305+ `if parseErr == nil && promptCtx != nil {`\n\n306+ `parseErr = validateReviewAgainstPromptContext(parsed, *promptCtx)`\n\n307+ `}`\n\n206308 `output = retryOutput`\n\n207309 `}`\n\n208310 `if parseErr != nil {`\n\n209311 `return existing, false, classifyReviewParseIssue(parseErr)`\n\n210312 `}`\n\n211313\n\n314+ `if promptCtx == nil && reviewVerdictIsGuardSuppressible(parsed) {`\n\n315+ `return existing, false, classifyReviewParseIssue(errors.New(\"review repair requires original verifier context for speculative blocker verdict\"))`\n\n316+ `}`\n\n317+ `parsed.Truncated = existing.Truncated`\n\n318+ `parsed.TruncationReason = existing.TruncationReason`\n\n212319 `parsed.Status = AIStatusUsed`\n\n213320 `parsed.Prompt = prompt`\n\n214321 `parsed.Output = output`\n\n…`func repairReviewFromPrompt(ctx context.Context, ai AIClient, existing ReviewRes`\n\n217324`}`\n\n218325\n\n219326`func runAIWithRetries(ctx context.Context, ai AIClient, prompt string) (string, error) {`\n\n220− `var lastErr error`\n\n221− `for attempt := 0; attempt <= len(coverageRepairRetryBackoffs); attempt++ {`\n\n222− `output, err := ai.Run(ctx, prompt)`\n\n223− `if err == nil {`\n\n224− `return output, nil`\n\n225− `}`\n\n226− `lastErr = err`\n\n227− `if errors.Is(err, ErrAIDisabled) || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {`\n\n228− `break`\n\n229− `}`\n\n230− `if attempt >= len(coverageRepairRetryBackoffs) {`\n\n231− `break`\n\n232− `}`\n\n233− `select {`\n\n234− `case <-ctx.Done():`\n\n235− `return \"\", ctx.Err()`\n\n236− `case <-time.After(coverageRepairRetryBackoffs[attempt]):`\n\n237− `}`\n\n327+ `if history, ok := ai.(*gradeAttemptClient); ok {`\n\n328+ `return history.Run(ctx, prompt)`\n\n238329 `}`\n\n239− `return \"\", lastErr`\n\n330+ `return newGradeAttemptClient(ai).Run(ctx, prompt)`\n\n240331`}`\n\n241332\n\n242333`func normalizedReviewParseRetries(parseRetries int) int {`\n\n…`func appendReviewNote(existing, extra string) string {`\n\n5353 `}`\n\n5454`}`\n\n5555\n\n56−`func RunReview(ctx context.Context, ai AIClient, promptCtx PromptContext) ReviewResult {`\n\n56+`func RunReview(ctx context.Context, ai AIClient, promptCtx PromptContext) (result ReviewResult) {`\n\n57+ `history := newGradeAttemptClient(ai)`\n\n58+ `defer func() { history.annotateReview(&result) }()`\n\n59+ `ai = history`\n\n60+\n\n5761 `return annotateReviewTruncation(runReview(ctx, ai, promptCtx), promptCtx)`\n\n5862`}`\n\n5963\n\n…`type TestVerifierEvidence struct {`\n\n712712`}`\n\n713713\n\n714714`type TestCommandResult struct {`\n\n715− `` Command           string             `json:\"command\"` ``\n\n716− `` GoldOutcome       TestOutcome        `json:\"gold_outcome,omitempty\"` ``\n\n717− `` GoldTrialName     string             `json:\"gold_tb_trial_name,omitempty\"` ``\n\n718− `` GoldFailureMode   string             `json:\"gold_failure_mode,omitempty\"` ``\n\n719− `` GoldRetried       bool               `json:\"gold_retried,omitempty\"` ``\n\n720− `` GoldRetryEvidence *TestRetryEvidence `json:\"gold_retry_evidence,omitempty\"` ``\n\n721− `` AgentOutcome      TestOutcome        `json:\"agent_outcome,omitempty\"` ``\n\n722− `` AgentTrialName    string             `json:\"agent_tb_trial_name,omitempty\"` ``\n\n723− `` AgentFailureMode  string             `json:\"agent_failure_mode,omitempty\"` ``\n\n724− `` AgentRetried      bool               `json:\"agent_retried,omitempty\"` ``\n\n715+ `// Counts cover dispatched gold and candidate verifier calls for this command.`\n\n716+ `` AttemptCount        int                `json:\"attempt_count,omitempty\"` ``\n\n717+ `` RetryCount          int                `json:\"retry_count,omitempty\"` ``\n\n718+ `` RetryReasons        []string           `json:\"retry_reasons,omitempty\"` ``\n\n719+ `` RecoveredAfterRetry bool               `json:\"recovered_after_retry,omitempty\"` ``\n\n720+ `` Command             string             `json:\"command\"` ``\n\n721+ `` GoldOutcome         TestOutcome        `json:\"gold_outcome,omitempty\"` ``\n\n722+ `` GoldTrialName       string             `json:\"gold_tb_trial_name,omitempty\"` ``\n\n723+ `` GoldFailureMode     string             `json:\"gold_failure_mode,omitempty\"` ``\n\n724+ `` GoldRetried         bool               `json:\"gold_retried,omitempty\"` ``\n\n725+ `` GoldRetryEvidence   *TestRetryEvidence `json:\"gold_retry_evidence,omitempty\"` ``\n\n726+ `` AgentOutcome        TestOutcome        `json:\"agent_outcome,omitempty\"` ``\n\n727+ `` AgentTrialName      string             `json:\"agent_tb_trial_name,omitempty\"` ``\n\n728+ `` AgentFailureMode    string             `json:\"agent_failure_mode,omitempty\"` ``\n\n729+ `` AgentRetried        bool               `json:\"agent_retried,omitempty\"` ``\n\n725730`}`\n\n726731\n\n727732`// TestRetryEvidence records the two verifier attempts that authorized and`\n\n…`type TestRetryAttemptEvidence struct {`\n\n742747`}`\n\n743748\n\n744749`type EquivalenceResult struct {`\n\n750+ `` AttemptCount        int      `json:\"attempt_count,omitempty\"` ``\n\n751+ `` RetryCount          int      `json:\"retry_count,omitempty\"` ``\n\n752+ `` RetryReasons        []string `json:\"retry_reasons,omitempty\"` ``\n\n753+ `` RecoveredAfterRetry bool     `json:\"recovered_after_retry,omitempty\"` ``\n\n754+\n\n745755 `` Outcome EquivalenceOutcome `json:\"outcome\"` ``\n\n746756 `` Status  AIStatus           `json:\"status\"` ``\n\n747757 `// PromptMode records the exact built-in equivalence template variant that`\n\n…`type EquivalenceObligation struct {`\n\n785795`}`\n\n786796\n\n787797`type ReviewResult struct {`\n\n798+ `` AttemptCount        int      `json:\"attempt_count,omitempty\"` ``\n\n799+ `` RetryCount          int      `json:\"retry_count,omitempty\"` ``\n\n800+ `` RetryReasons        []string `json:\"retry_reasons,omitempty\"` ``\n\n801+ `` RecoveredAfterRetry bool     `json:\"recovered_after_retry,omitempty\"` ``\n\n802+\n\n788803 `` Status                 AIStatus                     `json:\"status\"` ``\n\n789804 `` SchemaVersion          string                       `json:\"schema_version,omitempty\"` ``\n\n790805 `` ProfileID              string                       `json:\"profile_id,omitempty\"` ``\n\n…`Recovery rules:`\n\n793793 `Those repair steps preserve compare-critical metadata and existing custom`\n\n794794 `grader surfaces so the arm stays compare-compatible while recovering`\n\n795795 ``coverage. Add `--parse-retries N` when saved grader prompts failed``\n\n796− `JSON/schema parsing. Keep the original custom rubric file path on the regrade`\n\n797− `command.`\n\n796+ `JSON/schema parsing. Repair preserves recorded prompt mode, truncation, and`\n\n797+ `attempt history. Missing prompts require matching original task and artifact`\n\n798+ `provenance plus complete repository context; ambiguous reconstruction remains`\n\n799+ `unresolved. Runtime retries cover recognized transient failures only.`\n\n800+ `Keep the original custom rubric file path on the regrade command.`\n\n798801`` - If `stet runs repair-ai-coverage --cr-only` still leaves `code_review` ``\n\n799802 ``unresolved, use the summary's stable `unresolved[].reason` and optional``\n\n800803 `` `category`/` detail` fields to separate `model_output_shape`, ``\n\n…\n\n1+`package commandlist`\n\n2+\n\n3+`import (`\n\n4+ `\"path\"`\n\n5+ `\"regexp\"`\n\n6+ `\"strings\"`\n\n7+\n\n8+ `\"stet/internal/shellcmd\"`\n\n9+`)`\n\n10+\n\n11+``var assignment = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*=`)``\n\n12+``var duration = regexp.MustCompile(`^[0-9]+(?:\\.[0-9]+)?[smhd]?$`)``\n\n13+\n\n14+`// TestInvocation identifies exactly one known test command. The original text`\n\n15+`// is retained, including literal environment assignments and a relative cwd.`\n\n16+`// Opaque scripts and shell evaluation are deliberately not interpreted.`\n\n17+`func TestInvocation(command string) (string, []shellcmd.FieldSpan, bool) {`\n\n18+ `command = strings.TrimSpace(command)`\n\n19+ ``if command == \"\" || strings.ContainsAny(command, \"\\n\\r`$\") {``\n\n20+ `return \"\", nil, false`\n\n21+ `}`\n\n22+ `spans, ok := shellcmd.FieldSpans(command)`\n\n23+ `if !ok || len(spans) == 0 {`\n\n24+ `return \"\", nil, false`\n\n25+ `}`\n\n26+ `i := 0`\n\n27+ `if spans[0].Value == \"cd\" {`\n\n28+ `if len(spans) < 4 || spans[2].Value != \"&&\" || !relativeDirectory(spans[1].Value) {`\n\n29+ `return \"\", nil, false`\n\n30+ `}`\n\n31+ `if unsafeShellControl(command[:spans[2].Start]) {`\n\n32+ `return \"\", nil, false`\n\n33+ `}`\n\n34+ `i = 3`\n\n35+ `}`\n\n36+ `if i >= len(spans) || unsafeShellControl(command[spans[i].Start:]) {`\n\n37+ `return \"\", nil, false`\n\n38+ `}`\n\n39+ `// The sole supported control operator belongs to the proven cd prefix.`\n\n40+ `for _, span := range spans[i:] {`\n\n41+ `if span.Value == \"&&\" || span.Value == \"&\" {`\n\n42+ `return \"\", nil, false`\n\n43+ `}`\n\n44+ `}`\n\n45+ `for i < len(spans) && assignment.MatchString(spans[i].Value) {`\n\n46+ `if !assignment.MatchString(command[spans[i].Start:spans[i].End]) {`\n\n47+ `return \"\", nil, false`\n\n48+ `}`\n\n49+ `i++`\n\n50+ `}`\n\n51+ `if i < len(spans) && spans[i].Value == \"env\" {`\n\n52+ `i++`\n\n53+ `for i < len(spans) && assignment.MatchString(spans[i].Value) {`\n\n54+ `i++`\n\n55+ `}`\n\n56+ `}`\n\n57+ `if i < len(spans) && spans[i].Value == \"timeout\" {`\n\n58+ `i++`\n\n59+ `if i >= len(spans) || !duration.MatchString(spans[i].Value) {`\n\n60+ `return \"\", nil, false`\n\n61+ `}`\n\n62+ `i++`\n\n63+ `}`\n\n64+ `if i >= len(spans) {`\n\n65+ `return \"\", nil, false`\n\n66+ `}`\n\n67+ `fields := spans[i:]`\n\n68+ `value := func(n int) string {`\n\n69+ `if n >= len(fields) {`\n\n70+ `return \"\"`\n\n71+ `}`\n\n72+ `return fields[n].Value`\n\n73+ `}`\n\n74+ `switch value(0) {`\n\n75+ `case \"go\", \"cargo\":`\n\n76+ `ok = value(1) == \"test\"`\n\n77+ `case \"sh\", \"bash\":`\n\n78+ `ok = len(fields) == 2 && strings.HasSuffix(value(1), \".sh\") && relativeDirectory(value(1))`\n\n79+ `case \"pytest\", \"pytest-3\", \"vitest\", \"jest\", \"phpunit\":`\n\n80+ `ok = true`\n\n81+ `case \"python\", \"python3\":`\n\n82+ `ok = value(1) == \"-m\" && value(2) == \"pytest\"`\n\n83+ `case \"npm\", \"pnpm\", \"yarn\":`\n\n84+ `ok = value(1) == \"test\" || (value(1) == \"run\" && value(2) == \"test\")`\n\n85+ `case \"npx\":`\n\n86+ `ok = value(1) == \"vitest\" || value(1) == \"jest\"`\n\n87+ `default:`\n\n88+ `ok = false`\n\n89+ `}`\n\n90+ `if !ok {`\n\n91+ `return \"\", nil, false`\n\n92+ `}`\n\n93+ `return command, fields, true`\n\n94+`}`\n\n95+\n\n96+`func relativeDirectory(directory string) bool {`\n\n97+ `if directory == \"\" || strings.HasPrefix(directory, \"-\") || strings.Contains(directory, \"\\\\\") || path.IsAbs(directory) {`\n\n98+ `return false`\n\n99+ `}`\n\n100+ `clean := path.Clean(directory)`\n\n101+ `return clean != \"..\" && !strings.HasPrefix(clean, \"../\")`\n\n102+`}`\n\n103+\n\n104+`func unsafeShellControl(command string) bool {`\n\n105+ `var quote rune`\n\n106+ `escaped := false`\n\n107+ `for _, r := range command {`\n\n108+ `if escaped {`\n\n109+ `escaped = false`\n\n110+ `continue`\n\n111+ `}`\n\n112+ `if quote == '\\'' {`\n\n113+ `if r == '\\'' {`\n\n114+ `quote = 0`\n\n115+ `}`\n\n116+ `continue`\n\n117+ `}`\n\n118+ `if r == '\\\\' {`\n\n119+ `escaped = true`\n\n120+ `continue`\n\n121+ `}`\n\n122+ `if quote == '\"' {`\n\n123+ `if r == '\"' {`\n\n124+ `quote = 0`\n\n125+ `}`\n\n126+ `continue`\n\n127+ `}`\n\n128+ `if r == '\\'' || r == '\"' {`\n\n129+ `quote = r`\n\n130+ `continue`\n\n131+ `}`\n\n132+ `if strings.ContainsRune(\";|&<>#()\", r) {`\n\n133+ `return true`\n\n134+ `}`\n\n135+ `}`\n\n136+ `return false`\n\n137+`}`\n\n…\n\n1+`package commandlist`\n\n2+\n\n3+`import \"testing\"`\n\n4+\n\n5+`func TestTestInvocationPreservesProvenExecutionContext(t *testing.T) {`\n\n6+ `for _, command := range []string{`\n\n7+ `\"go test ./pkg -run 'TestOne|TestTwo' -count=1\",`\n\n8+ `\"cd packages/api && FOO='a b' env BAR=2 timeout 30s python3 -m pytest tests/test_api.py -q\",`\n\n9+ `\"npm run test -- --runInBand\", \"env CI=1 cargo test --locked\", \"pytest 'tests/test space.py'\",`\n\n10+ `} {`\n\n11+ `got, fields, ok := TestInvocation(command)`\n\n12+ `if !ok || got != command || len(fields) == 0 {`\n\n13+ `t.Errorf(\"TestInvocation(%q) = %q, %v\", command, got, ok)`\n\n14+ `}`\n\n15+ `}`\n\n16+`}`\n\n17+\n\n18+`func TestTestInvocationRejectsAmbiguousAndUnsafeWrappers(t *testing.T) {`\n\n19+ `for _, command := range []string{`\n\n20+ `\"echo go test ./pkg\", \"sh -c 'go test ./pkg'\", \"go test ./pkg && pytest tests\",`\n\n21+ `\"go test ./pkg; true\", \"go test ./pkg || true\", \"cd ../outside && go test ./pkg\",`\n\n22+ `\"env -i go test ./pkg\", \"timeout --signal=KILL 30 go test ./pkg\", \"FOO=$(touch sentinel) go test ./pkg\",`\n\n23+ `\"go test ./pkg > result\", \"cd pkg&&go test ./pkg\", \"go test ./pkg &\", \"make test\", \"\\\"CI=1\\\" go test ./pkg\", \"F\\\"OO\\\"=x go test ./pkg\",`\n\n24+ `\"cd pkg;touch && go test ./pkg\", \"echo tests/agent.sh\", \"cat tests/agent.sh\", \"rm tests/agent.sh\", \"sh -c tests/agent.sh\",`\n\n25+ `} {`\n\n26+ `if _, _, ok := TestInvocation(command); ok {`\n\n27+ `t.Errorf(\"accepted %q\", command)`\n\n28+ `}`\n\n29+ `}`\n\n30+`}`\n\n…\n\n1+`package h2h`\n\n2+\n\n3+`import (`\n\n4+ `\"fmt\"`\n\n5+ `\"path/filepath\"`\n\n6+ `\"strings\"`\n\n7+\n\n8+ `\"stet/internal/validate\"`\n\n9+`)`\n\n10+\n\n11+`// prepareCoverageRepairPrompts reconstructs only absent prompts. Stored prompts`\n\n12+`// retain their original evidence; reconstructed prompts use the sealed task`\n\n13+`// source and keep its extracted repository alive through grading.`\n\n14+`func prepareCoverageRepairPrompts(cfg *validate.CoverageRepairConfig, manifest *Manifest, artifacts TaskArtifacts) (func(), []validate.CoverageRepairIssue) {`\n\n15+ `missingEq := cfg.RepairEquivalence && strings.TrimSpace(cfg.Existing.Equivalence.Prompt) == \"\"`\n\n16+ `review := cfg.Existing.CodeReview`\n\n17+ `if strings.TrimSpace(review.Prompt) == \"\" && cfg.Existing.Review != nil {`\n\n18+ `review = *cfg.Existing.Review`\n\n19+ `}`\n\n20+ `missingReview := cfg.RepairCodeReview && strings.TrimSpace(review.Prompt) == \"\"`\n\n21+ `if !missingEq && !missingReview {`\n\n22+ `return nil, nil`\n\n23+ `}`\n\n24+ `var issues []validate.CoverageRepairIssue`\n\n25+ `fail := func(dimension string, err error) {`\n\n26+ `detail := \"prompt reconstruction failed: \" + err.Error()`\n\n27+ `if len(detail) > 240 {`\n\n28+ `detail = detail[:240]`\n\n29+ `}`\n\n30+ `issues = append(issues, validate.CoverageRepairIssue{Dimension: dimension, Reason: \"missing_prompt\", Detail: detail, Cause: err})`\n\n31+ `if dimension == TaskGraderEquivalence {`\n\n32+ `cfg.RepairEquivalence = false`\n\n33+ `} else {`\n\n34+ `cfg.RepairCodeReview = false`\n\n35+ `}`\n\n36+ `}`\n\n37+ `mode := cfg.Existing.Equivalence.PromptMode`\n\n38+ `if missingEq && mode != validate.EquivalencePromptModeStandard && mode != validate.EquivalencePromptModeGoldBlind {`\n\n39+ `fail(TaskGraderEquivalence, fmt.Errorf(\"missing or unsupported equivalence prompt mode %q\", mode))`\n\n40+ `missingEq = false`\n\n41+ `}`\n\n42+ `if !missingEq && !missingReview {`\n\n43+ `return nil, issues`\n\n44+ `}`\n\n45+ `truncation := cfg.Existing.PromptTruncation`\n\n46+ `if truncation.AgentPatch.Truncated || truncation.GoldPatch.Truncated || truncation.RepoTree.Truncated || truncation.RepoExcerpts.Truncated || cfg.Existing.Equivalence.Truncated || review.Truncated {`\n\n47+ `err := fmt.Errorf(\"original prompt was truncated; original context budgets are unavailable\")`\n\n48+ `if missingEq {`\n\n49+ `fail(TaskGraderEquivalence, err)`\n\n50+ `}`\n\n51+ `if missingReview {`\n\n52+ `fail(TaskGraderCodeReview, err)`\n\n53+ `}`\n\n54+ `return nil, issues`\n\n55+ `}`\n\n56+ `dataset := manifest.Dataset`\n\n57+ `goldPath := strings.TrimSpace(cfg.Existing.ArtifactsManifest.GoldPatch.Path)`\n\n58+ `if goldPath != \"\" {`\n\n59+ `taskDir := filepath.Dir(goldPath)`\n\n60+ `if filepath.Base(taskDir) != artifacts.TaskID {`\n\n61+ `err := fmt.Errorf(\"gold patch directory does not match task %q\", artifacts.TaskID)`\n\n62+ `if missingEq {`\n\n63+ `fail(TaskGraderEquivalence, err)`\n\n64+ `}`\n\n65+ `if missingReview {`\n\n66+ `fail(TaskGraderCodeReview, err)`\n\n67+ `}`\n\n68+ `return nil, issues`\n\n69+ `}`\n\n70+ `dataset = filepath.Dir(taskDir)`\n\n71+ `}`\n\n72+ `if err := verifyCoverageRepairTaskAuthority(manifest, artifacts, dataset); err != nil {`\n\n73+ `if missingEq {`\n\n74+ `fail(TaskGraderEquivalence, err)`\n\n75+ `}`\n\n76+ `if missingReview {`\n\n77+ `fail(TaskGraderCodeReview, err)`\n\n78+ `}`\n\n79+ `return nil, issues`\n\n80+ `}`\n\n81+ `resolved, _, _, err := resolveTaskArtifactsFromResults(artifacts)`\n\n82+ `if err == nil {`\n\n83+ `artifacts = resolved`\n\n84+ `}`\n\n85+ `patchPath := strings.TrimSpace(cfg.Existing.ArtifactsManifest.AgentPatch.Path)`\n\n86+ `if patchPath == \"\" {`\n\n87+ `patchPath = artifacts.ResolveAgentPatchPath()`\n\n88+ `}`\n\n89+ `promptCtx, cleanup, err := validate.BuildOfflinePromptContext(validate.OfflinePromptInputs{`\n\n90+ `DatasetPath: dataset, TaskID: artifacts.TaskID, AgentPatchPath: patchPath,`\n\n91+ `TBResultsRaw: validate.ReadTBResultsRaw(artifacts.ResultsPath(), artifacts.TaskID), RequireRepoContext: true,`\n\n92+ `EffectiveTestCommands: append([]string{}, cfg.Existing.Tests.Commands...),`\n\n93+ `ExpectedArtifacts:     &cfg.Existing.ArtifactsManifest,`\n\n94+ `RepoContextOptions:    validate.RepoContextOptions{ExcludePrefixes: cfg.Existing.ContextExcludePaths},`\n\n95+ `})`\n\n96+ `if err != nil {`\n\n97+ `if missingEq {`\n\n98+ `fail(TaskGraderEquivalence, err)`\n\n99+ `}`\n\n100+ `if missingReview {`\n\n101+ `fail(TaskGraderCodeReview, err)`\n\n102+ `}`\n\n103+ `return nil, issues`\n\n104+ `}`\n\n105+ `if strings.TrimSpace(promptCtx.AgentPatch) == \"\" || strings.TrimSpace(promptCtx.TaskYAML) == \"\" {`\n\n106+ `err := fmt.Errorf(\"task YAML and agent patch are required for prompt reconstruction\")`\n\n107+ `if missingEq {`\n\n108+ `fail(TaskGraderEquivalence, err)`\n\n109+ `}`\n\n110+ `if missingReview {`\n\n111+ `fail(TaskGraderCodeReview, err)`\n\n112+ `}`\n\n113+ `return cleanup, issues`\n\n114+ `}`\n\n115+ `client := cfg.AI`\n\n116+ `if shell, ok := client.(validate.ShellAIClient); ok {`\n\n117+ `shell.Dir = promptCtx.RepoPath`\n\n118+ `client = shell`\n\n119+ `}`\n\n120+ `if missingEq {`\n\n121+ `prompt, err := validate.RenderEquivalencePromptForMode(promptCtx, mode)`\n\n122+ `if err != nil {`\n\n123+ `fail(TaskGraderEquivalence, err)`\n\n124+ `} else {`\n\n125+ `cfg.EquivalencePrompt, cfg.EquivalencePromptMode = prompt, mode`\n\n126+ `cfg.EquivalencePromptContext, cfg.EquivalenceAI = &promptCtx, client`\n\n127+ `}`\n\n128+ `}`\n\n129+ `if missingReview {`\n\n130+ `prompt, err := validate.RenderReviewPrompt(promptCtx)`\n\n131+ `if err != nil {`\n\n132+ `fail(TaskGraderCodeReview, err)`\n\n133+ `} else {`\n\n134+ `cfg.ReviewPrompt, cfg.ReviewPromptContext, cfg.ReviewAI = prompt, &promptCtx, client`\n\n135+ `}`\n\n136+ `}`\n\n137+ `return cleanup, issues`\n\n138+`}`\n\n139+\n\n140+`func verifyCoverageRepairTaskAuthority(manifest *Manifest, artifacts TaskArtifacts, dataset string) error {`\n\n141+ `if manifest.TaskSelection == nil || manifest.TaskSelection.PromptProjectionReceipt == \"\" {`\n\n142+ `return fmt.Errorf(\"missing original task prompt projection authority\")`\n\n143+ `}`\n\n144+ `receipt, err := LoadTaskPromptProjectionReceipt(artifacts.OutputRoot)`\n\n145+ `if err != nil {`\n\n146+ `return err`\n\n147+ `}`\n\n148+ `if err := validateTaskSelectionProjection(manifest.TaskSelection, receipt); err != nil {`\n\n149+ `return err`\n\n150+ `}`\n\n151+ `digest, err := digestTaskContentTree(filepath.Join(dataset, artifacts.TaskID))`\n\n152+ `if err != nil {`\n\n153+ `return err`\n\n154+ `}`\n\n155+ `for _, row := range receipt.Tasks {`\n\n156+ `if row.TaskID == artifacts.TaskID {`\n\n157+ `if row.TaskContentSHA256 != digest {`\n\n158+ `return fmt.Errorf(\"original task content provenance mismatch for %q\", artifacts.TaskID)`\n\n159+ `}`\n\n160+ `return nil`\n\n161+ `}`\n\n162+ `}`\n\n163+ `return fmt.Errorf(\"missing original task content authority for %q\", artifacts.TaskID)`\n\n164+`}`\n\n…\n\n1+`package h2h`\n\n2+\n\n3+`import (`\n\n4+ `\"bytes\"`\n\n5+ `\"context\"`\n\n6+ `\"fmt\"`\n\n7+ `\"os\"`\n\n8+ `\"path/filepath\"`\n\n9+ `\"strings\"`\n\n10+ `\"testing\"`\n\n11+ `\"time\"`\n\n12+\n\n13+ `\"stet/internal/validate\"`\n\n14+`)`\n\n15+\n\n16+`func TestPrepareCoverageRepairPromptsPreservesStoredPrompts(t *testing.T) {`\n\n17+ `cfg := validate.CoverageRepairConfig{`\n\n18+ `Existing: validate.ValidationResult{`\n\n19+ `Equivalence: validate.EquivalenceResult{Prompt: \"stored equivalence\"},`\n\n20+ `Review:      &validate.ReviewResult{Prompt: \"legacy stored review\"},`\n\n21+ `}, RepairEquivalence: true, RepairCodeReview: true,`\n\n22+ `}`\n\n23+ `cleanup, issues := prepareCoverageRepairPrompts(&cfg, &Manifest{Dataset: \"missing\"}, TaskArtifacts{TaskID: \"task-a\"})`\n\n24+ `if cleanup != nil || len(issues) != 0 || cfg.EquivalencePromptContext != nil || cfg.ReviewPromptContext != nil {`\n\n25+ `t.Fatalf(\"stored prompts should require no reconstruction: %+v\", cfg)`\n\n26+ `}`\n\n27+`}`\n\n28+\n\n29+`func TestPrepareCoverageRepairPromptsRejectsUnknownModeWithoutBlockingStoredReview(t *testing.T) {`\n\n30+ `cfg := validate.CoverageRepairConfig{`\n\n31+ `Existing:          validate.ValidationResult{CodeReview: validate.ReviewResult{Prompt: \"stored review\"}},`\n\n32+ `RepairEquivalence: true, RepairCodeReview: true,`\n\n33+ `}`\n\n34+ `_, issues := prepareCoverageRepairPrompts(&cfg, &Manifest{Dataset: \"missing\"}, TaskArtifacts{TaskID: \"task-a\"})`\n\n35+ `if len(issues) != 1 || issues[0].Dimension != TaskGraderEquivalence || cfg.RepairEquivalence || !cfg.RepairCodeReview {`\n\n36+ `t.Fatalf(\"unexpected dimension selection: %+v, issues %+v\", cfg, issues)`\n\n37+ `}`\n\n38+`}`\n\n39+\n\n40+`func TestPrepareCoverageRepairPromptsRejectsUnknownOriginalTruncationBudget(t *testing.T) {`\n\n41+ `cfg := validate.CoverageRepairConfig{RepairCodeReview: true}`\n\n42+ `cfg.Existing.PromptTruncation.AgentPatch.Truncated = true`\n\n43+ `_, issues := prepareCoverageRepairPrompts(&cfg, &Manifest{Dataset: \"missing\"}, TaskArtifacts{TaskID: \"task-a\"})`\n\n44+ `if len(issues) != 1 || cfg.RepairCodeReview || !strings.Contains(issues[0].Detail, \"original context budgets\") {`\n\n45+ `t.Fatalf(\"unknown original truncation must prevent reconstruction: %+v\", issues)`\n\n46+ `}`\n\n47+`}`\n\n48+\n\n49+`func TestPrepareCoverageRepairPromptsReconstructsPersistedModeAndVerifierContext(t *testing.T) {`\n\n50+ `t.Parallel()`\n\n51+ `root := t.TempDir()`\n\n52+ `taskDir := filepath.Join(root, \"dataset\", \"task-a\")`\n\n53+ `mustWriteFile(t, filepath.Join(taskDir, \"task.yaml\"), \"id: task-a\\ninstruction: fix greeting\\n\")`\n\n54+ `mustWriteFile(t, filepath.Join(taskDir, \"gold.patch\"), \"gold reference secret\\n\")`\n\n55+ `mustWriteFile(t, filepath.Join(taskDir, \"tests\", \"test_outputs.py\"), \"COMMANDS = [\\\"dataset test\\\"]\\n\")`\n\n56+ `patch := filepath.Join(root, \"agent.patch\")`\n\n57+ `mustWriteFile(t, patch, \"diff --git a/hello.txt b/hello.txt\\n+hello\\n\")`\n\n58+ `writeAnswerSimilarityTar(t, filepath.Join(taskDir, \"repo.tar.gz\"), map[string]string{\"hello.txt\": \"old\\n\"}, nil)`\n\n59+ `existing := validate.ValidationResult{`\n\n60+ `Equivalence: validate.EquivalenceResult{PromptMode: validate.EquivalencePromptModeGoldBlind},`\n\n61+ `Tests:       validate.TestResult{Commands: []string{\"effective verifier\"}},`\n\n62+ `}`\n\n63+ `inputs, err := validate.ReadTaskInputs(filepath.Dir(taskDir), \"task-a\", patch)`\n\n64+ `if err != nil {`\n\n65+ `t.Fatal(err)`\n\n66+ `}`\n\n67+ `existing.ArtifactsManifest = validate.BuildArtifactManifest(inputs)`\n\n68+ `manifest := sealCoverageRepairTestTask(t, root, filepath.Dir(taskDir))`\n\n69+ `cfg := validate.CoverageRepairConfig{Existing: existing, AI: validate.ShellAIClient{}, RepairEquivalence: true, RepairCodeReview: true}`\n\n70+ `cleanup, issues := prepareCoverageRepairPrompts(&cfg, manifest, NewTaskArtifacts(root, VariantSpec{RunID: \"run-a\", ModelKey: \"model-a\"}, \"task-a\", \"\"))`\n\n71+ `if cleanup != nil {`\n\n72+ `defer cleanup()`\n\n73+ `}`\n\n74+ `if len(issues) != 0 {`\n\n75+ `t.Fatalf(\"reconstruction failed: %+v\", issues)`\n\n76+ `}`\n\n77+ `if cfg.EquivalencePromptMode != validate.EquivalencePromptModeGoldBlind || strings.Contains(cfg.EquivalencePrompt, \"gold reference secret\") {`\n\n78+ `t.Fatalf(\"gold-blind provenance was lost: %s\", cfg.EquivalencePrompt)`\n\n79+ `}`\n\n80+ `if !strings.Contains(cfg.ReviewPrompt, \"effective verifier\") || strings.Contains(cfg.ReviewPrompt, \"dataset test\") {`\n\n81+ `t.Fatalf(\"effective verifier context was lost: %s\", cfg.ReviewPrompt)`\n\n82+ `}`\n\n83+ `for _, client := range []validate.AIClient{cfg.EquivalenceAI, cfg.ReviewAI} {`\n\n84+ `shell := client.(validate.ShellAIClient)`\n\n85+ `if shell.Dir == \"\" || shell.Dir != cfg.ReviewPromptContext.RepoPath {`\n\n86+ `t.Fatalf(\"missing grading repository: %+v\", shell)`\n\n87+ `}`\n\n88+ `if _, err := os.Stat(filepath.Join(shell.Dir, \"hello.txt\")); err != nil {`\n\n89+ `t.Fatal(err)`\n\n90+ `}`\n\n91+ `}`\n\n92+`}`\n\n93+\n\n94+`func TestPrepareCoverageRepairPromptsRejectsUnavailableRepository(t *testing.T) {`\n\n95+ `root := t.TempDir()`\n\n96+ `taskDir := filepath.Join(root, \"dataset\", \"task-a\")`\n\n97+ `mustWriteFile(t, filepath.Join(taskDir, \"task.yaml\"), \"id: task-a\\n\")`\n\n98+ `mustWriteFile(t, filepath.Join(taskDir, \"gold.patch\"), \"gold\\n\")`\n\n99+ `mustWriteFile(t, filepath.Join(taskDir, \"repo.tar.gz\"), \"invalid gzip\")`\n\n100+ `patch := filepath.Join(root, \"agent.patch\")`\n\n101+ `mustWriteFile(t, patch, \"agent\\n\")`\n\n102+ `manifest := sealCoverageRepairTestTask(t, root, filepath.Dir(taskDir))`\n\n103+ `cfg := validate.CoverageRepairConfig{RepairCodeReview: true}`\n\n104+ `inputs, err := validate.ReadTaskInputs(filepath.Dir(taskDir), \"task-a\", patch)`\n\n105+ `if err != nil {`\n\n106+ `t.Fatal(err)`\n\n107+ `}`\n\n108+ `cfg.Existing.ArtifactsManifest = validate.BuildArtifactManifest(inputs)`\n\n109+ `cleanup, issues := prepareCoverageRepairPrompts(&cfg, manifest, NewTaskArtifacts(root, VariantSpec{RunID: \"run-a\"}, \"task-a\", \"\"))`\n\n110+ `if cleanup != nil {`\n\n111+ `defer cleanup()`\n\n112+ `}`\n\n113+ `if len(issues) != 1 || cfg.RepairCodeReview || cfg.ReviewPrompt != \"\" || !strings.Contains(issues[0].Detail, \"required repo context\") {`\n\n114+ `t.Fatalf(\"repository failure should prevent reconstructed grade: %+v, issues %+v\", cfg, issues)`\n\n115+ `}`\n\n116+`}`\n\n117+\n\n118+`func sealCoverageRepairTestTask(t *testing.T, root, dataset string) *Manifest {`\n\n119+ `t.Helper()`\n\n120+ `selection := &TaskSelectionRecord{RealizedTaskIDs: []string{\"task-a\"}}`\n\n121+ `receipt, err := BuildTaskPromptProjectionReceipt(dataset, root, selection, nil, \"dataset\", map[string]string{\"task-a\": \"fix greeting\"}, time.Time{})`\n\n122+ `if err != nil {`\n\n123+ `t.Fatal(err)`\n\n124+ `}`\n\n125+ `if _, err := WriteTaskPromptProjectionReceipt(root, receipt); err != nil {`\n\n126+ `t.Fatal(err)`\n\n127+ `}`\n\n128+ `return &Manifest{Dataset: dataset, TaskSelection: ApplyTaskPromptProjectionReceipt(selection, receipt)}`\n\n129+`}`\n\n130+\n\n131+`func TestPrepareCoverageRepairPromptsRejectsMutatedTaskYAML(t *testing.T) {`\n\n132+ `root := t.TempDir()`\n\n133+ `dataset := filepath.Join(root, \"dataset\")`\n\n134+ `mustWriteFile(t, filepath.Join(dataset, \"task-a\", \"task.yaml\"), \"id: task-a\\ninstruction: original\\n\")`\n\n135+ `manifest := sealCoverageRepairTestTask(t, root, dataset)`\n\n136+ `mustWriteFile(t, filepath.Join(dataset, \"task-a\", \"task.yaml\"), \"id: task-a\\ninstruction: mutated\\n\")`\n\n137+ `cfg := validate.CoverageRepairConfig{RepairCodeReview: true}`\n\n138+ `_, issues := prepareCoverageRepairPrompts(&cfg, manifest, NewTaskArtifacts(root, VariantSpec{RunID: \"run-a\"}, \"task-a\", \"\"))`\n\n139+ `if len(issues) != 1 || cfg.RepairCodeReview || !strings.Contains(issues[0].Detail, \"task content provenance mismatch\") {`\n\n140+ `t.Fatalf(\"changed task must not reconstruct: %+v\", issues)`\n\n141+ `}`\n\n142+`}`\n\n143+\n\n144+`func TestRunRepairAICoverageReconstructsBoundSourceAtSubprocessBoundary(t *testing.T) {`\n\n145+ `for _, mutate := range []bool{false, true} {`\n\n146+ `t.Run(fmt.Sprintf(\"mutate_%t\", mutate), func(t *testing.T) {`\n\n147+ `root := t.TempDir()`\n\n148+ `dataset := filepath.Join(root, \"dataset\")`\n\n149+ `taskDir := filepath.Join(dataset, \"task-a\")`\n\n150+ `mustWriteFile(t, filepath.Join(taskDir, \"task.yaml\"), \"id: task-a\\ninstruction: fix greeting\\n\")`\n\n151+ `mustWriteFile(t, filepath.Join(taskDir, \"gold.patch\"), \"gold reference secret\\n\")`\n\n152+ `writeAnswerSimilarityTar(t, filepath.Join(taskDir, \"repo.tar.gz\"), map[string]string{\"hello.txt\": \"old\\n\"}, nil)`\n\n153+ `patch := filepath.Join(root, \"agent.patch\")`\n\n154+ `mustWriteFile(t, patch, \"diff --git a/hello.txt b/hello.txt\\n--- a/hello.txt\\n+++ b/hello.txt\\n@@ -1 +1 @@\\n-old\\n+hello\\n\")`\n\n155+ `manifest := sealCoverageRepairTestTask(t, root, dataset)`\n\n156+ `manifest.Version, manifest.OutputRoot = ManifestVersion, root`\n\n157+ `manifest.Models = []string{\"model/a\"}`\n\n158+ `manifest.ModelKeys, manifest.RunIDs = map[string]string{\"model/a\": \"model-a\"}, map[string]string{\"model/a\": \"run-a\"}`\n\n159+ `if err := SaveManifest(filepath.Join(root, \"manifest.json\"), manifest); err != nil {`\n\n160+ `t.Fatal(err)`\n\n161+ `}`\n\n162+ `inputs, err := validate.ReadTaskInputs(dataset, \"task-a\", patch)`\n\n163+ `if err != nil {`\n\n164+ `t.Fatal(err)`\n\n165+ `}`\n\n166+ `original := validate.ValidationResult{`\n\n167+ `TaskID: \"task-a\", PrimarySignal: validate.PrimarySignalTests, Status: validate.StatusPass, MatrixStatus: validate.MatrixPass,`\n\n168+ `Guardrails:        validate.GuardrailResult{Status: validate.StatusPass},`\n\n169+ `Tests:             validate.TestResult{Outcome: validate.TestOutcomePass, Strength: validate.TestStrengthStrong, Commands: []string{\"effective verifier\"}},`\n\n170+ `Equivalence:       validate.EquivalenceResult{Status: validate.AIStatusUnavailable, Outcome: validate.EquivalenceUnknown, PromptMode: validate.EquivalencePromptModeGoldBlind},`\n\n171+ `CodeReview:        validate.ReviewResult{Status: validate.AIStatusUnavailable, Signal: validate.ReviewSignalUnsure},`\n\n172+ `ArtifactsManifest: validate.BuildArtifactManifest(inputs),`\n\n173+ `}`\n\n174+ `validationRoot := filepath.Join(root, \"validation\", \"model-a\")`\n\n175+ `mustWriteRepairValidation(t, validationRoot, \"task-a\", original)`\n\n176+ `if mutate {`\n\n177+ `mustWriteFile(t, patch, \"mutated agent evidence\\n\")`\n\n178+ `}`\n\n179+ `log := filepath.Join(root, \"calls\")`\n\n180+ `script := filepath.Join(root, \"judge.py\")`\n\n181+ ``mustWriteFile(t, script, `import json, pathlib, sys``\n\n182+`prompt = sys.stdin.read()`\n\n183+`assert pathlib.Path('hello.txt').read_text() == 'old\\n'`\n\n184+`with open(sys.argv[1], 'a') as log: log.write('call\\n')`\n\n185+`if 'You are reviewing' in prompt:`\n\n186+ `assert 'effective verifier' in prompt`\n\n187+ `scores = {name: {'score_0_4': 4, 'confidence_0_1': .95, 'rationale': 'ok', 'citations': [{'path':'hello.txt','side':'new','start_line':1,'end_line':1,'anchor_kind':'diff','head_sha':'abc1234'}]} for name in ['correctness','introduced_bug_risk','edge_case_handling','maintainability_idioms']}`\n\n188+ `print(json.dumps({'schema_version':'stet.review.v1','dimension_scores':scores,'findings':[],'overall':{'confidence_overall':.95,'summary':'repair passed'}}))`\n\n189+`else:`\n\n190+ `assert 'gold reference secret' not in prompt`\n\n191+ `print('obligations:\\n- fix greeting => MET\\n\\nequivalent: YES\\nconfidence: 0.95\\nissue_type: stylistic\\nnotes: repaired')`\n\n192+`` `) ``\n\n193+ `var stdout, stderr bytes.Buffer`\n\n194+ `err = RunRepairAICoverage(context.Background(), []string{\"--out\", root, \"--model-key\", \"model-a\", \"--ai-cmd\", \"python3 \" + shellQuote(script) + \" \" + shellQuote(log), \"--ai-model-id\", \"judge-a\"}, &stdout, &stderr)`\n\n195+ `if mutate {`\n\n196+ `if err == nil {`\n\n197+ `t.Fatal(\"mutated artifact repair must fail\")`\n\n198+ `}`\n\n199+ `if _, err := os.Stat(log); !os.IsNotExist(err) {`\n\n200+ `t.Fatalf(\"mutated source invoked grader: %v\", err)`\n\n201+ `}`\n\n202+ `if !strings.Contains(stdout.String(), \"agent patch provenance mismatch\") {`\n\n203+ `t.Fatalf(\"missing mutation reason: %s\", stdout.String())`\n\n204+ `}`\n\n205+ `return`\n\n206+ `}`\n\n207+ `if err != nil {`\n\n208+ `t.Fatalf(\"repair failed: %v\\nstdout %s\\nstderr %s\", err, stdout.String(), stderr.String())`\n\n209+ `}`\n\n210+ `updated := mustReadRepairValidation(t, filepath.Join(validationRoot, \"task-a\", \"validation.json\"))`\n\n211+ `if updated.Equivalence.PromptMode != validate.EquivalencePromptModeGoldBlind || updated.Equivalence.Status != validate.AIStatusUsed || updated.CodeReview.Status != validate.AIStatusUsed || updated.CodeReview.AttemptCount != 1 {`\n\n212+ `t.Fatalf(\"repair lost mode or attempt metadata: %+v\", updated)`\n\n213+ `}`\n\n214+ `if updated.ArtifactsManifest.AgentPatch.SHA256 != original.ArtifactsManifest.AgentPatch.SHA256 {`\n\n215+ `t.Fatal(\"repair changed original source authority\")`\n\n216+ `}`\n\n217+ `calls, err := os.ReadFile(log)`\n\n218+ `if err != nil || strings.Count(string(calls), \"call\") != 2 {`\n\n219+ `t.Fatalf(\"unexpected calls: %s, %v\", calls, err)`\n\n220+ `}`\n\n221+ `})`\n\n222+ `}`\n\n223+`}`\n\n…\n\n1+`package validate`\n\n2+\n\n3+`import (`\n\n4+ `\"context\"`\n\n5+ `\"errors\"`\n\n6+ `\"net\"`\n\n7+ `\"strings\"`\n\n8+ `\"time\"`\n\n9+\n\n10+ `\"stet/internal/runtimefailure\"`\n\n11+`)`\n\n12+\n\n13+`// gradeAttemptClient retains every dispatch, including runtime and parse retries.`\n\n14+`// One instance belongs to one measurement dimension and must not be shared.`\n\n15+`type gradeAttemptClient struct {`\n\n16+ `ai                AIClient`\n\n17+ `attempts          int`\n\n18+ `reasons           []string`\n\n19+ `previousSucceeded bool`\n\n20+`}`\n\n21+\n\n22+`func newGradeAttemptClient(ai AIClient) *gradeAttemptClient {`\n\n23+ `if ai == nil {`\n\n24+ `ai = NoopAIClient{}`\n\n25+ `}`\n\n26+ `return &gradeAttemptClient{ai: ai}`\n\n27+`}`\n\n28+\n\n29+`func (a *gradeAttemptClient) Run(ctx context.Context, prompt string) (string, error) {`\n\n30+ `if a.previousSucceeded {`\n\n31+ `a.reasons = append(a.reasons, \"model_output_shape\")`\n\n32+ `}`\n\n33+ `a.previousSucceeded = false`\n\n34+ `for attempt := 0; ; attempt++ {`\n\n35+ `if err := ctx.Err(); err != nil {`\n\n36+ `return \"\", err`\n\n37+ `}`\n\n38+ `a.attempts++`\n\n39+ `output, err := a.ai.Run(ctx, prompt)`\n\n40+ `if err == nil {`\n\n41+ `a.previousSucceeded = true`\n\n42+ `return output, nil`\n\n43+ `}`\n\n44+ `reason := recoverableGradeFailure(err)`\n\n45+ `if reason == \"\" || attempt >= len(coverageRepairRetryBackoffs) {`\n\n46+ `return \"\", err`\n\n47+ `}`\n\n48+ `select {`\n\n49+ `case <-ctx.Done():`\n\n50+ `return \"\", ctx.Err()`\n\n51+ `case <-time.After(coverageRepairRetryBackoffs[attempt]):`\n\n52+ `}`\n\n53+ `a.reasons = append(a.re`", "url": "https://wpnews.pro/news/i-ran-sol-6-vs-6-1-on-tasks-from-my-own-repo-it-s-cheaper-and-at-least-as-good", "canonical_source": "https://www.stet.sh/blog/gpt-6-1-sol-vs-gpt-6-sol", "published_at": "2026-10-12 16:29:52+00:00", "updated_at": "2026-10-12 16:49:31.202270+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models"], "entities": ["Sol 6", "Sol 6.1", "stet"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-ran-sol-6-vs-6-1-on-tasks-from-my-own-repo-it-s-cheaper-and-at-least-as-good", "markdown": "https://wpnews.pro/news/i-ran-sol-6-vs-6-1-on-tasks-from-my-own-repo-it-s-cheaper-and-at-least-as-good.md", "text": "https://wpnews.pro/news/i-ran-sol-6-vs-6-1-on-tasks-from-my-own-repo-it-s-cheaper-and-at-least-as-good.txt", "jsonld": "https://wpnews.pro/news/i-ran-sol-6-vs-6-1-on-tasks-from-my-own-repo-it-s-cheaper-and-at-least-as-good.jsonld"}}