GPT-5.6 Sol clarification vs guess benchmark

A developer benchmarked GPT-5.6 Sol on 40 tasks with one to four plausible choices to compare letting the model guess against asking one deterministic clarifying question first. Clarifying cut the average API cost per successful completion by 29.5% ($0.000543 versus $0.000771) and reduced median model latency to 1458 ms from 1660 ms, with savings rising from 6.5% at one choice to 45.2% at four choices. The author notes the test is a small synthetic benchmark and excludes human response time.

Question: When is it cheaper to ask one clarifying question than let the model guess? Model: GPT-5.6 Sol Reasoning effort: Low Tasks: 40 Ambiguity levels: 1 to 4 plausible choices Two strategies were compared: The model completes the task without the missing user preference. If the first answer is wrong, a simulated user correction is provided and the model gets a second API call. The system asks one deterministic clarification question first. The missing preference is then supplied before the model completes the task. Guess first: - First-pass success: 75% - Final success after correction: 100% - Rework rate: 25% - Average API calls/task: 1.25 - Average cost/task: $0.000771 - Median model latency: 1660 ms Clarify first: - Success: 100% - Average API calls/task: 1.00 - Average cost/task: $0.000543 - Median model latency: 1458 ms Clarification reduced API cost to successful completion by 29.5%. 1 choice: - Guess success: 100% - Clarification saving: 6.5% 2 choices: - Guess success: 80% - Clarification saving: 20.4% 3 choices: - Guess success: 70% - Clarification saving: 33.5% 4 choices: - Guess success: 50% - Clarification saving: 45.2% Clarification adds one user interaction. Human response time is not included in the latency numbers. latency figures are model/API latency only. This is a small synthetic benchmark.