When the fastest LLM reply skips the task: a proposed comparison protocol

A proposed comparison protocol from Sinaps Studio argues that LLM latency comparisons are misleading unless answers are first scored pass/fail/not run, citing one arithmetic request where a model replied in 2.47 seconds with a notebook workflow instead of the calculation while another returned the calculation after 4.11 seconds. The protocol calls for preserving exact prompt and system context, model ID/version, provider, settings, tool availability and cache condition per request, running tool-disabled and tool-enabled cases separately, and comparing latency and token usage only among passing answers. The follow-up test of three models across three cases with five repetitions each — 45 planned requests — has not been executed, and Sinaps Studio states the single integration example cannot establish a model ranking because system context and settings were not verified identical.

One arithmetic request, three model panes. The first response finished in 2.47 seconds , but proposed a notebook workflow instead of returning the calculation. Another returned a calculation after 4.11 seconds . The Sinaps Studio interface below puts the responses, full-response times and output-token counts alongside each other. This is one integration example : the complete system context and settings have not been verified identical, so it cannot establish a model ranking. Start with three small cases whose outcomes can be checked directly: Three configured models, five repetitions per case would give 45 planned requests . This follow-up has not been executed . For each request, preserve the exact prompt/system context, model ID/version, provider, settings, tool availability and cache condition. Run tool-disabled and tool-enabled cases separately. Record input/output tokens when reported; keep unavailable fields as N/A. Use pass / fail / not run before comparing latency. A request that fails before reaching the checker should remain visible in the attempted-request denominator. Record the complete application-request time separately from time to first token when that measurement is available. Then compare latency and token usage among passing answers , while still reporting failures and unscored requests. Keep raw answers and the checker version with each run so another person can inspect the classification. Which additional controls would you require before comparing models across providers? In particular, how would you report caching and a response that delegates the task instead of answering it? Pictured workspace: Sinaps Studio https://www.sinapsdata.com.br/ , Windows x64. A 30-day Professional evaluation is available; model-provider access and compute are separate.