One arithmetic request, three model panes. The first response finished in 2.47 seconds, but proposed a notebook workflow instead of returning the calculation. Another returned a calculation after 4.11 seconds.
The Sinaps Studio interface below puts the responses, full-response times and output-token counts alongside each other. This is one integration example: the complete system context and settings have not been verified identical, so it cannot establish a model ranking.
Start with three small cases whose outcomes can be checked directly:
Three configured models, five repetitions per case would give 45 planned requests. This follow-up has not been executed.
For each request, preserve the exact prompt/system context, model ID/version, provider, settings, tool availability and cache condition. Run tool-disabled and tool-enabled cases separately. Record input/output tokens when reported; keep unavailable fields as N/A.
Use **pass / fail / not run** before comparing latency. A request that fails before reaching the checker should remain visible in the attempted-request denominator. Record the complete application-request time separately from time to first token when that measurement is available.
Then compare latency and token usage among passing answers, while still reporting failures and unscored requests. Keep raw answers and the checker version with each run so another person can inspect the classification.
Which additional controls would you require before comparing models across providers? In particular, how would you report caching and a response that delegates the task instead of answering it?
Pictured workspace: Sinaps Studio, Windows x64. A 30-day Professional evaluation is available; model-provider access and compute are separate.