Trying GPT-6 Sol with AI SDK Evaluation API

Vercel's AI Gateway now lists GPT-6 Sol as usable through the AI SDK's experimental evaluation API, which lets application code ask named questions about shared input called `state` and return three answer types via the OpenAI evaluation adapter. The adapter uses the Responses API and structured output, and its Choice and Score answers omit probability distributions, so a boolean result of `0.9` should not be read as proof the model is correct nine times out of ten. The evaluation API is experimental and can change in patch releases, so developers are advised to keep a lockfile and recheck the evaluation contract when updating the SDK.

GPT-6 Sol https://vercel.com/ai-gateway/models/gpt-6-sol can answer typed questions through AI SDK's experimental evaluation API. Create an OpenAI evaluation model, supply the evidence to assess, and define the answers your application needs. The result can classify a proposed response, score it against a rubric, or estimate whether a statement is true. Copy link to heading what-does-evaluation-mean-in-the-ai-sdk What does evaluation mean in the AI SDK? The AI SDK evaluation API https://ai-sdk.dev/docs/ai-sdk-core/evaluation lets application code ask named questions about shared input, called state . For example, a support application could assess a draft reply against the customer's request and the relevant policy before showing it to a reviewer. The evaluation happens during the application's workflow. Measuring how accurately Sol performs that evaluation requires a separate comparison with examples that people have already reviewed. The OpenAI evaluation adapter https://ai-sdk.dev/providers/ai-sdk-providers/openai evaluation-models uses the Responses API and structured output to produce three answer types: Choice and Score answers from this adapter do not include probability distributions. Boolean probabilities are generated estimates, so a result of 0.9 should not be read as proof that the model is correct nine times out of ten on your task. Copy link to heading when-is-sol-useful-for-an-evaluation-step When is Sol useful for an evaluation step? Sol is designed for complex coding and agent workflows. It is worth testing when a decision requires interpreting several pieces of evidence, such as deciding whether a draft explanation accurately describes a code change or follows a policy with exceptions. Consider a customer asking whether an unused purchase qualifies for a return. The proposed reply says the refund has already been processed. Even if the customer is eligible, the policy alone cannot support a claim that money has been returned. An evaluation can separate whether the reply addresses the request from whether its factual claims are supported. Supply the customer message, the applicable policy, and the draft reply together. If the application has a payment record, include it as a distinct source. Keeping those inputs separate helps define what the model should judge and gives a reviewer a way to investigate a disputed result. Copy link to heading how-do-you-evaluate-a-draft-reply-with-sol How do you evaluate a draft reply with Sol? Copy link to heading 1.-install-the-ai-sdk-and-openai-provider 1. Install the AI SDK and OpenAI provider In a TypeScript application, install packages that include the experimental evaluation API: npm install ai @ai-sdk/openai Configure OPENAI API KEY as an environment variable available to the code that makes the request. These examples use the OpenAI provider directly. Its evaluationModel factory selects the Responses integration, so the model ID is gpt-6-sol without a Gateway prefix. The evaluation API is experimental and can change in patch releases. Keep a lockfile and check the evaluation contract when updating the SDK, particularly if your application depends on specific answer fields. Copy link to heading 2.-define-the-evidence-and-questions 2. Define the evidence and questions Put the shared example in src/reply-review.ts . The policy below is sample application data, and the questions describe how to assess a reply against it. js export const reviewState = { policy: 'Unused purchases can be returned within 30 days of delivery.', customerMessage: 'My order arrived 12 days ago and is unopened. Can I return it?', draftReply: 'Your purchase qualifies for a return. We have processed your refund.',}; export const reviewQuestions = { disposition: { type: 'choice' as const, instructions: 'Assess the draft reply using only the supplied evidence.', criteria: { ready: 'The reply answers the request and all factual claims are supported.', revise: 'The reply contains an unsupported claim, contradicts the evidence, or omits part of the request.', needs review: 'The request or applicable policy is too ambiguous to assess whether the reply answers it.', }, }, coverage: { type: 'score' as const, instructions: 'How fully does the reply address the customer request?', criteria: 'Does not address the request', 'Addresses part of the request', 'Addresses the full request', , }, unsupportedClaim: { type: 'boolean' as const, instructions: 'Does the reply make a factual claim not supported by the supplied evidence?', criteria: { true: 'At least one claim lacks support or contradicts the evidence.', false: 'Every factual claim is supported by the evidence.', }, },}; disposition provides a recommendation for the review workflow, while coverage measures how much of the request the reply addresses. unsupportedClaim focuses on factual support. Coverage alone should not determine whether a reply is ready because it may address the whole request and still make an unsupported promise. The OpenAI adapter evaluates these questions together in one prompt. If you change a question or its criteria, rerun the whole set against your reviewed examples to check whether the judgments remain useful. Copy link to heading 3.-call-sol-and-read-the-named-answers 3. Call Sol and read the named answers Import the shared inputs from a module that runs in your application's backend: js import { openai } from '@ai-sdk/openai';import { experimental evaluate as evaluate } from 'ai';import { reviewQuestions, reviewState } from './reply-review'; const result = await evaluate { model: openai.evaluationModel 'gpt-6-sol' , state: reviewState, questions: reviewQuestions, providerOptions: { openai: { reasoningEffort: 'medium' }, },} ; console.log result.answers.disposition.choice ;console.log result.answers.coverage.score ;console.log result.answers.unsupportedClaim.probability ; Named answers appear under answers , with disposition.choice containing one of the three configured option keys. The coverage score falls between 0 and 2 for this rubric and may be fractional. unsupportedClaim.probability estimates the truth of that specific statement, rather than confidence in the entire evaluation. Sol supports reasoning effort settings https://developers.openai.com/api/docs/models/gpt-6-sol from none through max . The evaluation adapter requests no reasoning by default, so this example explicitly selects medium . Compare effort settings on cases that require different amounts of interpretation before choosing a default for your workload. Copy link to heading how-should-your-application-use-the-result How should your application use the result? Display the disposition beside the original evidence and draft in the review interface. Route revise results back to the author for correction, and use needs review to request an additional record or a person's judgment. The ready option means the model found the reply acceptable under the supplied criteria. Sending it to the customer remains a separate operation. Keep the supporting fields visible when they disagree. If the disposition is ready but the unsupported-claim estimate is high, the reviewer needs to see that conflict. Decide how your application handles such combinations using labeled examples, and store the question definitions with the evaluation so you can reproduce what was asked. Avoid converting the coverage score into a probability. Its scale describes positions on your rubric, and changing the rubric changes the score's meaning. Adding a fourth level, for example, changes the upper bound from 2 to 3 . Copy link to heading can-i-use-other-openai-models-with-ai-sdk-evaluation Can I use other OpenAI models with AI SDK evaluation? Yes. The OpenAI evaluation adapter works with models that support its structured-output integration. GPT-6 Astra https://developers.openai.com/api/docs/models/gpt-6-astra and GPT-6 Luna https://developers.openai.com/api/docs/models/gpt-6-luna are alternatives you can compare using the same evidence and questions. Copy link to heading use-astra-for-complex-reasoning-across-supplied-evidence Use Astra for complex reasoning across supplied evidence Astra is a candidate when the review involves difficult reasoning across several records. Include those records in the shared state before making the call, since this evaluation assesses the supplied evidence without retrieving additional information. js import { openai } from '@ai-sdk/openai';import { experimental evaluate as evaluate } from 'ai';import { reviewQuestions, reviewState } from './reply-review'; const result = await evaluate { model: openai.evaluationModel 'gpt-6-astra' , state: reviewState, questions: reviewQuestions, providerOptions: { openai: { reasoningEffort: 'high' }, },} ; console.log result.answers.disposition.choice ; Astra does not support reasoning effort none . Without an explicit override, the evaluation wrapper requests none , but the OpenAI provider removes that setting and returns an unsupported-setting warning. Set a supported effort explicitly so the evaluation uses the level you intend and avoids the warning. high is an example for a demanding review; it is not required for every Astra evaluation. Copy link to heading use-luna-for-focused-evaluations-repeated-across-many-inputs Use Luna for focused evaluations repeated across many inputs Luna is a candidate for repeated checks with a narrow rubric, such as determining whether replies address a clearly stated request. Include ambiguous and incomplete examples when testing it, since routine successes can conceal weaknesses on the cases that need review. js import { openai } from '@ai-sdk/openai';import { experimental evaluate as evaluate } from 'ai';import { reviewQuestions, reviewState } from './reply-review'; const result = await evaluate { model: openai.evaluationModel 'gpt-6-luna' , state: reviewState, questions: reviewQuestions, providerOptions: { openai: { reasoningEffort: 'none' }, },} ; console.log result.answers.disposition.choice ; Luna supports none , and you can raise the effort if testing shows that reasoning improves the judgments you need. These examples demonstrate different configurations. For a comparison that isolates the model choice, use the same supported effort setting across all three, then tune each model separately. Copy link to heading what-happens-if-an-evaluation-cannot-return-usable-answers What happens if an evaluation cannot return usable answers? The OpenAI adapter fails the evaluation call when it receives a refusal, truncated output, or invalid answers. Your application should record that failure separately from a completed evaluation whose disposition is needs review . The latter is a judgment about the supplied evidence; the former means you have no usable evaluation result. Use abortSignal when a request needs a deadline or cancellation, and configure maxRetries for transient failures. Retries can recover from temporary provider errors; missing evidence needs to be added to state before another evaluation can assess it. Preserve the draft and offer another review path if the call fails. Evaluation returns one complete result and does not stream answers. An array passed as state is shared context for one evaluation, such as a conversation history. Evaluate unrelated replies separately so each decision has an unambiguous subject. Copy link to heading how-do-you-check-whether-the-evaluations-are-useful How do you check whether the evaluations are useful? Build a set of draft replies with expected dispositions established by reviewers. Include cases where the reply is fluent but unsupported, as well as cases where the policy itself is incomplete. Track false approvals separately from unnecessary revision requests because those mistakes have different consequences for the support team. Compare the model's judgments with those labels after changing the model, reasoning effort, or question definitions. Record request duration and token usage alongside the decisions to assess the cost of running the check across your expected volume. Keep a separate test for how application code responds to each answer and to failures; correct branching does not prove that the model chose the right answer. If your application needs native distributions over a defined set of answers, Jev with AI SDK https://vercel.com/kb/guide/typesafe-jev-and-ai-sdk provides another evaluation path. Jev's native probabilities and the OpenAI adapter's generated Boolean estimates have different origins. Test any acceptance threshold with the provider and task you intend to use. Copy link to heading frequently-asked-questions Frequently asked questions Copy link to heading does-this-benchmark-gpt-6-sol Does this benchmark GPT-6 Sol? No. These examples use Sol to judge supplied application data. Benchmarking those judgments requires expected answers and a process for comparing the model's results with them. Copy link to heading does-sol-return-a-confidence-score-for-every-evaluation-answer Does Sol return a confidence score for every evaluation answer? No. The OpenAI adapter returns Choice and Score judgments without probability distributions. Boolean questions return generated estimates of the probability that a statement is true, which need testing before use as decision thresholds. Copy link to heading can-i-disable-reasoning-for-every-gpt-6-model Can I disable reasoning for every GPT-6 model? No. Sol and Luna accept none, while Astra supports reasoning levels starting at low. Set Astra's effort explicitly to avoid the provider's unsupported-setting warning. Copy link to heading can-i-evaluate-a-reply-with-an-older-openai-model Can I evaluate a reply with an older OpenAI model? Yes, if the model supports the OpenAI adapter's structured-output integration. Check its supported reasoning settings and compare its judgments against reviewed examples before using it in the workflow.