# Beyond Fixed Probes: Making LLM API Evaluation Harder to Game

> Source: <https://dev.to/okenai_1ec49d66dbe8d17916/beyond-fixed-probes-making-llm-api-evaluation-harder-to-game-381d>
> Published: 2026-11-16 07:42:50+00:00

A model fingerprinting test can return a convincing result while missing an important failure mode:

What if the provider knows when it is being tested?

A provider could recognize a fixed fingerprinting prompt, route that request to the genuine model, and send ordinary production traffic to a cheaper or older model.

The test would pass. The endpoint would still behave differently for real users.

This possibility does not invalidate black-box evaluation. It defines a stronger requirement for the evaluation protocol:

The test should be difficult to recognize, reproducible across controls, and informative when request routing changes.

This question came up while working on the model evaluation workflow at Oken. Oken compares AI model and API behavior through controlled black-box tests. The next challenge is making those tests more robust against provider-side routing decisions and ordinary model drift.

A fixed probe is easy to recognize

A fixed fingerprinting task has several properties that make it vulnerable to pattern matching:

A detector should therefore avoid treating one successful probe as conclusive evidence.

Fresh content preserves the measurement target

The first improvement is to generate fresh probe content for every run.

The surface form can change while the statistical measurement target remains stable. For an integer-sequence task, this could include varying:

A useful protocol should preserve the behavioral dimensions being measured:

The design principle is:

Keep the measurement objective stable while changing the observable form of the probe.

Fresh content does not make a test impossible to target. It raises the cost of recognizing the test and reduces dependence on a single known prompt.

Request shape is another routing signal

Changing the prompt content may not be enough.

Providers can route requests based on metadata and request structure, including:

The same semantic probe can be sent using two request envelopes:

Request A

Minimal evaluation envelope

Request B

Production-shaped envelope with:

If the endpoint behaves differently across the two envelopes, that difference becomes a useful signal for investigation.

A simplified result could look like this:

Same probe content

Minimal envelope:      matches reference model A

Production envelope:   matches reference model B

Observed divergence:   substantial

This does not prove selective routing by itself. The request envelope can change model behavior naturally. A system prompt may alter instruction following, tool definitions may change the model’s context, and streaming may affect generation or post-processing.

The correct interpretation is:

The endpoint shows request-shape-dependent behavioral differences that are consistent with a routing or configuration effect and require further testing.

That wording preserves the signal without turning it into an unsupported accusation.

Same content, two shapes, repeated many times

A single paired comparison is still vulnerable to noise.

A stronger design repeats the same experiment across multiple fresh probes:

Fresh probe 1:

  minimal envelope

  production-shaped envelope

Fresh probe 2:

  minimal envelope

  production-shaped envelope

Fresh probe 3:

  minimal envelope

  production-shaped envelope

The evaluator can then measure:

The useful question is whether the difference between request shapes is larger and more stable than the normal variation expected under a trusted control.

A same-day trusted control

Reference-bank distance creates another interpretation problem.

Suppose a tested endpoint becomes more distant from its stored reference. Several explanations are possible:

One practical control is to call a trusted endpoint serving the claimed model on the same day and under the same protocol.

The evaluator can then compare:

Suspect endpoint → current observation

Trusted endpoint  → current control

Stored reference → historical baseline

The useful signal becomes differential behavior:

Difference(suspect, trusted control)

rather than only:

Difference(suspect, historical reference)

The trusted endpoint does not need to be perfectly current. It needs to provide a contemporary baseline for the model under the same testing conditions.

This helps distinguish provider-specific divergence from general model drift.

The control itself requires provenance:

For example, the same endpoint could be evaluated twice within a short interval:

Run 1: closest to Model A

Run 2: closest to Model A

Run 3: closest to Model B

This may indicate:

A result that changes dramatically within minutes should be treated differently from a result that remains stable across independent runs.

A useful report should therefore include:

This is more consistent with a provider-side routing or configuration difference.

Shared model drift

Both the tested endpoint and the trusted control move in a similar direction relative to the historical reference.

This is more consistent with a model update or a change shared by both deployments.

Measurement instability

The tested endpoint and the control both show large run-to-run variation under the same protocol.

This suggests that the probe, sampling process, or test configuration may not be stable enough for a strong conclusion.

These categories are not automatically identifiable from one run. They require repeated measurements and controls.

What Oken is designed to help with

Oken’s model evaluation workflow provides a practical way to compare black-box endpoints with reference model behavior.

The workflow can be used to:

It can help identify an endpoint that deserves further investigation. It does not independently reveal the provider’s internal routing configuration.

You can explore the evaluation workflow at [https://www.oken.ai/zh](https://www.oken.ai/zh).

A more complete result record

A production-oriented evaluator should preserve enough information to reproduce and audit the result.

For example:

{

  "provider_asserted_model": "model-a",

  "top_reference_match": "model-a",

  "closed_set_probability": 0.94,

  "absolute_compatibility": 0.81,

  "request_shape": "production_like",

  "trusted_control_status": "passed",

  "test_retest_spread": 0.07,

  "detector_version": "detector-1.2.0",

  "reference_bank_version": "bank-2026-11-01",

  "calibration_version": "calibration-2026-11-02",

  "decision": "behaviorally_consistent"

}

For a suspicious result:

{

  "provider_asserted_model": "model-a",

  "top_reference_match": "model-b",

  "closed_set_probability": 0.88,

  "absolute_compatibility": 0.42,

  "request_shape": "production_like",

  "trusted_control_status": "passed",

  "test_retest_spread": 0.31,

  "decision": "requires_investigation"

}

The output should preserve the difference between:

Stronger evidence would require a repeated pattern:

A suitable report might say:

Across repeated fresh probes, the endpoint showed a stable request-shape-dependent behavioral difference relative to a same-day trusted control. The pattern is consistent with routing or configuration differences and warrants further investigation.

This is a stronger and more defensible statement than:

The provider is definitely routing test requests to the real model.

Conclusion

A fixed probe can be measured, recognized, and targeted.

A stronger black-box evaluation protocol should use fresh probe content, multiple request shapes, same-day controls, and repeated runs. It should measure test-retest spread and preserve the versions and provenance required to interpret the result.

The objective is not to make the test impossible to game. No black-box test can guarantee that against a provider with complete control of the endpoint.

The objective is to make selective behavior more difficult to hide and easier to detect.

For developers comparing AI API providers, behavioral fingerprinting can provide a useful external signal when deployment logs and routing records are unavailable. Oken provides a practical workflow for running these comparisons and tracking provider-level differences over time.

The result should be read as evidence about observed behavior under a defined protocol. Its strength comes from fresh probes, controls, repeated measurements, and transparent uncertainty.

Disclosure: Oken is the platform discussed in this article. Its evaluation workflow reports behavioral similarity against reference samples and should not be interpreted as cryptographic proof of the server-side model identity.
