Free AI Endpoints Are Unreliable Dependencies. Test Them Like One.

Most glue code around a free AI endpoint fails for a very boring reason: the request returned a 200, but the body was not what the downstream code expected. A quota hit can truncate JSON. A proxy restart can return an HTML error page with the same status. A model can send valid JSON that is missing the one field your code reads.

If you are using MonkeyCode's free model access, this is still true. Treat a free endpoint as a third-party API, not as a trusted library call.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The fix is not a better model or a longer retry loop. It is a contract probe: a small deterministic request that checks the response shape, size, and error behavior at the boundary before the rest of the system touches it.

Most integrations stop at json.loads

inside a try block. That catches two failures:

It does not catch the worse failure: the response is valid JSON and structurally wrong.

For example, a prompt designed to return {"result": "..."}

may come back as {"text": "..."}

after a model update. A Python call like data.get('result').strip()

then raises AttributeError

three functions later, or worse, data.get('result')

returns None

and the code writes None

into a production record. The HTTP request succeeded, but the system still failed.

A contract probe moves the validation from the middle of the workflow to the first contact point.

A contract probe should check at least these properties:

The probe is not an evaluation of model quality. It only asks: Can this endpoint satisfy the response contract for a known prompt right now?

The following probe uses only the Python standard library. It is a prototype, not a production SDK.

import json
import os
import time
import urllib.error
import urllib.request

class ContractError(Exception):
    pass

def check_contract(data):
    if not isinstance(data, dict):
        raise ContractError('root must be an object')
    if 'result' not in data:
        raise ContractError('missing result')
    if not isinstance(data['result'], str):
        raise ContractError('result must be a string')
    if len(data['result']) > 2_000:
        raise ContractError('result is over the size budget')
    if 'usage' in data:
        usage = data['usage']
        if not isinstance(usage, dict):
            raise ContractError('usage must be an object')
        total = usage.get('total_tokens')
        if total is not None and (not isinstance(total, int) or total < 0):
            raise ContractError('usage.total_tokens must be a non-negative integer')

def run_probe(url, prompt, timeout=5.0):
    payload = json.dumps({'prompt': prompt, 'stream': False}).encode('utf-8')
    req = urllib.request.Request(
        url,
        data=payload,
        headers={
            'Content-Type': 'application/json',
            'Accept': 'application/json',
        },
        method='POST',
    )
    start = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            raw = resp.read(8_192)
            if resp.status != 200:
                raise ContractError(f'unexpected status {resp.status}')
            if len(raw) == 0:
                raise ContractError('empty response body')
            data = json.loads(raw)
    except urllib.error.HTTPError as exc:
        raise ContractError(f'HTTP error {exc.code}') from exc
    except (urllib.error.URLError, TimeoutError) as exc:
        raise ContractError(f'network failure: {exc}') from exc
    check_contract(data)
    elapsed_ms = (time.monotonic() - start) * 1000
    data['_probe_latency_ms'] = elapsed_ms
    return data

if __name__ == '__main__':
    endpoint = os.environ.get('AI_ENDPOINT_URL')
    if not endpoint:
        raise SystemExit('Set AI_ENDPOINT_URL before running this probe.')
    print(json.dumps(run_probe(endpoint, 'reply with exactly: ok'), indent=2))

Run the probe with a fixed prompt. The expected answer is not important; the contract is. If the probe fails on a known prompt, the endpoint should not receive production traffic until someone investigates.

A single probe result can feed a small policy table. The right response depends on the cost of a wrong answer.

The key point: the probe fails with a reason, and the policy acts on that reason. A timeout and a schema change are not the same problem.

If you run the probe against a public endpoint during every CI build, you are testing the provider's availability as much as your own code. Flakiness will teach people to ignore the probe.

A better setup is to point the same probe at a local server that implements the same response schema. If MonkeyCode's free server option is available to you, it can serve as that local test double: the server returns canned fixtures that satisfy the contract, so your test fails only when the contract changes, not when the network wobbles.

The public endpoint probe still runs separately, on a schedule or before a release, not on every commit.

This workflow catches structural and transport failures. It does not measure whether the model's answer is correct.

If the provider changes the response schema, the probe catches it. If the provider returns valid schema with a semantically wrong answer, the probe may pass. You still need separate evaluation, human review, or domain-specific checks for high-stakes outputs.

Do not use a free endpoint as the only dependency when the output affects money, health, safety, or irreparable user data. If you have no runbook for what to do when the contract fails, adding a probe will only turn silent failures into louder ones.

Free AI endpoints are useful for prototyping, but reliability is an explicit engineering choice. Add a contract probe at the boundary, make it deterministic with a local test double, and decide fail-open or fail-closed before a quota hit surprises you. If you already have free model access and a free server option, you can build and run this probe without spending anything first.