Free Endpoints Are a Contract, Not a Gift: A Fit Test for Agent Workloads

A developer argues that free model endpoints are a contract with rate limits and queueing, not a gift, and that cost-per-token benchmarks fail to predict agent workload performance. They propose a three-question fit test and provide a probe script to measure an endpoint's behavior under realistic bursty traffic, showing that traffic shape, data sensitivity, and operational slack determine whether free tiers or self-hosting are the right choice.

Free model access is not a gift. It is a contract with someone else's rate limits, queueing policy, and maintenance schedule. Self-hosting inverts that contract: you own the latency, the GPU, and the 2 a.m. page. Most teams choose between the two by comparing price per token, and that is exactly how they end up with a production agent that stalls at 9:15 every morning. Agent workloads are moving from demos to production, and the conversation has shifted from what models can do to what they cost to operate. The problem is that agent traffic does not look like chat traffic. A coding agent emits bursts of small requests — a tool call, a diff review, a short completion — separated by long idle gaps. That shape punishes endpoints optimized for steady throughput. A cost-per-token benchmark measures unit price, not whether the endpoint survives your burst pattern. The only honest test is to probe the endpoint the way your agent will actually call it. Three questions decide the fit before any pricing math. First, what is your traffic shape: steady, bursty, or spiky? Second, what happens to your data when it crosses a third-party boundary? Third, how much operational slack do you have — can you babysit a self-hosted model, or does the endpoint need to be someone else's problem? Free hosted tiers win when the answers are steady, non-sensitive, and no-slack; self-hosting wins when they are spiky, sensitive, and you have the time. Consider a concrete case. A background job that summarizes a few documents an hour is steady and forgiving; a free tier is almost certainly fine. An interactive coding agent that fires eight parallel tool calls while a developer waits is spiky and latency-sensitive; the same free tier can feel like a different product. Here is a probe you can run against any OpenAI-compatible endpoint. It fires a fixed number of requests at a fixed concurrency, retries once after a 429, and reports success rate, rate-limit events, and latency percentiles. Run it twice: once at concurrency 1 for a baseline, once at the concurrency your agent actually uses. The difference between those two runs is the real cost of the endpoint. '''probe endpoint.py — fit test for a free or cheap model endpoint. Usage: export ENDPOINT URL='https://...' export ENDPOINT KEY='your-key' export PROBE MODEL='model-name' python probe endpoint.py --requests 60 --concurrency 4 ''' import argparse import asyncio import os import statistics import time import httpx async def call once client, url, headers, payload : t0 = time.perf counter try: r = await client.post url, headers=headers, json=payload, timeout=60 return r.status code, time.perf counter - t0 1000 except Exception as exc: return 0, time.perf counter - t0 1000 async def worker client, url, headers, payload, sem, results : async with sem: status, ms = await call once client, url, headers, payload if status == 429: one recovery attempt: wait, then resend await asyncio.sleep 2 status, ms = await call once client, url, headers, payload results.append 'recovered', status, ms else: results.append 'direct', status, ms async def main : ap = argparse.ArgumentParser ap.add argument '--requests', type=int, default=60 ap.add argument '--concurrency', type=int, default=4 ap.add argument '--prompt', default='Reply with the single word ok.' args = ap.parse args url = os.environ 'ENDPOINT URL' key = os.environ 'ENDPOINT KEY' model = os.environ 'PROBE MODEL' headers = {'Authorization': f'Bearer {key}', 'Content-Type': 'application/json'} payload = { 'model': model, 'messages': {'role': 'user', 'content': args.prompt} , 'max tokens': 8, } sem = asyncio.Semaphore args.concurrency results = async with httpx.AsyncClient as client: tasks = worker client, url, headers, payload, sem, results for in range args.requests await asyncio.gather tasks ok = ms for , status, ms in results if status == 200 limited = ms for kind, , ms in results if kind == 'recovered' failed = ms for , status, ms in results if status not in 200, 429 print f'requests={len results } ok={len ok } rate limited={len limited } failed={len failed }' if ok: ok.sort p95 = ok min len ok - 1, int len ok 0.95 print f'latency ms p50={statistics.median ok :.0f} p95={p95:.0f} max={ok -1 :.0f}' if limited: print '429s observed; the probe waited 2s and retried once. If recovered entries are 429 again, the endpoint needs a longer cooldown than your agent timeout.' if name == ' main ': asyncio.run main Read the output like this. If p95 latency sits close to p50, the endpoint queues fairly under load. If p95 is three times p50, you are seeing contention, and your agent's timeouts will fire at the worst possible moment. If 429s appear at your expected burst size, the free tier's contract does not match your traffic shape. No retry logic fixes that; it only converts a rate limit into a token incinerator. Record four numbers for each run: success rate, 429 count, p50, and p95. That is your endpoint's signature. Compare the signature at concurrency 1 and concurrency 8; if p95 doubles while success rate drops, the endpoint is not built for your agent's parallel tool calls. Some agents fan out ten tool calls at once, and a free tier that handles one request gracefully can still fail that pattern. The probe results feed directly into the decision table below. A high 429 count at your working concurrency moves you to the "risky" row regardless of how cheap the tokens are. | Workload shape | Free hosted tier | Self-hosted | |---|---|---| | Dev/test, low volume | Fits | Overkill | | Steady production, non-sensitive | Fits with a fallback | Predictable but costly | | Bursty, latency-sensitive | Risky | Better control | | Regulated or private data | Avoid | Required | | No operational slack | Fits | Do not attempt | Notice what is missing from the table: price. Price decides which self-hosted option you pick, not whether you self-host. The free tier's real cost is coupling — your agent's availability inherits someone else's queue, and your p95 becomes their problem. A limitation like that is informative: it forces you to design backoff, fallbacks, and timeouts before you need them, which is exactly the discipline a production agent requires anyway. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option; at the time of writing, the free tier includes a 10M-token allowance. I have not benchmarked it here, and you should not trust a benchmark you cannot reproduce. What makes it relevant to this guide is that it is a legitimate candidate for the "free hosted tier" column — which means it deserves the same probe as any other endpoint. Point the script at its server, run the two concurrency passes, and record what you see. Who should not use this approach? Teams with regulated data, hard latency ceilings, or predictably spiky traffic. Also anyone who treats a free tier as a permanent contract: free tiers change quotas, models, and terms without notice. Re-run the probe on a schedule, and configure a fallback endpoint before you need it, not after the first 429 in production. The cheapest endpoint is the one that fails at the wrong moment. The probe costs twenty minutes and a few hundred tokens; the outage it prevents costs considerably more. Run it against MonkeyCode's free server if you want a concrete data point, but let the numbers — not the promise — make the decision.