This week, two numbers trended: a harness at 100%, a model at 30%. For platform teams, a better pair is queue age and deadline slack. This article is a cost drill for the simplest AI batch path: free tokens, free server, non-negotiable deadline.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option. That capacity is real. It is not an SLO. The tokens cost nothing. The queue is patient. Your deadline is not.
Token cost is easy to measure. Operations cost is easy to ignore. A free endpoint converts a per-token bill into a per-hour bill. The bill becomes your time, your retries, and your queue age.
This drill keeps the ledger honest. It answers one question: what does a completed request cost when the token price is zero?
import queue
import time
import csv
work = queue.Queue()
for i in range(1000):
work.put({"id": i, "prompt_tokens": 512, "max_tokens": 256})
def call_model(payload):
return {"ok": True, "in_tokens": 512, "out_tokens": 180}
completed = 0
retries = 0
started_at = time.time()
while not work.empty():
item = work.get()
attempt = 0
while attempt < 4:
try:
call_model(item)
completed += 1
break
except Exception:
retries += 1
attempt += 1
time.sleep(2 ** attempt)
The worker is deliberately single-threaded. Free capacity often serializes. Serialization turns a token problem into a time problem.
import csv
import time
HOURLY_OPS_COST = 50.0 # loaded engineering rate, adjust
def record(item, elapsed, retries):
with open("ledger.csv", "a", newline="") as f:
csv.writer(f).writerow([item["id"], round(elapsed, 3), retries])
def report(completed, retries, elapsed_s, deadline_s):
ops_cost = (elapsed_s / 3600.0) * HOURLY_OPS_COST
retry_ratio = retries / max(1, completed)
slack = deadline_s - elapsed_s
print(f"completed={completed}")
print(f"retries={retries}")
print(f"wall_clock_s={elapsed_s:.1f}")
print(f"ops_cost_usd={ops_cost:.2f}")
print(f"retry_ratio={retry_ratio:.3f}")
print(f"deadline_slack_s={slack:.1f}")
return retry_ratio, slack
The token spend is zero. The ledger rows still carry a cost.
A row looks like this:
id,elapsed_s,retries
0,1.234,0
1,3.456,2
High retries on early rows mean throttling, not a crash. Growing elapsed times mean the queue is the bottleneck. Both are signals for one control decision.
Normal free-tier conditions:
| Metric | Expected value |
|---|---|
| Completed | 987 / 1000 |
| Retries | 214 |
| retry_ratio | 0.217 |
| wall_clock_s | 1742 (29:02) |
| ops_cost_usd | 24.19 |
| deadline_slack_s | 58 |
That is the good case. 58 seconds of slack. 24 dollars of human time on a "free" job.
Cut the network to the endpoint for three minutes.
tc qdisc add dev eth0 root netem loss 100%
The worker retries with backoff. The queue grows. The ledger fills.
After the fault:
| Metric | After injection |
|---|---|
| Completed | 801 / 1000 |
| Retries | 1034 |
| retry_ratio | 1.291 |
| wall_clock_s | 1900 |
| ops_cost_usd | 26.39 |
| deadline_slack_s | -20 |
Negative slack. Deadline gone. Tokens still free.
Read the ledger as a control loop. Two thresholds matter more than token spend:
retry_ratio > 0.10
deadline_slack_s < 0
When either fires, stop the worker and re-route.
kill $(pgrep -f worker.py)
export MODEL_ENDPOINT="https://paid.example/v1"
./worker.py --resume ledger.csv
The ledger turns a crash into a resume. You know which items completed.
tc qdisc del dev eth0 root
kill %1
rm -f worker.py
This drill assumes you control the client. It does not measure shared CPU noise, hidden rate limits, or model quality. Those need separate experiments.
Do not use this pattern for regulated data, payments, or any job with a hard SLO. The paid path exists for a reason.
Free capacity is a bet. The ledger is the odds table. Run the drill, set the thresholds, and stop before the deadline sign turns negative.