My Discord bot has one job: when someone pastes a stack trace into our server's #help channel, it rewrites the trace into a plain-English explanation plus a likely fix. It ran for months on a paid LLM API at a few dollars a month — small money, but the kind that nags you when the workload is trivially bursty: silent for days, then thirty requests in an evening when a game update breaks everyone's mods.
So I pointed it at a free endpoint instead. The migration took an afternoon. The consequences of the migration took two weeks to fully shake out, because nothing failed loudly. Everything failed slightly. This is a log of the five things that broke, what each one taught me about provider portability, and the wrapper code that now sits between my bot and any LLM endpoint so I never have to debug these twice.
The endpoint I moved to is MonkeyCode, which offers free model access and a free server option — the right price shape for a hobby bot with spiky traffic. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Every failure below, though, is generic to switching OpenAI-compatible providers. I verified that by deliberately reproducing two of them against a second endpoint. Treat this as a portability field report, not a product review.
My config had the model name hardcoded in three places: the request builder, the logging line, and a cost-estimation function that divided tokens by a per-model price. When I swapped providers, I updated one of the three. The bot ran fine — and my logs confidently recorded the old model name for a week, which made every later debugging session confusing because I was reading telemetry for a model I wasn't using.
Fix: model identity is configuration, not code, and it flows from exactly one source.
from dataclasses import dataclass
import os
@dataclass(frozen=True)
class LLMConfig:
base_url: str
api_key: str
model: str # whatever the CURRENT provider's catalog calls it
max_output_tokens: int
timeout_s: float
def load_config() -> LLMConfig:
return LLMConfig(
base_url=os.environ["LLM_BASE_URL"],
api_key=os.environ["LLM_API_KEY"],
model=os.environ["LLM_MODEL"],
max_output_tokens=int(os.environ.get("LLM_MAX_OUTPUT", "600")),
timeout_s=float(os.environ.get("LLM_TIMEOUT_S", "20")),
)
Boring, obvious, and the root cause of the messiest week. A related trap: never assume a model name means the same thing across providers, or even across months on the same provider. Catalogs change. Check what your endpoint actually serves before writing it into an environment variable, and expect to change it again.
On the paid API, I had never once seen a 429. On a free tier, the evening burst pattern hit the limit within the first weekend. The truly bad part: my error handler treated non-200 responses as "log and move on," so during peak hours the bot silently ignored half the stack traces posted. Users thought it was ignoring them specifically and reposted, which made the burst worse. A rate limit had become a feedback loop.
Fix: bounded retries with jittered backoff, and — critically — a visible degraded state.
import asyncio
import random
import openai
MAX_ATTEMPTS = 4
async def call_with_backoff(client, cfg, messages) -> str | None:
for attempt in range(MAX_ATTEMPTS):
try:
resp = await asyncio.to_thread(
client.chat.completions.create,
model=cfg.model,
messages=messages,
temperature=0.2,
max_tokens=cfg.max_output_tokens,
timeout=cfg.timeout_s,
)
return resp.choices[0].message.content
except openai.RateLimitError:
if attempt == MAX_ATTEMPTS - 1:
return None # caller must say so out loud
base = min(2 ** attempt, 8)
await asyncio.sleep(base + random.uniform(0, base))
except openai.APITimeoutError:
return None # a slow answer in a chat channel is a wrong answer
return None
async def explain_trace(client, cfg, trace: str) -> str:
result = await call_with_backoff(client, cfg, [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": trace},
])
if result is None:
return ("⚠️ I'm rate-limited right now. Try again in a minute, "
"or trim the trace to the first error line — shorter "
"messages are more likely to get through.")
return result
The degraded message does two jobs: it resets user expectations, and it gives them an actionable workaround (shorter input). Silence is the worst possible response to a rate limit in an interactive product.
The old setup had headroom I never measured, so my prompt builder happily concatenated the system prompt, the full stack trace, and the last five channel messages for context. On the new model, long traces started returning explanations of the wrong error — because the actual exception line, at the end of the paste, had been truncated away by the context window.
No exception was raised. The model just answered a different question than the user asked. This is the nastiest class of provider-migration bug: output stays plausible while becoming wrong.
Fix: budget tokens explicitly and truncate deliberately, keeping the semantically important part. For stack traces, the important part is the last exception block, not the first frames:
def trim_trace(trace: str, char_budget: int) -> str:
"""Keep the tail of the trace: the exception line matters most."""
if len(trace) <= char_budget:
return trace
head_room = char_budget // 4
tail = trace[-(char_budget - head_room):]
nl = tail.find("\n")
if 0 < nl < 200:
tail = tail[nl + 1:]
return trace[:head_room] + "\n[...middle frames omitted...]\n" + tail
Character budgets are a crude proxy for tokens, but a 4:1 char-to-token estimate with a safety margin beats not thinking about it at all. The general principle: after any provider switch, find your longest realistic input and verify end-to-end what the model actually receives.
The bot edits its own Discord message progressively as tokens arrive — a nice touch that made it feel fast. The new endpoint delivered chunks on a different cadence: larger, less frequent batches. The visual result was a message that sat empty for seconds, then jumped. Worse, my progressive-edit logic issued a Discord API edit per chunk, and the chunkier cadence interacted badly with Discord's own rate limits on message edits.
Fix: I removed streaming for this use case. Deliberate regression, correct call. For a chat bot whose answers are a few hundred tokens, perceived speed comes more from a fast first reaction than from progressive rendering. The bot now posts "🔎 reading the trace…" immediately (a fast, guaranteed reaction), then replaces it with the full answer in one edit. Users rated it as feeling faster than the stuttering stream. The lesson generalizes: provider features that seem free — streaming, JSON modes, tool calling — are exactly where behavioral differences hide, because they're the least standardized parts of an API that is otherwise deliberately compatible.
I had an uptime monitor hitting the endpoint's root URL. It stayed green through every incident above, because the server was always up; it was the model path that rate-limited, truncated, or lagged. A TCP-level health check for an LLM-backed feature is close to useless.
Fix: a synthetic probe that exercises the real path, run on a schedule, alerting on latency and content sanity:
async def synthetic_probe(client, cfg) -> dict:
start = asyncio.get_event_loop().time()
result = await call_with_backoff(client, cfg, [
{"role": "user", "content":
"Reply with exactly: PROBE_OK"},
])
elapsed = asyncio.get_event_loop().time() - start
return {
"ok": result is not None and "PROBE_OK" in result,
"latency_s": round(elapsed, 2),
}
Two weeks of probe data also gave me something I never had on the paid API: an honest picture of the free tier's behavior under my real traffic pattern — when bursts collide with limits, what p50 latency actually feels like. If you run one thing from this article, run the probe. It converts "the free tier seems flaky" into data you can act on.
The switch was worth it: the bot's operating cost for this workload went to zero, and every failure above was a latent bug in my code that the paid tier's headroom had been papering over. Free capacity is unforgiving in a useful way.
But be honest about the fit:
If you want a concrete place to try this, MonkeyCode's free model access and free server option are what this bot currently runs on; the wrapper above means it could run somewhere else tomorrow, which is precisely the property worth engineering for. The migration took an afternoon. The engineering to make the next migration boring took two weeks — and it's the part that was actually worth doing.