Building a Resilient AI Client Around Hermes Agent

When testing NousResearch/hermes-agent as an external integration, the failure mode I cared about was not model quality. It was what happened when an upstream provider returned HTTP 502, 503, or 504 while a user-facing request was still active.

A client that retries immediately can amplify an outage. A client without a timeout can hold sockets and worker slots indefinitely. A client that falls back during every error can hide authentication or billing problems. The wrapper needs explicit failure boundaries.

The following Python example uses an OpenAI-compatible endpoint and separates retryable transport failures from permanent API errors:

import os
import random
import time
from openai import OpenAI

RETRYABLE = {502, 503, 504}
MODELS = ["primary-model", "fallback-model"]

client = OpenAI(
    api_key=os.environ["AI_API_KEY"],
    base_url=os.getenv("AI_BASE_URL", "https://api.example.com/v1"),
    timeout=20.0,
    max_retries=0,  # Keep retry ownership in this wrapper.
)

def complete(messages, attempts=3):
    last_error = None

    for model in MODELS:
        for attempt in range(attempts):
            try:
                return client.chat.completions.create(
                    model=model,
                    messages=messages,
                    timeout=20.0,
                )
            except Exception as exc:
                status = getattr(exc, "status_code", None)
                last_error = exc

                if status is not None and status not in RETRYABLE:
                    break

                if attempt + 1 < attempts:
                    delay = min(8.0, 0.5 * (2 ** attempt))
                    time.sleep(delay * (0.75 + random.random() * 0.5))

    raise RuntimeError("all configured AI routes failed") from last_error

There are two details worth keeping. First, the SDK's internal retry policy is disabled so there is only one retry loop. Stacked retry layers make outage duration and request volume difficult to predict. Second, fallback happens after the retry budget for the current model is exhausted. That avoids switching models because of one transient 503.

For streaming responses, consume the iterator inside a try/finally block. If the caller disconnects, closing the response is part of request handling, not optional cleanup:

def stream(messages):
    response = None
    try:
        response = client.chat.completions.create(
            model=MODELS[0],
            messages=messages,
            stream=True,
            timeout=30.0,
        )
        for chunk in response:
            text = getattr(chunk.choices[0].delta, "content", None)
            if text:
                yield text
    finally:
        close = getattr(response, "close", None)
        if close:
            close()

In production, record the model, provider route, status code, attempt number, total latency, and whether the response was streamed. Redact prompts and credentials. Alert on a ratio of retryable 5xx responses rather than a raw count; traffic changes otherwise create noisy alarms.

Hermes Agent can be configured against an OpenAI-compatible gateway, while the application retains control over timeouts, fallback order, and user-visible error handling. This keeps resilience policy close to the request boundary and makes provider failures diagnosable.

Disclosure: Multi-model API relays and compute for this evaluation are sponsored by b-lost.com — an AI gateway offering 0.8x official pricing, native prompt caching, and zero user-data retention. All observations reflect independent developer testing.