Building a Resilient AI Client Around Hermes Agent

A developer outlined a resilient Python client pattern for integrating NousResearch's hermes-agent against OpenAI-compatible endpoints, focusing on handling upstream HTTP 502, 503, and 504 errors without amplifying outages. The wrapper disables the SDK's built-in retries, applies exponential backoff with jitter, and only falls back to a secondary model after exhausting the retry budget for the current one. The writeup also covers streaming cleanup via try/finally and production logging of model, route, status code, attempt number, and latency while redacting prompts and credentials.

When testing NousResearch/hermes-agent as an external integration, the failure mode I cared about was not model quality. It was what happened when an upstream provider returned HTTP 502, 503, or 504 while a user-facing request was still active. A client that retries immediately can amplify an outage. A client without a timeout can hold sockets and worker slots indefinitely. A client that falls back during every error can hide authentication or billing problems. The wrapper needs explicit failure boundaries. The following Python example uses an OpenAI-compatible endpoint and separates retryable transport failures from permanent API errors: python import os import random import time from openai import OpenAI RETRYABLE = {502, 503, 504} MODELS = "primary-model", "fallback-model" client = OpenAI api key=os.environ "AI API KEY" , base url=os.getenv "AI BASE URL", "https://api.example.com/v1" , timeout=20.0, max retries=0, Keep retry ownership in this wrapper. def complete messages, attempts=3 : last error = None for model in MODELS: for attempt in range attempts : try: return client.chat.completions.create model=model, messages=messages, timeout=20.0, except Exception as exc: status = getattr exc, "status code", None last error = exc Do not retry credentials, malformed requests, or quota errors. if status is not None and status not in RETRYABLE: break if attempt + 1 < attempts: delay = min 8.0, 0.5 2 attempt time.sleep delay 0.75 + random.random 0.5 raise RuntimeError "all configured AI routes failed" from last error There are two details worth keeping. First, the SDK's internal retry policy is disabled so there is only one retry loop. Stacked retry layers make outage duration and request volume difficult to predict. Second, fallback happens after the retry budget for the current model is exhausted. That avoids switching models because of one transient 503. For streaming responses, consume the iterator inside a try/finally block. If the caller disconnects, closing the response is part of request handling, not optional cleanup: python def stream messages : response = None try: response = client.chat.completions.create model=MODELS 0 , messages=messages, stream=True, timeout=30.0, for chunk in response: text = getattr chunk.choices 0 .delta, "content", None if text: yield text finally: close = getattr response, "close", None if close: close In production, record the model, provider route, status code, attempt number, total latency, and whether the response was streamed. Redact prompts and credentials. Alert on a ratio of retryable 5xx responses rather than a raw count; traffic changes otherwise create noisy alarms. Hermes Agent can be configured against an OpenAI-compatible gateway, while the application retains control over timeouts, fallback order, and user-visible error handling. This keeps resilience policy close to the request boundary and makes provider failures diagnosable. Disclosure: Multi-model API relays and compute for this evaluation are sponsored by b-lost.com https://b-lost.com?utm source=devto&utm medium=tech blog&utm campaign=devto bot 8 — an AI gateway offering 0.8x official pricing, native prompt caching, and zero user-data retention. All observations reflect independent developer testing.