{"slug": "llm-gauze-fixes-llm-errors", "title": "LLM-gauze fixes LLM errors", "summary": "Llm-gauze, an HTTP gateway published as the container image ghcr.io/ggeorgovassilis/llm-gauze:latest, sits in front of a local OpenAI-compatible LLM and remediates transient errors, hung or looping models, empty responses, context-window overflows, runaway reasoning and malformed tool calls before the client sees them. The gateway listens on http://localhost:9317, forwards requests to the LLM_BASE_URL set in .env, and reports HTTP 502 with invalid_upstream_response instead of fabricating a completion when an upstream HTTP 200 carries no completion choices. The write-up warns that Hetzner Inference requires LLM_BASE_URL=https://inference.hetzner.com/api, that .env changes need docker compose up -d --force-recreate gateway, and that VS Code Copilot custom endpoints must set apiType: \"chat-completions\".", "body_md": "llm-gauze is an HTTP gateway that sits in front of a local LLM (served via an OpenAI-compatible API) and works around its shortcomings: transient errors without retries, silently hung or looping models, empty or sloppy responses, context-window overflows, runaway reasoning, and malformed tool calls. It logs every exchange and remediates what it can before the client ever sees it.\n\nllm-gauze ships as a published container image and runs with Docker Compose.\n\n1. \nCreate your configuration: \n\n```\ncp .env.example .env\n```\n\n2. \nPoint `LLM_BASE_URL` in`.env` at your local LLM's OpenAI-compatible\nendpoint (default`http://host.docker.internal:14434` , which reaches a\nhost-side LLM on port 14434 from inside the container).\n3. \nStart the gateway: \n\n```\ndocker compose up\n```\n\n This runs the published image `ghcr.io/ggeorgovassilis/llm-gauze:latest` (defined in`docker-compose.yml` ). To pin a specific release, override the\ntag — for example`ghcr.io/ggeorgovassilis/llm-gauze:7` .\n\nThe gateway listens on `http://localhost:9317` and exposes an\nOpenAI-compatible API (e.g. `POST /v1/chat/completions`), forwarding to the\n`LLM_BASE_URL` in `.env`.\n\nThe gateway appends the client's request path to `LLM_BASE_URL`. For Hetzner\nInference, set `LLM_BASE_URL=https://inference.hetzner.com/api` and point a\nchat-completions client at `http://localhost:9317` so `/v1/chat/completions`\nreaches `/api/v1/chat/completions` upstream. Do not include `/v1` twice or use\nthe provider's website root as the API base.\n\nAfter changing `.env`, recreate the service; restarting an existing container\ndoes not reload its environment:\n\n```\ndocker compose up -d --force-recreate gateway\n```\n\nFor local development, use\n`docker compose -f docker-compose.dev.yml up -d --build --force-recreate gateway`.\nCheck `/health` afterwards: its `upstream` must show the intended API base,\nincluding `/api` for Hetzner. Rebuilding an image alone does not correct a\nwrong value in `.env`.\n\nFor VS Code Copilot custom endpoints, explicitly select\nprovider-level `apiType: \"chat-completions\"` for both the direct provider and\ngateway entries.\nThe direct entry's URL is `https://inference.hetzner.com/api/v1`; the gateway\nentry's URL is `http://localhost:9317`. Supply the provider's bearer token in\nthe client's `Authorization` request header; the gateway forwards it unchanged.\nNever put credentials in committed configuration.\n\nThe current Copilot `customendpoint` implementation gives an explicit\n`Authorization` request header precedence over its provider API key. Without\nthat header, the provider API key must be the real token, not a placeholder.\n\nAn upstream HTTP 200 without completion choices is not a successful model\nreply. With streaming remediation enabled, the gateway reports HTTP 502 with\n`invalid_upstream_response` rather than fabricating a completion or an empty\nSSE success. Check the upstream API path and streaming support first. Upstream\nHTTP errors retain their status and body.\n\nFor an immediate VS Code 502, inspect `data/records.jsonl` structurally rather\nthan dumping recordings or private conversations. Match the client's\n`x-request-id` in `request_headers` to the gateway's `request_id`, then compare\nthe record timestamp and attempt `duration` with container completion logs.\n`GitHubCopilotChat` in the user agent distinguishes VS Code traffic from smoke\nprobes. A recorded `invalid_upstream_response` is a gateway protocol rejection,\nnot evidence that the provider returned HTTP 502: the rejected upstream status\nand content type are not retained in that record. Verify the running API base\nand compare a synthetic request directly when those details are needed.\nThe Hetzner website-root route has returned HTTP 200 with nine non-SSE bytes\nand `application/octet-stream`, producing this immediate rejection even for\na valid chat-completions request with tools.\n\nTemporary Compose overrides do not persist an `.env` correction. Before using\nthe ordinary start command again, privately set the correct `LLM_BASE_URL`\nand recreate the gateway. A corrected route alone does not prove a model\ncan complete the VS Code workflow. Compare the relevant request shape,\nincluding tools, message roles and `stream_options`, not just an OK-only curl.\nUse synthetic content and bounded time/output for live probes, and report any\nadded token cap as a difference from the recorded request.\n\nIf authenticated generation still times out, compare the same small request\ndirectly against the provider. Record time to response headers and first SSE\ndata separately from total duration: no response headers is not evidence of\nongoing reasoning. Authenticated `/models` discovery proves API reachability\nand token acceptance, but not model readiness. A successful completion from\nanother advertised model with the same token helps isolate a model-specific\nprovider failure; report that comparison to the provider rather than silently\nchanging models or treating a gateway health check as successful generation.\n\nThe container runs as your host user (`UID`/` GID`, default `1000`) so the\nfiles it writes to the bind-mounted `data/` directory are owned by you, not\nroot. If a `data/` directory was created earlier as root, fix ownership once\nbefore the non-root container can write to it:\n\n```\nsudo chown -R \"$(id -u):$(id -g)\" data\ncurl http://localhost:9317/health\n```\n\nReturns `{\"status\": \"ok\", \"upstream\": \"<LLM_BASE_URL>\"}` when the gateway is up.\nOperational metrics are exposed at `/metrics` (Prometheus text format, or JSON\nwith `Accept: application/json`).\n\nEvery setting is documented in [`docs/configuration.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/docs/configuration.md).\n\nllm-gauze sits between your client and the local LLM, recording every exchange:\n\n``` php\nflowchart LR\n    Client[Your client] -->|OpenAI-compatible API| Gauze[llm-gauze]\n    Gauze -->|forwards| LLM[Local LLM]\n    Gauze -.->|logs every exchange| Store[(data/*.jsonl)]\n```\n\nWhen the model misbehaves, llm-gauze detects it and remediates what it can before you ever see it — retrying transient failures, nudging empty replies, cleaning leaked thinking tags, breaking loops with varied sampling:\n\n```\nsequenceDiagram\n    participant C as Client\n    participant B as llm-gauze\n    participant L as Local LLM\n\n    C->>B: POST /v1/chat/completions\n    B->>L: forward\n    L-->>B: error, hang, loop, or sloppy reply\n    B->>B: detect & classify\n    B->>L: remediate (retry / nudge / repair)\n    L-->>B: clean completion\n    B-->>C: chat.completion\n```\n\nSee [`docs/architecture.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/docs/architecture.md) for the full design and\n[`docs/configuration.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/docs/configuration.md) for every setting.\n\nSee [`DEVELOPING.md`](https://github.com/ggeorgovassilis/llm-gauze/blob/main/DEVELOPING.md) for local development, running the tests,\nand adding a detector or remediation.", "url": "https://wpnews.pro/news/llm-gauze-fixes-llm-errors", "canonical_source": "https://github.com/ggeorgovassilis/llm-gauze", "published_at": "2026-10-15 08:00:08+00:00", "updated_at": "2026-10-15 08:18:33.086066+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-infrastructure", "developer-tools", "mlops"], "entities": ["llm-gauze", "Docker Compose", "Hetzner Inference", "VS Code Copilot", "ghcr.io/ggeorgovassilis/llm-gauze", "GitHubCopilotChat"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-gauze-fixes-llm-errors", "markdown": "https://wpnews.pro/news/llm-gauze-fixes-llm-errors.md", "text": "https://wpnews.pro/news/llm-gauze-fixes-llm-errors.txt", "jsonld": "https://wpnews.pro/news/llm-gauze-fixes-llm-errors.jsonld"}}