{"slug": "i-built-a-tiny-proxy-that-gives-glm-5-2-vision-or-any-text-llm-mit", "title": "I built a tiny proxy that gives GLM 5.2 vision (or any text LLM) – MIT", "summary": "Developer EekoSystems released VisionBridge, an open-source proxy that gives text-only large language models vision capabilities by routing image analysis to a separate vision model. The MIT-licensed tool sits between chat interfaces and reasoning models, enabling targeted visual inspection, OCR, and document scanning without retraining. It supports multiple backends including LM Studio, Ollama, and vLLM, and is available via Docker or Python.", "body_md": "Give text-only LLMs vision through a tiny OpenAI-compatible proxy.\n\nVisionBridge sits between your chat UI and your models. It lets a reasoning model from LM Studio, Ollama, vLLM, Z.ai, OpenRouter, or any OpenAI-compatible backend inspect images by asking a separate vision model targeted questions.\n\nNo training. No weights. Bring your own reasoning model and your own vision model.\n\n```\nOpenWebUI / LibreChat / app\n        |\n        v\nVisionBridge at /v1/chat/completions\n        |\n        +-- reasoning model: LM Studio / Ollama / vLLM / GLM / DeepSeek\n        |\n        +-- vision model: LLaVA / Qwen-VL / GLM-V / GPT-4o-mini / any VLM\n```\n\nVisionBridge receives an OpenAI-style chat request. If the request contains images, it stores the images, rewrites the prompt for the reasoning model, and lets the reasoning model call tools:\n\n`look(image_id, question)`\n\nfor targeted visual inspection`ocr(image_id)`\n\nfor text extraction`scan(image_id)`\n\nfor a tiled full-document text sweep of dense pages`crop_and_look(image_id, box, question)`\n\nfor zooming into a region (small crops are automatically upscaled before the VLM sees them)`compare(image_id_a, image_id_b, question)`\n\nfor judging two images together\n\nThe vision model does the seeing. The reasoning model does the thinking.\n\nBefore the loop starts, every image gets a one-shot **scene description** from\nthe vision model (in parallel, cached by content), so the reasoning model\nbegins with global context and spends its tool budget on details. Multiple\ntool calls in one turn run **concurrently**, and repeated images across a\nconversation are served from a content-addressed cache instead of being\nre-fetched and re-decoded.\n\nTool calling uses the backend's **native OpenAI function calling** when\navailable. In the default `TOOL_MODE=auto`\n\n, VisionBridge probes the reasoning\nbackend once and transparently falls back to a prompt-JSON protocol for\nbackends and models that don't support tools — so it works with strong hosted\nmodels and weak local ones alike.\n\nThe loop is built to survive imperfect models: hallucinated image ids, bad crop boxes, malformed JSON, and vision-backend hiccups are fed back to the reasoning model as observations so it can correct itself, and hitting the inspection limit triggers a final best-effort answer instead of an error.\n\n```\ndocker run -p 8080:8080 \\\n  -e REASONING_BASE_URL=http://host.docker.internal:1234/v1 \\\n  -e REASONING_API_KEY=lm-studio \\\n  -e REASONING_MODEL=local-model \\\n  -e VISION_BASE_URL=http://host.docker.internal:11434/v1 \\\n  -e VISION_API_KEY=ollama \\\n  -e VISION_MODEL=llava:13b \\\n  ghcr.io/eekosystems/visionbridge:latest\n```\n\nThen point your OpenAI-compatible client at:\n\n```\nBase URL: http://localhost:8080/v1\nAPI key: anything\nModel: visionbridge\ncd visionbridge\npython -m venv .venv\nsource .venv/bin/activate    # Windows: .venv\\Scripts\\activate\npip install -e \".[dev]\"\ncp .env.example .env         # Windows: copy .env.example .env\nvisionbridge serve --host 0.0.0.0 --port 8080\n```\n\nCheck your backend configuration any time with:\n\n```\nvisionbridge doctor\n```\n\nIt verifies both backends are reachable, that the configured models exist, and that a test completion succeeds.\n\nAll settings come from environment variables (or `.env`\n\n):\n\n| Variable | Default | Purpose |\n|---|---|---|\n`REASONING_BASE_URL` |\n`http://localhost:1234/v1` |\nReasoning backend |\n`REASONING_API_KEY` |\n`local` |\nReasoning API key |\n`REASONING_MODEL` |\n`local-model` |\nReasoning model id |\n`VISION_BASE_URL` |\n`http://localhost:11434/v1` |\nVision backend |\n`VISION_API_KEY` |\n`local` |\nVision API key |\n`VISION_MODEL` |\n`llava:13b` |\nVision model id |\n`BRIDGE_MODEL_NAME` |\n`visionbridge` |\nModel name the proxy advertises |\n`TOOL_MODE` |\n`auto` |\n`auto` , `native` , or `prompt` tool calling |\n`MAX_VISION_CALLS` |\n`8` |\nReasoning turns before a forced final answer |\n`REQUEST_TIMEOUT_SECONDS` |\n`120` |\nPer-backend-call timeout |\n`BACKEND_MAX_RETRIES` |\n`2` |\nRetries on 429/5xx/transport errors |\n`SCENE_PRIMING` |\n`true` |\nDescribe each image once before the loop starts |\n`ALLOW_FILE_URLS` |\n`false` |\nAllow `file://` image URLs |\n`ALLOW_PRIVATE_URLS` |\n`false` |\nAllow image fetches from private/internal hosts |\n`MAX_IMAGE_BYTES` |\n`20971520` |\nMax bytes per image |\n`MAX_IMAGES_PER_REQUEST` |\n`16` |\nMax images per request |\n`IMAGE_CACHE_ENTRIES` |\n`64` |\nContent-addressed image cache size |\n`ENABLE_TRACES` |\n`true` |\nKeep tool-loop traces for `/v1/traces/{id}` |\n`BRIDGE_API_KEYS` |\n(empty) |\nComma-separated keys; when set, `/v1/*` requires one |\n`EXTRA_MODELS` |\n(empty) |\nJSON map of extra model names to backend pairs |\n`LOG_JSON` |\n`false` |\nEmit JSON log lines |\n\nOne instance can serve several reasoning+vision pairs under different model names. Unspecified fields inherit from the default pair:\n\n```\nEXTRA_MODELS={\"visionbridge-fast\": {\"reasoning\": {\"model\": \"qwen3:8b\"}, \"vision\": {\"model\": \"llava:7b\"}}}\n```\n\n(or point `EXTRA_MODELS_FILE`\n\nat a JSON file). All names appear in\n`GET /v1/models`\n\n, and clients select a pair with the `model`\n\nfield.\n\n- Open LM Studio.\n- Load your reasoning model.\n- Go to the Developer tab and start the local server.\n- Use the default base URL:\n\n```\nREASONING_BASE_URL=http://localhost:1234/v1\nREASONING_API_KEY=lm-studio\nREASONING_MODEL=local-model\n```\n\nLM Studio can also be used as the vision backend if you load a vision-capable model and expose it through the same OpenAI-compatible server.\n\n```\nREASONING_BASE_URL=http://localhost:11434/v1\nREASONING_API_KEY=ollama\nREASONING_MODEL=qwen3:32b\n\nVISION_BASE_URL=http://localhost:11434/v1\nVISION_API_KEY=ollama\nVISION_MODEL=llava:13b\nREASONING_BASE_URL=http://localhost:8000/v1\nREASONING_API_KEY=token-abc123\nREASONING_MODEL=Qwen/Qwen3-32B\n\nVISION_BASE_URL=http://localhost:8001/v1\nVISION_API_KEY=token-abc123\nVISION_MODEL=Qwen/Qwen3-VL-8B-Instruct\n```\n\nIn OpenWebUI:\n\n``` php\nAdmin Panel -> Settings -> Connections -> OpenAI-compatible\nBase URL: http://host.docker.internal:8080/v1\nAPI Key: anything\nModel: visionbridge\n```\n\nVisionBridge intentionally exposes the small subset most clients need:\n\n`GET /health`\n\n(add`?probe=true`\n\nto verify both backends are reachable)`GET /v1/models`\n\n`POST /v1/chat/completions`\n\n`GET /v1/traces/{completion_id}`\n\n— the full tool-loop trace for a recent request: every tool call, its arguments, the vision model's observation, and timings. Invaluable for answering \"why did it say that?\"\n\nResponses include a `usage`\n\nblock aggregated across every reasoning and vision\ncall made for the request, and an `X-Request-ID`\n\nheader for correlating logs.\n\n**Streaming**: image-free requests are streamed token-by-token straight from\nthe reasoning backend. Requests with images stream too: SSE heartbeats keep\nthe connection alive while tools run, and in native tool-calling mode the\nfinal answer is forwarded token-by-token as the reasoning model produces it\n(prompt-JSON mode buffers the loop and streams the answer in chunks).\n\n**Errors**: backend failures return HTTP 502, invalid inputs 400.\n\n**Auth**: set `BRIDGE_API_KEYS=key1,key2`\n\nto require a Bearer key on `/v1/*`\n\nendpoints (`/health`\n\nstays open). For anything internet-facing, still prefer a\nreverse proxy with TLS in front.\n\nImage URL fetching is hardened by default: hosts resolving to private, loopback, or link-local addresses are rejected (SSRF guard), redirects are validated hop-by-hop, downloads are size-capped, and payloads must decode as real images. Run VisionBridge behind your own reverse proxy / auth layer if you expose it beyond localhost — it deliberately ships without built-in auth.\n\n`benchmarks/`\n\ncontains a harness that scores VisionBridge against a direct\nVLM (relaxed accuracy + ANLS) on DocVQA-style JSONL task sets. See\n[benchmarks/README.md](/thomasunise/visionbridge/blob/main/benchmarks/README.md).\n\nActively developed. See [CHANGELOG.md](/thomasunise/visionbridge/blob/main/CHANGELOG.md) for what's new in 0.2.0.\nNext up:\n\n- published benchmark numbers against caption-only prompting\n- persistent on-disk image/primer cache\n- OpenWebUI and LibreChat screenshots\n\nContributions welcome — see [CONTRIBUTING.md](/thomasunise/visionbridge/blob/main/CONTRIBUTING.md).\n\nMIT", "url": "https://wpnews.pro/news/i-built-a-tiny-proxy-that-gives-glm-5-2-vision-or-any-text-llm-mit", "canonical_source": "https://github.com/thomasunise/visionbridge", "published_at": "2026-07-07 19:37:55+00:00", "updated_at": "2026-07-07 20:00:56.172500+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "computer-vision", "developer-tools"], "entities": ["EekoSystems", "VisionBridge", "LM Studio", "Ollama", "vLLM", "OpenAI", "GLM", "DeepSeek"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-tiny-proxy-that-gives-glm-5-2-vision-or-any-text-llm-mit", "markdown": "https://wpnews.pro/news/i-built-a-tiny-proxy-that-gives-glm-5-2-vision-or-any-text-llm-mit.md", "text": "https://wpnews.pro/news/i-built-a-tiny-proxy-that-gives-glm-5-2-vision-or-any-text-llm-mit.txt", "jsonld": "https://wpnews.pro/news/i-built-a-tiny-proxy-that-gives-glm-5-2-vision-or-any-text-llm-mit.jsonld"}}