{"slug": "vlmhub-simple-unified-python-interface-for-vision-language-models", "title": "Vlmhub – Simple Unified Python interface for vision-language models", "summary": "A developer released Vlmhub, an open-source Python library that provides a single Model.generate() interface for running interleaved text-and-image prompts across Ollama, MLX-VLM, vLLM, HuggingFace Transformers, Google Gemini, Vertex AI, OpenAI, Anthropic and OpenRouter. Vlmhub routes cloud and local-server backends through LiteLLM, while its Transformers backend runs models in-process with optional 4-bit quantization and a fallback_dtype for pre-Ampere GPUs such as the V100 that lack native bfloat16. The library ships a JSON model registry with pre-configured entries including Gemma 3 (4B, 12B) and Qwen3-VL (4B, 8B), and installs without PyTorch unless the transformers extra is added.", "body_md": "A lightweight, config-driven framework for **unified vision-language model inference** across local and cloud backends. Run multimodal prompts — interleaved text and images — against Ollama, MLX-VLM, vLLM, HuggingFace Transformers, Gemini, Vertex AI, OpenAI, or Anthropic without rewriting any inference code.\n\n``` python\nfrom vlmhub import Model, TextBlock, ImageBlock\n\nmodel = Model(\"ollama/gemma3-4b\")            # local server\n# model = Model(\"gemini/flash-3.5lite\")      # cloud API\n# model = Model(\"transformers/qwen3vl-8b\")   # in-process, no server\n\nreply = model.generate([\n    TextBlock(\"Here are two images:\"),\n    ImageBlock(image_path=\"image1.png\"),     # an image file on disk\n    ImageBlock(image=pil_image),             # or a PIL image in memory\n    TextBlock(\"What are the main differences between them?\"),\n])\nprint(reply[\"text\"])\n```\n\nAn `ImageBlock` takes an image file (`image_path=`) or a PIL image in memory (` image=`), such as a video frame.\n\n- **One interface, every backend** — a single`Model.generate()` call over`TextBlock` and`ImageBlock` , independent of model or backend.\n- **Interleaved multimodal prompts** — mix`TextBlock` and`ImageBlock` in any order; each backend encodes as it needs (base64 data URI over the wire, PIL in-process).\n- **A model registry** — models, settings and generation defaults live in one editable JSON file; give a model any short name you like alongside its real model id.\n- **A hardware-aware Transformers backend** —`transformers` runs a model in-process with an optional 4-bit quantized load and a`fallback_dtype` for pre-Ampere GPUs (e.g. V100) that lack native bfloat16;[LiteLLM](https://docs.litellm.ai) covers every other hosting.\n\n| Category | Provider | Hosting Key | Backend | How it runs | \n|---|---|---|---|---|\n| **Local Server** | Ollama | `ollama` | `litellm` | Local server on port 11434 | \n| **Local Server** | MLX-VLM | `mlx_vlm` | `litellm` | Apple Silicon, port 8080 | \n| **Local Server** | vLLM | `vllm` | `litellm` | CUDA GPU, port 8000 | \n| **In-Process** | HuggingFace Transformers | `transformers` | `transformers` | Direct model loading (CUDA / MPS / CPU) | \n| **Cloud API** | Google Gemini (AI Studio) | `gemini` | `litellm` | LiteLLM `gemini/` provider;`thinking_budget` (2.5) or`reasoning_effort` (3.x) | \n| **Cloud API** | Google Gemini via Vertex AI | `vertex_ai` | `litellm` | LiteLLM `vertex_ai/` , Application Default Credentials | \n| **Cloud API** | OpenAI | `openai` | `litellm` | GPT-4o, GPT-4o-mini, GPT-4.1, GPT-5.x | \n| **Cloud API** | Anthropic | `anthropic` | `litellm` | LiteLLM `anthropic/` provider | \n| **Cloud API** | OpenRouter | `openrouter` | `litellm` | GPT, Claude and Gemini with one key | \n\nPre-configured models include **Gemma 3** (4B, 12B) and **Qwen3-VL** (4B, 8B) across all local hostings — plus Qwen2.5-VL, InternVL3.5, mPLUG-Owl3, Molmo2, Idefics3 and LLaVA-OneVision under `transformers` — open-weight judges (Qwen3.6/3.8, Prometheus, CompassJudger, UnifiedReward, gpt-oss) under `vllm`, and Gemini (2.5 / 3.x), GPT-4o/4.1/5.x, Claude 4.5+ and Gemma 4 (via Gemini) for cloud.\n\nPick whichever fits: **A** to try vlmhub out or work on it, **B** to install it as a library into another project.\n\nThe repo carries the test scripts in [`tests/`](https://github.com/dbiswas55/vlmhub/blob/main/tests), so this is the way to see it running end to end.\n\n```\ngit clone https://github.com/dbiswas55/vlmhub.git\ncd vlmhub\n\npython3 -m venv venv312\nsource venv312/bin/activate      # macOS / Linux\n# venv312\\Scripts\\activate       # Windows\n\npip install -e .\n```\n\nThat covers every **API-hosted** backend — LiteLLM handles Gemini, Vertex AI, OpenAI, Anthropic\nand all OpenAI-compatible local servers (Ollama, MLX-VLM, vLLM), so there are no per-hosting SDKs\nto add. PyTorch is not installed: only the in-process `transformers` hosting needs it.\n\n```\npip install -e \".[transformers]\"   # torch + transformers, for in-process model loading\npip install -e \".[quantization]\"   # adds bitsandbytes, for 4-bit loads under `transformers`\npip install -e \".[ollama]\"         # only to manage Ollama's store via local_models.py\npip install -e \".[all]\"            # everything above\n```\n\nAdd vlmhub as a submodule, then install it from there. The source sits in your project, readable and\nupgradeable, and `pip` handles the dependencies and the import path.\n\n```\ncd ~/my-project\ngit init                          # skip if your project is already a git repository\ngit submodule add https://github.com/dbiswas55/vlmhub.git src/_libs/vlmhub\n\npython3 -m venv venv312 && source venv312/bin/activate\npip install -e src/_libs/vlmhub                    # add \"[transformers]\" for the in-process hosting\n```\n\nEverything vlmhub brings stays inside that one folder:\n\n```\nmy-project/\n├── .env                    # your keys\n└── src/_libs/vlmhub/       # the repository, pinned to one commit\n```\n\nImport it anywhere in your project — the editable install handles the path:\n\n``` python\nfrom vlmhub import Model, TextBlock, ImageBlock\n\nmodel = Model(\"gemini/flash-3.5\")\n```\n\n**Upgrading** — move the submodule to the latest upstream commit:\n\n```\ngit submodule update --remote --merge src/_libs/vlmhub\n```\n\nThen add and commit it as you would any other change — your repo records the new commit pointer, not\nvlmhub's files. The next run picks up the new code; re-run `pip install -e src/_libs/vlmhub` only when\nan upgrade adds a dependency.\n\n```\ncp .env.example .env                    # under A\ncp src/_libs/vlmhub/.env.example .env   # under B\n```\n\nThen fill in only what you use — `GEMINI_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, `GCP_PROJECT`/` GCP_LOCATION`, `HF_TOKEN`/` HF_HOME`. A purely local Ollama or vLLM setup needs none of them. **Different variable names** — edit `api_key_env` in the registry instead.\n\n**Ollama** (easiest to start with):\n\n```\nbrew install ollama                # macOS\nollama pull gemma3:4b\nollama serve                       # http://localhost:11434/v1\n```\n\n**MLX-VLM** (Apple Silicon):\n\n```\npython -m mlx_vlm.server --model mlx-community/gemma-3-4b-it-qat-4bit --port 8080\n```\n\n**vLLM** (CUDA):\n\n```\npip install vllm\nvllm serve Qwen/Qwen3-VL-4B-Instruct --port 8000\n```\n\nSee [`src/vlmhub/backends/README.md`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md) for the full setup guide, and [`src/vlmhub/utils/local_models.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local_models.py) for model download utilities.\n\nThe scripts in [`tests/`](https://github.com/dbiswas55/vlmhub/blob/main/tests) are both smoke tests and usage examples — the task scripts download a\nfew real dataset samples, run them through a client, and print the output next to the ground truth;\n`test_judge.py` grades built-in answers as an LLM judge. They ship with the repository, so run them\nfrom a clone (**A**):\n\n```\npython tests/test_task_captioning.py                                 # models.json's active client\npython tests/test_task_vqa.py --client ollama/gemma3-4b\npython tests/test_task_vqa.py --client gemini/flash-2.5 --n 5\npython tests/test_task_captioning.py --client transformers/gemma3-4b\npython tests/test_judge.py --client openai/gpt-4o-mini               # text-only judge check\nsbatch test_vllm.sh qwen3vl-4b                                       # Slurm: serve a vllm model, run the tests\n```\n\n`Model` is the whole surface: it resolves a client from the registry and holds the backend that serves it.\n\n``` python\nfrom vlmhub import Model, TextBlock, ImageBlock\n\nmodel = Model()                      # models.json's \"active\" client\nmodel = Model(\"ollama/gemma3-4b\")    # a specific one, as \"hosting/model\"\n\nresponse = model.generate(\n    [\n        ImageBlock(image_path=\"input/images/slide_1.png\"),\n        TextBlock(\"What does this diagram show?\"),\n        ImageBlock(image_path=\"input/images/slide_2.png\"),\n        TextBlock(\"How does this compare to the previous slide?\"),\n    ],\n    system_prompt=\"You are a helpful assistant.\",\n)\n\nprint(response[\"text\"])              # the generation\nprint(response[\"logs\"])              # e.g. [\"MODEL gemma3:4b\", \"USAGE prompt=288 output=304 total=592\"]\nprint(response[\"model\"])             # the model that actually served the request\nprint(response[\"params\"])            # the generation settings actually sent, e.g. {\"temperature\": 0.3}\n```\n\nGeneration parameters come from the registry (`defaults`, overridden per hosting, then per model) and can be overridden for one call:\n\n```\nresponse = model.generate(content, max_new_tokens=1024, temperature=0.0, top_p=1.0,\n                          top_k=40, reasoning_effort=\"medium\")\n```\n\nA `None` value is not sent, so the provider's default applies (for Transformers, the model's own `generation_config`); `top_p`, `top_k` and `reasoning_effort` are `None` by default. `reasoning_effort` (`\"low\"`, `\"medium\"`, `\"high\"`, …) is mapped by LiteLLM to each provider's own control: effort on Claude, `thinking_level` on Gemini 3, `reasoning_effort` on OpenAI. Models marked `\"sampling\": false` in the registry reject `temperature`/` top_p`/` top_k` (e.g. Claude Sonnet 5, Opus 4.8+, GPT-5.6); those values are dropped for them, with a `DROPPED ...` line in `logs`, and `params` shows what was really sent. For reproducible runs, prefer dated model IDs where the provider offers them and record `model` and `params` with each result.\n\nUseful properties: `model.name` (`\"ollama/gemma3-4b\"`), `model.model_id` (`\"gemma3:4b\"`), `model.short_name` (`\"gemma3-4b\"`), and `model.report()` to print both.\n\nImages are encoded automatically — base64 data URI for every API-hosted backend, PIL for Transformers. An `ImageBlock(image=...)` is encoded as PNG, so nothing is re-compressed on the way out.\n\n```\nvlmhub/\n├── pyproject.toml\n├── .env.example                 # API keys and HF token (copy to .env)\n├── test_vllm.sh                 # Slurm job: serve a vllm model, run tests/ against it\n├── src/vlmhub/\n│   ├── __init__.py              # Public API: Model, Config, TextBlock, ImageBlock, …\n│   ├── models.json              # The model/hosting registry\n│   ├── model.py                 # Model — a client's config plus its backend\n│   ├── utils/\n│   │   ├── config.py            # Config — resolves \"hosting/model\" into a client dict\n│   │   └── local_models.py      # List / download / delete HF-cache and Ollama models\n│   └── backends/\n│       ├── __init__.py          # Backend factory (get_backend_from_config)\n│       ├── backends.py          # BaseBackend, LiteLLMBackend, TransformersBackend\n│       ├── request.py           # TextBlock, ImageBlock, InferenceRequest\n│       └── README.md            # Backend setup guide\n└── tests/\n    ├── test_task_vqa.py         # VQAv2 samples — image + questions vs. ground-truth answers\n    ├── test_task_captioning.py  # COCO samples — image + generated vs. reference captions\n    ├── test_clients.py          # One image, one prompt — checks a list of clients end to end\n    └── test_judge.py            # Built-in answers graded on a 1-5 rubric vs. expected scores\n```\n\nThe model registry lives in [`src/vlmhub/models.json`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/models.json), which ships with the package:\n\n```\n{\n  \"active\": { \"hosting\": \"ollama\", \"model\": \"gemma3-4b\" },      // default client\n  \"defaults\": { \"max_new_tokens\": 4096, \"temperature\": 0.3, \"top_p\": null, \"top_k\": null, \"reasoning_effort\": null },\n  \"hostings\": {\n    \"ollama\": {\n      \"backend\": \"litellm\",\n      \"litellm_prefix\": \"openai\",\n      \"api_base\": \"http://localhost:11434/v1\",\n      \"models\": [\n        { \"name\": \"gemma3-4b\", \"model_id\": \"gemma3:4b\" }\n        // ...\n      ]\n    }\n    // gemini, vertex_ai, openai, anthropic, openrouter, mlx_vlm, vllm, transformers ...\n  }\n}\n```\n\n**Selecting a client** — pass `\"hosting/model\"` to `Model(...)`, or set `active` in the registry and call `Model()`. Model-level fields override hosting-level fields, which override `defaults`.\n\n**Editing it in place** works for a quick change. Git keeps your edits, but they land in vlmhub's\nhistory rather than your project's, and a release that also touches `models.json` blocks the next\n`git submodule update` until you stash or commit them.\n\n**Extending the registry** — for anything you mean to keep, pass your own file instead. It holds only\nwhat you add or change:\n\n```\n// my_models.json\n{\n  \"active\": { \"hosting\": \"my_server\", \"model\": \"my-finetune\" },\n  \"defaults\": { \"temperature\": 0.0 },\n  \"hostings\": {\n    \"my_server\": {\n      \"backend\": \"litellm\",\n      \"litellm_prefix\": \"openai\",\n      \"api_base\": \"http://localhost:8000/v1\",\n      \"models\": [{ \"name\": \"my-finetune\", \"model_id\": \"org/my-finetuned-vlm\" }]\n    }\n  }\n}\nmodel = Model(\"my_server/my-finetune\", models_path=\"my_models.json\")\n```\n\nIt has the same shape as the bundled file and is deep-merged over it, so you can add hostings and models or override single fields. One catch: lists are replaced, not appended — naming an existing hosting's `models` drops the rest, so add a new hosting key instead. See [Adding a New Model](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md#adding-a-new-model) and [Adding a New Hosting](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md#adding-a-new-hosting).\n\n**Transformers models** take extra per-model fields — `model_class` and `fallback_dtype`, plus an optional `processor_kwargs` — described in [the backend guide](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md#transformers).\n\n[`src/vlmhub/utils/local_models.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local_models.py) manages both the HuggingFace cache and Ollama's store:\n\n```\npython -m vlmhub.utils.local_models\n```\n\n| Function | Description | \n|---|---|\n| `list_hf_cache_models()` /`list_ollama_cache_models()` | List models with sizes | \n| `download_hf_model(ids)` /`download_ollama_model(ids)` | Download one model or a list | \n| `delete_hf_cache_model(id)` /`delete_ollama_cache_model(id)` | Delete a specific model | \n| `delete_hf_cache_model_interactive()` /`delete_ollama_cache_model_interactive()` | Interactive picker to delete models | \n\nThe Ollama functions need the `ollama` extra; the HuggingFace ones work out of the box.\n\nMost new providers need **no code at all** — add a hosting to `models.json` naming a [LiteLLM provider prefix](https://docs.litellm.ai/docs/providers), and the existing `LiteLLMBackend` serves it. See [Adding a New Hosting](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md#adding-a-new-hosting).\n\nA genuinely new *backend type* — something LiteLLM cannot reach, the way in-process Transformers isn't a wire protocol — means subclassing `BaseBackend` in [`backends.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py) and registering it in [`get_backend_from_config`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/__init__.py). See [Adding a New Backend](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md#adding-a-new-backend).\n\nReleased under the MIT License — see [LICENSE](https://github.com/dbiswas55/vlmhub/blob/main/LICENSE).", "url": "https://wpnews.pro/news/vlmhub-simple-unified-python-interface-for-vision-language-models", "canonical_source": "https://github.com/dbiswas55/vlmhub", "published_at": "2026-10-19 17:33:03+00:00", "updated_at": "2026-10-19 17:46:31.213415+00:00", "lang": "en", "topics": ["computer-vision", "large-language-models", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["Vlmhub", "LiteLLM", "Ollama", "MLX-VLM", "vLLM", "HuggingFace Transformers", "Google Gemini", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/vlmhub-simple-unified-python-interface-for-vision-language-models", "markdown": "https://wpnews.pro/news/vlmhub-simple-unified-python-interface-for-vision-language-models.md", "text": "https://wpnews.pro/news/vlmhub-simple-unified-python-interface-for-vision-language-models.txt", "jsonld": "https://wpnews.pro/news/vlmhub-simple-unified-python-interface-for-vision-language-models.jsonld"}}