Vlmhub – Simple Unified Python interface for vision-language models

A developer released Vlmhub, an open-source Python library that provides a single Model.generate() interface for running interleaved text-and-image prompts across Ollama, MLX-VLM, vLLM, HuggingFace Transformers, Google Gemini, Vertex AI, OpenAI, Anthropic and OpenRouter. Vlmhub routes cloud and local-server backends through LiteLLM, while its Transformers backend runs models in-process with optional 4-bit quantization and a fallback_dtype for pre-Ampere GPUs such as the V100 that lack native bfloat16. The library ships a JSON model registry with pre-configured entries including Gemma 3 (4B, 12B) and Qwen3-VL (4B, 8B), and installs without PyTorch unless the transformers extra is added.

A lightweight, config-driven framework for unified vision-language model inference across local and cloud backends. Run multimodal prompts — interleaved text and images — against Ollama, MLX-VLM, vLLM, HuggingFace Transformers, Gemini, Vertex AI, OpenAI, or Anthropic without rewriting any inference code. python from vlmhub import Model, TextBlock, ImageBlock model = Model "ollama/gemma3-4b" local server model = Model "gemini/flash-3.5lite" cloud API model = Model "transformers/qwen3vl-8b" in-process, no server reply = model.generate TextBlock "Here are two images:" , ImageBlock image path="image1.png" , an image file on disk ImageBlock image=pil image , or a PIL image in memory TextBlock "What are the main differences between them?" , print reply "text" An ImageBlock takes an image file image path= or a PIL image in memory image= , such as a video frame. - One interface, every backend — a single Model.generate call over TextBlock and ImageBlock , independent of model or backend. - Interleaved multimodal prompts — mix TextBlock and ImageBlock in any order; each backend encodes as it needs base64 data URI over the wire, PIL in-process . - A model registry — models, settings and generation defaults live in one editable JSON file; give a model any short name you like alongside its real model id. - A hardware-aware Transformers backend — transformers runs a model in-process with an optional 4-bit quantized load and a fallback dtype for pre-Ampere GPUs e.g. V100 that lack native bfloat16; LiteLLM https://docs.litellm.ai covers every other hosting. | Category | Provider | Hosting Key | Backend | How it runs | |---|---|---|---|---| | Local Server | Ollama | ollama | litellm | Local server on port 11434 | | Local Server | MLX-VLM | mlx vlm | litellm | Apple Silicon, port 8080 | | Local Server | vLLM | vllm | litellm | CUDA GPU, port 8000 | | In-Process | HuggingFace Transformers | transformers | transformers | Direct model loading CUDA / MPS / CPU | | Cloud API | Google Gemini AI Studio | gemini | litellm | LiteLLM gemini/ provider; thinking budget 2.5 or reasoning effort 3.x | | Cloud API | Google Gemini via Vertex AI | vertex ai | litellm | LiteLLM vertex ai/ , Application Default Credentials | | Cloud API | OpenAI | openai | litellm | GPT-4o, GPT-4o-mini, GPT-4.1, GPT-5.x | | Cloud API | Anthropic | anthropic | litellm | LiteLLM anthropic/ provider | | Cloud API | OpenRouter | openrouter | litellm | GPT, Claude and Gemini with one key | Pre-configured models include Gemma 3 4B, 12B and Qwen3-VL 4B, 8B across all local hostings — plus Qwen2.5-VL, InternVL3.5, mPLUG-Owl3, Molmo2, Idefics3 and LLaVA-OneVision under transformers — open-weight judges Qwen3.6/3.8, Prometheus, CompassJudger, UnifiedReward, gpt-oss under vllm , and Gemini 2.5 / 3.x , GPT-4o/4.1/5.x, Claude 4.5+ and Gemma 4 via Gemini for cloud. Pick whichever fits: A to try vlmhub out or work on it, B to install it as a library into another project. The repo carries the test scripts in tests/ https://github.com/dbiswas55/vlmhub/blob/main/tests , so this is the way to see it running end to end. git clone https://github.com/dbiswas55/vlmhub.git cd vlmhub python3 -m venv venv312 source venv312/bin/activate macOS / Linux venv312\Scripts\activate Windows pip install -e . That covers every API-hosted backend — LiteLLM handles Gemini, Vertex AI, OpenAI, Anthropic and all OpenAI-compatible local servers Ollama, MLX-VLM, vLLM , so there are no per-hosting SDKs to add. PyTorch is not installed: only the in-process transformers hosting needs it. pip install -e ". transformers " torch + transformers, for in-process model loading pip install -e ". quantization " adds bitsandbytes, for 4-bit loads under transformers pip install -e ". ollama " only to manage Ollama's store via local models.py pip install -e ". all " everything above Add vlmhub as a submodule, then install it from there. The source sits in your project, readable and upgradeable, and pip handles the dependencies and the import path. cd ~/my-project git init skip if your project is already a git repository git submodule add https://github.com/dbiswas55/vlmhub.git src/ libs/vlmhub python3 -m venv venv312 && source venv312/bin/activate pip install -e src/ libs/vlmhub add " transformers " for the in-process hosting Everything vlmhub brings stays inside that one folder: my-project/ ├── .env your keys └── src/ libs/vlmhub/ the repository, pinned to one commit Import it anywhere in your project — the editable install handles the path: python from vlmhub import Model, TextBlock, ImageBlock model = Model "gemini/flash-3.5" Upgrading — move the submodule to the latest upstream commit: git submodule update --remote --merge src/ libs/vlmhub Then add and commit it as you would any other change — your repo records the new commit pointer, not vlmhub's files. The next run picks up the new code; re-run pip install -e src/ libs/vlmhub only when an upgrade adds a dependency. cp .env.example .env under A cp src/ libs/vlmhub/.env.example .env under B Then fill in only what you use — GEMINI API KEY , OPENAI API KEY , ANTHROPIC API KEY , OPENROUTER API KEY , GCP PROJECT / GCP LOCATION , HF TOKEN / HF HOME . A purely local Ollama or vLLM setup needs none of them. Different variable names — edit api key env in the registry instead. Ollama easiest to start with : brew install ollama macOS ollama pull gemma3:4b ollama serve http://localhost:11434/v1 MLX-VLM Apple Silicon : python -m mlx vlm.server --model mlx-community/gemma-3-4b-it-qat-4bit --port 8080 vLLM CUDA : pip install vllm vllm serve Qwen/Qwen3-VL-4B-Instruct --port 8000 See src/vlmhub/backends/README.md https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md for the full setup guide, and src/vlmhub/utils/local models.py https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local models.py for model download utilities. The scripts in tests/ https://github.com/dbiswas55/vlmhub/blob/main/tests are both smoke tests and usage examples — the task scripts download a few real dataset samples, run them through a client, and print the output next to the ground truth; test judge.py grades built-in answers as an LLM judge. They ship with the repository, so run them from a clone A : python tests/test task captioning.py models.json's active client python tests/test task vqa.py --client ollama/gemma3-4b python tests/test task vqa.py --client gemini/flash-2.5 --n 5 python tests/test task captioning.py --client transformers/gemma3-4b python tests/test judge.py --client openai/gpt-4o-mini text-only judge check sbatch test vllm.sh qwen3vl-4b Slurm: serve a vllm model, run the tests Model is the whole surface: it resolves a client from the registry and holds the backend that serves it. python from vlmhub import Model, TextBlock, ImageBlock model = Model models.json's "active" client model = Model "ollama/gemma3-4b" a specific one, as "hosting/model" response = model.generate ImageBlock image path="input/images/slide 1.png" , TextBlock "What does this diagram show?" , ImageBlock image path="input/images/slide 2.png" , TextBlock "How does this compare to the previous slide?" , , system prompt="You are a helpful assistant.", print response "text" the generation print response "logs" e.g. "MODEL gemma3:4b", "USAGE prompt=288 output=304 total=592" print response "model" the model that actually served the request print response "params" the generation settings actually sent, e.g. {"temperature": 0.3} Generation parameters come from the registry defaults , overridden per hosting, then per model and can be overridden for one call: response = model.generate content, max new tokens=1024, temperature=0.0, top p=1.0, top k=40, reasoning effort="medium" A None value is not sent, so the provider's default applies for Transformers, the model's own generation config ; top p , top k and reasoning effort are None by default. reasoning effort "low" , "medium" , "high" , … is mapped by LiteLLM to each provider's own control: effort on Claude, thinking level on Gemini 3, reasoning effort on OpenAI. Models marked "sampling": false in the registry reject temperature / top p / top k e.g. Claude Sonnet 5, Opus 4.8+, GPT-5.6 ; those values are dropped for them, with a DROPPED ... line in logs , and params shows what was really sent. For reproducible runs, prefer dated model IDs where the provider offers them and record model and params with each result. Useful properties: model.name "ollama/gemma3-4b" , model.model id "gemma3:4b" , model.short name "gemma3-4b" , and model.report to print both. Images are encoded automatically — base64 data URI for every API-hosted backend, PIL for Transformers. An ImageBlock image=... is encoded as PNG, so nothing is re-compressed on the way out. vlmhub/ ├── pyproject.toml ├── .env.example API keys and HF token copy to .env ├── test vllm.sh Slurm job: serve a vllm model, run tests/ against it ├── src/vlmhub/ │ ├── init .py Public API: Model, Config, TextBlock, ImageBlock, … │ ├── models.json The model/hosting registry │ ├── model.py Model — a client's config plus its backend │ ├── utils/ │ │ ├── config.py Config — resolves "hosting/model" into a client dict │ │ └── local models.py List / download / delete HF-cache and Ollama models │ └── backends/ │ ├── init .py Backend factory get backend from config │ ├── backends.py BaseBackend, LiteLLMBackend, TransformersBackend │ ├── request.py TextBlock, ImageBlock, InferenceRequest │ └── README.md Backend setup guide └── tests/ ├── test task vqa.py VQAv2 samples — image + questions vs. ground-truth answers ├── test task captioning.py COCO samples — image + generated vs. reference captions ├── test clients.py One image, one prompt — checks a list of clients end to end └── test judge.py Built-in answers graded on a 1-5 rubric vs. expected scores The model registry lives in src/vlmhub/models.json https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/models.json , which ships with the package: { "active": { "hosting": "ollama", "model": "gemma3-4b" }, // default client "defaults": { "max new tokens": 4096, "temperature": 0.3, "top p": null, "top k": null, "reasoning effort": null }, "hostings": { "ollama": { "backend": "litellm", "litellm prefix": "openai", "api base": "http://localhost:11434/v1", "models": { "name": "gemma3-4b", "model id": "gemma3:4b" } // ... } // gemini, vertex ai, openai, anthropic, openrouter, mlx vlm, vllm, transformers ... } } Selecting a client — pass "hosting/model" to Model ... , or set active in the registry and call Model . Model-level fields override hosting-level fields, which override defaults . Editing it in place works for a quick change. Git keeps your edits, but they land in vlmhub's history rather than your project's, and a release that also touches models.json blocks the next git submodule update until you stash or commit them. Extending the registry — for anything you mean to keep, pass your own file instead. It holds only what you add or change: // my models.json { "active": { "hosting": "my server", "model": "my-finetune" }, "defaults": { "temperature": 0.0 }, "hostings": { "my server": { "backend": "litellm", "litellm prefix": "openai", "api base": "http://localhost:8000/v1", "models": { "name": "my-finetune", "model id": "org/my-finetuned-vlm" } } } } model = Model "my server/my-finetune", models path="my models.json" It has the same shape as the bundled file and is deep-merged over it, so you can add hostings and models or override single fields. One catch: lists are replaced, not appended — naming an existing hosting's models drops the rest, so add a new hosting key instead. See Adding a New Model https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md adding-a-new-model and Adding a New Hosting https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md adding-a-new-hosting . Transformers models take extra per-model fields — model class and fallback dtype , plus an optional processor kwargs — described in the backend guide https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md transformers . src/vlmhub/utils/local models.py https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local models.py manages both the HuggingFace cache and Ollama's store: python -m vlmhub.utils.local models | Function | Description | |---|---| | list hf cache models / list ollama cache models | List models with sizes | | download hf model ids / download ollama model ids | Download one model or a list | | delete hf cache model id / delete ollama cache model id | Delete a specific model | | delete hf cache model interactive / delete ollama cache model interactive | Interactive picker to delete models | The Ollama functions need the ollama extra; the HuggingFace ones work out of the box. Most new providers need no code at all — add a hosting to models.json naming a LiteLLM provider prefix https://docs.litellm.ai/docs/providers , and the existing LiteLLMBackend serves it. See Adding a New Hosting https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md adding-a-new-hosting . A genuinely new backend type — something LiteLLM cannot reach, the way in-process Transformers isn't a wire protocol — means subclassing BaseBackend in backends.py https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py and registering it in get backend from config https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/ init .py . See Adding a New Backend https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/README.md adding-a-new-backend . Released under the MIT License — see LICENSE https://github.com/dbiswas55/vlmhub/blob/main/LICENSE .