Vlmhub – Simple Unified Python interface for vision-language models

A lightweight, config-driven framework for unified vision-language model inference across local and cloud backends. Run multimodal prompts — interleaved text and images — against Ollama, MLX-VLM, vLLM, HuggingFace Transformers, Gemini, Vertex AI, OpenAI, or Anthropic without rewriting any inference code.

from vlmhub import Model, TextBlock, ImageBlock

model = Model("ollama/gemma3-4b")            # local server

reply = model.generate([
    TextBlock("Here are two images:"),
    ImageBlock(image_path="image1.png"),     # an image file on disk
    ImageBlock(image=pil_image),             # or a PIL image in memory
    TextBlock("What are the main differences between them?"),
])
print(reply["text"])

An ImageBlock takes an image file (image_path=) or a PIL image in memory ( image=), such as a video frame.

• One interface, every backend — a singleModel.generate() call overTextBlock andImageBlock , independent of model or backend.
• Interleaved multimodal prompts — mixTextBlock andImageBlock in any order; each backend encodes as it needs (base64 data URI over the wire, PIL in-process).
• A model registry — models, settings and generation defaults live in one editable JSON file; give a model any short name you like alongside its real model id.
• A hardware-aware Transformers backend —transformers runs a model in-process with an optional 4-bit quantized load and afallback_dtype for pre-Ampere GPUs (e.g. V100) that lack native bfloat16;LiteLLM covers every other hosting.

Pre-configured models include Gemma 3 (4B, 12B) and Qwen3-VL (4B, 8B) across all local hostings — plus Qwen2.5-VL, InternVL3.5, mPLUG-Owl3, Molmo2, Idefics3 and LLaVA-OneVision under transformers — open-weight judges (Qwen3.6/3.8, Prometheus, CompassJudger, UnifiedReward, gpt-oss) under vllm, and Gemini (2.5 / 3.x), GPT-4o/4.1/5.x, Claude 4.5+ and Gemma 4 (via Gemini) for cloud.

Pick whichever fits: A to try vlmhub out or work on it, B to install it as a library into another project.

The repo carries the test scripts in tests/, so this is the way to see it running end to end.

git clone https://github.com/dbiswas55/vlmhub.git
cd vlmhub

python3 -m venv venv312
source venv312/bin/activate      # macOS / Linux

pip install -e .

That covers every API-hosted backend — LiteLLM handles Gemini, Vertex AI, OpenAI, Anthropic and all OpenAI-compatible local servers (Ollama, MLX-VLM, vLLM), so there are no per-hosting SDKs to add. PyTorch is not installed: only the in-process transformers hosting needs it.

pip install -e ".[transformers]"   # torch + transformers, for in-process model 
pip install -e ".[quantization]"   # adds bitsandbytes, for 4-bit loads under `transformers`
pip install -e ".[ollama]"         # only to manage Ollama's store via local_models.py
pip install -e ".[all]"            # everything above

Add vlmhub as a submodule, then install it from there. The source sits in your project, readable and upgradeable, and pip handles the dependencies and the import path.

cd ~/my-project
git init                          # skip if your project is already a git repository
git submodule add https://github.com/dbiswas55/vlmhub.git src/_libs/vlmhub

python3 -m venv venv312 && source venv312/bin/activate
pip install -e src/_libs/vlmhub                    # add "[transformers]" for the in-process hosting

Everything vlmhub brings stays inside that one folder:

my-project/
├── .env                    # your keys
└── src/_libs/vlmhub/       # the repository, pinned to one commit

Import it anywhere in your project — the editable install handles the path:

from vlmhub import Model, TextBlock, ImageBlock

model = Model("gemini/flash-3.5")

Upgrading — move the submodule to the latest upstream commit:

git submodule update --remote --merge src/_libs/vlmhub

Then add and commit it as you would any other change — your repo records the new commit pointer, not vlmhub's files. The next run picks up the new code; re-run pip install -e src/_libs/vlmhub only when an upgrade adds a dependency.

cp .env.example .env                    # under A
cp src/_libs/vlmhub/.env.example .env   # under B

Then fill in only what you use — GEMINI_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, OPENROUTER_API_KEY, GCP_PROJECT/ GCP_LOCATION, HF_TOKEN/ HF_HOME. A purely local Ollama or vLLM setup needs none of them. Different variable names — edit api_key_env in the registry instead.

Ollama (easiest to start with):

brew install ollama                # macOS
ollama pull gemma3:4b
ollama serve                       # http://localhost:11434/v1

MLX-VLM (Apple Silicon):

python -m mlx_vlm.server --model mlx-community/gemma-3-4b-it-qat-4bit --port 8080

vLLM (CUDA):

pip install vllm
vllm serve Qwen/Qwen3-VL-4B-Instruct --port 8000

See src/vlmhub/backends/README.md for the full setup guide, and src/vlmhub/utils/local_models.py for model download utilities.

The scripts in tests/ are both smoke tests and usage examples — the task scripts download a few real dataset samples, run them through a client, and print the output next to the ground truth; test_judge.py grades built-in answers as an LLM judge. They ship with the repository, so run them from a clone (A):

python tests/test_task_captioning.py                                 # models.json's active client
python tests/test_task_vqa.py --client ollama/gemma3-4b
python tests/test_task_vqa.py --client gemini/flash-2.5 --n 5
python tests/test_task_captioning.py --client transformers/gemma3-4b
python tests/test_judge.py --client openai/gpt-4o-mini               # text-only judge check
sbatch test_vllm.sh qwen3vl-4b                                       # Slurm: serve a vllm model, run the tests

Model is the whole surface: it resolves a client from the registry and holds the backend that serves it.

from vlmhub import Model, TextBlock, ImageBlock

model = Model()                      # models.json's "active" client
model = Model("ollama/gemma3-4b")    # a specific one, as "hosting/model"

response = model.generate(
    [
        ImageBlock(image_path="input/images/slide_1.png"),
        TextBlock("What does this diagram show?"),
        ImageBlock(image_path="input/images/slide_2.png"),
        TextBlock("How does this compare to the previous slide?"),
    ],
    system_prompt="You are a helpful assistant.",
)

print(response["text"])              # the generation
print(response["logs"])              # e.g. ["MODEL gemma3:4b", "USAGE prompt=288 output=304 total=592"]
print(response["model"])             # the model that actually served the request
print(response["params"])            # the generation settings actually sent, e.g. {"temperature": 0.3}

Generation parameters come from the registry (defaults, overridden per hosting, then per model) and can be overridden for one call:

response = model.generate(content, max_new_tokens=1024, temperature=0.0, top_p=1.0,
                          top_k=40, reasoning_effort="medium")

A None value is not sent, so the provider's default applies (for Transformers, the model's own generation_config); top_p, top_k and reasoning_effort are None by default. reasoning_effort ("low", "medium", "high", …) is mapped by LiteLLM to each provider's own control: effort on Claude, thinking_level on Gemini 3, reasoning_effort on OpenAI. Models marked "sampling": false in the registry reject temperature/ top_p/ top_k (e.g. Claude Sonnet 5, Opus 4.8+, GPT-5.6); those values are dropped for them, with a DROPPED ... line in logs, and params shows what was really sent. For reproducible runs, prefer dated model IDs where the provider offers them and record model and params with each result.

Useful properties: model.name ("ollama/gemma3-4b"), model.model_id ("gemma3:4b"), model.short_name ("gemma3-4b"), and model.report() to print both.

Images are encoded automatically — base64 data URI for every API-hosted backend, PIL for Transformers. An ImageBlock(image=...) is encoded as PNG, so nothing is re-compressed on the way out.

vlmhub/
├── pyproject.toml
├── .env.example                 # API keys and HF token (copy to .env)
├── test_vllm.sh                 # Slurm job: serve a vllm model, run tests/ against it
├── src/vlmhub/
│   ├── __init__.py              # Public API: Model, Config, TextBlock, ImageBlock, …
│   ├── models.json              # The model/hosting registry
│   ├── model.py                 # Model — a client's config plus its backend
│   ├── utils/
│   │   ├── config.py            # Config — resolves "hosting/model" into a client dict
│   │   └── local_models.py      # List / download / delete HF-cache and Ollama models
│   └── backends/
│       ├── __init__.py          # Backend factory (get_backend_from_config)
│       ├── backends.py          # BaseBackend, LiteLLMBackend, TransformersBackend
│       ├── request.py           # TextBlock, ImageBlock, InferenceRequest
│       └── README.md            # Backend setup guide
└── tests/
    ├── test_task_vqa.py         # VQAv2 samples — image + questions vs. ground-truth answers
    ├── test_task_captioning.py  # COCO samples — image + generated vs. reference captions
    ├── test_clients.py          # One image, one prompt — checks a list of clients end to end
    └── test_judge.py            # Built-in answers graded on a 1-5 rubric vs. expected scores

The model registry lives in src/vlmhub/models.json, which ships with the package:

{
  "active": { "hosting": "ollama", "model": "gemma3-4b" },      // default client
  "defaults": { "max_new_tokens": 4096, "temperature": 0.3, "top_p": null, "top_k": null, "reasoning_effort": null },
  "hostings": {
    "ollama": {
      "backend": "litellm",
      "litellm_prefix": "openai",
      "api_base": "http://localhost:11434/v1",
      "models": [
        { "name": "gemma3-4b", "model_id": "gemma3:4b" }
        // ...
      ]
    }
    // gemini, vertex_ai, openai, anthropic, openrouter, mlx_vlm, vllm, transformers ...
  }
}

Selecting a client — pass "hosting/model" to Model(...), or set active in the registry and call Model(). Model-level fields override hosting-level fields, which override defaults.

Editing it in place works for a quick change. Git keeps your edits, but they land in vlmhub's history rather than your project's, and a release that also touches models.json blocks the next git submodule update until you stash or commit them.

Extending the registry — for anything you mean to keep, pass your own file instead. It holds only what you add or change:

// my_models.json
{
  "active": { "hosting": "my_server", "model": "my-finetune" },
  "defaults": { "temperature": 0.0 },
  "hostings": {
    "my_server": {
      "backend": "litellm",
      "litellm_prefix": "openai",
      "api_base": "http://localhost:8000/v1",
      "models": [{ "name": "my-finetune", "model_id": "org/my-finetuned-vlm" }]
    }
  }
}
model = Model("my_server/my-finetune", models_path="my_models.json")

It has the same shape as the bundled file and is deep-merged over it, so you can add hostings and models or override single fields. One catch: lists are replaced, not appended — naming an existing hosting's models drops the rest, so add a new hosting key instead. See Adding a New Model and Adding a New Hosting.

Transformers models take extra per-model fields — model_class and fallback_dtype, plus an optional processor_kwargs — described in the backend guide.

src/vlmhub/utils/local_models.py manages both the HuggingFace cache and Ollama's store:

python -m vlmhub.utils.local_models

The Ollama functions need the ollama extra; the HuggingFace ones work out of the box.

Most new providers need no code at all — add a hosting to models.json naming a LiteLLM provider prefix, and the existing LiteLLMBackend serves it. See Adding a New Hosting.

A genuinely new backend type — something LiteLLM cannot reach, the way in-process Transformers isn't a wire protocol — means subclassing BaseBackend in backends.py and registering it in get_backend_from_config. See Adding a New Backend.

Released under the MIT License — see LICENSE.