A single function Jev-like wrapper for LLMs, including vision models

A developer extended the Jev-style single-function LLM wrapper to support vision models by adding an `attachments` field for base64 images to the request format, then built a webcam demo that classifies frames with plain-text questions. Running Gemma 4 12B on an RTX 3090, the setup processed roughly 1 frame per second with three questions per frame, versus about 0.2 FPS against OpenAI's gpt-6-luna, which the developer attributed to a separate connection per question per frame.

I was intrigued by Jev https://docs.typesafe.ai/introduction and the self-hostable projects appearing around it, such as OpenJev https://huggingface.co/openjev/openjev and SemIf https://github.com/TheoLeeCJ/SemIf-OpenJev . Reading about them introduced me to a neat trick: reading an LLM's token probabilities. Apparently this is an old trick for some people. See e.g. OpenAI's logprobs cookbook https://developers.openai.com/cookbook/examples/using logprobs . But it was new to me. I believe the basic idea is to write a prompt like this: State: My order arrived broken and I want a refund. Question: Which team should handle this? A billing B shipping C returns Answer with the letter of the best option only. Then add a few JSON request parameters to a compatible Chat Completions request: { "max completion tokens": 1, "logprobs": true, "top logprobs": 20 } The LLM API will return the letter plus the model's log probabilities for alternative tokens. Repeat for each question. Forcing it to generating only one token avoids a lengthy answer and is super quick, though processing the input still costs time. Though for each of the questions a shared state prefix can be KV-cached if the backend supports it. The fun part: this works with vision models too. Jev's documented request format https://docs.typesafe.ai/introduction/quickstart currently describes only text/JSON state. I added an attachments field for images for my local experiments. My example captures webcam frames, sends base64 JPEGs, and prints a table: is a person visible, are we indoors or outdoors, and how bright is the scene? With Gemma 4 12B on my RTX 3090, I get around 1 frames per second , with three questions per frame. I also ran it against OpenAI gpt-6-luna and got around 0.2 FPS. Presumably because I didn't make any effort to avoid the cost of a separate connection through their system per question per frame. Specialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text. Here's the standalone Python example OpenCV is just used for convenient access to the webcam, not for any actual computer vision : bash /usr/bin/env -S uv run --script /// script dependencies = "opencv-python" /// """Preview and score webcam frames with llama.cpp or OpenAI. uv run webcam.py uv run webcam.py https://api.openai.com/v1 gpt-6-luna OpenAI reads OPENAI API KEY. """ import argparse import base64 import concurrent.futures import datetime import json import math import mimetypes import os import pathlib import time import urllib.parse import urllib.request import cv2 attachments is our custom addition to the Jev request format. data = json.loads """ { "state": "Inspect this webcam frame. Judge only what is visibly present.", "attachments": , "questions": { "person": { "type": "noul", "instructions": "Is a person visible?" }, "plant": { "type": "noul", "instructions": "Is a plant visible?" }, "setting": { "type": "choice", "instructions": "Where is the camera?", "criteria": { "indoors": null, "outdoors": null, "unclear": null } }, "light": { "type": "score", "instructions": "How bright is the scene?", "criteria": "dark", "dim", "bright" } } } """ def score data, url, model : state = data "state" if not isinstance state, str : state = json.dumps state Attachments are our extension to the Jev-style request format: image file paths or base64 data URLs. Load them once for all questions. images = for attachment in data.get "attachments", : if attachment.startswith "data:image/" : images.append attachment continue path = pathlib.Path attachment .expanduser mime type, = mimetypes.guess type path if mime type not in {"image/png", "image/jpeg", "image/webp", "image/gif"}: raise ValueError f"Unsupported image file: {path}" encoded = base64.b64encode path.read bytes .decode images.append f"data:{mime type};base64,{encoded}" Send the API key only to OpenAI. is openai = urllib.parse.urlsplit url .hostname == "api.openai.com" headers = {"Content-Type": "application/json"} if is openai: headers "Authorization" = "Bearer " + os.environ "OPENAI API KEY" answers = {} for name, question in data "questions" .items : Represent choices, booleans, and ordinal levels as lettered options. if question "type" == "choice": options = question "criteria" elif question "type" == "noul": options = {"true": None, "false": None} | question.get "criteria", {} elif question "type" == "score": options = {str i : description for i, description in enumerate question "criteria" } else: raise ValueError f"Unknown question type: {question 'type' }" if not 2 <= len options <= 20: raise ValueError "Provide 2 to 20 criteria per question." letters = "ABCDEFGHIJKLMNOPQRST" :len options Ask for a single option letter, so its logprob represents that option. instructions = question "instructions" if not isinstance instructions, str : instructions = json.dumps instructions lines = f"State:\n{state}\n\nQuestion: {instructions}\nOptions:" for letter, key, description in zip letters, options.items : line = f" {letter} {key}" if description is not None: line += f": {description}" lines.append line prompt = "\n".join lines + "\n\nAnswer with the letter of the best option only." OpenAI needs Responses for enough alternatives; llama.cpp needs Chat for logprobs. top p=1 avoids pruning alternatives. if is openai: endpoint = "/responses" content = {"type": "input text", "text": prompt} content.extend {"type": "input image", "image url": image} for image in images body = { "model": model, "input": {"role": "user", "content": content} , "reasoning": {"effort": "none"}, "max output tokens": 16, "top p": 1, "top logprobs": 20, "include": "message.output text.logprobs" , } else: endpoint = "/chat/completions" content = {"type": "text", "text": prompt} content.extend {"type": "image url", "image url": {"url": image}} for image in images body = { "model": model, "messages": {"role": "user", "content": content} , "max completion tokens": 1, "temperature": 0, "reasoning effort": "none", "logprobs": True, "top logprobs": 1024, } Send the request and read the first output token's alternatives. request = urllib.request.Request url.rstrip "/" + endpoint, headers=headers, data=json.dumps body .encode , with urllib.request.urlopen request as response: result = json.load response if is openai: message = next item for item in result "output" if item "type" == "message" candidates = message "content" 0 "logprobs" 0 "top logprobs" else: candidates = result "choices" 0 "logprobs" "content" 0 "top logprobs" logprobs = {item "token" : item "logprob" for item in candidates} Normalize the returned option scores; missing options initially get zero. missing = letter for letter in letters if letter not in logprobs or logprobs letter <= -9999 if len missing == len letters : raise ValueError "API did not return usable scores for any option" peak = max logprobs letter for letter in letters if letter not in missing weights = math.exp logprobs letter - peak if letter not in missing else 0 for letter in letters total = sum weights An omitted token cannot outrank the last returned alternative. Allow zero only when their combined normalized probability is below 1e-6. if missing: cutoff = min value for value in logprobs.values if value -9999 missing weight = len missing math.exp cutoff - peak if missing weight / total + missing weight = 1e-6: raise ValueError f"API omitted non-negligible option scores for: {', '.join missing }" probabilities = {key: weight / total for key, weight in zip options, weights } Return the winning choice, probability of true, or expected ordinal level. if question "type" == "choice": answers name = { "type": "choice", "choice": max probabilities, key=probabilities.get , "probabilities": probabilities, } elif question "type" == "noul": answers name = {"type": "noul", "noul": probabilities "true" } else: answers name = { "type": "score", "score": sum int key probability for key, probability in probabilities.items , "legend": options, "probabilities": probabilities, } return {"answers": answers} Choose the server and model before opening the camera. parser = argparse.ArgumentParser description= doc parser.add argument "url", nargs="?", default="http://localhost:8060/v1" parser.add argument "model", nargs="?", default="gemma-4-12b" args = parser.parse args Point OpenCV's bundled Qt at the installed system fonts. os.environ "QT QPA FONTDIR" = "/usr/share/fonts/truetype/noto" Open the default Linux webcam with a small capture buffer. camera = cv2.VideoCapture 0, cv2.CAP V4L2 if not camera.isOpened : raise RuntimeError "Could not open /dev/video0" camera.set cv2.CAP PROP BUFFERSIZE, 1 print f"Webcam - {args.model}. Noul: yes %; score: value/max. Ctrl-C or Esc to stop.", flush=True print f"{'time':<8}" + "".join f"{name: 10}" for name in data "questions" + f"{'fps': 10}", flush=True Preview continuously while a background worker scores one frame at a time. executor = concurrent.futures.ThreadPoolExecutor max workers=1 pending = None try: while True: ok, frame = camera.read if not ok: raise RuntimeError "Could not read a webcam frame" cv2.imshow "Webcam", frame if cv2.waitKey 1 == 27 or cv2.getWindowProperty "Webcam", cv2.WND PROP VISIBLE < 1: break Print a completed result, then submit the latest frame. if pending is not None: if not pending.done : continue result = pending.result columns = for name in data "questions" : answer = result "answers" name if answer "type" == "noul": value = f"{answer 'noul' :.1%}" elif answer "type" == "choice": value = answer "choice" else: value = f"{answer 'score' :.2f}/{len data 'questions' name 'criteria' - 1}" columns.append f"{value: 10}" columns.append f"{1 / time.perf counter - started : 10.2f}" print captured + "".join columns , flush=True Measure throughput for evaluated frames, including image encoding. started = time.perf counter captured = datetime.datetime.now .strftime "%H:%M:%S" ok, jpeg = cv2.imencode ".jpg", frame if not ok: raise RuntimeError "Could not encode the webcam frame" image = "data:image/jpeg;base64," + base64.b64encode jpeg.tobytes .decode data "attachments" = image pending = executor.submit score, data, args.url, args.model except KeyboardInterrupt: print "\nStopped." finally: camera.release cv2.destroyAllWindows executor.shutdown The script handles the API differences: llama.cpp uses Chat Completions and OpenAI uses Responses to get it to show alternatives. I ran Gemma 4 12B QAT through llama.cpp. On Linux with NVIDIA drivers, curl , zstd , and uv installed: Model ~7 GB and multimodal projector ~175 MB . mkdir -p ~/models/gemma-4-12b/ cd ~/models/gemma-4-12b/ curl -fL -C - -o gemma-4-12b-it-qat-q4 0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4 0-gguf/resolve/main/gemma-4-12b-it-qat-q4 0.gguf curl -fL -C - -o mmproj-gemma-4-12b-it-qat-q4 0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4 0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4 0.gguf Standalone llama.cpp binary for RTX 3090 CUDA architecture 86 . curl -fL -o llama.zst https://huggingface.co/buckets/ggml-org/install.sh/resolve/b11160/x86 64/linux/cuda/86/llama-app.zst mkdir -p ~/bin/ zstd -d llama.zst -o ~/bin/llama chmod +x ~/bin/llama ~/bin/llama serve --models-dir ~/models/ --port 8060 Save the Python example as webcam.py . In another terminal, from that directory: uv run webcam.py http://localhost:8060/v1 gemma-4-12b Or use OpenAI, with OPENAI API KEY set in your environment. uv run webcam.py https://api.openai.com/v1 gpt-6-luna