{"slug": "a-single-function-jev-like-wrapper-for-llms-including-vision-models", "title": "A single function Jev-like wrapper for LLMs, including vision models", "summary": "A developer extended the Jev-style single-function LLM wrapper to support vision models by adding an `attachments` field for base64 images to the request format, then built a webcam demo that classifies frames with plain-text questions. Running Gemma 4 12B on an RTX 3090, the setup processed roughly 1 frame per second with three questions per frame, versus about 0.2 FPS against OpenAI's gpt-6-luna, which the developer attributed to a separate connection per question per frame.", "body_md": "I was intrigued by [Jev](https://docs.typesafe.ai/introduction) and the self-hostable projects appearing around it, such as [OpenJev](https://huggingface.co/openjev/openjev) and [SemIf](https://github.com/TheoLeeCJ/SemIf-OpenJev). Reading about them introduced me to a neat trick: reading an LLM's token probabilities.\n\nApparently this is an old trick for some people. See e.g. [OpenAI's logprobs cookbook](https://developers.openai.com/cookbook/examples/using_logprobs). But it was new to me.\n\nI believe the basic idea is to write a prompt like this:\n\n```\nState: My order arrived broken and I want a refund.\nQuestion: Which team should handle this?\n[A] billing\n[B] shipping\n[C] returns\nAnswer with the letter of the best option only.\n```\n\nThen add a few JSON request parameters to a compatible Chat Completions request:\n\n```\n{\n  \"max_completion_tokens\": 1,\n  \"logprobs\": true,\n  \"top_logprobs\": 20\n}\n```\n\nThe LLM API will return the letter plus the model's log probabilities for alternative tokens.\n\nRepeat for each question. Forcing it to generating only one token avoids a lengthy answer and is super quick, though processing the input still costs time. Though for each of the questions a shared state prefix can be KV-cached if the backend supports it.\n\nThe fun part: this works with vision models too. Jev's [documented request format](https://docs.typesafe.ai/introduction/quickstart) currently describes only text/JSON state. I added an `attachments` field for images for my local experiments.\n\nMy example captures webcam frames, sends base64 JPEGs, and prints a table: is a person visible, are we indoors or outdoors, and how bright is the scene? With Gemma 4 12B on my RTX 3090, I get around **1 frames per second**, with three questions per frame. I also ran it against OpenAI gpt-6-luna and got around 0.2 FPS. Presumably because I didn't make any effort to avoid the cost of a separate connection through their system per question per frame.\n\nSpecialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text.\n\nHere's the standalone Python example (OpenCV is just used for convenient access to the webcam, not for any actual computer vision):\n\n``` bash\n#!/usr/bin/env -S uv run --script\n# /// script\n# dependencies = [\"opencv-python\"]\n# ///\n\"\"\"Preview and score webcam frames with llama.cpp or OpenAI.\n\nuv run webcam.py\nuv run webcam.py https://api.openai.com/v1 gpt-6-luna\nOpenAI reads OPENAI_API_KEY.\n\"\"\"\nimport argparse\nimport base64\nimport concurrent.futures\nimport datetime\nimport json\nimport math\nimport mimetypes\nimport os\nimport pathlib\nimport time\nimport urllib.parse\nimport urllib.request\n\nimport cv2\n\n# attachments is our custom addition to the Jev request format.\ndata = json.loads(\"\"\"\n{\n    \"state\": \"Inspect this webcam frame. Judge only what is visibly present.\",\n    \"attachments\": [],\n    \"questions\": {\n        \"person\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Is a person visible?\"\n        },\n        \"plant\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Is a plant visible?\"\n        },\n        \"setting\": {\n            \"type\": \"choice\",\n            \"instructions\": \"Where is the camera?\",\n            \"criteria\": {\n                \"indoors\": null,\n                \"outdoors\": null,\n                \"unclear\": null\n            }\n        },\n        \"light\": {\n            \"type\": \"score\",\n            \"instructions\": \"How bright is the scene?\",\n            \"criteria\": [\n                \"dark\",\n                \"dim\",\n                \"bright\"\n            ]\n        }\n    }\n}\n\"\"\")\n\ndef score(data, url, model):\n    state = data[\"state\"]\n    if not isinstance(state, str):\n        state = json.dumps(state)\n\n    # Attachments are our extension to the Jev-style request format:\n    # image file paths or base64 data URLs. Load them once for all questions.\n    images = []\n    for attachment in data.get(\"attachments\", []):\n        if attachment.startswith(\"data:image/\"):\n            images.append(attachment)\n            continue\n        path = pathlib.Path(attachment).expanduser()\n        mime_type, _ = mimetypes.guess_type(path)\n        if mime_type not in {\"image/png\", \"image/jpeg\", \"image/webp\", \"image/gif\"}:\n            raise ValueError(f\"Unsupported image file: {path}\")\n        encoded = base64.b64encode(path.read_bytes()).decode()\n        images.append(f\"data:{mime_type};base64,{encoded}\")\n\n    # Send the API key only to OpenAI.\n    is_openai = urllib.parse.urlsplit(url).hostname == \"api.openai.com\"\n    headers = {\"Content-Type\": \"application/json\"}\n    if is_openai:\n        headers[\"Authorization\"] = \"Bearer \" + os.environ[\"OPENAI_API_KEY\"]\n\n    answers = {}\n    for name, question in data[\"questions\"].items():\n        # Represent choices, booleans, and ordinal levels as lettered options.\n        if question[\"type\"] == \"choice\":\n            options = question[\"criteria\"]\n        elif question[\"type\"] == \"noul\":\n            options = {\"true\": None, \"false\": None} | question.get(\"criteria\", {})\n        elif question[\"type\"] == \"score\":\n            options = {str(i): description for i, description in enumerate(question[\"criteria\"])}\n        else:\n            raise ValueError(f\"Unknown question type: {question['type']}\")\n        if not 2 <= len(options) <= 20:\n            raise ValueError(\"Provide 2 to 20 criteria per question.\")\n        letters = \"ABCDEFGHIJKLMNOPQRST\"[:len(options)]\n\n        # Ask for a single option letter, so its logprob represents that option.\n        instructions = question[\"instructions\"]\n        if not isinstance(instructions, str):\n            instructions = json.dumps(instructions)\n        lines = [f\"State:\\n{state}\\n\\nQuestion: {instructions}\\nOptions:\"]\n        for letter, (key, description) in zip(letters, options.items()):\n            line = f\"[{letter}] {key}\"\n            if description is not None:\n                line += f\": {description}\"\n            lines.append(line)\n        prompt = \"\\n\".join(lines) + \"\\n\\nAnswer with the letter of the best option only.\"\n\n        # OpenAI needs Responses for enough alternatives; llama.cpp needs Chat for logprobs.\n        # top_p=1 avoids pruning alternatives.\n        if is_openai:\n            endpoint = \"/responses\"\n            content = [{\"type\": \"input_text\", \"text\": prompt}]\n            content.extend({\"type\": \"input_image\", \"image_url\": image} for image in images)\n            body = {\n                \"model\": model,\n                \"input\": [{\"role\": \"user\", \"content\": content}],\n                \"reasoning\": {\"effort\": \"none\"},\n                \"max_output_tokens\": 16,\n                \"top_p\": 1,\n                \"top_logprobs\": 20,\n                \"include\": [\"message.output_text.logprobs\"],\n            }\n        else:\n            endpoint = \"/chat/completions\"\n            content = [{\"type\": \"text\", \"text\": prompt}]\n            content.extend({\"type\": \"image_url\", \"image_url\": {\"url\": image}} for image in images)\n            body = {\n                \"model\": model,\n                \"messages\": [{\"role\": \"user\", \"content\": content}],\n                \"max_completion_tokens\": 1,\n                \"temperature\": 0,\n                \"reasoning_effort\": \"none\",\n                \"logprobs\": True,\n                \"top_logprobs\": 1024,\n            }\n\n        # Send the request and read the first output token's alternatives.\n        request = urllib.request.Request(\n            url.rstrip(\"/\") + endpoint,\n            headers=headers,\n            data=json.dumps(body).encode(),\n        )\n        with urllib.request.urlopen(request) as response:\n            result = json.load(response)\n        if is_openai:\n            message = next(item for item in result[\"output\"] if item[\"type\"] == \"message\")\n            candidates = message[\"content\"][0][\"logprobs\"][0][\"top_logprobs\"]\n        else:\n            candidates = result[\"choices\"][0][\"logprobs\"][\"content\"][0][\"top_logprobs\"]\n        logprobs = {item[\"token\"]: item[\"logprob\"] for item in candidates}\n\n        # Normalize the returned option scores; missing options initially get zero.\n        missing = [letter for letter in letters if letter not in logprobs or logprobs[letter] <= -9999]\n        if len(missing) == len(letters):\n            raise ValueError(\"API did not return usable scores for any option\")\n        peak = max(logprobs[letter] for letter in letters if letter not in missing)\n        weights = [math.exp(logprobs[letter] - peak) if letter not in missing else 0 for letter in letters]\n        total = sum(weights)\n\n        # An omitted token cannot outrank the last returned alternative.\n        # Allow zero only when their combined normalized probability is below 1e-6.\n        if missing:\n            cutoff = min(value for value in logprobs.values() if value > -9999)\n            missing_weight = len(missing) * math.exp(cutoff - peak)\n            if missing_weight / (total + missing_weight) >= 1e-6:\n                raise ValueError(f\"API omitted non-negligible option scores for: {', '.join(missing)}\")\n        probabilities = {key: weight / total for key, weight in zip(options, weights)}\n\n        # Return the winning choice, probability of true, or expected ordinal level.\n        if question[\"type\"] == \"choice\":\n            answers[name] = {\n                \"type\": \"choice\",\n                \"choice\": max(probabilities, key=probabilities.get),\n                \"probabilities\": probabilities,\n            }\n        elif question[\"type\"] == \"noul\":\n            answers[name] = {\"type\": \"noul\", \"noul\": probabilities[\"true\"]}\n        else:\n            answers[name] = {\n                \"type\": \"score\",\n                \"score\": sum(int(key) * probability for key, probability in probabilities.items()),\n                \"legend\": options,\n                \"probabilities\": probabilities,\n            }\n\n    return {\"answers\": answers}\n\n# Choose the server and model before opening the camera.\nparser = argparse.ArgumentParser(description=__doc__)\nparser.add_argument(\"url\", nargs=\"?\", default=\"http://localhost:8060/v1\")\nparser.add_argument(\"model\", nargs=\"?\", default=\"gemma-4-12b\")\nargs = parser.parse_args()\n\n# Point OpenCV's bundled Qt at the installed system fonts.\nos.environ[\"QT_QPA_FONTDIR\"] = \"/usr/share/fonts/truetype/noto\"\n\n# Open the default Linux webcam with a small capture buffer.\ncamera = cv2.VideoCapture(0, cv2.CAP_V4L2)\nif not camera.isOpened():\n    raise RuntimeError(\"Could not open /dev/video0\")\ncamera.set(cv2.CAP_PROP_BUFFERSIZE, 1)\nprint(f\"Webcam -> {args.model}. Noul: yes %; score: value/max. Ctrl-C or Esc to stop.\", flush=True)\nprint(f\"{'time':<8}\" + \"\".join(f\"{name:>10}\" for name in data[\"questions\"]) + f\"{'fps':>10}\", flush=True)\n\n# Preview continuously while a background worker scores one frame at a time.\nexecutor = concurrent.futures.ThreadPoolExecutor(max_workers=1)\npending = None\ntry:\n    while True:\n        ok, frame = camera.read()\n        if not ok:\n            raise RuntimeError(\"Could not read a webcam frame\")\n        cv2.imshow(\"Webcam\", frame)\n        if cv2.waitKey(1) == 27 or cv2.getWindowProperty(\"Webcam\", cv2.WND_PROP_VISIBLE) < 1:\n            break\n\n        # Print a completed result, then submit the latest frame.\n        if pending is not None:\n            if not pending.done():\n                continue\n            result = pending.result()\n            columns = []\n            for name in data[\"questions\"]:\n                answer = result[\"answers\"][name]\n                if answer[\"type\"] == \"noul\":\n                    value = f\"{answer['noul']:.1%}\"\n                elif answer[\"type\"] == \"choice\":\n                    value = answer[\"choice\"]\n                else:\n                    value = f\"{answer['score']:.2f}/{len(data['questions'][name]['criteria']) - 1}\"\n                columns.append(f\"{value:>10}\")\n            columns.append(f\"{1 / (time.perf_counter() - started):>10.2f}\")\n            print(captured + \"\".join(columns), flush=True)\n        # Measure throughput for evaluated frames, including image encoding.\n        started = time.perf_counter()\n        captured = datetime.datetime.now().strftime(\"%H:%M:%S\")\n        ok, jpeg = cv2.imencode(\".jpg\", frame)\n        if not ok:\n            raise RuntimeError(\"Could not encode the webcam frame\")\n        image = \"data:image/jpeg;base64,\" + base64.b64encode(jpeg.tobytes()).decode()\n        data[\"attachments\"] = [image]\n        pending = executor.submit(score, data, args.url, args.model)\nexcept KeyboardInterrupt:\n    print(\"\\nStopped.\")\nfinally:\n    camera.release()\n    cv2.destroyAllWindows()\n    executor.shutdown()\n```\n\nThe script handles the API differences: llama.cpp uses Chat Completions and OpenAI uses Responses to get it to show alternatives.\n\nI ran Gemma 4 12B QAT through llama.cpp. On Linux with NVIDIA drivers, `curl`, `zstd`, and `uv` installed:\n\n```\n# Model (~7 GB) and multimodal projector (~175 MB).\nmkdir -p ~/models/gemma-4-12b/\ncd ~/models/gemma-4-12b/\ncurl -fL -C - -o gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/gemma-4-12b-it-qat-q4_0.gguf\ncurl -fL -C - -o mmproj-gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4_0.gguf\n\n# Standalone llama.cpp binary for RTX 3090 (CUDA architecture 86).\ncurl -fL -o llama.zst https://huggingface.co/buckets/ggml-org/install.sh/resolve/b11160/x86_64/linux/cuda/86/llama-app.zst\nmkdir -p ~/bin/\nzstd -d llama.zst -o ~/bin/llama\nchmod +x ~/bin/llama\n~/bin/llama serve --models-dir ~/models/ --port 8060\n```\n\nSave the Python example as `webcam.py`. In another terminal, from that directory:\n\n```\nuv run webcam.py http://localhost:8060/v1 gemma-4-12b\n# Or use OpenAI, with OPENAI_API_KEY set in your environment.\nuv run webcam.py https://api.openai.com/v1 gpt-6-luna\n```\n\n", "url": "https://wpnews.pro/news/a-single-function-jev-like-wrapper-for-llms-including-vision-models", "canonical_source": "http://allanrbo.blogspot.com/2026/09/a-jev-like-wrapper-for-llms-including.html", "published_at": "2026-09-26 04:20:58+00:00", "updated_at": "2026-09-26 04:31:13.381339+00:00", "lang": "en", "topics": ["large-language-models", "computer-vision", "ai-tools", "generative-ai"], "entities": ["Jev", "OpenJev", "SemIf", "OpenAI", "Gemma 4 12B", "RTX 3090", "gpt-6-luna", "llama.cpp"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-single-function-jev-like-wrapper-for-llms-including-vision-models", "markdown": "https://wpnews.pro/news/a-single-function-jev-like-wrapper-for-llms-including-vision-models.md", "text": "https://wpnews.pro/news/a-single-function-jev-like-wrapper-for-llms-including-vision-models.txt", "jsonld": "https://wpnews.pro/news/a-single-function-jev-like-wrapper-for-llms-including-vision-models.jsonld"}}