{"slug": "show-hn-peekaboolean-image-and-jev-like-typed-questions-in-typed-anwers-out", "title": "Show HN: Peekaboolean – image and jev-like typed questions in typed anwers out", "summary": "Developer bykof released Peekaboolean v0.1.0, a 500M-parameter vision-language model that answers caller-supplied typed questions about an image without generating free text, returning choice, ordinal score, and yes/no probability outputs. The model builds on SmolVLM-500M-Instruct with a LoRA adapter on the language model and a frozen vision tower, distilled from Qwen3-VL-30B-A3B plus public VQA data, and runs a six-question, 28-option request at about 400 ms p95 on an M1 Pro (MPS, 512 px) and about 60 ms on a desktop GPU. Weights are published on GitHub under CC BY-NC 4.0.", "body_md": "*Peek at an image, get booleans back* (and choices, and scores).\n\nA small vision-language model that answers typed questions about an image, fast enough\nfor a laptop. You send one image, a `state` (context) and any number of named\nquestions. Each question is one of three types:\n\n- **choice** : pick one of the options you supply, each with its own description\n- **score** : place the image on an ordinal rubric you supply (any number of levels)\n- **noul** : yes/no, returned as a probability\n\nThe model never generates text. It scores the options the caller wrote, so it only returns answers you asked for, the probabilities are calibrated, and a rubric the model never saw in training works as well as a familiar one.\n\n- Backbone: [SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct) ,\nLoRA on the language model, vision tower frozen\n- Head: the backbone's own `logit(Yes) − logit(No)` for \"is this proposed answer correct?\"\n- Training: distilled from Qwen3-VL-30B-A3B (a local teacher that wrote and labelled requests) plus public VQA data\n- Latency: about 400 ms p95 for a six-question request (28 options) on an M1 Pro (MPS, 512 px); about 60 ms on a desktop GPU\n- Weights: [GitHub release v0.1.0](https://github.com/bykof/peekaboolean/releases/tag/v0.1.0) (CC BY-NC 4.0, see[Licence](#licence) )\n\nHow it was built and what did and did not work: [docs/REPORT.md](https://github.com/bykof/peekaboolean/blob/main/docs/REPORT.md).\n\n```\ngit clone https://github.com/bykof/peekaboolean && cd peekaboolean\nuv sync --python 3.13\ncurl -L https://github.com/bykof/peekaboolean/releases/download/v0.1.0/peekaboolean-500m.tar.gz | tar xz\nuv run python -m peekaboolean.serve --adapter peekaboolean-500m \\\n  --image photo.jpg --request requests/general.json --max-edge 512\n```\n\n`--adapter` takes a local checkpoint directory or a Hugging Face repo id. The base model\ndownloads on first use. `--device` picks `cuda`, `mps` or `cpu` (default: auto).\n\nA request (`requests/general.json`):\n\n```\n{\n  \"state\": \"Answer using only what is visible in the image.\",\n  \"questions\": {\n    \"kind\": {\n      \"type\": \"choice\",\n      \"instructions\": \"What kind of image is this?\",\n      \"criteria\": {\n        \"photo\": \"A photograph of a real scene\",\n        \"screenshot\": \"An application or website screenshot\",\n        \"document\": \"A scanned or photographed document page\",\n        \"chart\": \"A chart, graph, or diagram\"\n      }\n    },\n    \"sharpness\": {\n      \"type\": \"score\",\n      \"instructions\": \"How sharp is the image?\",\n      \"criteria\": [\n        \"very blurry\",\n        \"somewhat blurry\",\n        \"acceptable\",\n        \"sharp\"\n      ]\n    },\n    \"person_visible\": {\n      \"type\": \"noul\",\n      \"instructions\": \"Is a person visible in the image?\"\n    }\n  }\n}\n```\n\nThe answer for a chart image:\n\n```\n{\n  \"answers\": {\n    \"kind\": {\"type\": \"choice\", \"choice\": \"chart\", \"confidence\": 0.755,\n             \"probabilities\": {\"photo\": 0.0002, \"screenshot\": 0.180, \"document\": 0.004, \"chart\": 0.817}},\n    \"sharpness\": {\"type\": \"score\", \"score\": 1.96, \"confidence\": 0.151,\n                  \"probabilities\": {\"0\": 0.114, \"1\": 0.176, \"2\": 0.346, \"3\": 0.363},\n                  \"legend\": {\"0\": \"very blurry\", \"1\": \"somewhat blurry\", \"2\": \"acceptable\", \"3\": \"sharp\"}},\n    \"person_visible\": {\"type\": \"noul\", \"noul\": 0.047}\n  }\n}\n```\n\n`score` is the expected level index (0 = first level). `noul` is P(yes).\n\nFrom Python, load once and reuse:\n\n``` python\nfrom peekaboolean.serve import load, evaluate\nmodel, processor, calibration = load(\"peekaboolean-500m\", device=\"mps\", merge=True)\nresult = evaluate(model, processor, state, questions, \"photo.jpg\", calibration, max_edge=512)\n```\n\n`state`, `instructions` and every option may be a string, an object or a list. Noul\nquestions may carry their own wording: `\"criteria\": {\"true\": \"...\", \"false\": \"...\"}`.\nEach question is scored independently, so answers do not depend on option order or on\nthe other questions in the request.\n\n`--mode` (default `auto`); all return the same answers in fp32:\n\n- `shared` : encode image and state once, then score every option as a suffix against\nthe KV cache\n- `single` : everything in one forward pass; faster for small requests on MPS\n- `auto` :`single` up to 8 options,`shared` above\n- `naive` : one forward pass per question; the reference path\n\n`serve --check` asserts that the three agree. `python -m peekaboolean.benchmark` measures warm\nlatency per image size and request shape (`--breakdown` for per-stage times).\n\nCalibration temperatures were fitted separately for 256, 384 and 512 px. Serve at 512 px unless latency forces a smaller size.\n\nHeld-out test split, 512 px. Image splits are by content hash, so no test image was seen in training. \"Teacher\" groups are requests written by the teacher model and measure agreement with the teacher, not with ground truth.\n\n| Group | untrained 500M (yes/no head) | v6 (256M, scalar head) | **v8b (this release)** | \n|---|---|---|---|\n| teacher choice, accuracy | 0.51 | 0.77 | **0.78** | \n| teacher noul, balanced accuracy | 0.65 | 0.87 | **0.94** | \n| teacher score, Spearman | 0.37 | 0.71 | **0.73** | \n| VQAv2 choice / noul | 0.73 / 0.83 | 0.91 / 0.79 | **0.91 / 0.79** | \n| DocVQA / ChartQA / TextVQA choice | 0.69 / 0.60 / 0.85 | 0.85 / 0.86 / 0.96 | **0.87 / 0.87 / 0.96** | \n| AI2D / CLEVR choice | 0.76 / 0.48 | 0.79 / 0.74 | **0.90 / 0.80** | \n| counting rubrics VQAv2 / CLEVR, Spearman | 0.44 / 0.47 | 0.79 / 0.70 | **0.79 / 0.79** | \n| FairFace age (10 bins), Spearman | – | – | **0.81** | \n| FairFace \"is this a child\" / gender, balanced accuracy | – | – | **0.97 / 0.96** | \n\nThe untrained column comes from a 4,000-row validation sample. v6 is measured on its own (v6) test split, v8b on the full v7/v8 test split (21k questions); both use the same image splits. Per-group JSON reports are attached to the GitHub release.\n\nMac latency (M1 Pro, MPS, fp32, `--mode auto`, 512 px): a six-question request with 28\noptions measured about 380–400 ms p95 for the 500M backbone. The 256M v6 model: 116 ms\nfor one noul, 230 ms for the six-question request.\n\n- Accuracy against the teacher is not accuracy against people. There is no public human-labelled acceptance set for request-shaped questions yet.\n- Photo aesthetics were dropped from training in v7; the model's aesthetic scores are no better than a text-only prior.\n- The model estimates age, gender and \"child or adult\" from faces (FairFace training). These estimates carry the biases of the data and are wrong for individual people often enough that they must not be used to decide anything about a person. It also answers such questions for drawings and cartoon characters.\n- Options are scored independently, so the model cannot compare options that differ only by contrast (\"the larger one\").\n\nEverything that produced this checkpoint is in `src/peekaboolean`. The pipeline, in order:\n\n1. `prepare_general.py` : typed questions from[The Cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) (VQAv2, CLEVR, TextVQA, DocVQA, ChartQA, Screen2Words, AI2D), training partitions only\n2. `prepare_teacher.py` : a local Qwen3-VL-30B-A3B (vLLM) writes one request per image\nand labels it (see the module docstring; runs in its own vLLM environment)\n3. `prepare_v6.py` : mixes public and teacher rows, adds counting rubrics, tempers the\nteacher's probabilities against rows with known answers\n4. `prepare_age.py` : FairFace age, gender and child/adult questions\n5. `pipeline.py` :`train_general.py` → per-size calibration and test report\n(`postprocess.py` ) → latency benchmark, as detached background jobs\n6. `full_test.py` : score a checkpoint on every row of a test split\n\nThe exact settings for each version are in [docs/REPORT.md](https://github.com/bykof/peekaboolean/blob/main/docs/REPORT.md). All runs\nused one RTX PRO 6000 (96 GB).\n`docs/legacy-qwen-v1-v2.md` describes the earlier Qwen3-VL-4B aesthetics experiments\n(`train.py`, `prepare_ava.py`, `prepare_aadb.py`, ...).\n\n- Code: Apache-2.0 ([LICENSE](https://github.com/bykof/peekaboolean/blob/main/LICENSE) )\n- Weights: CC BY-NC 4.0. The base model and the teacher are Apache-2.0, but some training data only allows research use (DocVQA; AVA and AADB images, which the teacher wrote questions about) or carries GPL-3.0 (ChartQA). The weights are therefore released for research and non-commercial use. Details are in the model card.\n\nSee [CITATION.cff](https://github.com/bykof/peekaboolean/blob/main/CITATION.cff).", "url": "https://wpnews.pro/news/show-hn-peekaboolean-image-and-jev-like-typed-questions-in-typed-anwers-out", "canonical_source": "https://github.com/bykof/peekaboolean", "published_at": "2026-09-27 09:50:05+00:00", "updated_at": "2026-09-27 10:31:25.338209+00:00", "lang": "en", "topics": ["computer-vision", "large-language-models", "ai-tools", "ai-research"], "entities": ["Peekaboolean", "bykof", "SmolVLM-500M-Instruct", "Qwen3-VL-30B-A3B", "GitHub", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-peekaboolean-image-and-jev-like-typed-questions-in-typed-anwers-out", "markdown": "https://wpnews.pro/news/show-hn-peekaboolean-image-and-jev-like-typed-questions-in-typed-anwers-out.md", "text": "https://wpnews.pro/news/show-hn-peekaboolean-image-and-jev-like-typed-questions-in-typed-anwers-out.txt", "jsonld": "https://wpnews.pro/news/show-hn-peekaboolean-image-and-jev-like-typed-questions-in-typed-anwers-out.jsonld"}}