{"slug": "open-decision-model-that-answers-typed-questions-about-audio", "title": "Open decision model that answers typed questions about audio", "summary": "Egma AI released homi, an open decision model built on frozen Qwen3-Omni-30B-A3B-Instruct weights with a decision head replacing the language-model head, 17M parameters of LoRA and one calibration temperature, which answers typed questions about audio clips with a probability per option rather than generating text. On the 7,200-question VocalGrad benchmark homi scored 73.9 versus 63.0 for Gemini 3.8 Flash, at $0.12 per 10k questions against $12.10 for Gemini 3.8 Flash, and on the 4,981-question MMSU benchmark homi scored 83.1 versus Gemini 3.8 Flash's 85.7 at $0.28 versus $14.80. The model supports three question types — noul (yes/no), choice (2 to 255 options) and score (2 to 10 ordered levels) — and the repository includes code to run, evaluate, rebuild training data and train it.", "body_md": "A Pareto-frontier open decision model that answers typed questions about a given audio clip (inspired by [Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev)). Unlike\nJev, [Clef](https://blog.cloudflare.com/clef-decision-models/) and [pplx-decider](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b), which only take in text, homi takes an audio clip and a question, and returns a\nprobability for every option. It beats Gemini Flash on [VocalGrad](https://github.com/ynklab/vocalgrad) and is comparable to Gemini\nFlash on [MMSU](https://github.com/dingdongwang/MMSU). The model can be used\nfor monitoring and evaluating large-scale audio at low cost.\n\nWeights: `hf`\n\nA lot of paralinguistic and phonological questions about audio cannot be answered by looking at a transcript. This model takes in audio natively and answers those questions, e.g. \"How does the speaker feel?\" with the options angry / happy / sad / fear / disgust.\n\nIt is [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) with frozen weights, a\ndecision head in place of the language-model head, 17M parameters of [LoRA](https://arxiv.org/abs/2106.09685), and one calibration temperature. It does\nnot generate text.\n\nWeights: [`egma-ai/homi`](https://huggingface.co/egma-ai/homi). This repository has the code to run\nit, evaluate it, rebuild its training data and train it.\n\nJust like Jev, there are three question types:\n\n- `noul` : yes/no, returns P(yes)\n- `choice` : 2 to 255 options, returns a probability per option\n- `score` : 2 to 10 ordered levels (e.g. slow / normal / fast), returns a probability per level\n\nSame audio, question and options for every model. Unanswered counts as wrong. Our cost is GPU time on one H200 ($4.54/h, batched); API costs are reported tokens at list prices (Gemini 3.8 Flash at its promotional price).\n\n**[VocalGrad](https://github.com/ynklab/vocalgrad)** (7,200 questions: does pitch, speed or volume rise or fall within a clip? Chance is 50%.)\n\n| Model | VocalGrad | Speed | Pitch | Volume | $ per 10k questions | \n|---|---|---|---|---|---|\n| homi | 73.9 | 74 | 61 | 87 | 0.12 | \n| Gemini 3.8 Flash | 63.0 | 78 | 56 | 55 | 12.10 | \n| Qwen3-Omni-30B-A3B | 50.3 | 50 | 50 | 50 | 0.11 | \n| Gemini 3.5 Flash-Lite | 50.9 | 51 | 51 | 50 | 1.30 | \n| OpenAI gpt-audio | 33.8 | 32 | 35 | 34 | 37.03 | \n\n**[MMSU](https://github.com/dingdongwang/MMSU)** (4,981 questions, 47 tasks)\n\n| Model | MMSU | Perception | Reasoning | $ per 10k questions | \n|---|---|---|---|---|\n| Gemini 3.8 Flash | 85.7 | 82.4 | 89.2 | 14.80 | \n| homi | 83.1 | 80.4 | 86.0 | 0.28 | \n| Qwen3-Omni-30B-A3B | 72.2 | 60.5 | 84.5 | 0.19 | \n| Gemini 3.5 Flash-Lite | 67.4 | 55.1 | 80.3 | 2.07 | \n| OpenAI gpt-audio | 60.3 | 46.5 | 74.9 | 41.69 | \n\ngpt-audio did not answer 13% of MMSU and 34% of VocalGrad (it declined or asked us to \"play the audio\"); on the\nquestions it did answer it scores 69.5 and 51.2. Raw answers and per-task tables: [`eval/results/`](https://github.com/egma-ai/homi/blob/main/eval/results).\n\n**Calibration** (does confidence match accuracy?). \"Sure\" means confidence of at least 0.9.\n\n| Model | VocalGrad ECE | VocalGrad Brier | VocalGrad: accuracy when sure (share sure) | MMSU ECE | MMSU Brier | MMSU: accuracy when sure (share sure) | \n|---|---|---|---|---|---|---|\n| homi | 0.157 | 0.399 | 82.8% (67%) | 0.076 | 0.259 | 91.7% (73%) | \n| Gemini 3.8 Flash | 0.237 | 0.570 | 70.2% (37%) | 0.052 | 0.238 | 93.7% (63%) | \n| Qwen3-Omni-30B-A3B | 0.337 | 0.740 | 52.5% (38%) | 0.182 | 0.439 | 84.4% (73%) | \n| Gemini 3.5 Flash-Lite | 0.229 | 0.622 | 55.0% (15%) | 0.133 | 0.512 | 72.1% (14%) | \n| OpenAI gpt-audio | 0.269 | 0.657 | 50.1% (8%) | 0.081 | 0.459 | 84.8% (11%) | \n\nGemini and OpenAI probabilities are the ones the models report in their answers. gpt-audio is scored only on the\nquestions it answered. On VocalGrad, homi is the only model with a\nBrier score better than guessing evenly, but it is overconfident: its most confident answers (98% on average) are\nright 85% of the time. Full tables:\n[`eval/results/CALIBRATION.md`](https://github.com/egma-ai/homi/blob/main/eval/results/CALIBRATION.md).\n\nNo MMSU audio or text is in the training data, and neither are the datasets MMSU builds its audio from.\n\nWeak spots: counting distinct speakers (it usually answers one too few), long conversations with several people, rhyming couplets, reasoning from pauses.\n\n| Folder | Contents | \n|---|---|\n| `inference/` | `predict.py` (`AudioDecider` and a CLI), the decision head, prompt and answer-code helpers | \n| `eval/` | benchmark preparation, runners for our model, Gemini and OpenAI, scoring scripts, `results/` | \n| `training/` | data loading and mixture, training loop, temperature fit (Modal) | \n| `data/` | one folder per training-data builder, and [`DATA.md`](https://github.com/egma-ai/homi/blob/main/data/DATA.md) with every source and licence | \n\nData building, training and evaluation run on [Modal](https://modal.com), using one Volume (`homi`, mounted at\n`/vol`).\n\nYou need a GPU with about 70 GB free (bf16).\n\n```\npip install torch==2.14.1 transformers==5.18.0 peft==0.21.2 accelerate==1.15.0 safetensors==0.8.0 \\\n    soundfile==0.14.0 scipy==1.18.1 numpy==2.5.3 huggingface_hub\n\nhf download egma-ai/homi --local-dir homi\n\npython homi/predict.py clip.wav\npython\nimport sys; sys.path.insert(0, \"homi\")\nfrom predict import AudioDecider\n\ndecider = AudioDecider(\"homi\")\ndecider.ask(\"clip.wav\", {\n    \"angry\": {\"type\": \"noul\", \"instructions\": \"Does the speaker sound angry?\"},\n    \"emotion\": {\"type\": \"choice\", \"instructions\": \"Which emotion does the speaker express?\",\n                \"criteria\": {\"anger\": None, \"happy\": None, \"neutral\": None, \"sad\": None}},\n    \"pace\": {\"type\": \"score\", \"instructions\": \"How fast is the person speaking?\",\n             \"criteria\": [\"slow\", \"normal pace\", \"fast\"]},\n})\n```\n\nAll questions about a clip run in one batched forward pass (about 75 ms for three questions on a short clip on an\nH200). Load the adapter with `predict.py`; it targets the Qwen3-Omni thinker, so `AutoPeftModel` will not find it.\n\n```\npip install modal && modal setup\nmodal volume put homi homi /models/homi       # the downloaded weights\n\nmodal run eval/modal_eval.py::mmsu --step prepare\nmodal run --detach eval/modal_eval.py::mmsu --step run --which homi --adapter-dir /vol/models/homi\nmodal run --detach eval/modal_eval.py::mmsu --step run --which qwen-stock,gemini-3.8-flash,gemini-3.5-flash-lite\nmodal volume get homi /bench_mmsu ./bench_mmsu\npython eval/mmsu_report.py --dir ./bench_mmsu --out MMSU.md\n\nmodal run --detach eval/bench_prepare.py::main                # VocalGrad (also VoxParadox, MSU-Bench)\nmodal run --detach eval/modal_eval.py::bench --name vocalgrad --which homi --adapter-dir /vol/models/homi\npython eval/bench_report.py --dir ./bench_vocalgrad --name vocalgrad --out VOCALGRAD.md\n```\n\nGemini needs a Modal secret `gemini` (`GEMINI_API_KEY`), OpenAI a secret `openai` (`OPENAI_API_KEY`). The exact\nprompts, revisions and scoring are in `eval/bench_lock.json` and `eval/modal_eval.py`. [VoxParadox](https://github.com/ihp-lab/VoxParadox) and [MSU-Bench](https://github.com/ASLP-lab/MSU-Bench) are\nresearch-only licences; check them before publishing scores.\n\nEvery training question has optional audio, an optional text context, a question, 2 to 40 options and a target\nprobability per option. The data is public and permissively licensed, or synthetic audio whose labels are exact by\nconstruction. [`data/DATA.md`](https://github.com/egma-ai/homi/blob/main/data/DATA.md) lists every source and licence.\n\n| Builder | Teaches | Main sources | \n|---|---|---|\n| `training/prepare_data.py` | emotion, spoken intent | [CREMA-D](https://github.com/CheyneyComputerScience/CREMA-D) ,[MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) | \n| `sb_voice` | emotion, pitch, rate, loudness, pauses, gender, age, pronunciation | CREMA-D, [LibriTTS-R](https://www.openslr.org/141/) ,[speechocean762](https://www.openslr.org/101/) | \n| `sb_relative` | louder / faster / higher than the speaker's usual voice | LibriTTS-R, CREMA-D | \n| `sb_count` ,`sb_speakers` | counting turns and speakers, same speaker | LibriTTS-R, [LibriSpeech](https://www.openslr.org/12/) ,[HarperValleyBank](https://github.com/cricketclub/gridspace-stanford-harper-valley) ,[AMI](https://groups.inf.ed.ac.uk/ami/corpus/) | \n| `sb_compare` | pitch / speed / volume / range comparisons, durations, pauses | LibriTTS-R, [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) | \n| `sb_phon` | syllables, stress, emphasis, minimal pairs, consonants and vowels | Kokoro-82M, [CMUdict](https://github.com/cmusphinx/cmudict) | \n| `sb_intonation` | intonation contours, meaning of pauses | LibriTTS-R, Kokoro-82M, [WORLD](https://github.com/mmorise/World) | \n| `sb_disfluency` | disfluency types, verse rhythm and rhyme | Kokoro-82M, CMUdict | \n| `sb_synthetic` | real vs synthetic speech | LibriSpeech, [Apollo](https://huggingface.co/datasets/BrunoHays/apollo-telephony-en) , HarperValleyBank, AMI; Kokoro,[Piper](https://github.com/rhasspy/piper) ,[Parler-TTS](https://huggingface.co/parler-tts/parler-tts-mini-v1) ,[SpeechT5](https://huggingface.co/microsoft/speecht5_tts) ,[Chatterbox](https://huggingface.co/ResembleAI/chatterbox) ,[Vocos](https://github.com/gemelo-ai/vocos) | \n| `sb_sounds` ,`sb_sounds2` | sound events, music genre, instruments | [FSD50K](https://zenodo.org/records/4060432) ,[NSynth](https://magenta.tensorflow.org/datasets/nsynth) ,[FMA](https://github.com/mdeff/fma) ,[OpenMIC-2018](https://github.com/cosmir/openmic-2018) | \n| `sb_calls` ,`sb_aicall` | who answered a call; AI agent with a person, voicemail or machine | Kokoro-82M, HarperValleyBank, [FTC robocalls](https://github.com/wspr-ncsu/robocall-audio-dataset) | \n| `sb_dialog` ,`sb_calltext` | talk-over, silences, turn order; call content | HarperValleyBank, AMI | \n| `sb_phone` ,`sb_phone_pitch` | phone line quality, noise, live vs recorded, language; pitch on phone audio | Apollo, MInDS-14, FTC robocalls, LibriTTS-R, CREMA-D, [MUSAN](https://www.openslr.org/17/) | \n| `sb_spoken` ,`sb_spoken2` | spoken questions, intents, dialog acts | HarperValleyBank, [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) ,[QASC](https://huggingface.co/datasets/allenai/qasc) ,[QuaRTz](https://huggingface.co/datasets/allenai/quartz) ,[JADE-Data](https://huggingface.co/datasets/theunnecessarythings/JADE-Data) | \n| `sb_tasksource` , JADE (`training/` ) | text-only decisions | [tasksource](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (commercial, non-share-alike tasks), JADE-Data | \n| `teacher` | soft labels for text questions | [perplexity-ai/pplx-decider-v1.1-27b](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b) | \n\nEach builder runs with its `modal_build.py` and writes `/vol/stageb/<builder>/`. Some builders reuse raw copies made by\nothers (e.g. `sb_aicall` reads `sb_dialog`'s HarperValleyBank copy), so build in table order. Rebuilt data follows the same\nrecipe, but the seeded samples will not be byte-identical to ours.\n\n```\nmodal run training/modal_train.py::prepare                     # base items, Qwen3-Omni weights\nmodal run training/modal_train.py::jade                        # JADE-Data text items\n# run the data builders and data/teacher\nmodal run training/modal_train.py::audit --run pass1           # counts; fails if an excluded item is present\nmodal run --detach training/modal_train.py::train --run pass1  # 8 x H200 or B200\nmodal run --detach training/modal_train.py::train --run pass2\nmodal run --detach training/modal_train.py::calibrate --run pass2\n```\n\n- LoRA rank 16 (alpha 32, dropout 0.05) on q/k/v/o of the language model's 48 layers and q/k/v of the audio encoder's 32 layers. All base weights frozen.\n- Decision head initialised from the base model's output rows for the answer codes; read at the last prompt position; soft-target cross-entropy.\n- AdamW, 64 questions per step, cosine schedule, 3% warmup, gradient clipping at 1.0, data parallel on 8 GPUs.\n- Two passes: 5,654 steps at learning rate 1e-4 on the core builders, then 2,136 steps at 5e-5 on the newer builders\nplus a 20% replay of the core data. Mixtures and exclusions are in `training/recipe.py` .\n- Text targets blend 50/50 with the teacher's probabilities where the teacher is right.\n- One temperature, fitted on every builder's calibration split with sources weighted equally.\n\nOutputs go to `/vol/runs/<run>/adapter/` and `/vol/runs/<run>/temperature_lora.json`. Put those files next to\n`inference/` (temperature file renamed to `temperature.json`) to get the Hugging Face folder.\n\nCode: Apache-2.0 ([LICENSE](https://github.com/egma-ai/homi/blob/main/LICENSE), [NOTICE](https://github.com/egma-ai/homi/blob/main/NOTICE)). Weights: Apache-2.0 on Hugging Face. Datasets keep their own\nlicences ([data/DATA.md](https://github.com/egma-ai/homi/blob/main/data/DATA.md)).\n\n```\n@misc{homi_2026,\n  title        = {homi: typed questions about audio, with probabilities},\n  author       = {{Outlier Humans Inc.}},\n  year         = {2026},\n  howpublished = {\\url{https://github.com/egma-ai/homi}}\n}\n```\n\n**Inspiration**\n\n- The name is a nod to [Homi J. Bhabha](https://en.wikipedia.org/wiki/Homi_J._Bhabha) .\n- Jev (TypeSafe AI): [launch post](https://typesafe.ai/blog/introducing-system-one-models-and-jev)\n- Clef (Cloudflare): [blog post](https://blog.cloudflare.com/clef-decision-models/) ,[weights](https://huggingface.co/Cloudflare/clef)\n- pplx-decider (Perplexity): [weights](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b)\n\n**Model and method**\n\n- Qwen3-Omni-30B-A3B-Instruct: [weights](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) ,[Qwen3-Omni Technical Report](https://arxiv.org/abs/2509.17765)\n- Hu et al., 2021. [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)\n\n**Benchmarks**\n\n- MMSU (ICLR 2026): [paper](https://arxiv.org/abs/2506.04779) ,[code](https://github.com/dingdongwang/MMSU) ,[data](https://huggingface.co/datasets/ddwang2000/MMSU)\n- VocalGrad: [code](https://github.com/ynklab/vocalgrad) ,[data](https://huggingface.co/datasets/ynklab/vocalgrad)\n- VoxParadox: [paper](https://arxiv.org/abs/2605.27772) ,[code](https://github.com/ihp-lab/VoxParadox)\n- MSU-Bench: [paper](https://arxiv.org/abs/2606.22868) ,[code](https://github.com/ASLP-lab/MSU-Bench)\n\n**Compared models**\n\n- Gemini 3.8 Flash, Gemini 3.5 Flash-Lite: [Gemini API models](https://ai.google.dev/gemini-api/docs/models)\n- OpenAI gpt-audio: [model page](https://platform.openai.com/docs/models/gpt-audio)\n\n**Training data** (licences in [data/DATA.md](https://github.com/egma-ai/homi/blob/main/data/DATA.md))\n\n- Speech: [LibriTTS-R](https://www.openslr.org/141/) ,[LibriSpeech](https://www.openslr.org/12/) ,[CREMA-D](https://github.com/CheyneyComputerScience/CREMA-D) (Cao et al., 2014),[speechocean762](https://www.openslr.org/101/) ,[HarperValleyBank](https://github.com/cricketclub/gridspace-stanford-harper-valley) ,[AMI](https://groups.inf.ed.ac.uk/ami/corpus/) ,[Apollo telephony](https://huggingface.co/datasets/BrunoHays/apollo-telephony-en) ,[MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) ,[FTC robocall audio](https://github.com/wspr-ncsu/robocall-audio-dataset)\n- Sounds and music: [FSD50K](https://zenodo.org/records/4060432) ,[NSynth](https://magenta.tensorflow.org/datasets/nsynth) ,[FMA](https://github.com/mdeff/fma) ,[OpenMIC-2018](https://github.com/cosmir/openmic-2018) ,[MUSAN](https://www.openslr.org/17/) ,[OpenSLR 28 room responses](https://www.openslr.org/28/)\n- Text: [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) ,[QASC](https://huggingface.co/datasets/allenai/qasc) ,[QuaRTz](https://huggingface.co/datasets/allenai/quartz) ,[JADE-Data](https://huggingface.co/datasets/theunnecessarythings/JADE-Data) ,[tasksource](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions)\n- Pronunciation: [CMUdict](https://github.com/cmusphinx/cmudict) ,[misaki](https://github.com/hexgrad/misaki)\n- Synthetic speech and audio tools: [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) ,[Piper](https://github.com/rhasspy/piper) ,[Parler-TTS mini v1](https://huggingface.co/parler-tts/parler-tts-mini-v1) ,[SpeechT5](https://huggingface.co/microsoft/speecht5_tts) with[HiFi-GAN](https://huggingface.co/microsoft/speecht5_hifigan) and[CMU ARCTIC x-vectors](https://huggingface.co/datasets/Matthijs/cmu-arctic-xvectors) ,[Chatterbox](https://huggingface.co/ResembleAI/chatterbox) ,[Vocos](https://github.com/gemelo-ai/vocos) ,[WORLD](https://github.com/mmorise/World) /[pyworld](https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoder) ,[Rubber Band](https://breakfastquay.com/rubberband/)", "url": "https://wpnews.pro/news/open-decision-model-that-answers-typed-questions-about-audio", "canonical_source": "https://github.com/egma-ai/homi", "published_at": "2026-11-04 18:39:11+00:00", "updated_at": "2026-11-04 18:50:31.145334+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-tools"], "entities": ["Egma AI", "homi", "Qwen3-Omni-30B-A3B-Instruct", "Gemini 3.8 Flash", "OpenAI gpt-audio", "VocalGrad", "MMSU", "LoRA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/open-decision-model-that-answers-typed-questions-about-audio", "markdown": "https://wpnews.pro/news/open-decision-model-that-answers-typed-questions-about-audio.md", "text": "https://wpnews.pro/news/open-decision-model-that-answers-typed-questions-about-audio.txt", "jsonld": "https://wpnews.pro/news/open-decision-model-that-answers-typed-questions-about-audio.jsonld"}}