{"slug": "show-hn-minnarone-multimodal-agents-that-watch-listen-and-react-live", "title": "Show HN: Minnarone – Multimodal agents that watch, listen, and react live", "summary": "Carlos A. Rueda released Minnarone, an open-source framework for building multimodal AI agents that perceive live audio, video, and chat context and react proactively, initially supporting a chat-only shadow mode for Twitch. The framework generalizes a bot that participated in Twitch live streams indistinguishably from a human, and is available on GitHub under the MIT license.", "body_md": "*Cover of enkk's origin video.*\n\nA reusable framework for building AI agents that **perceive a live multimodal context** (audio, video/screen, chat, platform events) and **react proactively** — both as a public participant (streamer co-host, group commentator) and as a private assistant (suggestions for sellers, presenters, meeting participants).\n\nIt grew out of the generalization of **Minnarone**, a bot that watched Twitch live streams and interacted in chat in a way indistinguishable from a human.\n\n*An English Twitch shadow run: live chat, speech, and video become context, then Minnarone produces a candidate reaction without sending it.*\n\nThe first supported task is a **chat-only shadow** rehearsal: Minnarone reads\nthe channel, builds candidate replies, records local evidence, and sends no\nTwitch message. From a clean clone:\n\n```\ngit clone https://github.com/carlitose/minnarone.git\ncd minnarone\nuv sync --extra tui\ncp .env.example .env\nmkdir -p .local/minnarone/onboarding/examplechannel\ncp -R examples/onboarding/. .local/minnarone/onboarding/examplechannel/\n```\n\nIn the local copy, change `examplechannel`\n\n, `soul.md`\n\n, and `facts/channel.md`\n\n.\nPut the dedicated bot account's read credentials and the LLM key in `.env`\n\n;\nnever paste tokens into YAML. Then validate and run:\n\n```\nuv run python -m minnarone .local/minnarone/onboarding/examplechannel/twitch-chat-shadow.it.yaml --check\nuv run python -m minnarone .local/minnarone/onboarding/examplechannel/twitch-chat-shadow.it.yaml --tui\n```\n\n`--check`\n\nis local and lazy: it validates construction but does not prove a\nTwitch connection, hardware capacity, or first model inference. The run itself\nstarts in shadow and requires an attended stop with `Ctrl-C`\n\n.\n\nThese inputs have deliberately different jobs:\n\n| Input | Responsibility | Not its job |\n|---|---|---|\nYAML configuration |\nSelect adapters, modes, models, providers, safety budgets, and cadences. | Persona prose or stable channel knowledge. |\n`soul.md` |\nDefine identity, voice, values, and behavioral boundaries. | Runtime wiring or claims about the channel. |\n`facts/` |\nHold manually curated, sourced, stable facts about channels or interlocutors. | Learned memory, secrets, or prompt instructions. |\nBuilt-in prompt templates / `prompts_dir` |\nShape runtime instructions; an explicit `prompts_dir` overrides packaged templates. |\nGeneral configuration, soul, or facts. |\n\nMinnarone does not auto-learn persistent facts. Review each change. Use the prompt skill below for template overrides; onboarding must not blur that safety boundary.\n\n**Chat-only shadow (P0).** Start with, run`twitch-chat-shadow.it.yaml`\n\n`--check`\n\n, then observe candidate sends in the TUI. No write token is needed or read.**Media smoke (P1/P2).** Prove capture components separately before adding them to the agent. Follow the bounded chat,`--no-chat --audio`\n\n, VAD, and`--no-chat --video`\n\ncommands in the[Twitch operator guide](/carlitose/minnarone/blob/main/docs/twitch-operator.md#smoke-commands). Inspect`.smoke/*/stats.json`\n\n; a smoke is evidence for that component only.**Full multimodal shadow (P3-P5).** Choose a hardware profile in the[runtime model research](/carlitose/minnarone/blob/main/docs/research/runtime-model-profiles.md), acquire artifacts explicitly at the revisions in the[model manifest](/carlitose/minnarone/blob/main/docs/runtime-model-manifest.json), and verify the complete bundle: required files, published SHA-256 digests, and a local loader smoke. A primary weight digest alone is not a readiness result. The Italianis a`twitch-full-shadow.it.yaml`\n\n**P3-only MPS example**. It uses English VoxCeleb CAM++`3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx`\n\n, dimension`512`\n\n, and threshold`0.5`\n\n; the older Mandarin 192-dimension artifact is not the public default.**P4 CUDA and P5 llama.cpp** require their profile-specific backend, device, model, and runtime settings; do not run it unchanged. Run every isolated smoke, then`--check`\n\n, then a bounded shadow TUI run while watching memory, queues, and latency.**Attended live.** Use a separate local config only after every gate below passes. A new`live`\n\n-configured session still starts in shadow. Inspect the candidate output, then press`p`\n\ntwice within three seconds to promote. Press`k`\n\nfor the immediate kill-switch; returning live always requires pressing`p`\n\ntwice again. Never leave a live session unattended.\n\nLive is not the quickstart. It requires recorded **broadcaster consent** for\nthe target channel, a **dedicated bot account**, an allow-list, and read/write\ntokens belonging to that account. The allow-list is a technical defense, not\nevidence of consent. Complete **token validation** for account, required scope,\nexpiry, and revocation at startup and no later than hourly. A mismatch, `401`\n\n,\nrevocation, missing scope, block, ban, or stop request means fallback to shadow\nor stop, never a best-effort send.\n\nChoose an honest **disclosure** mechanism appropriate to the channel: the bot\nprofile, channel description, a clear response, or an announcement. Do not\ninstruct the system to deny what it is. Minnarone currently sends through IRC;\ndo not promise a Twitch **Chat Bot Badge**, which requires the supported Send\nChat Message API/App-token authorization path and is not provided by IRC.\n\nThe defaults `max_per_minute: 1`\n\nand `max_per_hour: 20`\n\nare conservative\n**Minnarone budgets**. They are not official **Twitch limits** and do not grant\npermission to fill Twitch's rate bucket. Keep both layers of constraints.\n\nShadow still reads and stores data. A run can contain `perceptions.jsonl`\n\n,\n`debug/prompts/`\n\n, `debug/events.jsonl`\n\n, summaries, and derived copies of chat.\n`retention.perceptions_days`\n\nis reserved and inert; it does not delete data.\nChoose a purpose-bound duration, document an **opt-out** contact, stop the run,\nand perform **manual deletion** of the complete run directory and derived\ncopies when requested, consent is withdrawn, or the data is no longer needed.\n\nThe **portable canonical** skills are under `.agents/skills/`\n\non every checkout.\nRelative `.claude/skills/`\n\nsymlink aliases are optional conveniences on\nsymlink-capable checkouts. With `core.symlinks=false`\n\n, Git may materialize an\nalias as a plain file containing only its target; that plain file is not a\nvalid skill. Code agents must load the matching canonical `.agents`\n\n`SKILL.md`\n\nbefore acting.\n\n| Skill | Trigger | Actions | Boundary |\n|---|---|---|---|\n`minnarone-prompts` |\n\n`minnarone-twitch-onboarding`\n\n`minnarone-runtime-doctor`\n\nHumans can follow the paths above directly. Code agents should start at\n[ AGENTS.md](/carlitose/minnarone/blob/main/AGENTS.md) and contributors at\n\n[; both route architecture, quality, prompt safety, dirty-worktree handling, and skills.](/carlitose/minnarone/blob/main/CONTRIBUTING.md)\n\n`CONTRIBUTING.md`\n\nMinnarone was conceived and built by **enkk**, its original author and designer: a bot able to listen to and watch a Twitch live stream and to interact in chat with other users and with the streamer without anyone realizing they were talking to an artificial intelligence (a kind of Turing test applied to chat).\n\n**Origin video**:[https://www.youtube.com/watch?v=EkunaRO0uKg](https://www.youtube.com/watch?v=EkunaRO0uKg)** Transcript**:— the transcript of enkk's video from which the specification of this framework was derived.`docs/source/transcript.md`\n\nThis repository generalizes that idea into a reusable framework: the same perception + reaction engine serves different use cases (Twitch, Teams meetings) by changing only the configuration.\n\nDisclaimer: This project was inspired by Enkk's idea and video. However, it is not affiliated with him, and he is not involved in the maintenance of this repository.\n\n— requirements, user stories, use cases, edge cases, system design and roadmap.[Project specification](/carlitose/minnarone/blob/main/docs/SPECIFICATION.md)— capture smoke, VAD diagnostics,[Twitch operator guide](/carlitose/minnarone/blob/main/docs/twitch-operator.md)`adapter: twitch`\n\nruntime and enabling public send (shadow/live).— synthesizer and suggester profiles on Teams via[Meeting assistant guide](/carlitose/minnarone/blob/main/docs/meeting-assistant-operator.md)`adapter: os_capture`\n\n.— transcript and screenshots from which the specification was derived.[Source material](/carlitose/minnarone/blob/main/docs/source)\n\nThe \"Minnarone\" app starts from a YAML **configuration file** (soul, facts,\nadapter, provider, cadences, mode) — without writing code.\n\n**Prerequisites**: Python 3.11+ (3.12 recommended — see `.python-version`\n\n).\n\nFirst create and activate a virtual environment. With\n[uv](https://docs.astral.sh/uv/) (recommended):\n\n```\nuv venv                             # create .venv\n# activate it (pick your shell):\nsource .venv/bin/activate           # macOS / Linux\nsource .venv/Scripts/activate       # Windows — Git Bash\n# .venv\\Scripts\\Activate.ps1        # Windows — PowerShell\n# .venv\\Scripts\\activate.bat        # Windows — cmd\n\nuv pip install -e .                 # core\nuv pip install -e '.[tui]'          # + observability dashboard (textual)\nuv pip install -e '.[audio]'        # + Twitch audio runtime: faster-whisper + sherpa-onnx (ASR + speaker)\nuv pip install -e '.[video]'        # + Twitch video runtime: Streamlink Python + PyAV\nuv pip install -e '.[vlm]'          # + captioning `vlm.backend: qwen`: transformers + torch/torchvision + Pillow\nuv pip install -e '.[vlm-llamacpp]' # + captioning `vlm.backend: llamacpp`: Pillow only (via multimodal llama-server, no torch)\nuv pip install -e '.[os-capture]'   # + system audio capture (soundcard) and screen capture (mss)\n\n# Validate the config and build the agent (dry-run, no loop, no network):\npython -m minnarone path/to/config.yaml --check\n\n# Start the live reaction loop:\npython -m minnarone path/to/config.yaml\n\n# Start with the observability TUI dashboard (requires the `tui` extra):\npython -m minnarone path/to/config.yaml --tui\n\n# Replay a past run or a perceptions.jsonl offline (replay dashboard):\npython -m minnarone --replay <run_dir_or_perceptions.jsonl>\n```\n\nPrefer plain `pip`\n\n? Create the venv with the stdlib\n(`python -m venv .venv`\n\n), activate it, then drop the `uv`\n\nprefix\n(`pip install -e '.[tui]'`\n\n). Note: a `uv venv`\n\ndoes **not** ship `pip`\n\n, so use\n`uv pip`\n\nwith it. The run commands above assume the venv is activated (or\nprefix them with `uv run`\n\n).\n\nThe extras can be combined (e.g. `uv pip install -e '.[os-capture,audio,vlm]'`\n\n)\nand can also be installed with `uv sync --extra <name>`\n\n.\n\nFirst: the Twitch/Teams examples validate that setup exists, so they error out of the box. Twitch adapters need`--check`\n\n`TWITCH_BOT_USERNAME`\n\n/`TWITCH_OAUTH_TOKEN`\n\nin`.env`\n\n(`cp .env.example .env`\n\n); the OS-capture and Teams examples need a local speaker-embedding ONNX model (`speaker_embedding.model_path`\n\n) — see the sections below.`examples/llamacpp-local.example.yaml`\n\npasses`--check`\n\nwith no extra setup.\n\nThe CLI loads a `.env`\n\nfile **at startup**, before reading the secrets: first\nnext to the config file, then in the `cwd`\n\n. Variables already exported in the\nterminal take precedence over the file (standard dotenv semantics). The loader is\nminimal and zero-dependency; `.env`\n\nis gitignored — never commit real\nvalues. The template is [ .env.example](/carlitose/minnarone/blob/main/.env.example) (\n\n`cp .env.example .env`\n\n).| Variable | When it is needed |\n|---|---|\n`OPENROUTER_API_KEY` |\nWith `llm_provider: grok` /`deepseek` (OpenRouter). NOT needed with `llm_provider: llamacpp` (local LLM). |\n`TWITCH_BOT_USERNAME` |\nWith `adapter: twitch` + `twitch.chat: true` (read-side IRC ingestion). |\n`TWITCH_OAUTH_TOKEN` |\nWith `adapter: twitch` + `twitch.chat: true` or live send — read token (`chat:read` ). |\n`TWITCH_SEND_OAUTH_TOKEN` |\nOnly for `twitch.send.mode: live` — write token (`chat:edit` ) of the dedicated bot account. |\n\nThe read token and the write token are deliberately distinct: a read-only\nconfig must never have the power to send messages. In `live`\n\n, both tokens must\nbelong to `TWITCH_BOT_USERNAME`\n\n: the read token needs `chat:read`\n\n, while the send\ntoken needs `chat:edit`\n\n. Minnarone validates them at startup, no later than\nhourly, and before a short `expires_in`\n\ndeadline using a safety margin. Absolute\ndeadlines prevent HTTP latency drift; tokens already inside the margin fail\nclosed instead of retrying rapidly. A read\nfailure disarms live, stops the sender, and stops the run; a send failure stops\nthe sender and permanently keeps that running session in shadow. `off`\n\nand\n`shadow`\n\nnever read the send token. Token values never enter logs, errors, or\nartifacts; `--check`\n\nremains offline.\n\n```\nuv sync --extra dev\nmake quality\n\n# run every hook against the whole repo on demand\nuv run --extra dev pre-commit run --all-files\n\n# make the hooks run automatically at commit — either:\ngit config core.hooksPath .githooks   # bridge to .pre-commit-config.yaml\n# ...or the standard pre-commit install:\nuv run --extra dev pre-commit install\n```\n\n`make quality`\n\nruns `ruff format --check`\n\n, `ruff check`\n\n, Vulture, Deptry and\nPylint limited to `duplicate-code`\n\n(`R0801`\n\n, on `src`\n\nonly — test setup is\nallowed to repeat). The same tools run at commit time via\n[ .pre-commit-config.yaml](/carlitose/minnarone/blob/main/.pre-commit-config.yaml) (local\n\n`uv run`\n\nhooks;\n`ruff check --fix`\n\nand `ruff format`\n\nalso auto-fix). The `spike/`\n\nreference code\nis excluded from all of them.For contributors and code agents working on the externalized prompts there is a\nrepo-local Claude Code skill in [ .agents/skills/minnarone-prompts/](/carlitose/minnarone/blob/main/.agents/skills/minnarone-prompts/SKILL.md)\n(also reachable via the optional\n\n`.claude/skills/`\n\nsymlink alias):\nit documents the file map and the non-obvious constraints (required\nplaceholders, control tokens, the hard-coded security boundary, byte-invariance\nof the packaged defaults) and prescribes the safe flow — edit →\n`minnarone validate-prompts`\n\n→ targeted tests → preview render.: put it in`OPENROUTER_API_KEY`\n\n`.env`\n\n(or export it into the environment). Not needed with`llm_provider: llamacpp`\n\n(see[local LLM](#local-llm-llamacpp)).**macOS permissions**: perception capture requires authorizing the process (e.g. the terminal) in*System Settings → Privacy & Security*for**Microphone**(audio) and** Screen Recording**(video/screen). System audio may require additional tooling (loopback). Without the permissions the reaction loop runs but receives no perceptions.\n\n`Agent.run()`\n\nruns three things CONCURRENTLY: the reaction loop, the Summarizer\nloop (short-term memory, `summarizer_interval`\n\ncadence) and the\n*perception pump*, which routes every adapter `RawEvent`\n\nto the perceiver of\nits channel (`chat`\n\n/`audio`\n\n/`video`\n\n) → store.\n\n- The\n**chat** channel is always wired (no model):`adapter: twitch`\n\nbuilds the Twitch runtime from the config and the credentials in the environment. - The\n**audio** channel uses local VAD + faster-whisper +`sherpa-onnx`\n\nspeaker embedding when`twitch.audio: true`\n\n(or`os_capture.audio: true`\n\n) and the local backends are installed/configured. - The\n**video** channel uses Streamlink + PyAV + a local captioning backend when`twitch.video: true`\n\nand the`video`\n\nextra is installed. The backend is chosen by`vlm.backend`\n\n:`qwen`\n\n(Qwen2-VL torch, requires`vlm.model`\n\n+ the`vlm`\n\nextra) or`llamacpp`\n\n(multimodal llama-server, lightweight`vlm-llamacpp`\n\nextra). - The\nadapter (mic + system loopback audio + screen recording) observes the local machine instead of a remote stream.`os_capture`\n\nThe ** mode switch** is only configuration (same engine):\n\n`public`\n\nroutes the output to the public channel (console and, if enabled,`twitch.send`\n\n). On Twitch in public mode the persona is**always**`original_chat`\n\n(see below).`private`\n\nkeeps the output on only the**local console**(`[PRIVATE]`\n\n): no public message is ever sent, regardless of`twitch.send`\n\n.\n\nThe local commentator is configured with `commentator.profiles`\n\n: a dictionary of\n**profiles** indexed by style. A present profile activates the corresponding\nreactor; an empty profiles dictionary = commentator off.\n\nStyle (`commentator.profiles.<style>` ) |\nWhat it does | Mode |\n|---|---|---|\n`operator` |\nPrivate local play-by-play/commentary for the operator. | `private` |\n`original_chat` |\nPublic persona for Twitch chat (`RE:` /`MSG:` contract). |\n`public` (Twitch) or `private` (local dry-run) |\n`meeting_synthesizer` |\nPeriodic structured summaries of a meeting. | `private` |\n`suggester` |\nContextual suggestions on every perception. | `private` |\n\n`meeting_synthesizer`\n\nand `suggester`\n\nrequire `mode: private`\n\n(validated at\n`--check`\n\n). On `adapter: twitch`\n\n+ `mode: public`\n\nonly `original_chat`\n\nis allowed:\na different profile is rejected with a clear error.\n\nPublic send in chat is **gated** and off by default. The\n`twitch.send`\n\nblock (inside `twitch:`\n\n) has three modes:\n\n`off`\n\n(default): no`PRIVMSG`\n\nsent.`shadow`\n\n: a dry run without network — the agent decides what it would write but sends nothing. It is the recommended break-in step (shadow-first).`live`\n\n: real send to chat.\n\nGuardrails (conservative defaults): channel **allow-list** (`allowed_channels`\n\n;\n`mode: live`\n\nrequires the channel to be in the list), **budget** well below\nTwitch's IRC limits (`max_per_minute: 1`\n\n, `max_per_hour: 20`\n\n), **kill-switch** with\nauto-degrade to shadow after `failure_threshold`\n\nconsecutive failed sends (default\n3), and a **separate write token** (`TWITCH_SEND_OAUTH_TOKEN`\n\n) of a\ndedicated bot account. In the TUI, press `p`\n\ntwice within 3 seconds to promote\nand `k`\n\nonce for the kill-switch. Full procedure in the\n[Twitch operator guide](/carlitose/minnarone/blob/main/docs/twitch-operator.md).\n\nMinnarone can act as a **local commentator** or **meeting assistant** on a\nTeams call you take part in: it observes the **system audio** (the voices of the other\nparticipants, captured from the audio-output loopback) and the **screen** (slides,\nfaces, shared text), and produces output only on the **local console**\n(`[PRIVATE]`\n\n). It sends nothing into the meeting: no message, no\naudio, no public output.\n\nReady-to-use presets:\n\n[examples/teams-commentator.yaml](/carlitose/minnarone/blob/main/examples/teams-commentator.yaml)—`operator`\n\nprofile.[examples/teams-meeting-assistant.yaml](/carlitose/minnarone/blob/main/examples/teams-meeting-assistant.yaml)—`meeting_synthesizer`\n\n+`suggester`\n\nprofiles.[examples/teams-meeting-full.yaml](/carlitose/minnarone/blob/main/examples/teams-meeting-full.yaml)— full configuration.\n\nOperational details (profiles, TUI, troubleshooting) in the\n[meeting assistant guide](/carlitose/minnarone/blob/main/docs/meeting-assistant-operator.md).\n\nOS capture lives in the `os-capture`\n\nextra (system audio via `soundcard`\n\n,\nscreen via `mss`\n\n):\n\n```\npip install -e '.[os-capture]'   # or: uv sync --extra os-capture\n```\n\nThe `os-capture`\n\nextra covers only the **raw capture**. To actually run the\nmodels you also need:\n\n- the\n`audio`\n\nextra (faster-whisper + sherpa-onnx) so that the audio is transcribed and diarized (ASR/speaker); - the\n`vlm`\n\nextra (transformers + torch) so that the screen is described by the Qwen2-VL captioner (lazy import: the model loads at the first description).\n\n```\npip install -e '.[os-capture,audio,vlm]'\n```\n\nWith `os-capture`\n\nalone you can do capture diagnostics (below) but no ASR/VLM.\n\n**Default audio output**: the loopback captures the** system default audio output**. Set as the default output device the one on which Teams plays audio (Windows:*Settings → Sound → Output*; Linux: the corresponding PulseAudio sink). If Teams plays on another device, the loopback will capture silence.**Screen capture permission**: authorize the process (e.g. the terminal) to record the screen. On macOS it is*System Settings → Privacy & Security → Screen Recording*; without the permission the frames come out empty/black.**Monitor selection**: choose which screen to capture with`os_capture.monitor`\n\n(index`>= 1`\n\n; 1 = primary monitor). The same index is exposed by the smoke as`--monitor`\n\n.\n\nBefore enabling ASR/VLM it is worth verifying that audio and screen are\nactually captured. The OS capture smoke is **capture-only** (no ASR/VLM,\ndoes not require `OPENROUTER_API_KEY`\n\n) and writes bounded artifacts to the\n`--output`\n\ndirectory: `raw/audio/*.pcm`\n\n(PCM mono 16 kHz s16le), `raw/video/*.jpg`\n\n, and\n`stats.json`\n\nwith counts and any failures.\n\nThe\n\n`minnarone-oscapture-smoke`\n\nentry point lives in the virtualenv, so it is only on your`PATH`\n\nwhen the venv isactivated(see the install section). Otherwise invoke it without activation via`python -m minnarone.oscapture_smoke ...`\n\nor`uv run minnarone-oscapture-smoke ...`\n\n. Requires the`os-capture`\n\nextra.\n\nVerify audio capture from the default output loopback:\n\n```\nminnarone-oscapture-smoke \\\n  --duration 30 \\\n  --output ./.smoke/os-audio \\\n  --audio \\\n  --audio-chunk-seconds 1.0\n```\n\nVerify screen capture from the chosen monitor:\n\n```\nminnarone-oscapture-smoke \\\n  --duration 30 \\\n  --output ./.smoke/os-video \\\n  --video \\\n  --video-fps 1.0 \\\n  --monitor 1\n```\n\nTo check only the VAD segmentation on the audio (counts/durations without\nASR), use `--vad-diagnostic`\n\n(it also enables audio): `stats.json`\n\nwill include\n`vad_utterances`\n\nand `vad_utterance_durations_ms`\n\n.\n\n```\nminnarone-oscapture-smoke \\\n  --duration 30 \\\n  --output ./.smoke/os-vad \\\n  --vad-diagnostic\n```\n\nValidate dry first (no hardware opened, no network), then start the loop:\n\n```\npython -m minnarone examples/teams-commentator.yaml --check\npython -m minnarone examples/teams-commentator.yaml\n```\n\n**Windows**(WASAPI) and** Linux**(PulseAudio monitor):** native**loopback of the default output, no additional tooling.** macOS**:`soundcard`\n\ndoes**not** support loopback. You need an external loopback device (e.g. BlackHole) set as the default output to get the system audio to the capture.\n\nThe audio pipeline (VAD → ASR → speaker tagging) labels every utterance with one\nof **three canonical labels**:\n\n`streamer`\n\n— the local operator / whoever runs the session;`altro`\n\n— any other voice (guests, audio from a played video, etc.); the internal clustering stays per-cluster, but the exposed label collapses into a single \"altro\" identity;`?`\n\n— utterance too short or not attributable.\n\nThe old `speaker_N`\n\nlabels no longer exist. The operator can **manually mark\nthe streamer** during a run with the TUI by pressing `s`\n\n(\"Mark\nstreamer\"): it pins the cluster of the last assigned utterance as streamer and\ndisables the automatic choice for that cluster (it also supports multiple streamers).\n\nThe speaker embedding model must be chosen **consistent with the language** of the\naudio. `speaker_embedding.dimension`\n\nmust **match the chosen model**:\n\n- recommended English CAM++ model\n`3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx`\n\n→`dimension: 512`\n\n.\n\nMinnarone does not download any model: point `speaker_embedding.model_path`\n\nto a\nlocal ONNX file. Start the recommended VoxCeleb model at\n`speaker_clustering.threshold: 0.5`\n\n; higher means more splitting, so tune it on\nrepresentative audio.\n\nTwitch chat-only example (based on\n[examples/twitch.example.yaml](/carlitose/minnarone/blob/main/examples/twitch.example.yaml)). For other scenarios\nsee the examples in [examples/](/carlitose/minnarone/blob/main/examples): `twitch-commentator.example.yaml`\n\n,\n`twitch-original-chat.example.yaml`\n\n, `teams-commentator.yaml`\n\n,\n`teams-meeting-assistant.yaml`\n\n, `teams-meeting-full.yaml`\n\n.\n\n```\nmode: public              # public | private (private = local console only)\nsoul_path: soul.md        # agent identity\nfacts_dir: facts          # persistent facts directory (one or more files)\nadapter: twitch           # perception source (twitch | os_capture)\nllm_provider: grok        # grok | deepseek (slug via llm_params.model) | llamacpp (local, model fixed by the server)\nagent_name: minnarone     # name the agent responds to (mention detection)\n\ntwitch:\n  channel: minnarone\n  quality: best\n  chat: true\n  audio: false            # true = local audio perception (requires audio extra + model_path)\n  video: false            # true = video frames (requires video/vlm extras + vlm.model)\n  audio_chunk_seconds: 1.0\n  video_fps: 1.0\n  # Gated public sending. Default: off. Read docs/twitch-operator.md before live.\n  send:\n    mode: off             # off | shadow | live (quote it: YAML reads on/off as bool)\n    allowed_channels: []\n    max_per_minute: 1\n    max_per_hour: 20\n    failure_threshold: 3\n\nllm_params:\n  model: x-ai/grok-4.5\n  reasoning:\n    effort: low             # low | medium | high\n\nsenser_interval: 0.5\nidle_interval: 150.0\nsummarizer_interval: 30.0   # Summarizer cadence (short-term memory)\nrecent_chat_window: 15\nperception_queue_size: 32   # perception work-queue cap (backpressure)\nperception_shutdown_timeout: 5.0\n\nvad:\n  mode: 2                   # 0 least aggressive, 3 most aggressive\n  frame_ms: 30              # 10 | 20 | 30\n  padding_ms: 300           # ring/hangover VAD\n  max_utterance_seconds: 30.0\n\nasr:\n  model: large-v3-turbo\n  device: auto\n  compute_type: default\n  language: null\n  beam_size: 5\n  condition_on_previous_text: false\n\nspeaker_embedding:\n  model_path: null          # recommended: 3dspeaker_speech_campplus_sv_en_voxceleb_16k.onnx\n  provider: cpu\n  num_threads: 1\n  dimension: 512            # English VoxCeleb CAM++; must match the model.\n\nspeaker_clustering:\n  threshold: 0.5            # VoxCeleb starting point; tune on representative audio.\n  warmup_seconds: 60.0\n  min_update_seconds: 1.0\n\nvideo:\n  sample_every: 1              # additional sampling before the captioner\n  dedup_change_threshold: 0.0  # 0 = skip only byte-identical frames\n\nvlm:\n  backend: qwen                # qwen (local torch) | llamacpp (multimodal llama-server)\n  model: null                  # local Qwen2-VL-compatible path/id (qwen backend only)\n  device: auto\n  device_map: auto\n  torch_dtype: auto\n  max_new_tokens: 48\n  timeout_seconds: 30.0\n  language: en                 # concise English captions by default\n\ncommentator:\n  language: it\n  # No profiles = commentator disabled. Add a profile to enable it, for example:\n  #   profiles:\n  #     original_chat:        # public-chat persona (only one allowed with twitch + public)\n  #       idle_interval: 30.0\n\n# --- v2 extension points (present but INERT in the MVP) ---\ndisclosure:\n  announce_ai: false    # the only active field: disclosure stance in the prompt\nretention:\n  perceptions_days: 7   # inert in the MVP\nauto_memory: false      # inert in the MVP\n```\n\nThe `retention`\n\nand `auto_memory`\n\nitems are present in the schema but do not alter\nbehavior (v2 extension). Local runs may contain `perceptions.jsonl`\n\n, summaries,\nand `debug/prompts`\n\n/ `debug/events.jsonl`\n\n; delete the relevant\n`.local/<agent>/runs/run-*`\n\ndirectory manually after an opt-out or revocation.\nDo not publish these artifacts.\n\nThe **tunable** prompt text (persona, per-style rules, response format, the\nsituation variants and the summarizer instruction) lives in external Markdown\nfiles, not in Python. They ship packaged with the wheel under\n`src/minnarone/prompts/`\n\nand are read at startup:\n\n| File | What it is |\n|---|---|\n`rules.md` |\nOriginal-chat persona/style rules. Uses `{{channel}}` . |\n`intro.md` |\n\"Current situation\" banner + channel line. Uses `{{channel}}` . |\n`situations.md` |\nThe 6 situation variants (keyed by `## <key>` ). Validated per section: `{{user}}` /`{{mention}}` only in `chat-mention` /`chat-continuation` , `{{reason}}` only in `generic` ; `#end_conv` required in `idle` , `chat-mention` , `chat-continuation` and `streamer-continuation` . Bodies cite section headers via `{{header_memoria}}` , `{{header_tuoi_ultimi_messaggi}}` , `{{header_conversazione_recente}}` (resolved from `headers.md` , allowed in every section). |\n`headers.md` |\nThe section headers and framing lines of the reaction prompt (keyed by `## <key>` : `regole` , `memoria` , `memoria_suffix` , `situazione` , `chat_recente` , …, plus `riassunto_std` /`conversazione_recente_std` /`situazione_std` for the non-original-chat styles, where the `## ` markdown prefix stays structural). `{{channel}}` required in `cosa_sai` only. In-body references resolve from the same values, so renaming a header here updates every text that cites it. |\n`format.md` |\nThe `RE:` /`MSG:` response contract. Must keep `RE:` , `MSG:` , `#end_conv` . |\n`operator.md` |\nLocal-commentator rules. Uses `{{language}}` . |\n`meeting_synthesizer.md` |\nMeeting-notes rules. Uses `{{language}}` . |\n`suggester.md` |\nPrivate-suggester rules. Uses `{{language}}` ; must keep the `#nothing` control token. |\n`summarizer.md` |\nShort-term-memory summarizer text (keyed sections). |\n\nSet `prompts_dir`\n\nin the config, the same spirit as `soul_path`\n\n/ `facts_dir`\n\n(the path is resolved relative to the config file):\n\n```\nprompts_dir: my-prompts   # a directory next to the config file\n```\n\nResolution is **per file**: for each prompt file, if it exists under\n`prompts_dir`\n\nit wins, otherwise the packaged default is used. You can override\njust one file and let the rest fall back. If `prompts_dir`\n\nis absent, only the\npackaged defaults are used, so a fresh install works with no configuration.\n\nThe loader is **fail-fast**: a missing file, a missing/unknown placeholder, a\nmissing control token or an empty required section aborts startup — a tunable\nprompt is never allowed to degrade to empty text. For keyed files\n(`situations.md`\n\n, `summarizer.md`\n\n) the constraints are enforced **per section**:\nremoving `#end_conv`\n\nfrom a single situation, or using a placeholder in a\nsection whose render path never supplies it, fails at startup with an error\nnaming the file and the section (not at runtime on the first unlucky trigger).\n\nTo validate an override without starting the app, run\n`minnarone validate-prompts --prompts-dir my-prompts`\n\n(or `--config config.yaml`\n\nto read `prompts_dir`\n\nfrom the config): exit 0 on success, one line per broken\nfile otherwise. The summary lists each file's origin (override vs default) and\nprints an explicit notice when the override is partial — per-file fallback is a\nfeature, but a half-translated set should be intentional, not silent.\n\nSubstitution uses double braces `{{name}}`\n\n. The whitelisted names are\n`{{channel}}`\n\n, `{{language}}`\n\n, `{{user}}`\n\n, `{{mention}}`\n\n, `{{reason}}`\n\nand the\nheader references `{{header_memoria}}`\n\n, `{{header_tuoi_ultimi_messaggi}}`\n\n,\n`{{header_conversazione_recente}}`\n\n(resolved from `headers.md`\n\n). Their\nvalues come from config/code (trusted data, never perceived content), single\nbraces `{ }`\n\nand `<...>`\n\nsurvive untouched, and an injected value is never\nre-scanned (no recursive template injection).\n\n`{{channel}}`\n\nfollows `twitch.channel`\n\nfrom your config file — do not hard-code\na channel name inside the prompt files.\n\nExternalizing the prompts **is** the localization mechanism — there is no i18n\nengine and the project does not ship translated sets. To run a channel in\nanother language: copy `src/minnarone/prompts/`\n\nto a new directory, rewrite the\n`.md`\n\nfiles in your language (keeping the placeholders and control tokens),\nand point `prompts_dir`\n\nat it. No code changes. A minimal, partial example lives\nin [examples/prompts-en/](/carlitose/minnarone/blob/main/examples/prompts-en).\n\nThe **anti-injection** and **disclosure** rules are hard-coded in `prompt.py`\n\nand\nare intentionally NOT among the editable files, together with the untrusted-data\nfence mechanics. Tunable style rules are rendered first and the configured\ndisclosure stance is appended afterward, so an override cannot become the last,\ncontradictory instruction. With `announce_ai: false`\n\nMinnarone does not announce\nitself proactively but must not lie when asked; `true`\n\npermits an explicit answer.\n\nWith `llm_provider: llamacpp`\n\nthe Reactor generates the reactions against a\nlocal `llama-server`\n\n([llama.cpp](https://github.com/ggml-org/llama.cpp)) with\nan OpenAI-compatible API: **no OPENROUTER_API_KEY**, no new runtime\ndependency. The server must be started\n\n**by hand** before the live loop (minnarone does not manage the process, it only does a health-check on\n\n`GET /health`\n\nat startup):\n\n```\nllama-server -m gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf --port 8080 -ngl 99 -c 8192 --reasoning off --parallel 1\n```\n\nConfig (full example in\n[examples/llamacpp-local.example.yaml](/carlitose/minnarone/blob/main/examples/llamacpp-local.example.yaml)):\n\n```\nllm_provider: llamacpp\nllamacpp:\n  base_url: http://127.0.0.1:8080   # default; explicit port required\n```\n\nNotes:\n\n- No\n`model`\n\nin config nor in the body: the server serves the single loaded model (the real slug appears in the observability meta of the response). - The\n`llm_params`\n\n(`temperature`\n\n,`max_tokens`\n\n,`timeout`\n\n, ...) pass as for the cloud providers;`thinking`\n\nis dropped (reasoning is turned off server-side with`--reasoning off`\n\n). `--check`\n\nstays a dry run without network: it validates only the shape of`base_url`\n\n. If at live startup the server is down or is still loading the model (503), the CLI exits with an error that includes the command above.\n\nThe video channel can describe the frames using a **multimodal** `llama-server`\n\n(model + `--mmproj`\n\nprojector, e.g. multimodal Gemma) instead of the\ntorch Qwen2-VL backend. Decisive advantage on small GPUs (~4 GB): **a single\nmultimodal llama-server instance serves both the text reactions (llm_provider: llamacpp) and the captioning**, avoiding the double VRAM residency of\ntorch-VLM + LLM. No new runtime dependency (transformers/torch are not\nneeded with this backend): the transport is the same urllib as the local LLM\nprovider.\n\nStart the multimodal instance by hand, adding the `--mmproj`\n\nprojector and\n`--parallel 2`\n\n(so that text and vision run concurrently on the same\ninstance, cost ~10 MiB VRAM):\n\n```\nllama-server -m <model.gguf> --mmproj <mmproj.gguf> --port 8080 -ngl 99 -c 16384 --reasoning off --parallel 2\n```\n\nContext and:`--parallel`\n\n`llama-server`\n\nsplits`-c`\n\nacross the slots, so the per-request context is`n_ctx / n_slots`\n\n. With`--parallel 2`\n\nyou need`-c 16384`\n\nto have 8192 tokens per slot: a multi-channel prompt (chat + audio + video + soul/facts) easily exceeds the 2048 that`-c 4096 --parallel 2`\n\nwould give, and llama-server would respond`400 \"exceeds the available context size\"`\n\n. The KV cache of E2B is small: quadrupling the context costs ~+80 MiB VRAM.\n\nConfig: the backend reuses `llamacpp.base_url`\n\n(same instance as the LLM provider),\nwhile `prompt`\n\n/`language`\n\n/`max_new_tokens`\n\n/downscale/`max_caption_chars`\n\nstay in the `vlm:`\n\nblock:\n\n```\nvlm:\n  backend: llamacpp     # caption frames through the multimodal llama-server instance\nllamacpp:\n  base_url: http://127.0.0.1:8080   # shared with the local LLM provider\n```\n\nNotes:\n\n- At live loop startup (never in\n`--check`\n\n) the CLI verifies via`GET /props`\n\nthat the instance exposes vision (`modalities.vision == true`\n\n). If the projector is missing, it exits with an actionable error that reminds you of`--mmproj`\n\n. The check also runs with a cloud`llm_provider`\n\n(the captioner uses`llamacpp.base_url`\n\nanyway). - Best-effort contract: on a transport/HTTP error at runtime the captioner\nreturns an empty caption (skips the frame) and logs the event, without killing\nthe video channel. The\n`qwen`\n\nbackend stays unchanged for whoever selects it. **Lightweight install**: this backend requires only the`vlm-llamacpp`\n\nextra (`pip install -e '.[vlm-llamacpp]'`\n\n→ only Pillow), not the heavy`vlm`\n\nextra (torch/transformers), which is only needed by the`qwen`\n\nbackend.\n\nThe Twitch smoke is separate from the agent CLI and does not require\n`OPENROUTER_API_KEY`\n\n. The full guide for operators, artifacts, troubleshooting\nand the chat-only `adapter: twitch`\n\nruntime with console output is in\n[docs/twitch-operator.md](/carlitose/minnarone/blob/main/docs/twitch-operator.md).\nFor chat you need the bot credentials in the environment (via `.env`\n\nor exported):\n`TWITCH_BOT_USERNAME`\n\nand `TWITCH_OAUTH_TOKEN`\n\n.\n\n```\nminnarone-twitch-smoke \\\n  --channel nomecanale \\\n  --duration 30 \\\n  --output ./.smoke/twitch-chat\n```\n\nTo also enable raw audio capture you need `streamlink`\n\nand `ffmpeg`\n\ninstalled on the system and available on `PATH`\n\n:\n\n```\nstreamlink --version\nffmpeg -version\n\nminnarone-twitch-smoke \\\n  --channel nomecanale \\\n  --duration 30 \\\n  --output ./.smoke/twitch-audio \\\n  --audio \\\n  --audio-chunk-seconds 1.0 \\\n  --quality audio_only\n```\n\nTo validate only the VAD segmentation on the Twitch audio, without ASR:\n\n```\nminnarone-twitch-smoke \\\n  --channel nomecanale \\\n  --duration 30 \\\n  --output ./.smoke/twitch-vad \\\n  --no-chat \\\n  --vad-diagnostic \\\n  --quality audio_only\n```\n\nTo also sample low-frequency JPEG video frames:\n\n```\nminnarone-twitch-smoke \\\n  --channel nomecanale \\\n  --duration 30 \\\n  --output ./.smoke/twitch-video \\\n  --video \\\n  --video-fps 1.0 \\\n  --quality best\n```\n\nThe artifacts are written to the directory passed to `--output`\n\n:\n`perceptions.jsonl`\n\nfor the chat, `raw/audio/*.pcm`\n\nfor a limited number of\nPCM mono 16 kHz signed 16-bit little-endian samples, `raw/video/*.jpg`\n\nfor a\nlimited number of JPEG frames, and `stats.json`\n\nwith counts and any failures.\nWith `--vad-diagnostic`\n\n, `stats.json`\n\nalso includes `vad_utterances`\n\nand\n`vad_utterance_durations_ms`\n\n. The `.pcm`\n\nand `.jpg`\n\nfiles prove only the raw capture\nfrom FFmpeg: the capture-only smoke does not run ASR, diarization or VLM captioning.\nThe operator guide also includes manual smokes to transcribe a `.pcm`\n\nwith\n`faster-whisper`\n\n, extract speaker embeddings with `sherpa-onnx`\n\n, and start the\nconsole runtime with `twitch.audio: true`\n\n. A dedicated chat-only smoke is\nalso available: `minnarone-twitch-chat-smoke`\n\n.\n\nThe core runtime is **implemented**: Twitch perception (chat/audio/video) with\ngated public send (shadow/live), local commentator and meeting assistant on\nTeams (`operator`\n\n, `meeting_synthesizer`\n\n, `suggester`\n\nprofiles), speaker\ndiarization (`streamer`\n\n/`altro`\n\n/`?`\n\n+ manual marking), observability TUI\ndashboard and offline replay of the runs.\n\nThe remaining work is centered on the **live acceptance runs with\nhuman-in-the-loop** (HITL). See the\n[roadmap](/carlitose/minnarone/blob/main/docs/SPECIFICATION.md#10-roadmap-per-priorit%C3%A0) for MVP / v2 / v3.\n\nIf this project is useful to you, you can buy me a coffee ☕\n\nDistributed under the [MIT](/carlitose/minnarone/blob/main/LICENSE) license.", "url": "https://wpnews.pro/news/show-hn-minnarone-multimodal-agents-that-watch-listen-and-react-live", "canonical_source": "https://github.com/carlitose/minnarone", "published_at": "2026-07-26 15:10:49+00:00", "updated_at": "2026-07-26 15:22:18.972024+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents"], "entities": ["Minnarone", "Carlos A. Rueda", "Twitch", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/show-hn-minnarone-multimodal-agents-that-watch-listen-and-react-live", "markdown": "https://wpnews.pro/news/show-hn-minnarone-multimodal-agents-that-watch-listen-and-react-live.md", "text": "https://wpnews.pro/news/show-hn-minnarone-multimodal-agents-that-watch-listen-and-react-live.txt", "jsonld": "https://wpnews.pro/news/show-hn-minnarone-multimodal-agents-that-watch-listen-and-react-live.jsonld"}}