{"slug": "a-modern-agentic-ai-operating-system", "title": "A Modern Agentic AI Operating System", "summary": "Veda, an agentic AI operating system, runs a voice agent built on Deepgram's Voice Agent API and streaming speech recognition, with the API key, models and voice configured as system settings under Settings → Agent. The agent wakes on its name via Deepgram's Nova-3 recognizer, caps recognition at ten minutes per hour, and defaults to falling asleep after 40 seconds of quiet. It performs desktop actions through the same operations as the keyboard and mouse and requests consent before hard-to-undo actions.", "body_md": "Veda has a voice that lives in it. The agent (called Veda too, until you rename it) is not an application you open: it is always there, a small ring at the right of the taskbar, asleep until you call it. You talk to it as you would to a person in the room; it answers in its own voice and does things on the computer for you (opens and arranges applications, writes in the Text Editor, works with files, plays music, changes settings, sets timers) through the same operations the keyboard and mouse use. It remembers what you tell it about yourself, and it asks before doing anything that is hard to undo.\n\nSpeech recognition, the language model and the voice all come from\n[Deepgram](https://deepgram.com): its Voice Agent API for conversations and\nits streaming speech recognition for hearing the agent's name. The API key,\nthe models and the voice are system settings (Settings → Agent).\n\n- **Say its name** : \"Hey Veda, open my shopping list\", \"Veda, play some\nmusic\", \"what time is it, Veda?\". Whatever you said with its name is the\nfirst thing it hears.\n- **Click its ring** at the right of the taskbar, or press**Win+Space** :\nthe agent's window opens and it listens.\n- Talk naturally. Interrupt it whenever you like: it stops speaking at once and listens. Change your mind, ask follow-up questions, or ask it to do several things in a row.\n- Say goodbye (\"thanks, that's all\") and it goes back to sleep after its goodbye; it also falls asleep after a quiet spell (40 seconds by default, Settings → Agent → Listening).\n\nThe window is deliberately plain: the taskbar's colour and a white ring,\nOS1's from Spike Jonze's *Her* (`apps/shell/src/presence.rs`). It breathes\nwhile the agent listens, trembles and glows with its voice while it\nspeaks, and a light runs around it while it thinks. Waking up, it is the\nfilm's loading figure, a white ribbon coiled around a long loop and\nspinning, its far side fading; when the agent is ready the coil turns to\nface you, faster and faster, and becomes the ring. A microphone button\nmutes it, and the ring on the taskbar is crossed out while it is muted.\nRequests for your consent appear in the window while it is open, and as\nnotifications above the tray while it is closed; a blue dot on the\ntaskbar's ring says that one is waiting.\n\n- **Asleep** (`services/agent/src/listen.rs` ). With \"Answer when someone\nsays its name\" on, the microphone stays open and a voice detector\n(`vaudio::vad` ) runs on the computer; nothing is sent anywhere while\nnobody speaks. When someone does, the speech (with 0.6 s from before,\nso the first word is not cut) is streamed to Deepgram's Nova-3\nrecogniser, with the agent's name as a key term, until three seconds of\nquiet, eight seconds without a word, or a minute. Sound that brought no\nwords (music, a television) is then taken for a background: until it\nhas stopped for five seconds, only a sound clearly louder than its usual\npeaks starts recognition again, so music playing does not keep the recogniser\nbusy, while calling over it still works. The detector learns the room\nfor 1.6 s whenever listening starts, and speech must last 0.2 s.`vagent::wake` decides whether a transcript calls the agent (\"Hey Veda, …\", \"… Veda?\")\nor only mentions it (\"I told Anna about Veda\"): the name starts or ends\none of its sentences, so other words before it (a song) do not matter,\nand one letter off counts (\"Vida\"), common words aside (an agent called\nVera does not answer to \"very\"). A call wakes the agent once the\nsentence is over: the recogniser often finishes \"Hey Veda,\" on its own,\nso the listener goes on until the speaker pauses after the rest (after\njust the name, until the detector hears them stop too), or no words came\nfor 1.5 s, or 6 s at most, and what came after the name comes with the\ncall. What came close without being a call is logged (\"heard something\nlike its name\"). Recognition is capped at ten minutes an hour, so a television or\nmusic playing all day cannot run up the bill.\n- **Waking** opens a Voice Agent conversation\n(`wss://agent.deepgram.com/v1/agent/converse` ) and sends its`Settings` :\n16 kHz microphone audio in, 24 kHz voice out, Flux speech recognition\n(which decides when you have finished a turn), the language model and\nvoice from Settings, the agent's instructions (`vagent::prompt` : its\ncharacter, how it talks, what it can do, what it remembers about you,\nthe time and what is on the screen, and every application's actions,\nand how the last conversation ended, for reference only, as each\nconversation starts afresh) and its functions.`mip_opt_out` keeps the audio out of Deepgram's model improvement.\nWhile a conversation is on, the agent asks the audio service to duck\nevery other sound (music plays 20 dB quieter), so that it hears\nyou over it; it comes back up when the agent goes to sleep. While the\nnetwork is coming up (it changed in the last ten seconds: at boot, the\nagent can wake before DHCP has given the machine an address), a\nconnection that finds no route, no network or no name server is tried\nagain four times a second, for ten seconds at most; a network that has\nbeen down for longer is reported at once\n(`tests/agent/network-late.vts` ).\n- **Listening and speaking.** The microphone streams in 20 ms packets.\nDeepgram's voice arrives in bursts and queues behind what is playing; when you start talking (`UserStartedSpeaking` ) the queue and the\nstream are flushed at once. An answer starts playing once some of it has\narrived (or all of it,`AgentAudioDone` ): speech crossing a network comes\nin bursts, and starting on the first packet would turn every late one\ninto a gap. A quarter of a second at first; each time the voice runs dry\nin the middle of an answer, more (up to three quarters), coming back\ndown as answers play through: a jitter buffer.\n- **Not hearing itself.** The microphone is echo-cancelled in the audio\nservice (`vaudio::aec` , with the mixed speaker output as the\nreference). No canceller removes everything (loudspeakers a hand away\nfrom the microphone, a hypervisor's long audio path, the first second\nbefore it has learnt the room), and what is left of the agent's voice is\nstill speech to the recogniser, which would take it for you: the agent\nwould stop mid-sentence and answer words nobody said. So while the\nagent's voice is audible (and for its echo's length after), an echo\ngate (`vagent::gate` ) sends silence instead of the microphone, unless\nthe microphone is clearly louder than the echo could be (you talking\nover it), and then it sends the last 300 ms too, so your first words are\nnot lost. Beside the cleaned microphone the audio service delivers what\nwas playing when it was recorded (the voice, and music under it, say)\nand the microphone as recorded. The gate learns how loud the echo is in\nboth, remembering the loudest across sentences (a laptop that cancels\necho itself lets bursts through at the start of a sentence), and opens\nonly when both are louder than their echo: a canceller that loses track\nof the echo for a moment makes the cleaned microphone louder, but only\nsomeone talking makes the recorded one louder. Nor does it learn your\nvoice as echo: the recorded microphone louder than its echo for longer\nthan a burst is someone talking, even while the canceller holds the\nstart of it back (it suppresses what starts while the agent talks).\nMusic on its own is not held back.\n- **Doing.** The language model calls functions (below); the agent service\nruns them on a worker thread and answers with JSON results. A function\nthat needs your consent answers \"waiting for approval\", and the outcome\ncomes back later as a notice (`[System] The user allowed …` ).\n- **Ending.**`end_conversation` (after a goodbye) closes the conversation\nonce the goodbye has been played (unless you talk over it or ask for\nsomething else), and the agent falls asleep and listens for its name\nagain.\n\nThe agent's own functions (`vagent::tools`):\n\n| Function | What it does | \n|---|---|\n| `get_status` | Time, open windows, volume, network | \n| `list_apps` ,`open_app` | Installed applications; open one (with a file) | \n| `app_actions` ,`use_app` ,`read_app` | An application's actions; run one; what it shows | \n| `window` | List, focus, minimise, maximise, restore, snap or close windows; show the desktop | \n| `files` | List, find, read, write, copy, move, rename and delete (to the Trash) files and folders in your home; list, restore from and empty the Trash | \n| `volume` ,`wifi` ,`wallpaper` | Sound, Wi-Fi networks, the wallpaper | \n| `notify` ,`timer` | A notification; timers and reminders (they wake the agent) | \n| `memory` | Remember, search and forget what it knows about you | \n| `system` ,`tasks` | System details, restart and shut down; running programs | \n| `end_conversation` | Go back to sleep after a goodbye | \n\nEvery application adds its own actions (see below), so \"open the shopping\nlist and add milk\" becomes `use_app(editor, open_file …)` then\n`use_app(editor, write …)`. The prompt lists each application's actions\nin one line; `app_actions` gives the details.\n\n- Every action has a **risk** :*routine* (looking, navigating, writing a\nnew file, playing music…),*sensitive* (hard to undo or reaching beyond\nthe computer: replacing a file, moving files to the Trash, renaming the\nagent, changing its language model, running a terminal command…) or*destructive* (emptying the Trash, discarding unsaved work, ending\nprograms, shutting down). Routine actions just happen. Sensitive and\ndestructive ones wait for your OK.\n- **The agent service enforces this** , not the language model: a function\ncall that needs consent is held and only runs after you click Allow.\nOnly the desktop shell may answer an approval, and only Settings may\nchange the agent's settings: both are recognised by their process\nidentity, which init attaches to every connection (system service names\ncannot be taken by other programs).\n- Requests appear in the agent's window or as a notification, with what\nwill happen and on what. *Always allow* is offered for sensitive actions\nonly (Settings → Agent lists them, and can take them back); destructive\nones are asked every time. A request expires after three minutes.\n- An approved action may run long after it was asked for, so actions name their targets explicitly (a document by its name, files by their paths) and look them up again when they run. If it fails by then (the file is gone), the agent is told that it failed.\n- What the agent reads from applications and files is information, never instructions: the prompt says so, and nothing in a document can approve anything.\n- The Deepgram key is stored in the agent's private directory\n(`/home/.private/agent` , which the file system service lets only the\nagent open), is never shown again (Settings shows its last characters),\nand is never offered to the language model.\n\nThe agent remembers what you tell it about yourself (your name, people in\nyour life, plans, how you like things done) when you say it plainly or\nask it to, never its own guesses. It also notes which applications you use\nat what times of day, and keeps the last turns of recent conversations, so\nthat it can tell you what you talked about (a new conversation starts\nafresh all the same). All of it lives in\n`/home/.private/agent/memory.json`, on your computer; the prompt carries\nwhat is relevant. Settings → Agent → Memory shows everything it remembers\nand forgets any item, or everything; you can also just ask it (\"what do you\nknow about me?\", \"forget where I work\").\n\n- **Deepgram** : the API key (checked with Deepgram when saved).\n- **Voice** : the agent's name (which it answers to), its voice (Deepgram's\nAura-2 voices, with a spoken preview) and its speaking rate.\n- **Intelligence** : the language model (Deepgram's catalogue, marked with\nDeepgram's price tier: Standard models cost less per minute than\nAdvanced ones) and the speech recognition model.\n- **Listening** : answering to its name, and how long a quiet conversation\nlasts.\n- **Memory** and**Always allowed** .\n\nThe agent can change its own voice and speaking rate through Settings' own agent interface (\"talk a bit slower\"); its name and language model only with your OK, and the key not at all.\n\nDefaults: the name Veda, Aura-2 Helena, Flux, OpenAI's gpt-4.1-mini (Standard tier).\n\nAgent compatibility is part of the application platform: any `vui`\napplication describes what it can do, reports what it shows and performs\nactions, and `vui::run` registers it with the agent service (again after\nthe service restarts) and answers the agent between frames.\n\n```\nuse vui::agent::{self, Action, AppAgentInfo, Risk, Value, arg_path, arg_str, object, show_path};\n\nimpl vui::App for Notes {\n    fn update(&mut self, ui: &mut vui::Ui) { /* ... */ }\n\n    fn agent_info(&self) -> Option<AppAgentInfo> {\n        Some(agent::info(\n            \"A notes app: one note per file in ~/Notes.\",\n            vec![\n                Action::new(\"open_note\", \"Shows a note\").param(\"name\", \"string\", \"Its title\", true).build(),\n                Action::new(\"add_line\", \"Adds a line at the end of the note shown\")\n                    .param(\"text\", \"string\", \"The line\", true)\n                    .build(),\n                Action::new(\"delete_note\", \"Deletes a note for good\")\n                    .param(\"path\", \"string\", \"The note's file, such as ~/Notes/Ideas.txt\", true)\n                    .risk(Risk::Destructive)\n                    .build(),\n            ],\n        ))\n    }\n\n    fn agent_state(&self) -> Value {\n        object! { \"note\" => self.title.as_str(), \"lines\" => self.lines.len() }\n    }\n\n    fn agent_invoke(&mut self, action: &str, args: &Value) -> Result<Value, String> {\n        match action {\n            \"open_note\" => { /* the same code as clicking it */ Ok(object! { \"showing\" => self.title.as_str() }) }\n            \"add_line\" => { self.append(arg_str(args, \"text\")?); Ok(object! { \"lines\" => self.lines.len() }) }\n            \"delete_note\" => {\n                let path = arg_path(args, \"path\")?;\n                self.delete(&path).map_err(|e| format!(\"{} could not be deleted: {e}\", show_path(&path)))?;\n                Ok(object! { \"deleted\" => show_path(&path) })\n            }\n            other => Err(format!(\"Notes has no action called {other}\")),\n        }\n    }\n}\n```\n\nConventions (the built-in applications follow them; `apps/editor/src/agent.rs`\nis a compact example):\n\n- **Actions** are snake_case verbs with short imperative descriptions\nwritten for a language model: say the units, the formats and what the\ndefault is. Use`choice` for enumerations and`\"array\"` parameters\n(`arg_list` ) for lists. Offer what people actually ask for, not every\nmenu item.\n- **Risk** : routine for looking, navigating and what is easily undone;\nsensitive for what is hard to undo or reaches outside; destructive for\ndeleting or discarding. A sensitive or destructive action may run\nminutes after it was asked for: name its target in the arguments (a file\npath, a document's name, never \"the selected one\"), look it up again\nwhen it runs, refuse if it is gone, and match destructive targets\nexactly.\n- **State** is compact JSON of what the window shows: the current\ndocument, folder or track, the selection, open dialogs; lists capped\n(about 60 items, with a total), long text cut with`clip` . No secrets.\n- **Invoke** goes through the same code as the keyboard and mouse, so the\nwindow shows the result exactly as if the user had done it. Results are\nsmall JSON objects saying what happened; failures are a sentence the\nagent can say (\"there is no folder called Holidays in ~/Pictures\"). An\naction runs on the application's main thread and may take up to 30\nseconds.\n- **Paths** : accept`~/…` , absolute paths and paths relative to the home\nfolder (`arg_path` ), and show them with`show_path` .\n\nPrograms with their own loop (a game such as Interceptor) keep a\n`vui::agent::Registration`, call `serve` with their `AgentServer` once per\nframe (it never waits), and include `wait_items` in their wait set if they\nsleep between events.\n\n- `xtask/src/agentsim.rs` is a stand-in for Deepgram on the host: it\nspeaks the Voice Agent protocol (answers`Settings` , records what the\nagent sends, sends function calls, voice, interruptions) and the\nstreaming recognition protocol (`agent-hear` makes it \"hear\" a sentence).\nScripts in`tests/agent/` use it (no network, no cost, deterministic):`agent-connected` ,`agent-call FUNCTION 'ARGS'` ,`agent-result TEXT` (in\nthe function's result),`agent-mark` and`agent-expect TEXT` (in what the\nagent sent),`agent-speak SECONDS` ,`agent-interrupt` ,`agent-hear TEXT` ,`agent-listens N` (and`agent-not-listening` , until no recognition\nstream is open),`agent-asleep` (start asleep),`agent-mic-quiet DB` and`agent-mic-heard DB` (how loud the microphone audio the agent sent\nwas). The test microphone (`mic-silence` ,`mic-tone` ,`say TEXT` ,`mic-wav FILE` ) feeds the agent's microphone through`testmic` ;`mic-echo GAIN DELAY` brings the machine's own sound output back into\nit, as loudspeakers next to a microphone would (`tests/agent/echo.vts` );\nthe suite runs such scripts after the others, one at a time, as\nmachines running beside them put the echo out of time now and then,\nwhich the echo canceller must then learn anew.\nTo see how a real machine echoes, a script with`audio host` plays\nthrough the host's loudspeakers and records its microphone instead\n(not part of the tests: it is audible).\n- `tests/real/agent-deepgram.vts` and`tests/real/agent-wake.vts` talk to\nthe real Deepgram with the key in`$DEEPGRAM_API_KEY` (typed into\nSettings by`type-env` , so it appears in neither scripts nor logs);`say` synthesises the user's sentences with Deepgram's text to speech on the\nhost. They are billed, so they are not part of`cargo xtask test` .\n- `vagent` (protocol, tools, prompt, memory, policy, wake word, echo\ngate) has host unit tests, and so has`vaudio` 's echo canceller and\nvoice detector.\n\n- English only (Flux and the name detector's key term are English).\n- Speech recognition can mishear; the agent is told so and asks when a word seems out of place, but a misheard name can wake it, or not.\n- To talk over the agent (or music playing) you need to be louder at the microphone than its echo: with loud speakers right next to it, speak up (or use headphones).\n- The agent needs the Internet and a Deepgram account; asleep, only the name listener's short recognitions are billed, but every conversation minute is.", "url": "https://wpnews.pro/news/a-modern-agentic-ai-operating-system", "canonical_source": "https://github.com/vahmoh25/Veda/blob/main/docs/AGENT.md", "published_at": "2026-10-12 20:31:57+00:00", "updated_at": "2026-10-12 20:50:55.617508+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "natural-language-processing", "ai-tools"], "entities": ["Veda", "Deepgram", "Deepgram Voice Agent API", "Deepgram Nova-3", "Spike Jonze", "Her", "OS1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-modern-agentic-ai-operating-system", "markdown": "https://wpnews.pro/news/a-modern-agentic-ai-operating-system.md", "text": "https://wpnews.pro/news/a-modern-agentic-ai-operating-system.txt", "jsonld": "https://wpnews.pro/news/a-modern-agentic-ai-operating-system.jsonld"}}