{"slug": "an-ios-app-that-runs-ai-agents-and-a-complete-voice-pipeline-on-the-device", "title": "An iOS app that runs AI agents and a complete voice pipeline on the device", "summary": "Rob Sandhu has built an iOS app that runs AI agents and a complete voice pipeline entirely on-device, with no account, model API, or cloud inference, using Apple Intelligence's on-device model for reasoning and Sherpa-onnx for speech. The app features background agents that plan searches, read pages, and cite sources, plus streaming speech-to-text and zero-shot voice cloning from a ten-second sample. A future voice-call app would transmit text (~50 bytes/sec) instead of audio (6–24 kbps Opus), re-synthesizing the sender's cloned voice on the receiver's device.", "body_md": "An iOS app that runs **AI agents and a complete voice pipeline entirely on the\ndevice**. No account, no model API, no inference in the cloud. The one thing\nthat leaves the phone is web research — the agent searches and downloads pages\nso the on-device model has something current to reason over, and that switch\ncan be turned off.\n\nTwo halves that reinforce each other:\n\n**Background agents.** Describe a task (\"find the best summer camps for my 9-year-old near Chicago\"), and an agent works on it in the background: it plans search queries, reads the pages it finds, then reasons over them with Apple Intelligence's on-device model and cites what it used. Come back later and press play — the result is read aloud.**On-device speech.** Streaming speech-to-text (used both for dictating tasks and for live transcription) and zero-shot voice-cloning text-to-speech, which is what gives agent results a voice — optionally*your*voice, cloned from a ten-second sample.\n\nThe speech half also exists as the local building block for a longer-term\nidea: a voice-call app where audio is transcribed on the sender's device, sent\nover the network as **text** (~50 bytes/sec instead of 6–24 kbps of Opus), and\nre-synthesized on the receiver's device in the sender's cloned voice. The\nnetworking layer isn't built yet; the *Live Transcription* screen's Echo\ntoggle is a local loopback of that pipeline.\n\nFive layers. The UI never touches the native runtime directly, every model call happens off the main thread, and the model never reaches the network itself — the web layer does the fetching and hands it text.\n\n```\nflowchart TB\n    subgraph UI[\"UI — SwiftUI\"]\n        A[\"AgentsView<br/>list · composer · detail\"]\n        S[\"SettingsView<br/>My Voice · Transcribe · Speak\"]\n        D[\"DesignSystem<br/>shared components\"]\n    end\n\n    subgraph Domain[\"Agents — domain\"]\n        R[\"AgentRunner<br/>BGProcessingTask + foreground\"]\n        B[\"AgentBrain<br/>protocol\"]\n        ST[(\"AgentStore<br/>ModelActor\")]\n        J[\"AgentJob<br/>SwiftData Model\"]\n    end\n\n    subgraph Web[\"Web research\"]\n        WR[\"WebResearcher<br/>search · read · excerpt\"]\n        SP[\"WebSearchProvider<br/>DuckDuckGo · Brave\"]\n        PR[\"PageReader<br/>+ HTMLText\"]\n    end\n\n    subgraph Speech[\"Speech — engines\"]\n        STT[\"SttEngine<br/>serial queue\"]\n        TTS[\"TtsEngine<br/>serial queue\"]\n        CAP[\"AudioCapture\"]\n        PLAY[\"AudioPlayer\"]\n        VP[\"VoiceProfileStore\"]\n    end\n\n    subgraph Native[\"Native bridge\"]\n        W[\"SherpaOnnx.swift<br/>+ Zipvoice extension\"]\n        C[\"c-api.h via bridging header\"]\n        X[\"sherpa-onnx.xcframework<br/>onnxruntime.xcframework\"]\n    end\n\n    A --> R\n    A --> TTS\n    A --> STT\n    S --> STT\n    S --> TTS\n    S --> VP\n    S --> WR\n    R --> B\n    R --> ST\n    R --> WR\n    B --> WR\n    WR --> SP\n    WR --> PR\n    ST --- J\n    B -.->|iOS 26+| FM[\"FoundationModels\"]\n    B -.->|fallback| MOCK[\"MockAgentBrain\"]\n    STT --> CAP\n    TTS --> PLAY\n    TTS --> VP\n    STT --> W\n    TTS --> W\n    W --> C --> X\n```\n\nAn agent is a row in SwiftData with a status (`queued → running → completed/failed`\n\n), a progress log appended as it works, and a result split\ninto a spoken-style `resultSummary`\n\nand a longer `resultDetail`\n\n.\n\n`AgentRunner`\n\nis a singleton configured at launch (BGTaskScheduler *requires*\nregistration before launch finishes). It drains the queue from two entry\npoints:\n\n| Path | Trigger | Notes |\n|---|---|---|\n| Foreground | App becomes active, or you spawn an agent | The interactive path, and the only one that works in the simulator |\n| Background | `BGProcessingTask` id `com.robsandhu.Agent.agentwork` |\niOS schedules it at its discretion (typically idle/charging); the handler re-chains the next slot before starting work |\n\nInterruption is handled explicitly: the task's `expirationHandler`\n\ncancels the\nwork, `drainQueue`\n\ncatches `CancellationError`\n\nand requeues the in-flight job,\nand any job left stranded in `running`\n\nby a process death is requeued at next\nlaunch (`requeueOrphanedRunningJobs`\n\n).\n\nThe model behind an agent is swappable via the `AgentBrain`\n\nprotocol, so the\npersistence, scheduling, and playback machinery is independent of which model\nruns:\n\n— Apple Intelligence's on-device model, compiled in behind`FoundationModelsBrain`\n\n`#if canImport(FoundationModels)`\n\nand offered only when`SystemLanguageModel.default.availability`\n\nreports available. Runs three passes: plan the search queries, write the findings from what was read, then a three-sentence summary written to be read aloud.— the fallback where Apple Intelligence isn't available. With web research on it still searches and reads for real and reports a digest of what it found; with research off it simulates staged research, so the whole pipeline is testable on any device.`MockAgentBrain`\n\nAgents search *before* they think. The model never touches the network itself\n— the app runs the searches, decides which pages to download, and hands back\nexcerpts — so what the model can see is bounded by policy rather than by its\nown tool calls.\n\n```\nplan queries → search → interleave + dedupe → read N pages → excerpt → ground\n```\n\n| Step | Where | Notes |\n|---|---|---|\n| Plan queries | `FoundationModelsBrain.planQueries` |\nGuided generation (`@Generable` ) forces a list of query strings. Left to free text, a small model answers the task instead of writing queries for it — `WebResearcher.normalize` scrubs the leftovers of that habit (markdown, dash clauses, over-long lines) |\n| Search | `WebSearchProvider` |\n`DuckDuckGoSearch` (keyless) or `BraveSearch` (API key in the keychain) |\n| Merge | `WebResearcher` |\nRound-robins across queries so one query can't monopolize the read budget, and dedupes by canonical host+path |\n| Read | `PageReader` + `HTMLText` |\nCapped at 1.2 MB per page, `<article>` /`<main>` preferred, tags and entities reduced to plain text. A page that won't load degrades to its search snippet rather than failing the job |\n| Ground | `FoundationModelsBrain.findings` |\nNumbered excerpts in the prompt, citations required inline. On a context-window overflow it retries at 1100 → 600 → 300 chars per source, then falls back to unresearched general knowledge |\n\nSources are persisted on the job (`sourcesJSON`\n\n) and listed as tappable links\nunder the findings, so every claim can be traced back to the page it came\nfrom. Settings › Web Research holds the master switch, the provider choice,\nthe depth knobs, and a live test button.\n\n`AudioCapture`\n\ntaps `AVAudioEngine`\n\nand converts the hardware format\n(44.1/48 kHz) to the 16 kHz mono Float32 the models expect. Its output feeds\neither the recognizer or the enrollment recorder — never both, since there's\none capture session.\n\n`SttEngine`\n\nwraps a streaming Zipformer transducer with endpoint detection: it\npublishes a live `partial`\n\nand appends a finalized `TranscriptSegment`\n\non each\ndetected pause. It has a **dictation mode** — when `dictationOnPartial`\n\n/\n`dictationOnUtterance`\n\nare set, recognized speech routes to those callbacks\ninstead of the main transcript, which is how the composer pill takes spoken\ninput without polluting the transcription screen.\n\n`TtsEngine`\n\nruns ZipVoice zero-shot synthesis on a serial queue. Voice cloning\nneeds no training: a `VoiceProfile`\n\nis just a reference wav plus its\ntranscript, and synthesis conditions on that pair at call time. Prompt audio\nis cached per file path; enrollment writes a new filename each time so the\ncache can never go stale.\n\n| Component | Executor |\n|---|---|\nViews, `@Published` state |\nMain |\n`AudioCapture` callbacks |\n`AVAudioEngine` render thread |\n`SttEngine` decode loop |\nPrivate serial `DispatchQueue` |\n`TtsEngine` synthesis |\nPrivate serial `DispatchQueue` |\n`AgentStore` (all SwiftData writes) |\n`@ModelActor` |\n`AgentRunner.drainQueue` |\nSwift `Task` , awaits the store actor |\n\nSwiftData contexts aren't thread-safe, so every mutation from a background\ntask goes through `AgentStore`\n\n; the UI reads through `@Query`\n\non its own\nmain-thread context.\n\n**There are no Swift Package Manager or CocoaPods dependencies.** The one\nthird-party runtime is vendored as prebuilt `.xcframework`\n\nbinaries, and the\nproject file is generated rather than committed.\n\n| Package | Version | License | Role |\n|---|---|---|---|\n|\n\n`csukuangfj/sherpa-onnx-libs`\n\n[ONNX Runtime](https://github.com/microsoft/onnxruntime)Pinned to 1.12.21 because that's the newest version with **prebuilt iOS\nframeworks published** — 1.13.x has no iOS build. The Swift wrapper, C\nheaders, and binaries must all come from the same release.\n\n| Model | Size | License / training data | Role |\n|---|---|---|---|\n|\n\n[ZipVoice-Distill int8, zh-en](https://github.com/k2-fsa/ZipVoice)(`sherpa-onnx-zipvoice-distill-int8-zh-en-emilia`\n\n)[vocos 24 kHz vocoder](https://github.com/k2-fsa/sherpa-onnx/releases/download/vocoder-models/vocos_24khz.onnx)229 MB of models in total, which puts the installed app at ~257 MB. `vendor/`\n\nis gitignored and fully reproducible from `scripts/fetch-deps.sh`\n\n.\n\n| Framework | Used for |\n|---|---|\n| SwiftUI | Entire UI |\n| SwiftData | Agent persistence (`@Model` , `@Query` , `@ModelActor` ) |\n| BackgroundTasks | `BGProcessingTask` scheduling and execution |\n| AVFoundation | Mic capture, format conversion, PCM playback, audio session |\n| FoundationModels | Apple Intelligence on-device LLM — weak, conditional (`#if canImport` ), iOS 26+ |\n| URLSession | Web search and page fetching (ephemeral session, no cookie or cache persistence) |\n| Security | Keychain storage for the optional Brave API key |\n| Combine | `ObservableObject` engines |\n\n| Tool | Role |\n|---|---|\n|\n\n`Agent.xcodeproj`\n\nfrom [— the project file is disposable,](/hsandhu/agent/blob/main/project.yml)`project.yml`\n\n`project.yml`\n\nis the source of truthDeployment target iOS 17.0, Swift 5.9, iPhone only. The C API is reached\nthrough an Objective-C bridging header\n(`Agent/Support/SherpaOnnx-Bridging-Header.h`\n\n) with `-lc++`\n\nlinked.\n\n```\n./scripts/fetch-deps.sh\nxcodegen generate && open Agent.xcodeproj\n```\n\nThe fetch script downloads the frameworks and models (~330 MB of archives)\ninto `vendor/`\n\n, and is idempotent — it skips anything already present.\n\n```\nAgent/\n  AgentApp.swift              App entry; builds ModelContainer, registers AgentRunner,\n                              warms the TTS model, schedules background work on phase change\n  ContentView.swift           Root (AgentsView)\n  Agents/\n    AgentJob.swift            @Model: status, prompt, timestamps, progress log, result\n    AgentStore.swift          @ModelActor: all background SwiftData access\n    AgentRunner.swift         BGProcessingTask registration/scheduling + the work loop\n    AgentBrain.swift          Brain protocol, MockAgentBrain, FoundationModelsBrain\n  Web/\n    WebSearch.swift           WebResult/WebSource, provider protocol, WebSearchConfig, keychain\n    WebResearcher.swift       Orchestration: queries → search → dedupe → read → excerpts\n    DuckDuckGoSearch.swift    Keyless HTML endpoint + Instant Answer fallback\n    BraveSearch.swift         Keyed JSON API\n    PageReader.swift          Fetch, size-cap, main-content extraction\n    HTMLText.swift            HTML → plain text, entity decoding, regex helpers\n  Speech/\n    AudioCapture.swift        AVAudioEngine tap → 16 kHz mono Float32\n    AudioPlayer.swift         Queued Float32 PCM playback\n    SttEngine.swift           Streaming recognizer, endpointing, dictation mode\n    TtsEngine.swift           ZipVoice synthesis queue, prompt cache, RTF stats\n    VoiceProfile.swift        VoiceProfile, VoiceProfileStore, EnrollRecorder\n    ModelPaths.swift          Bundle paths for every model file\n    SherpaOnnx.swift          Upstream wrapper, verbatim from v1.12.21 swift-api-examples\n    SherpaOnnx+Zipvoice.swift The zero-shot generate call upstream doesn't expose\n  Views/\n    AgentsView.swift          List, composer pill, agent detail, MarkdownText\n                              (headings/lists/rules + inline markup)\n    SettingsView.swift        Settings hub\n    WebSearchView.swift       Web research: switch, provider, depth, test search\n    VoiceView.swift           Voice enrollment + selection\n    TranscribeView.swift      Live transcription + echo\n    SpeakView.swift           Type-to-speak\n    DesignSystem.swift        PillButton, Card, SectionLabel, RowGroup, SettingsRow,\n                              InfoRow, SelectionRow, StatusDot\n  Support/                    Bridging header, generated Info.plist\nscripts/fetch-deps.sh         Reproduces vendor/\nproject.yml                   XcodeGen project definition\n```\n\nThe design language is flat and monochrome (modeled on Cursor's mobile app):\nhairline-separated rows, gray content cards, status dots, and exactly one\nhigh-contrast pill button per screen. Everything routes through\n`DesignSystem.swift`\n\n, and `PillButton`\n\n's `Color.primary`\n\nfill inverts\ncorrectly in dark mode.\n\n, copied verbatim from the v1.12.21`SherpaOnnx.swift`\n\nis vendored upstream code`swift-api-examples`\n\n. Local additions go in`SherpaOnnx+Zipvoice.swift`\n\nso the wrapper can be replaced wholesale on a version bump. The upstream wrapper doesn't expose`SherpaOnnxOfflineTtsGenerateWithZipvoice`\n\n, which is why that extension exists.**The simulator can't run**—`BGProcessingTask`\n\n`BGTaskScheduler`\n\nis unavailable there, which the UI surfaces rather than failing silently. To test on a real device, pause in the debugger and run:`e -l objc -- (void)[[BGTaskScheduler sharedScheduler] _simulateLaunchForTaskWithIdentifier:@\"com.robsandhu.Agent.agentwork\"]`\n\n**The DuckDuckGo provider parses HTML, not an API**— there is no free keyless search API, so it reads the same no-JavaScript results page a browser would get. Markup changes will break it; the parse accepts the classes used by both the`html.`\n\nand`lite.`\n\nfront ends, falls back to the (supported, but narrow) Instant Answer API, and Brave is there as a documented alternative for anyone willing to hold a key.**Research is stuffed into the prompt rather than exposed as a tool.** FoundationModels can do tool calling, but the on-device model's context window is small enough that a search-tool round trip competes with the evidence itself for space. Retrieving first and handing over fixed excerpts keeps the budget predictable and makes citation numbering deterministic.**Web research is the one thing that leaves the device.** The task text goes to the search provider and the pages it points at get downloaded; nothing else does, and the switch in Settings turns it off entirely — agents then fall back to the model's own knowledge.**ZipVoice settings**:`numSteps = 4`\n\nis the speed/quality sweet spot for the distilled model. Measured RTF ≈ 1.0 in the simulator on an M-series Mac — roughly real time.**The 20M Zipformer emits unpunctuated, uppercase text.**`SttEngine.prettify`\n\nnormalizes casing, and`TtsEngine`\n\nappends terminal punctuation before synthesis because ZipVoice's prosody depends on it.**Enrollment rejects near-silent recordings**(peak amplitude below 0.02) — cloning from silence produces an unusable voice, and a muted or covered mic is otherwise invisible until playback.**Model licenses are separate from the runtime's.** sherpa-onnx and ZipVoice are Apache 2.0 and the bundled models are permissive, but check each model card before shipping anything commercially.", "url": "https://wpnews.pro/news/an-ios-app-that-runs-ai-agents-and-a-complete-voice-pipeline-on-the-device", "canonical_source": "https://github.com/hsandhu/agent", "published_at": "2026-08-22 16:52:38+00:00", "updated_at": "2026-08-22 17:13:57.474061+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "ai-infrastructure", "natural-language-processing"], "entities": ["Rob Sandhu", "Apple Intelligence", "Sherpa-onnx", "SwiftData", "DuckDuckGo", "Brave", "BGTaskScheduler", "FoundationModels"], "alternates": {"html": "https://wpnews.pro/news/an-ios-app-that-runs-ai-agents-and-a-complete-voice-pipeline-on-the-device", "markdown": "https://wpnews.pro/news/an-ios-app-that-runs-ai-agents-and-a-complete-voice-pipeline-on-the-device.md", "text": "https://wpnews.pro/news/an-ios-app-that-runs-ai-agents-and-a-complete-voice-pipeline-on-the-device.txt", "jsonld": "https://wpnews.pro/news/an-ios-app-that-runs-ai-agents-and-a-complete-voice-pipeline-on-the-device.jsonld"}}