An iOS app that runs AI agents and a complete voice pipeline on the device

An iOS app that runs AI agents and a complete voice pipeline entirely on the device. No account, no model API, no inference in the cloud. The one thing that leaves the phone is web research — the agent searches and downloads pages so the on-device model has something current to reason over, and that switch can be turned off.

Two halves that reinforce each other:

Background agents. Describe a task ("find the best summer camps for my 9-year-old near Chicago"), and an agent works on it in the background: it plans search queries, reads the pages it finds, then reasons over them with Apple Intelligence's on-device model and cites what it used. Come back later and press play — the result is read aloud.On-device speech. Streaming speech-to-text (used both for dictating tasks and for live transcription) and zero-shot voice-cloning text-to-speech, which is what gives agent results a voice — optionallyyourvoice, cloned from a ten-second sample.

The speech half also exists as the local building block for a longer-term idea: a voice-call app where audio is transcribed on the sender's device, sent over the network as text (~50 bytes/sec instead of 6–24 kbps of Opus), and re-synthesized on the receiver's device in the sender's cloned voice. The networking layer isn't built yet; the Live Transcription screen's Echo toggle is a local loopback of that pipeline.

Five layers. The UI never touches the native runtime directly, every model call happens off the main thread, and the model never reaches the network itself — the web layer does the fetching and hands it text.

flowchart TB
    subgraph UI["UI — SwiftUI"]
        A["AgentsView<br/>list · composer · detail"]
        S["SettingsView<br/>My Voice · Transcribe · Speak"]
        D["DesignSystem<br/>shared components"]
    end

    subgraph Domain["Agents — domain"]
        R["AgentRunner<br/>BGProcessingTask + foreground"]
        B["AgentBrain<br/>protocol"]
        ST[("AgentStore<br/>ModelActor")]
        J["AgentJob<br/>SwiftData Model"]
    end

    subgraph Web["Web research"]
        WR["WebResearcher<br/>search · read · excerpt"]
        SP["WebSearchProvider<br/>DuckDuckGo · Brave"]
        PR["PageReader<br/>+ HTMLText"]
    end

    subgraph Speech["Speech — engines"]
        STT["SttEngine<br/>serial queue"]
        TTS["TtsEngine<br/>serial queue"]
        CAP["AudioCapture"]
        PLAY["AudioPlayer"]
        VP["VoiceProfileStore"]
    end

    subgraph Native["Native bridge"]
        W["SherpaOnnx.swift<br/>+ Zipvoice extension"]
        C["c-api.h via bridging header"]
        X["sherpa-onnx.xcframework<br/>onnxruntime.xcframework"]
    end

    A --> R
    A --> TTS
    A --> STT
    S --> STT
    S --> TTS
    S --> VP
    S --> WR
    R --> B
    R --> ST
    R --> WR
    B --> WR
    WR --> SP
    WR --> PR
    ST --- J
    B -.->|iOS 26+| FM["FoundationModels"]
    B -.->|fallback| MOCK["MockAgentBrain"]
    STT --> CAP
    TTS --> PLAY
    TTS --> VP
    STT --> W
    TTS --> W
    W --> C --> X

An agent is a row in SwiftData with a status (queued → running → completed/failed

), a progress log appended as it works, and a result split into a spoken-style resultSummary

and a longer resultDetail

.

AgentRunner

is a singleton configured at launch (BGTaskScheduler requires registration before launch finishes). It drains the queue from two entry points:

Path	Trigger	Notes
Foreground	App becomes active, or you spawn an agent	The interactive path, and the only one that works in the simulator
Background	BGProcessingTask id com.robsandhu.Agent.agentwork
iOS schedules it at its discretion (typically idle/charging); the handler re-chains the next slot before starting work

Interruption is handled explicitly: the task's expirationHandler

cancels the work, drainQueue

catches CancellationError

and requeues the in-flight job, and any job left stranded in running

by a process death is requeued at next launch (requeueOrphanedRunningJobs

).

The model behind an agent is swappable via the AgentBrain

protocol, so the persistence, scheduling, and playback machinery is independent of which model runs:

— Apple Intelligence's on-device model, compiled in behindFoundationModelsBrain

#if canImport(FoundationModels)

and offered only whenSystemLanguageModel.default.availability

reports available. Runs three passes: plan the search queries, write the findings from what was read, then a three-sentence summary written to be read aloud.— the fallback where Apple Intelligence isn't available. With web research on it still searches and reads for real and reports a digest of what it found; with research off it simulates staged research, so the whole pipeline is testable on any device.MockAgentBrain

Agents search before they think. The model never touches the network itself — the app runs the searches, decides which pages to download, and hands back excerpts — so what the model can see is bounded by policy rather than by its own tool calls.

plan queries → search → interleave + dedupe → read N pages → excerpt → ground

Step	Where	Notes
Plan queries	FoundationModelsBrain.planQueries
Guided generation (@Generable ) forces a list of query strings. Left to free text, a small model answers the task instead of writing queries for it — WebResearcher.normalize scrubs the leftovers of that habit (markdown, dash clauses, over-long lines)
Search	WebSearchProvider
DuckDuckGoSearch (keyless) or BraveSearch (API key in the keychain)
Merge	WebResearcher
Round-robins across queries so one query can't monopolize the read budget, and dedupes by canonical host+path
Read	PageReader + HTMLText
Capped at 1.2 MB per page, <article> /<main> preferred, tags and entities reduced to plain text. A page that won't load degrades to its search snippet rather than failing the job
Ground	FoundationModelsBrain.findings
Numbered excerpts in the prompt, citations required inline. On a context-window overflow it retries at 1100 → 600 → 300 chars per source, then falls back to unresearched general knowledge

Sources are persisted on the job (sourcesJSON

) and listed as tappable links under the findings, so every claim can be traced back to the page it came from. Settings › Web Research holds the master switch, the provider choice, the depth knobs, and a live test button.

AudioCapture

taps AVAudioEngine

and converts the hardware format (44.1/48 kHz) to the 16 kHz mono Float32 the models expect. Its output feeds either the recognizer or the enrollment recorder — never both, since there's one capture session.

SttEngine

wraps a streaming Zipformer transducer with endpoint detection: it publishes a live partial

and appends a finalized TranscriptSegment

on each detected . It has a dictation mode — when dictationOnPartial

/ dictationOnUtterance

are set, recognized speech routes to those callbacks instead of the main transcript, which is how the composer pill takes spoken input without polluting the transcription screen.

TtsEngine

runs ZipVoice zero-shot synthesis on a serial queue. Voice cloning needs no training: a VoiceProfile

is just a reference wav plus its transcript, and synthesis conditions on that pair at call time. Prompt audio is cached per file path; enrollment writes a new filename each time so the cache can never go stale.

Component	Executor
Views, @Published state
Main
AudioCapture callbacks
AVAudioEngine render thread
SttEngine decode loop
Private serial DispatchQueue
TtsEngine synthesis
Private serial DispatchQueue
AgentStore (all SwiftData writes)
@ModelActor
AgentRunner.drainQueue
Swift Task , awaits the store actor

SwiftData contexts aren't thread-safe, so every mutation from a background task goes through AgentStore

; the UI reads through @Query

on its own main-thread context.

There are no Swift Package Manager or CocoaPods dependencies. The one third-party runtime is vendored as prebuilt .xcframework

binaries, and the project file is generated rather than committed.

Package	Version	License	Role

csukuangfj/sherpa-onnx-libs

ONNX RuntimePinned to 1.12.21 because that's the newest version with prebuilt iOS frameworks published — 1.13.x has no iOS build. The Swift wrapper, C headers, and binaries must all come from the same release.

Model	Size	License / training data	Role

ZipVoice-Distill int8, zh-en(sherpa-onnx-zipvoice-distill-int8-zh-en-emilia

)vocos 24 kHz vocoder229 MB of models in total, which puts the installed app at ~257 MB. vendor/

is gitignored and fully reproducible from scripts/fetch-deps.sh

.

Framework	Used for
SwiftUI	Entire UI
SwiftData	Agent persistence (@Model , @Query , @ModelActor )
BackgroundTasks	BGProcessingTask scheduling and execution
AVFoundation	Mic capture, format conversion, PCM playback, audio session
FoundationModels	Apple Intelligence on-device LLM — weak, conditional (#if canImport ), iOS 26+
URLSession	Web search and page fetching (ephemeral session, no cookie or cache persistence)
Security	Keychain storage for the optional Brave API key
Combine	ObservableObject engines

Tool	Role

Agent.xcodeproj

from — the project file is disposable,project.yml

project.yml

is the source of truthDeployment target iOS 17.0, Swift 5.9, iPhone only. The C API is reached through an Objective-C bridging header (Agent/Support/SherpaOnnx-Bridging-Header.h

) with -lc++

linked.

./scripts/fetch-deps.sh
xcodegen generate && open Agent.xcodeproj

The fetch script downloads the frameworks and models (~330 MB of archives) into vendor/

, and is idempotent — it skips anything already present.

Agent/
  AgentApp.swift              App entry; builds ModelContainer, registers AgentRunner,
                              warms the TTS model, schedules background work on phase change
  ContentView.swift           Root (AgentsView)
  Agents/
    AgentJob.swift            @Model: status, prompt, timestamps, progress log, result
    AgentStore.swift          @ModelActor: all background SwiftData access
    AgentRunner.swift         BGProcessingTask registration/scheduling + the work loop
    AgentBrain.swift          Brain protocol, MockAgentBrain, FoundationModelsBrain
  Web/
    WebSearch.swift           WebResult/WebSource, provider protocol, WebSearchConfig, keychain
    WebResearcher.swift       Orchestration: queries → search → dedupe → read → excerpts
    DuckDuckGoSearch.swift    Keyless HTML endpoint + Instant Answer fallback
    BraveSearch.swift         Keyed JSON API
    PageReader.swift          Fetch, size-cap, main-content extraction
    HTMLText.swift            HTML → plain text, entity decoding, regex helpers
  Speech/
    AudioCapture.swift        AVAudioEngine tap → 16 kHz mono Float32
    AudioPlayer.swift         Queued Float32 PCM playback
    SttEngine.swift           Streaming recognizer, endpointing, dictation mode
    TtsEngine.swift           ZipVoice synthesis queue, prompt cache, RTF stats
    VoiceProfile.swift        VoiceProfile, VoiceProfileStore, EnrollRecorder
    ModelPaths.swift          Bundle paths for every model file
    SherpaOnnx.swift          Upstream wrapper, verbatim from v1.12.21 swift-api-examples
    SherpaOnnx+Zipvoice.swift The zero-shot generate call upstream doesn't expose
  Views/
    AgentsView.swift          List, composer pill, agent detail, MarkdownText
                              (headings/lists/rules + inline markup)
    SettingsView.swift        Settings hub
    WebSearchView.swift       Web research: switch, provider, depth, test search
    VoiceView.swift           Voice enrollment + selection
    TranscribeView.swift      Live transcription + echo
    SpeakView.swift           Type-to-speak
    DesignSystem.swift        PillButton, Card, SectionLabel, RowGroup, SettingsRow,
                              InfoRow, SelectionRow, StatusDot
  Support/                    Bridging header, generated Info.plist
scripts/fetch-deps.sh         Reproduces vendor/
project.yml                   XcodeGen project definition

The design language is flat and monochrome (modeled on Cursor's mobile app): hairline-separated rows, gray content cards, status dots, and exactly one high-contrast pill button per screen. Everything routes through DesignSystem.swift

, and PillButton

's Color.primary

fill inverts correctly in dark mode.

, copied verbatim from the v1.12.21SherpaOnnx.swift

is vendored upstream codeswift-api-examples

. Local additions go inSherpaOnnx+Zipvoice.swift

so the wrapper can be replaced wholesale on a version bump. The upstream wrapper doesn't exposeSherpaOnnxOfflineTtsGenerateWithZipvoice

, which is why that extension exists.The simulator can't run—BGProcessingTask

BGTaskScheduler

is unavailable there, which the UI surfaces rather than failing silently. To test on a real device, in the debugger and run:e -l objc -- (void)[[BGTaskScheduler sharedScheduler] _simulateLaunchForTaskWithIdentifier:@"com.robsandhu.Agent.agentwork"]

The DuckDuckGo provider parses HTML, not an API— there is no free keyless search API, so it reads the same no-JavaScript results page a browser would get. Markup changes will break it; the parse accepts the classes used by both thehtml.

andlite.

front ends, falls back to the (supported, but narrow) Instant Answer API, and Brave is there as a documented alternative for anyone willing to hold a key.Research is stuffed into the prompt rather than exposed as a tool. FoundationModels can do tool calling, but the on-device model's context window is small enough that a search-tool round trip competes with the evidence itself for space. Retrieving first and handing over fixed excerpts keeps the budget predictable and makes citation numbering deterministic.Web research is the one thing that leaves the device. The task text goes to the search provider and the pages it points at get downloaded; nothing else does, and the switch in Settings turns it off entirely — agents then fall back to the model's own knowledge.ZipVoice settings:numSteps = 4

is the speed/quality sweet spot for the distilled model. Measured RTF ≈ 1.0 in the simulator on an M-series Mac — roughly real time.The 20M Zipformer emits unpunctuated, uppercase text.SttEngine.prettify

normalizes casing, andTtsEngine

appends terminal punctuation before synthesis because ZipVoice's prosody depends on it.Enrollment rejects near-silent recordings(peak amplitude below 0.02) — cloning from silence produces an unusable voice, and a muted or covered mic is otherwise invisible until playback.Model licenses are separate from the runtime's. sherpa-onnx and ZipVoice are Apache 2.0 and the bundled models are permissive, but check each model card before shipping anything commercially.

source & further reading

github.com — original article Multi-Agent Harness for Visual Design TT-AMX, a zero-copy Tensor-Train inference engine for Apple Silicon Show HN: Knowl – agent memory that retires stale facts when they change