In this Swift tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.
Each concept is explained before the code, so you can follow along whether you're new to on-device AI.
Most AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.
Running the model directly on the device avoids all of them:
The tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases like summarization, chatbots, or local search, they're more than good enough.
We'll use the NobodyWho library throughout this tutorial. It wraps llama.cpp in Rust and exposes a clean Swift API for running any model locally in .gguf
format, across iOS, macOS, visionOS, and watchOS.
Add it with Swift Package Manager. In Xcode, go to File → Add Package Dependencies and enter:
https://github.com/nobodywho-ooo/nobodywho-swift.git
Or add it to your Package.swift
:
dependencies: [
.package(url: "https://github.com/nobodywho-ooo/nobodywho-swift.git", from: "2.1.0")
]
NobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to bundle anything into your app or manage downloads yourself:
import NobodyWho
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
The first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.
modelPath
accepts a few different forms:
| Form | Example | Notes |
|---|---|---|
| HuggingFace reference | hf://owner/repo/file.gguf |
|
| Downloaded and cached on first use | ||
| HTTPS URL | https://example.com/model.gguf |
|
| Downloaded and cached on first use | ||
| Local path | /path/to/model.gguf |
|
| Used as-is, no download |
The HuggingFace prefix is case-insensitive, so hf://
and huggingface://
are equivalent. You can also track a remote download by passing a progress closure to Chat.fromPath
, which receives (downloaded, total)
byte counts and is skipped for cached or local files:
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
) { downloaded, total in
print("Downloaded \(downloaded)/\(total) bytes")
}
You can find thousands of LLM in .gguf format on Hugging Face here.
With a model loaded, you're ready to start a conversation:
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf"
)
let response = try await chat.ask("Is water wet?").completed()
print(response) // Yes, indeed, water is wet!
chat.ask()
sends your message and returns a TokenStream
, which conforms to AsyncSequence
. Calling .completed()
waits for the whole response and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.
let response = chat.ask("What is the capital of Denmark?")
for await token in response {
print(token)
}
A token is the smallest unit a model generates, typically a word, or a fragment of a word.
If you need to cancel a response mid-generation, for example when the user taps a "Stop" button, call chat.stopGeneration()
. It's synchronous and safe to call from any thread; the tokens already produced stay in the stream and are kept in the chat history, so the conversation stays coherent.
Some models can natively ingest images and audio. To use them, you need two things: a multimodal LLM, and its projection model that converts images and/or audio into tokens the LLM can consume (usually named with mmproj
in it). A solid default that handles both image and audio is Gemma 4 with its BF16 projection model.
import NobodyWho
let chat = try await Chat.fromPath(
modelPath: "/path/to/vision-model.gguf",
projectionModelPath: "/path/to/mmproj.gguf"
)
To actually send image or audio content, build a Prompt
mixing text, images, and audio, and pass it to chat.ask()
instead of a plain string:
let prompt = Prompt([
Prompt.text("Tell me what you see in the image and what you hear in the audio."),
Prompt.image("/path/to/dog.png"),
Prompt.audio("/path/to/sound.mp3"),
])
let response = try await chat.ask(prompt).completed()
Keep in mind that images and audio consume context fast, so you'll likely want a bigger contextSize
than you'd use for text-only chat. Also note that the language model and its projection model have to be trained together.
If you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through SpeechToText
.
import NobodyWho
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")
let text = try await stt.transcribeFile(path: "recording.mp3").completed()
print(text)
source
is a Hugging Face repo (hf://owner/repo
) or a local directory laid out the same way. Browse the Whisper ONNX models on Hugging Face to find one that fits your accuracy and speed needs.
If your audio comes from a buffer rather than a file, use transcribePcm
:
let text = try await stt.transcribePcm(samples: samples, sampleRate: 16000).completed()
The buffer needs to be mono i16 PCM samples. The sample rate can be anything, NobodyWho resamples internally to what Whisper expects. And just like chat, transcription can be streamed piece by piece instead of waiting for the full result:
for try await piece in stt.transcribeFile(path: "recording.mp3") {
print(piece)
}
Going the other direction, TextToSpeech
turns text into WAV audio you can play back or save.
import Foundation
import NobodyWho
let tts = try await TextToSpeech.load(
source: "hf://NobodyWho/Kokoro-82M",
voice: "bf_emma",
language: "en-gb"
)
let wav = try await tts.synthesize("Hello from NobodyWho!")
try wav.write(to: URL(fileURLWithPath: "out.wav"))
Three architectures are supported, all ONNX-based: Kokoro, Pocket TTS, and Supertonic. NobodyWho infers which one you're using from the source
string, so you only need to set architecture
explicitly when from a custom local folder.
Each architecture has its own voice
and language
options that need to agree with what the model supports.
Before transcribing audio, it helps to know when someone is actually speaking rather than relying on a fixed silence timeout. VoiceActivityDetection
uses a small model to reliably tell speech and silence apart, and pairs naturally with SpeechToText
.
For streaming microphone input, push chunks in as they arrive:
import NobodyWho
let vad = try await VoiceActivityDetection.load(sampleRate: 16000, source: "hf://onnx-community/silero-vad")
let stt = try await SpeechToText.load(source: "hf://onnx-community/whisper-base")
while let chunk = readMic() {
if try vad.push(chunk: chunk) == .speechEnded {
break
}
}
let speech = vad.finish()
let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()
print(transcription)
Each push
call reports the current state (.speechStarted
, .speechEnded
, .speech
, or .silence
), and finish()
hands you back the buffered speech segment while resetting internal state for the next turn.
If you already have a full recording and just want to pull out the speech segments from it, segment
does that in one pass:
let audio = readWavPcm(path: "recording.wav")
for speech in try vad.segment(samples: audio) {
let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()
print(transcription)
}
Sensitivity is tunable via threshold
, minSpeechDurationMs
, minSilenceDurationMs
, and prerollDurationMs
(how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.
Tools let the model call out to real functions in your app rather than just generating text. The easiest way to create one is with the @DeclareTool
macro: annotate any top-level or type-member function with a description, and NobodyWho generates the tool for you.
import NobodyWho
@DeclareTool("Calculates the area of a circle given its radius")
func circleArea(radius: Double) -> String {
let area = Double.pi * radius * radius
return "Circle with radius \(radius) has area \(String(format: "%.2f", area))"
}
let chat = try await Chat.fromPath(
modelPath: "hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf",
tools: [circleAreaTool]
)
The macro names the generated variable <functionName>Tool
, so circleArea
becomes circleAreaTool
. It doesn't work inside function bodies though, for a tool that needs to capture local state, use the manual Tool
initializer instead.
Not every model supports tool calling well, the Qwen family is a solid choice if you need it to be reliable. See the Tool Calling documentation for more.
Retrieval-Augmented Generation combines document search with LLM generation, so the model grounds its answers in your own knowledge base instead of what it happened to learn during training. NobodyWho provides an Encoder
for embeddings and a CrossEncoder
for reranking:
import NobodyWho
let encoder = try await Encoder.fromPath(modelPath: "/path/to/embeddings.gguf", contextSize: 512, useGpu: true)
let queryEmbedding = try await encoder.encode("What is the return policy?")
let docEmbeddings = try await encoder.encodeBatch(knowledge)
let similarities = docEmbeddings.map { cosineSimilarity(a: queryEmbedding, b: $0) }
For better precision, rerank the top candidates with a CrossEncoder
before handing them to the model:
let crossEncoder = try await CrossEncoder.fromPath(modelPath: "/path/to/reranker.gguf", contextSize: 512, useGpu: true)
let ranked = try await crossEncoder.rankAndSort(query: "What is the return policy?", documents: topDocs)
See the Embeddings & RAG documentation for the full walkthrough, including how to pass the reranked documents into the chat's system prompt.
You now have a complete foundation for building on-device AI features in Swift:
You can also have a look at the iOS & macOS starter examples to see a full implementation.