{"slug": "on-device-ai-for-ios-macos", "title": "On-Device AI for iOS & macOS", "summary": "A new Swift tutorial demonstrates how to run large language models directly on iOS and macOS devices using the NobodyWho library, which wraps llama.cpp in Rust and provides a clean Swift API. The tutorial covers streaming responses, model downloads from Hugging Face, and advanced features like multimodal input and tool calling, highlighting the benefits of on-device AI such as privacy, offline capability, and cost savings.", "body_md": "In this Swift tutorial, you'll learn how to run a large language model (LLM) directly on a user's device: no server, no API key needed. We'll start from scratch with a simple chat exchange, and progressively introduce more advanced features: multimodal input, speech-to-text, text-to-speech, voice activity detection, tool calling and RAG.\n\nEach concept is explained before the code, so you can follow along whether you're new to on-device AI.\n\nMost AI features rely on a cloud API: you send a request to a remote server, it runs the model, and sends a response back. That works well, but it comes with tradeoffs.\n\nRunning the model directly on the device avoids all of them:\n\nThe tradeoff is raw capability: on-device models are smaller and less powerful than frontier cloud models. But for many use cases like summarization, chatbots, or local search, they're more than good enough.\n\nWe'll use the [NobodyWho](https://github.com/nobodywho-ooo/nobodywho) library throughout this tutorial. It wraps [llama.cpp](https://github.com/ggerganov/llama.cpp) in Rust and exposes a clean Swift API for running any model locally in `.gguf`\n\nformat, across iOS, macOS, visionOS, and watchOS.\n\nAdd it with Swift Package Manager. In Xcode, go to **File → Add Package Dependencies** and enter:\n\n```\nhttps://github.com/nobodywho-ooo/nobodywho-swift.git\n```\n\nOr add it to your `Package.swift`\n\n:\n\n```\ndependencies: [\n    .package(url: \"https://github.com/nobodywho-ooo/nobodywho-swift.git\", from: \"2.1.0\")\n]\n```\n\nNobodyWho can download a GGUF model for you directly from Hugging Face, cache it, and reuse it on every subsequent launch. That means you don't need to bundle anything into your app or manage downloads yourself:\n\n``` python\nimport NobodyWho\n\nlet chat = try await Chat.fromPath(\n    modelPath: \"hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf\"\n)\n```\n\nThe first time this runs, the model is downloaded to the platform cache directory. Every call after that loads the model directly.\n\n`modelPath`\n\naccepts a few different forms:\n\n| Form | Example | Notes |\n|---|---|---|\n| HuggingFace reference | `hf://owner/repo/file.gguf` |\nDownloaded and cached on first use |\n| HTTPS URL | `https://example.com/model.gguf` |\nDownloaded and cached on first use |\n| Local path | `/path/to/model.gguf` |\nUsed as-is, no download |\n\nThe HuggingFace prefix is case-insensitive, so `hf://`\n\nand `huggingface://`\n\nare equivalent. You can also track a remote download by passing a progress closure to `Chat.fromPath`\n\n, which receives `(downloaded, total)`\n\nbyte counts and is skipped for cached or local files:\n\n``` js\nlet chat = try await Chat.fromPath(\n    modelPath: \"hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf\"\n) { downloaded, total in\n    print(\"Downloaded \\(downloaded)/\\(total) bytes\")\n}\n```\n\nYou can find thousands of LLM in .gguf format on Hugging Face [here](https://huggingface.co/models?library=gguf&sort=trending).\n\nWith a model loaded, you're ready to start a conversation:\n\n``` js\nlet chat = try await Chat.fromPath(\n    modelPath: \"hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf\"\n)\nlet response = try await chat.ask(\"Is water wet?\").completed()\nprint(response) // Yes, indeed, water is wet!\n```\n\n`chat.ask()`\n\nsends your message and returns a `TokenStream`\n\n, which conforms to `AsyncSequence`\n\n. Calling `.completed()`\n\nwaits for the whole response and gives you back the final string, which is fine for a one-off question. But a real chat interface needs to stream tokens as they arrive, otherwise users stare at a blank screen until generation finishes.\n\n``` js\nlet response = chat.ask(\"What is the capital of Denmark?\")\n\nfor await token in response {\n    print(token)\n}\n```\n\nA *token* is the smallest unit a model generates, typically a word, or a fragment of a word.\n\nIf you need to cancel a response mid-generation, for example when the user taps a \"Stop\" button, call `chat.stopGeneration()`\n\n. It's synchronous and safe to call from any thread; the tokens already produced stay in the stream and are kept in the chat history, so the conversation stays coherent.\n\nSome models can natively ingest images and audio. To use them, you need two things: a multimodal LLM, and its projection model that converts images and/or audio into tokens the LLM can consume (usually named with `mmproj`\n\nin it). A solid default that handles both image and audio is Gemma 4 with its BF16 projection model.\n\n``` python\nimport NobodyWho\n\nlet chat = try await Chat.fromPath(\n    modelPath: \"/path/to/vision-model.gguf\",\n    projectionModelPath: \"/path/to/mmproj.gguf\"\n)\n```\n\nTo actually send image or audio content, build a `Prompt`\n\nmixing text, images, and audio, and pass it to `chat.ask()`\n\ninstead of a plain string:\n\n``` js\nlet prompt = Prompt([\n    Prompt.text(\"Tell me what you see in the image and what you hear in the audio.\"),\n    Prompt.image(\"/path/to/dog.png\"),\n    Prompt.audio(\"/path/to/sound.mp3\"),\n])\nlet response = try await chat.ask(prompt).completed()\n```\n\nKeep in mind that images and audio consume context fast, so you'll likely want a bigger `contextSize`\n\nthan you'd use for text-only chat. Also note that the language model and its projection model have to be trained together.\n\nIf you'd rather transcribe spoken audio into text than have the model listen to it directly, NobodyWho integrates Whisper models in ONNX format through `SpeechToText`\n\n.\n\n``` python\nimport NobodyWho\n\nlet stt = try await SpeechToText.load(source: \"hf://onnx-community/whisper-base\")\n\nlet text = try await stt.transcribeFile(path: \"recording.mp3\").completed()\nprint(text)\n```\n\n`source`\n\nis a Hugging Face repo (`hf://owner/repo`\n\n) or a local directory laid out the same way. Browse the [Whisper ONNX models on Hugging Face](https://huggingface.co/models?library=onnx&search=whisper) to find one that fits your accuracy and speed needs.\n\nIf your audio comes from a buffer rather than a file, use `transcribePcm`\n\n:\n\n``` js\nlet text = try await stt.transcribePcm(samples: samples, sampleRate: 16000).completed()\n```\n\nThe buffer needs to be mono i16 PCM samples. The sample rate can be anything, NobodyWho resamples internally to what Whisper expects. And just like chat, transcription can be streamed piece by piece instead of waiting for the full result:\n\n```\nfor try await piece in stt.transcribeFile(path: \"recording.mp3\") {\n    print(piece)\n}\n```\n\nGoing the other direction, `TextToSpeech`\n\nturns text into WAV audio you can play back or save.\n\n``` python\nimport Foundation\nimport NobodyWho\n\nlet tts = try await TextToSpeech.load(\n    source: \"hf://NobodyWho/Kokoro-82M\",\n    voice: \"bf_emma\",\n    language: \"en-gb\"\n)\n\nlet wav = try await tts.synthesize(\"Hello from NobodyWho!\")\ntry wav.write(to: URL(fileURLWithPath: \"out.wav\"))\n```\n\nThree architectures are supported, all ONNX-based: [Kokoro](https://github.com/hexgrad/kokoro), [Pocket TTS](https://github.com/kyutai-labs/pocket-tts), and [Supertonic](https://github.com/supertone-inc/supertonic). NobodyWho infers which one you're using from the `source`\n\nstring, so you only need to set `architecture`\n\nexplicitly when loading from a custom local folder.\n\nEach architecture has its own `voice`\n\nand `language`\n\noptions that need to agree with what the model supports.\n\nBefore transcribing audio, it helps to know when someone is actually speaking rather than relying on a fixed silence timeout. `VoiceActivityDetection`\n\nuses a small model to reliably tell speech and silence apart, and pairs naturally with `SpeechToText`\n\n.\n\nFor streaming microphone input, push chunks in as they arrive:\n\n``` python\nimport NobodyWho\n\nlet vad = try await VoiceActivityDetection.load(sampleRate: 16000, source: \"hf://onnx-community/silero-vad\")\nlet stt = try await SpeechToText.load(source: \"hf://onnx-community/whisper-base\")\n\nwhile let chunk = readMic() {\n    if try vad.push(chunk: chunk) == .speechEnded {\n        break\n    }\n}\n\nlet speech = vad.finish()\nlet transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()\nprint(transcription)\n```\n\nEach `push`\n\ncall reports the current state (`.speechStarted`\n\n, `.speechEnded`\n\n, `.speech`\n\n, or `.silence`\n\n), and `finish()`\n\nhands you back the buffered speech segment while resetting internal state for the next turn.\n\nIf you already have a full recording and just want to pull out the speech segments from it, `segment`\n\ndoes that in one pass:\n\n``` js\nlet audio = readWavPcm(path: \"recording.wav\")\n\nfor speech in try vad.segment(samples: audio) {\n    let transcription = try await stt.transcribePcm(samples: speech, sampleRate: 16000).completed()\n    print(transcription)\n}\n```\n\nSensitivity is tunable via `threshold`\n\n, `minSpeechDurationMs`\n\n, `minSilenceDurationMs`\n\n, and `prerollDurationMs`\n\n(how much audio to keep before the detected start, so you don't clip the beginning of a sentence). The defaults are a reasonable starting point, but VAD is one of those things that usually benefits from tuning to your actual environment.\n\nTools let the model call out to real functions in your app rather than just generating text. The easiest way to create one is with the `@DeclareTool`\n\nmacro: annotate any top-level or type-member function with a description, and NobodyWho generates the tool for you.\n\n``` python\nimport NobodyWho\n\n@DeclareTool(\"Calculates the area of a circle given its radius\")\nfunc circleArea(radius: Double) -> String {\n    let area = Double.pi * radius * radius\n    return \"Circle with radius \\(radius) has area \\(String(format: \"%.2f\", area))\"\n}\n\nlet chat = try await Chat.fromPath(\n    modelPath: \"hf://NobodyWho/Qwen_Qwen3-0.6B-GGUF/Qwen_Qwen3-0.6B-Q4_K_M.gguf\",\n    tools: [circleAreaTool]\n)\n```\n\nThe macro names the generated variable `<functionName>Tool`\n\n, so `circleArea`\n\nbecomes `circleAreaTool`\n\n. It doesn't work inside function bodies though, for a tool that needs to capture local state, use the manual `Tool`\n\ninitializer instead.\n\nNot every model supports tool calling well, the [Qwen](https://huggingface.co/collections/NobodyWho/qwen-3) family is a solid choice if you need it to be reliable. See the [Tool Calling documentation](https://docs.nobodywho.ooo/swift/tool-calling/) for more.\n\nRetrieval-Augmented Generation combines document search with LLM generation, so the model grounds its answers in your own knowledge base instead of what it happened to learn during training. NobodyWho provides an `Encoder`\n\nfor embeddings and a `CrossEncoder`\n\nfor reranking:\n\n``` python\nimport NobodyWho\n\nlet encoder = try await Encoder.fromPath(modelPath: \"/path/to/embeddings.gguf\", contextSize: 512, useGpu: true)\n\nlet queryEmbedding = try await encoder.encode(\"What is the return policy?\")\nlet docEmbeddings = try await encoder.encodeBatch(knowledge)\nlet similarities = docEmbeddings.map { cosineSimilarity(a: queryEmbedding, b: $0) }\n```\n\nFor better precision, rerank the top candidates with a `CrossEncoder`\n\nbefore handing them to the model:\n\n``` js\nlet crossEncoder = try await CrossEncoder.fromPath(modelPath: \"/path/to/reranker.gguf\", contextSize: 512, useGpu: true)\nlet ranked = try await crossEncoder.rankAndSort(query: \"What is the return policy?\", documents: topDocs)\n```\n\nSee the [Embeddings & RAG documentation](https://docs.nobodywho.ooo/swift/embeddings-and-rag/) for the full walkthrough, including how to pass the reranked documents into the chat's system prompt.\n\nYou now have a complete foundation for building on-device AI features in Swift:\n\nYou can also have a look at the [iOS & macOS starter examples](https://github.com/nobodywho-ooo/swift-starter-example) to see a full implementation.", "url": "https://wpnews.pro/news/on-device-ai-for-ios-macos", "canonical_source": "https://dev.to/pielounw/on-device-ai-for-ios-macos-2glm", "published_at": "2026-09-01 13:42:20+00:00", "updated_at": "2026-09-01 13:53:34.275494+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-products"], "entities": ["NobodyWho", "llama.cpp", "Hugging Face", "Swift", "iOS", "macOS"], "alternates": {"html": "https://wpnews.pro/news/on-device-ai-for-ios-macos", "markdown": "https://wpnews.pro/news/on-device-ai-for-ios-macos.md", "text": "https://wpnews.pro/news/on-device-ai-for-ios-macos.txt", "jsonld": "https://wpnews.pro/news/on-device-ai-for-ios-macos.jsonld"}}