Wiring MLX and Core ML ANE Pipelines in Swift 6: On-Device Inference Without the Latency Cliff

A developer built a Swift 6 actor topology that routes on-device inference requests between MLX's GPU compute and Core ML's ANE scheduler on Apple Silicon, achieving sub-50ms first-token latency on quantized 3B models. The architecture uses strict actor isolation with one actor per compute backend, avoiding main actor blocking.

--- title: "Wiring MLX and Core ML ANE Pipelines in Swift 6: On-Device Inference Without the Latency Cliff" published: true description: "Architect Swift 6 actors for MLX GPU and Core ML ANE to achieve sub-50ms first-token latency on Apple Silicon. Benchmarks and routing logic included." tags: swift, ios, mobile, architecture canonical url: https://mvpfactory.co/blog/swift6-mlx-coreml-ane-inference --- What We Are Building By the end of this tutorial, you will have a production-ready Swift 6 actor topology that routes on-device inference requests between MLX's GPU compute and Core ML's ANE scheduler — without ever blocking the main actor. We will wire up two isolated actors, a coordinator with runtime routing logic, and validate the setup against real benchmarks on M-series chips. Target outcome: sub-50ms first-token latency on quantized 3B models. That is achievable, and here is the architecture that gets you there. --- Prerequisites - Xcode 16+ with Swift 6 mode enabled - macOS 15+ running on M-series hardware M1 or later - Familiarity with Swift's async/await and basic actor concepts - MLX Swift package github.com/ml-explore/mlx-swift https://github.com/ml-explore/mlx-swift - A Core ML compiled model .mlmodelc — Llama 3.2 3B 4-bit AWQ works well as a reference --- Step 1 — One Actor Per Compute Backend Let me show you a pattern I use in every project: strict actor isolation, one backend per actor, zero shared mutable state. Most teams treat inference like a network call — async/await , fire and forget, handle on the main actor. That mental model breaks the moment your model is 2GB and the ANE scheduler starts competing with Core Animation for memory bandwidth. Swift 6 makes data races a compile error, which forces the architectural conversation you should have had earlier. You cannot share an MLModel instance across actor boundaries without explicit sendability guarantees. Here is the minimal setup to get this working: swift // MLX GPU actor — owns the model graph and KV cache @globalActor actor MLXInferenceActor { static let shared = MLXInferenceActor private var model: LLMModel? private var kvCache: KVCache? func load config: ModelConfig async throws { model = try await LLMModel.load config kvCache = KVCache capacity: config.contextLength } func generate prompt: MLXArray, maxTokens: Int - AsyncStream<Token { AsyncStream { continuation in Task { guard let model else { continuation.finish ; return } for token in model.stream prompt, cache: kvCache { continuation.yield token } continuation.finish } } } } // Core ML ANE actor — owns the compiled model and prediction context @globalActor actor ANEInferenceActor { static let shared = ANEInferenceActor private var compiledModel: MLModel? js func load url: URL async throws { let config = MLModelConfiguration config.computeUnits = .cpuAndNeuralEngine // ANE-first, CPU fallback compiledModel = try MLModel contentsOf: url, configuration: config } func predict input: MLFeatureProvider async throws - MLFeatureProvider { guard let compiledModel else { throw InferenceError.notLoaded } return try compiledModel.prediction from: input } func stream request: InferenceRequest - AsyncStream<Token { AsyncStream { continuation in Task { do { let output = try await predict input: request.toMLFeatureProvider for token in output.toTokenSequence { continuation.yield token } } catch { } continuation.finish } } } } Neither actor touches MainActor . Your UI layer subscribes to AsyncStream<Token and updates SwiftUI state via @MainActor task groups. The inference path never blocks the run loop. --- Step 2 — Know Which Backend to Route To Here is the benchmark data from Llama 3.2 3B 4-bit AWQ, ~1.8GB on-disk on an M3 Pro with 18GB unified memory, macOS 15.2. Medians from 20 runs, first two cold-start runs excluded: | Workload | Backend | First Token ms | Tokens/sec | Power Draw | |---|---|---|---|---| | Single prompt, 512 ctx | ANE | 38 | 42 | 2.1W | | Single prompt, 512 ctx | GPU | 61 | 38 | 4.8W | | Batch=4, 512 ctx | ANE | 44 | 31 | 2.3W | | Batch=4, 512 ctx | GPU | 63 | 89 | 6.1W | | 4K context window | ANE | 112 | 28 | 2.6W | | 4K context window | GPU | 74 | 44 | 5.4W | Rule of thumb: if your p95 context length stays under 1K tokens and you are serving one request at a time, route to ANE. If you are building a document summarization pipeline or supporting concurrent requests, MLX on GPU gives you the throughput. --- Step 3 — Wire the Coordinator swift actor InferenceCoordinator { private let aneActor = ANEInferenceActor.shared private let mlxActor = MLXInferenceActor.shared php func route request: InferenceRequest async - AsyncStream<Token { switch request.contextLength { case ..<1024 where request.isBatched: return await aneActor.stream request default: return await mlxActor.generate prompt: request.toMLXArray , maxTokens: request.maxTokens } } } Changing the cutoff threshold is a one-line change, not a refactor. All routing logic lives here and nowhere else. --- Gotchas The docs do not mention this, but Core ML's ANE scheduler silently falls back to CPU for unsupported operator types. These four account for the majority of unexpected CPU fallback in transformer pipelines: - LayerNorm with dynamic sequence length — most frequent culprit; static-shape variants compile cleanly - RoPE with variable context — dynamic sequence indexing breaks ANE dispatch - Grouped-query attention GQA — multi-head key/value grouping patterns are not fully supported on current ANE hardware - Custom activation functions — anything beyond ReLU, GELU, and Swish typically falls back to CPU Run mlmodelc compilation with --compute-units cpuAndNeuralEngine and inspect the compilation report. Any op marked cpu only is killing your ANE gains. If more than 15% of ops fall back to CPU — especially LayerNorm, RoPE, or GQA — your model architecture needs modification before ANE routing delivers real gains. Also: MLX GPU and Core ML ANE have incompatible threading models. Mixing them on a shared actor is a deadlock waiting to happen. Swift 6 enforces the boundary at compile time — use it. --- Conclusion One actor per compute backend. Compile the Core ML model and audit the ops report. Let context length and batch size drive routing, not intuition. Build a coordinator that makes the routing decision at runtime based on request shape, and benchmark against your actual p95 context distribution before picking a fixed cutoff. The architecture here gets you to sub-50ms first-token latency — but only if you do the operator audit first.