Wiring MLX to Swift: Running Fine-Tuned Models on Apple Silicon with Zero CoreML Overhead

A developer demonstrates how to run fine-tuned large language models on Apple Silicon using MLX Swift bindings, bypassing CoreML's compilation latency. The approach uses lazy evaluation, direct loading of .safetensors weights, and Swift 6 actors for KV-cache management, with benchmarks showing MLX outperforming CoreML on M-series chips.

--- title: "Wiring MLX to Swift: Running Fine-Tuned Models on Apple Silicon with Zero CoreML Overhead" published: true description: "MLX Swift bindings bypass CoreML's compilation latency. Load quantized fine-tuned LLMs via swift-transformers, manage KV-cache with Swift 6 actors, and see where MLX beats CoreML on M-series chips." tags: swift, mobile, architecture, ios canonical url: https://mvpfactory.co/blog/mlx-swift-fine-tuned-llms-apple-silicon --- What You Will Build By the end of this tutorial, you will have a working Swift setup that loads a fine-tuned LLM directly via MLX Swift bindings — no CoreML compilation step, no ANE scheduling overhead, no app bundle bloat from pre-compiled .mlmodelc packages. Let me show you a pattern I use in every project that ships on-device language models on Apple Silicon. Prerequisites - macOS on Apple Silicon M1, M2, or M3 — MLX targets macOS as its primary environment - Xcode 15+ with Swift 6 concurrency enabled - Python environment for weight conversion mlx-lm toolchain - A fine-tuned model checkpoint exported to .safetensors Why Not Just Use CoreML? CoreML is the default assumption for on-device inference, and for vision and classification tasks it earns that position. For autoregressive LLM generation — especially models you are iterating on — it compounds friction fast. First-load compilation for a 3B parameter model takes 15–45 seconds on M2. Every fine-tune iteration requires a new compilation cycle. Your app bundle grows with each compiled artifact. For teams shipping model updates from a server, this is a real cost. MLX sidesteps this entirely. It operates on a lazy computation graph backed by Metal, fusing kernels across the unified memory architecture M-series chips use. Step 1: Understand MLX's Lazy Evaluation Model Operations on MLXArray are not executed until you call MLX.eval . This is different from CoreML's eager model, where each layer boundary is a potential synchronization point. swift import MLX import MLXRandom // Lazy — no compute happens here let weights = MLXArray converting: loadedFloatArray let input = MLXArray tokenIds, dtype: .int32 // Compute graph is built, then evaluated in one fused pass let logits = model input, weights MLX.eval logits Step 2: Load Your Fine-Tuned Weights Convert your checkpoint to .safetensors once using the mlx-lm Python toolchain. Then load directly — no per-device compilation, no format migration between chip generations. swift import Transformers import MLX // Tokenizer loaded via swift-transformers let tokenizer = try await AutoTokenizer.from pretrained: "your-org/your-finetuned-model" // Weights loaded from locally converted .safetensors checkpoint let weightsURL = localCheckpointURL.appendingPathComponent "weights.safetensors" let arrays = try MLX.loadArrays url: weightsURL // see mlx-swift docs for current API The same checkpoint loads on M1, M2, and M3 without changes. That portability compounds over time when you are shipping model updates from a server. Step 3: Wrap Your KV-Cache in a Swift 6 Actor Autoregressive generation needs KV-cache management to avoid redundant computation. Here is the minimal setup to get this working safely with Swift 6 concurrency: swift actor KVCache { private var keys: MLXArray = private var values: MLXArray = func append key: MLXArray, value: MLXArray { keys.append key values.append value } func concatenated - MLXArray, MLXArray { return MLX.concatenated keys, axis: 1 , MLX.concatenated values, axis: 1 } } The actor boundary gives you Sendable conformance and data-race safety for free. The isolation cost is negligible against generation latency. Benchmark: MLX vs CoreML on M-Series These figures are representative of community benchmarks on M2 Pro macOS 14.x , Mistral-7B-Instruct at 4-bit quantization, single-sequence generation. Treat them as directional — your results will vary by chip SKU and thermal state. | Metric | CoreML ANE | MLX Metal GPU | |---|---|---| | First-load compile 3B model | 15–45s | 0s | | Time-to-first-token 4-bit quant | ~120ms | ~60–80ms | | Tokens/sec 7B 4-bit, sustained | ~25 tok/s | ~40–55 tok/s | | Fine-tune iteration cycle | Recompile required | Reload weights only | Gotchas iOS memory budget will stop you cold. A 7B model at 4-bit quantization needs approximately 4GB of RAM. The iPhone 15 Pro tops out at 8GB — leaving almost no headroom alongside the OS and app stack. MLX Swift is the right call for macOS apps and developer tooling. For iPhone, look at sub-2B models with CoreML or llama.cpp via C interop. Thermal throttling on iPhone is aggressive. MLX runs GPU-primary, trading ANE efficiency for flexibility. On sustained generation workloads, battery-sensitive iOS apps will feel this in ways a MacBook will not. The API surface is evolving fast. The docs do not always reflect current method names in mlx-swift and swift-transformers. Pin your versions explicitly and check the respective repositories before assuming any snippet compiles as-is. Profile time-to-first-token, not throughput. For interactive applications, TTFT is the user-perceived metric that matters. This is where MLX's lazy graph evaluation delivers its most visible win over CoreML's eager dispatch. When CoreML Still Wins MLX is not a universal replacement. CoreML retains the edge when your model ships statically with the app, when you need ANE efficiency for battery-sensitive sustained inference, or when you are targeting older A-series chips. The right production answer is usually both — CoreML for static, battery-sensitive workloads; MLX for dynamic fine-tuned models updating from a server on macOS. Conclusion Three things to take away: 1. Convert weights to .safetensors via mlx-lm once. It eliminates the CoreML compilation bottleneck from your iteration cycle and keeps your checkpoint portable across chip generations. 2. Wrap your KV-cache in a Swift 6 actor. Data-race safety at negligible runtime cost. 3. Benchmark time-to-first-token before committing to an inference backend. It is the metric that matters for interactive use, and it is where MLX's lazy evaluation delivers the most visible win. --- Resources: - mlx-swift on GitHub https://github.com/ml-explore/mlx-swift - swift-transformers on GitHub https://github.com/huggingface/swift-transformers - mlx-lm Python toolchain https://github.com/ml-explore/mlx-examples/tree/main/llms