Wiring MLX to Swift: Running Fine-Tuned Models on Apple Silicon with Zero CoreML Overhead

---
title: "Wiring MLX to Swift: Running Fine-Tuned Models on Apple Silicon with Zero CoreML Overhead"
published: true
description: "MLX Swift bindings bypass CoreML's compilation latency. Load quantized fine-tuned LLMs via swift-transformers, manage KV-cache with Swift 6 actors, and see where MLX beats CoreML on M-series chips."
tags: swift, mobile, architecture, ios
canonical_url: https://mvpfactory.co/blog/mlx-swift-fine-tuned-llms-apple-silicon
---

## What You Will Build

By the end of this tutorial, you will have a working Swift setup that loads a fine-tuned LLM directly via MLX Swift bindings — no CoreML compilation step, no ANE scheduling overhead, no app bundle bloat from pre-compiled `.mlmodelc` packages.

Let me show you a pattern I use in every project that ships on-device language models on Apple Silicon.

## Prerequisites

- macOS on Apple Silicon (M1, M2, or M3) — MLX targets macOS as its primary environment
- Xcode 15+ with Swift 6 concurrency enabled
- Python environment for weight conversion (mlx-lm toolchain)
- A fine-tuned model checkpoint exported to `.safetensors`

## Why Not Just Use CoreML?

CoreML is the default assumption for on-device inference, and for vision and classification tasks it earns that position. For autoregressive LLM generation — especially models you are iterating on — it compounds friction fast.

First-load compilation for a 3B parameter model takes 15–45 seconds on M2. Every fine-tune iteration requires a new compilation cycle. Your app bundle grows with each compiled artifact. For teams shipping model updates from a server, this is a real cost.

MLX sidesteps this entirely. It operates on a lazy computation graph backed by Metal, fusing kernels across the unified memory architecture M-series chips use.

## Step 1: Understand MLX's Lazy Evaluation Model

Operations on `MLXArray` are not executed until you call `MLX.eval()`. This is different from CoreML's eager model, where each layer boundary is a potential synchronization point.

swift

import MLX

import MLXRandom

// Lazy — no compute happens here

let weights = MLXArray(converting: loadedFloatArray)

let input = MLXArray(tokenIds, dtype: .int32)

// Compute graph is built, then evaluated in one fused pass

let logits = model(input, weights)

MLX.eval(logits)

## Step 2: Load Your Fine-Tuned Weights

Convert your checkpoint to `.safetensors` once using the mlx-lm Python toolchain. Then load directly — no per-device compilation, no format migration between chip generations.

swift

import Transformers

import MLX

// Tokenizer loaded via swift-transformers

let tokenizer = try await AutoTokenizer.from(pretrained: "your-org/your-finetuned-model")

// Weights loaded from locally converted .safetensors checkpoint

let weightsURL = localCheckpointURL.appendingPathComponent("weights.safetensors")

let arrays = try MLX.loadArrays(url: weightsURL) // see mlx-swift docs for current API

The same checkpoint loads on M1, M2, and M3 without changes. That portability compounds over time when you are shipping model updates from a server.

## Step 3: Wrap Your KV-Cache in a Swift 6 Actor

Autoregressive generation needs KV-cache management to avoid redundant computation. Here is the minimal setup to get this working safely with Swift 6 concurrency:

swift

actor KVCache {

private var keys: [MLXArray] = []

private var values: [MLXArray] = []

func append(key: MLXArray, value: MLXArray) {
    keys.append(key)
    values.append(value)
}

func concatenated() -> (MLXArray, MLXArray) {
    return (MLX.concatenated(keys, axis: 1),
            MLX.concatenated(values, axis: 1))
}

}

The actor boundary gives you `Sendable` conformance and data-race safety for free. The isolation cost is negligible against generation latency.

## Benchmark: MLX vs CoreML on M-Series

These figures are representative of community benchmarks on M2 Pro (macOS 14.x), Mistral-7B-Instruct at 4-bit quantization, single-sequence generation. Treat them as directional — your results will vary by chip SKU and thermal state.

| Metric | CoreML (ANE) | MLX (Metal GPU) |
|---|---|---|
| First-load compile (3B model) | 15–45s | 0s |
| Time-to-first-token (4-bit quant) | ~120ms | ~60–80ms |
| Tokens/sec (7B 4-bit, sustained) | ~25 tok/s | ~40–55 tok/s |
| Fine-tune iteration cycle | Recompile required | Reload weights only |

## Gotchas

**iOS memory budget will stop you cold.** A 7B model at 4-bit quantization needs approximately 4GB of RAM. The iPhone 15 Pro tops out at 8GB — leaving almost no headroom alongside the OS and app stack. MLX Swift is the right call for macOS apps and developer tooling. For iPhone, look at sub-2B models with CoreML or llama.cpp via C interop.

**Thermal throttling on iPhone is aggressive.** MLX runs GPU-primary, trading ANE efficiency for flexibility. On sustained generation workloads, battery-sensitive iOS apps will feel this in ways a MacBook will not.

**The API surface is evolving fast.** The docs do not always reflect current method names in mlx-swift and swift-transformers. Pin your versions explicitly and check the respective repositories before assuming any snippet compiles as-is.

**Profile time-to-first-token, not throughput.** For interactive applications, TTFT is the user-perceived metric that matters. This is where MLX's lazy graph evaluation delivers its most visible win over CoreML's eager dispatch.

## When CoreML Still Wins

MLX is not a universal replacement. CoreML retains the edge when your model ships statically with the app, when you need ANE efficiency for battery-sensitive sustained inference, or when you are targeting older A-series chips. The right production answer is usually both — CoreML for static, battery-sensitive workloads; MLX for dynamic fine-tuned models updating from a server on macOS.

## Conclusion

Three things to take away:

1. **Convert weights to `.safetensors` via mlx-lm once.** It eliminates the CoreML compilation bottleneck from your iteration cycle and keeps your checkpoint portable across chip generations.
2. **Wrap your KV-cache in a Swift 6 actor.** Data-race safety at negligible runtime cost.
3. **Benchmark time-to-first-token before committing to an inference backend.** It is the metric that matters for interactive use, and it is where MLX's lazy evaluation delivers the most visible win.

---

**Resources:**
- [mlx-swift on GitHub](https://github.com/ml-explore/mlx-swift)
- [swift-transformers on GitHub](https://github.com/huggingface/swift-transformers)
- [mlx-lm Python toolchain](https://github.com/ml-explore/mlx-examples/tree/main/llms)