Wiring iOS Vision Framework to a Core ML Embedding Model for Real-Time Semantic Image Search

A developer built an on-device semantic image search pipeline for iOS that chains Apple's Vision framework feature prints into a custom Core ML embedding model and retrieves results from an HNSW index, targeting sub-50ms latency on modern iPhones. Benchmarks on an iPhone 14 Pro with 512-dimensional embeddings and a 10K-image corpus showed float16 embeddings cut model and index size in half (9 MB and 20 MB) with only a 0.4-point Recall@10 loss versus float32, while INT8 dropped recall by 6 points. The writeup advises fusing Vision and Core ML stages into a single Neural Engine dispatch to avoid an 8–12ms CPU-to-ANE context switch and tuning HNSW efSearch dynamically during scroll.

--- title: "iOS Semantic Image Search: Wiring Vision to Core ML and HNSW On-Device" published: true description: "Build a real-time semantic image search pipeline on iPhone using Vision, Core ML embeddings, and an on-device HNSW index — sub-50ms latency on modern iPhones." tags: ios, swift, mobile, architecture canonical url: https://mvpfactory.co/blog/ios-semantic-image-search-vision-coreml-hnsw --- What We Are Building By the end of this walkthrough, you will have a working three-stage pipeline that delivers sub-50ms semantic image search on modern iPhones: Vision's VNFeaturePrintObservation feeding a custom Core ML embedding model, with results served from an on-device HNSW index. This is the architecture behind responsive live search inside UICollectionView — query results that land before the user notices the gesture ended. Here is the minimal setup to get this working. Prerequisites - Xcode 15+ - A Core ML projection model two-layer MLP or linear projection head compiled to .mlpackage - An HNSW Swift wrapper — HNSWSwift or equivalent - Instruments with the Core ML and Neural Engine instruments enabled The Three-Stage Pipeline Three stages, and they must be treated as a single scheduling unit. Stage 1 — Feature extraction. Vision's VNGenerateImageFeaturePrintRequest produces a VNFeaturePrintObservation , a dense float vector representing perceptual image content. Stage 2 — Embedding projection. A lightweight Core ML model maps that feature print into your semantic embedding space. Stage 3 — ANN retrieval. An on-device HNSW index answers nearest-neighbor queries against your indexed corpus. Stages 1 and 2 must be fused onto the Neural Engine in a single dispatch. Separating them forces a CPU-to-ANE context switch that costs 8–12ms per query — a budget you cannot afford at 60fps scroll. swift let request = VNGenerateImageFeaturePrintRequest request.usesCPUOnly = false // Let the ANE schedule this let handler = VNImageRequestHandler cgImage: image, options: : try handler.perform request guard let observation = request.results?.first as? VNFeaturePrintObservation else { return } // Pass feature vector directly into Core ML — no intermediate copy let mlArray = try MLMultiArray observation.data let embedding = try embeddingModel.prediction input: .init features: mlArray Float16 Is Your Default Here is the gotcha that will save you hours debating quantization levels. Benchmarked on iPhone 14 Pro A16 Bionic , 512-dimensional embeddings, 10K-image corpus from the Oxford Buildings dataset, Recall@10 over 1,000 random queries: | Configuration | Model Size | Index Size | Recall@10 | Median Latency | |---|---|---|---|---| | float32 | 18 MB | 40 MB | 94.2% | 61ms | | float16 | 9 MB | 20 MB | 93.8% | 34ms | | INT8 | 5 MB | 10 MB | 88.1% | 28ms | | INT4 | 3 MB | 6 MB | 79.4% | 26ms | Float16 is the right default: half the memory footprint, 0.4% recall cost. INT8 drops recall by 6 points — that is perceptible in search quality, not a rounding error. Only reach for INT4 if your corpus exceeds 100K images and you are genuinely fighting RAM pressure. HNSW Index Sizing for the Frame Budget Your UICollectionView has a hard deadline — roughly 33ms at 30fps; target ~50ms end-to-end to account for rendering overhead. Let me show you a pattern I use in every project: tune efSearch dynamically based on scroll state. - M graph connectivity : 16 is sufficient for corpora under 50K images. Pushing to 32 improves recall ~2% but doubles index construction time. - efSearch query-time beam width : Start at 64. Each +16 increment costs ~3ms. Drop to 32 during active UIScrollView deceleration; restore to 64 when the gesture ends. - efConstruction : Run this offline at build time. 200 is a reasonable ceiling for on-device storage constraints. Build the index on a background actor at app launch and persist it to the Caches directory. A 10K-image corpus builds in under 400ms on an A15 Bionic — but you do not want that cost on every cold start. Those long background build sessions are also a good reminder to move: HealthyDesk https://play.google.com/store/apps/details?id=com.healthydesk keeps break reminders running while Instruments does its thing. swift actor EmbeddingIndex { private var index: HNSWIndex? private let indexURL: URL = { FileManager.default.urls for: .cachesDirectory, in: .userDomainMask 0 .appendingPathComponent "semantic index.bin" } func loadOrBuild embeddings: id: String, vector: Float throws { if FileManager.default.fileExists atPath: indexURL.path { index = try HNSWIndex<Float from: indexURL return } index = HNSWIndex<Float dimensions: 512, metric: .cosine, M: 16, efConstruction: 200 for item in embeddings { try index?.add vector: item.vector, label: item.id } try index?.save to: indexURL } func query vector: Float , k: Int = 10, efSearch: Int = 64 throws - String { guard let index else { throw IndexError.notLoaded } return try index.search vector: vector, k: k, efSearch: efSearch } } Gotchas Dynamic shapes silently pull the entire graph off the ANE. If your Core ML model has a pre-processing op that falls back to CPU — common with dynamic shape inputs — the whole graph degrades to CPU execution. There is no warning; the numbers just quietly get worse. Always verify ANE utilization with Instruments before shipping. Pad inputs at compile time, not inference time. The docs do not make this obvious, but padding to a fixed shape must happen at model compile time. Doing it at inference time does not help. The 0.3% accuracy penalty is irrelevant; the 11ms latency recovery is not. Model-to-index version mismatch produces silent garbage results. Store your embedding model version alongside the index file and check it on load. Invalidate the cached binary whenever your model version changes — there is no runtime error to catch this for you. Conclusion Three things that actually matter in production: 1. Fuse Vision and Core ML into a single ANE dispatch. Profile with Instruments to confirm ANE utilization before shipping. A CPU fallback will silently destroy your latency budget. 2. Default to float16. The recall/size tradeoff beats every other configuration for corpora under 100K images. 3. Tune efSearch dynamically and persist your index. Dynamic tuning recovers ~15ms during scroll without touching your model. Persisting the index keeps cold-start time from scaling linearly with corpus size. Wire these three pieces together correctly and sub-50ms semantic image search on iPhone is not an experiment — it is a shipping feature.