# 10x Faster Inference on Edge Deployments > Source: > Published: 2026-07-27 18:14:23+00:00 # 10x faster inference on edge deployments The current software (specifically software-as-a-service) landscape heavily leans into cloud-based machine-learning services, whether that be for tools like CapCut or for tools that do one thing, like a subtitle generator online. This is easily the easiest way to get ML models into production since you only have to think about your own server infrastructure and the user has minimal setup. It’s understandable why engineering teams choose the cloud-based approach, but the simplest solution isn’t always the best, you might: - lose control over your data; - wait in a queue; - increase latency; - increase cost; All of these are valid reasons to turn away from cloud services and try something else instead. ## How we deploy to edge devices We at PostSlate are building next-generation video editing tools that use on-device machine-learning, efficient pipelining and smart UX to make video editing faster, more efficient and more reliable while retaining creative control for the user. Since I just said that we use on-device machine-learning, that means we have to deal with a huge variety of hardware configurations. Whenever you ship ML models, you should most likely try and offload to the GPU, since it’s more optimized for inference. This is why it is incredibly important to benchmark, test and optimize our models for a wide range of hardware, not just NVIDIA CUDA. Although NVIDIA discrete GPUs are common, a large share of users — especially on laptops — run AMD or Intel integrated graphics, and Apple Silicon on macOS. None of these are compatible with CUDA (ZLUDA offers a partial AMD compatibility shim, but it doesn’t cover Intel or Apple Silicon), so we have to find a cross-vendor solution. For our use case we always try to deploy models using [ncnn](https://github.com/tencent/ncnn), a high-performance neural network inference framework that provides a Vulkan GPU backend as well as a CPU backend. Vulkan is an open-standard cross-vendor compute API that runs on any modern GPU, drivers exist for every major vendor, so we avoid any dependency on proprietary CUDA infrastructure. Unfortunately, not every model converts cleanly to ncnn: some ops or graph patterns (such as rotary position embeddings in transformer models) are mis-lowered by the converter and require manual graph rewrites. For models where a clean conversion isn’t achievable, we fall back to ONNX Runtime directly. But the benefits are there: we’ve seen **inference speedups of up to 10x** when switching to ncnn Vulkan, for example, ArcFace R50 (face embedding) goes from 30 ms on ONNX CPU to 3 ms on ncnn with a 4070 (fp16, Vulkan), and SCRFD face detection drops from 25 ms to 2.5 ms. ncnn’s fp16 weight storage also cuts binary size by roughly 50%: ArcFace goes from 174 MB (ONNX fp32) to 87 MB (ncnn fp16). It’s worth being precise about where the speedup comes from: it is entirely the Vulkan GPU. On CPU alone, ncnn is roughly on par with ONNX Runtime, around 1.2× faster at best for these models, and actually slower for detection models like SCRFD where ORT has more finely-tuned CPU kernels. If a user has no GPU, the advantage over ONNX is small, but in situations where you don’t want to force the user to download heavy drivers, ncnn Vulkan is a great approach. Another framework worth mentioning is Apache’s [TVM](https://github.com/apache/tvm), a high-performance inference compiler with broad hardware support, we haven’t benchmarked it yet but it’s on our radar. ## Conclusion On-device inference isn’t without tradeoffs, it requires careful benchmarking across hardware, manual model conversion work, and accepting that some models need to stay on ONNX Runtime when conversion quality is insufficient. But the payoff is real: potentially faster inference, half the model footprint on disk, no cloud dependency, and full data privacy. For a video editing tool where users expect near-realtime feedback, those gains matter more than the convenience of offloading to a server.