TIL: What’s actually different about GGUF and ONNX?

A technical comparison explains that GGUF stores model weights and metadata for llama.cpp, which implements the architecture itself, while ONNX stores a computation graph that any compatible runtime can execute. The article notes GGUF's limitation is runtime architecture support, whereas ONNX's challenges include export compatibility and operator support, with ONNX Runtime offering hardware-specific execution providers.

TIL: What’s actually different about GGUF and ONNX? I’ve been looking into running models on constrained hardware and wanted to get clearer on the difference between GGUF and ONNX. I was particularly interested in what each format stores, how the runtime uses it, and what I’d need to check before choosing one. What is GGUF? GGUF is a binary file format for storing models for inference. It’s widely used in the GGML ecosystem, including llama.cpp, and packages model weights alongside metadata. For a language model, that metadata can include the architecture, layer counts, tokeniser information and a chat template. When llama.cpp loads a GGUF file, it reads those weights and metadata and builds the computation using its own implementation of the model’s architecture. The attention layers, feed-forward operations and other architectural details are implemented in the runtime’s code. This is convenient for supported models because the weights and associated metadata can be distributed together. The limitation is architecture support: converting a model to GGUF won’t make it runnable in an engine that hasn’t implemented that architecture. What is ONNX? ONNX stands for Open Neural Network Exchange . It’s an open specification for representing machine-learning models across frameworks and runtimes. An ONNX model includes a computation graph, its parameters and metadata. Large parameter tensors can also be stored in external files. The graph describes the operations and their connections. For example, it might specify that an input is multiplied by a weight matrix, a bias is added, and the result passes through an activation function. A compatible runtime executes that graph using implementations of the required operators. This gives ONNX more flexibility for representing different architectures. A runtime can execute an unfamiliar architecture if the exported graph uses operators, versions and data types it supports. Export compatibility and custom operators can still cause problems, so I’d check those early. ONNX Runtime is one engine that executes ONNX models. Its execution providers connect it to different hardware acceleration libraries. Whether an operation runs on a particular accelerator depends on the provider’s support for it. How they compare The runtime examples here are llama.cpp for GGUF and ONNX Runtime for ONNX. | Aspect | GGUF | ONNX | |---|---|---| | What it stores | Model weights and metadata, including architecture and tokeniser information. | A computation graph, parameters and metadata. | | How computation is defined | The runtime implements the model architecture and builds its computation. | The exported graph specifies the operations and their connections. | | Main advantage | Convenient packaging for supported models, with established local LLM and low-bit quantisation tooling. | Graph-based interoperability across frameworks and runtimes, with hardware-specific execution options. | | Main drawback | Each model architecture needs support in the runtime. | Export compatibility, operator versions and execution-provider support need checking. | | Quantisation | Supports floating-point and quantised tensors. llama.cpp provides several low-bit quantisation options. | Supports quantised models. Available schemes depend on the tooling, operators and backend. | | Hardware execution | llama.cpp supports CPU, GPU and hybrid CPU/GPU inference. | ONNX Runtime supports CPUs, GPUs and other accelerators through execution providers. | What about quantisation? GGUF files can contain floating-point weights as well as quantised ones. In a filename such as model-Q4 K M.gguf , Q4 K M identifies the quantisation configuration and .gguf identifies the file format. ONNX models can also be quantised. ONNX Runtime supports 8-bit quantisation and 4-bit weight-only quantisation for certain operators. The available options depend on the model and execution backend, and quantisation’s effects on speed and accuracy need testing. Which would I use? For a local LLM on a machine with limited GPU memory, I’d start by checking llama.cpp’s support for the model and the available GGUF quantisations. Its CPU/GPU offloading would be useful to investigate when the model doesn’t fit entirely in VRAM. I’d investigate ONNX when I need to export a model from a training framework into a separate application runtime, particularly when a specific hardware execution provider is part of the deployment. I’d check the exported graph against that provider before committing to it. ONNX is also an option for LLMs. ONNX Runtime’s GenAI library provides functionality such as tokenisation, the generation loop, sampling and KV-cache management. For the LLM deployment I’m considering, I’d compare the same model at similar quantisation and output quality, then measure time to first token, generation speed and peak memory use at the context length the application needs. No spam, no sharing to third party. Only you and me.