Dissecting llama.cpp, Part 1: From GGML to GGUF, and Why llama.cpp Is Just the Wrapper

GGML, a small math library created by Georgi Gerganov, underpins llama.cpp and is designed for CPU inference on constrained hardware, using a computation-graph-first approach that pre-allocates memory to avoid repeated allocation. In a benchmark, PyTorch's mature kernels beat GGML at batch size 1024 (compute-bound), but GGML outperformed at batch size 1 (memory-bound), the shape typical in autoregressive token generation. The article explains that llama.cpp is a thin wrapper around GGML, and the GGUF file format grew out of GGML.

Most explanations of llama.cpp start with the tool itself: download this, run that command, here’s your chatbot. That skips the actual origin story, and the origin story is where the interesting engineering lives. So this piece starts one layer down, at the thing llama.cpp is built on : a small math library called GGML, and the file format that grew out of it. Only once that foundation is solid does it make sense to talk about llama.cpp itself, because by the end, you’ll see llama.cpp for what it really is: a thin, purpose-built wrapper around GGML. GGML stands for Georgi Gerganov Machine Learning , named after its creator. It began as, and still is, two things at once: We’ll come back to the format later. First, the library. Gerganov built the GGML framework because he wanted to run LLaMA on his Mac’s CPU, with none of the GPU or heavy Python dependencies the official release assumed. GGML is what made that possible, and its design reflects that goal directly: PyTorch tensors are built for training : dynamic shapes, autograd bookkeeping, dispatch across arbitrary backends, and eager execution by default each op runs immediately, with allocation and deallocation cycles as tensors come and go — modern PyTorch softens this with caching allocators, but the underlying model is still fundamentally dynamic . GGML tensors are built for inference on constrained hardware , and the struct itself shows it: struct ggml tensor { enum ggml type type; // Data type e.g., GGML TYPE F32, GGML TYPE Q4 0 struct ggml backend buffer buffer; // Pointer to hardware buffer CPU RAM or VRAM int64 t ne GGML MAX DIMS ; // Number of elements per dimension shape size t nb GGML MAX DIMS ; // Stride/bytes per dimension // Computation graph metadata enum ggml op op; // The op that produced this tensor, if any int32 t flags; // GGML TENSOR FLAG INPUT, PARAM, etc. struct ggml tensor src GGML MAX SRC ; // Input tensors to op // View optimization struct ggml tensor view src; // Non-NULL if this is a slice/view of another tensor size t view offs; void data; // Raw pointer to the weights char name GGML MAX NAME ; // e.g. "blk.0.attn v.weight"}; Two design choices stand out: Because GGML builds the full computation graph before running anything, it knows in advance exactly how much memory every intermediate tensor needs. It reserves that memory once and reuses the same addresses across every token and every batch. It can even fuse some operations together for extra efficiency. PyTorch’s eager mode, historically, worked the opposite way: allocate as you go, free when done, repeat for every operation in every forward pass. Repeated allocation and deallocation isn’t free — the allocator has to hunt for a suitably sized contiguous block each time. Modern PyTorch narrows this gap with caching allocators that hold onto memory between iterations, but GGML’s graph-first approach sidesteps the problem entirely rather than mitigating it after the fact. This isn’t just a theoretical claim — it shows up directly in a simple matrix multiplication benchmark. I ran the same 1024×1024 quantized weight matrix through both GGML Q8 0 weights, quantized-on-the-fly activations and PyTorch quantize dynamic, the same int8-weight-and-activation scheme , across two shapes: multiplying against a 1024×1024 batch, and multiplying against a single 1024×1 vector — the shape that actually occurs during one step of autoregressive token generation. The two rows aren’t just different numbers, they’re different bottlenecks entirely. At batch size 1024, every weight gets reused 1024 times before it’s evicted from cache, so the workload is compute-bound, and PyTorch’s mature, heavily-tuned kernels win. At batch size 1, each weight byte is read from memory and used exactly once — there’s no reuse to amortize, so the workload becomes bound by memory bandwidth and by how much per-call overhead each framework carries. That’s where GGML’s graph-once, allocate-once design pays off: there’s no allocator to consult, no Python-to-C++ dispatch to cross, no tensor bookkeeping to redo, just the same pre-built graph running again with the same buffers. That second row is the one that actually matters for local inference. Generating text one token at a time is a sequence of matrix-vector multiplies, not matrix-matrix ones, so the batch-size-1 result is the more representative predictor of how each framework performs during real chat-style generation, not the batch-size-1024 one. Numbers above are from a single machine, one CPU, one matrix size — the crossover point between “PyTorch wins” and “GGML wins” will shift with matrix size, thread count, and hardware. Worth keeping that caveat in mind before generalizing too far from one benchmark. This is the part that’s easy to get wrong, so it’s worth being precise. “GGML” as a file format wasn’t one static thing, it was three formats, released in sequence, each patching a problem with the last: In practice, if you go download an old model file today with “ggml” in its filename, what you’re actually holding is almost always GGJT . That’s exactly what happened in the case study below. When people say “GGML format” in the context of old llama.cpp models, they usually mean the old binary model-file format used by the GGML/llama.cpp ecosystem — not to be confused with the ggml tensor C structure used by the GGML runtime itself. Those are two different things sharing one name. The old format is essentially a binary serialization of a model’s configuration, vocabulary, and tensors . Its structure is rigid: the reader has to already know what each field means and exactly where it appears in the file, since nothing in the file itself is self-describing. At a high level, an old GGML/GGJT model file looks like: GGML/GGJT file│├── Header│ ├── magic│ └── version│├── fixed hyperparameters│├── Vocabulary│ ├── token length│ ├── token bytes│ └── token score│└── Tensors ├── tensor header ├── dimensions ├── tensor name ├── alignment padding └── raw tensor data A Header The header begins with a 4-byte magic number that identifies the file format, followed by a version field. For a GGJT v3 file, the relevant layout is: 4 bytes → magic number4 bytes → version B Hyperparameter list The seven hyperparameters are: n vocab → vocabulary sizen embd → model embedding / hidden dimensionn mult → architecture-specific feed-forward multiplen head → number of attention headsn layer → number of Transformer layersn rot → RoPE dimensionftype → integer identifier for the model's tensor/quantization format Since each hyperparameter occupies 4 bytes, the seven fields consume: 7 × 4 = 28 bytes right after the initial 8-byte magic/version portion. The old format has a fixed binary header layout, and it reads fields strictly in that predetermined order. C Vocabulary The vocabulary immediately follows the hyperparameter list. For each of the n vocab entries, the reader consumes: 4 bytes → token lengthtoken length → token bytes4 bytes → tokenizer score float32 Conceptually, the layout in memory looks like: token length token bytes score token length token bytes score token length token bytes score ... The token itself is stored as raw bytes rather than in a fixed-size field, so the parser reads the first 4 bytes to learn how many bytes the token occupies, then advances by exactly that amount. The score belongs to the tokenizer’s vocabulary — it is not a neural-network weight, logit, or attention score. D Tensor data After the vocabulary, tensors are stored one after another. Each tensor has a small binary descriptor followed by its actual data: 4 bytes → n dims4 bytes → name len4 bytes → dtype4 × n dims → dimensionsname len → tensor name0–31 bytes → alignment paddingn bytes → raw tensor data For example, a tensor might be: name = "layers.0.attention.wq.weight"shape = 4096, 4096 dtype = Q4 0 The number of elements is: n elems = dimensions 0 × dimensions 1 × ... For a quantized tensor, its storage size is determined by the quantization type: n bytes = n elems × tysize / blksize where: blksize = number of elements represented by one quantization blocktysize = number of bytes occupied by that block For Q4 0, each block represents a fixed number of weights and stores both the quantized values and the associated quantization scale. The tensor loader doesn’t separately parse those scales — it treats the whole block as raw tensor data. For standard Q4 0 quantization, blksize = 32 and tysize = 18 bytes. Each quantization scheme fixes its own blksize/tysize. Here's how it's actually defined, straight from ggml-common.h: typedef uint16 t ggml half; define QK4 0 32typedef struct { ggml half d; // delta uint8 t qs QK4 0 / 2 ; // nibbles / quants} block q4 0;static assert sizeof block q4 0 == sizeof ggml half + QK4 0 / 2, "wrong q4 0 block size/padding" ; used in ggml.c as: js static const struct ggml type traits type traits GGML TYPE COUNT = { ... GGML TYPE Q4 0 = { .type name = "q4 0", .blck size = QK4 0, .type size = sizeof block q4 0 , .is quantized = true, .to float = ggml to float t dequantize row q4 0, .from float ref = ggml from float t quantize row q4 0 ref, }, ... In the Q4 0 scheme, the tensor data is organized as blksize = 32 and type size = 16 + 2 bytes, meaning a single block holds 32 weights and occupies 18 bytes. E Alignment padding Before the tensor data begins, the parser may skip some padding bytes so that the tensor data starts at an aligned address, typically a multiple of 32 bytes in the old format. tensor metadata ↓padding ↓32-byte-aligned tensor data This alignment matters because it lets the tensor data be accessed efficiently, and it’s specifically what makes memory-mapped loading possible. F Important distinction: file format vs. GGML tensor types The old file format tells the loader how to find and interpret tensors. The ggml type enum tells it what kind of data each tensor actually contains: GGML TYPE F32GGML TYPE F16GGML TYPE Q4 0GGML TYPE Q5 0GGML TYPE Q8 0GGML TYPE Q4 K... For example: dtype = 2 ↓GGML TYPE Q4 0 The number 2 here is a type identifier, not “2-bit quantization.” I downloaded llama-2-7b.ggmlv3.q4 0.bin from a legacy TheBloke/Llama-2-7B-GGML repository. Its file size: Size : 3,791,725,184 bytes 3.53 GiB The script used to decode the following info from the file is linked here . The file begins as: ==========================================================================================HEADER offset, raw bytes, decoded field ========================================================================================== 0x000000-0x000003 74 6a 67 67 - magic number : 'tjgg' 0x000004-0x000007 03 00 00 00 - version : 3 The first four bytes are the magic bytes 74 6a 67 67, which correspond to 'tjgg'. The bytes look reversed when read as ASCII because the integer magic value is stored in little-endian order — the important point is that these bytes identify the file as the GGJT variant. The next four bytes, 03 00 00 00, decode as the integer version = 3. The seven fixed hyperparameters follow: 0x000008-0x00000b 00 7d 00 00 → n vocab : 32000 0x00000c-0x00000f 00 10 00 00 → n embd : 4096 0x000010-0x000013 00 01 00 00 → n mult : 256 0x000014-0x000017 20 00 00 00 → n head : 32 0x000018-0x00001b 20 00 00 00 → n layer : 32 0x00001c-0x00001f 80 00 00 00 → n rot : 128 0x000020-0x000023 02 00 00 00 → ftype : 2 Which implies: Vocabulary size = 32000Hidden dimension = 4096Attention heads = 32Transformer layers = 32RoPE dimension = 128Tensor format = MOSTLY Q4 0 There’s also a useful consistency check: n embd / n head = 4096 / 32 = 128 which matches n rot = 128. The important thing to notice is that these values are stored without descriptive key names anywhere in the file. The reader knows the first 4-byte integer means n vocab, the next means n embd, and so on purely because the format specification hard-codes that order. The vocabulary begins immediately after the fixed hyperparameter list, at 0x000024. For each token, the parser performs: read 4-byte token length ↓read that many token bytes to get the token value ↓read 4-byte float score ↓move to the next token For token 0: 0x000024-0x000027 05 00 00 00 → tok len : 5 0x000028-0x00002c 20 e2 81 87 20 → token value : ' ⁇ ' 0x00002d-0x000030 00 00 00 00 → score : 0.0 For token 3: 0x000041-0x000044 01 00 00 00 → tok len : 1 0x000045-0x000045 00 → token : '\x00' 0x000046-0x000049 00 00 00 00 → score : 0.0 The important observation is that the token length alone determines exactly how far the parser advances — there’s no fixed-width field to fall back on. A few more examples: php token 0 0x000024-0x000027 05 00 00 00 - tok len : 5 0x000028-0x00002c 20 e2 81 87 20 - token : ' ⁇ ' 0x00002d-0x000030 00 00 00 00 - score : 0.0 token 1 0x000031-0x000034 00 00 00 00 - tok len : 0 0x000035-0x000034 - token : '' 0x000035-0x000038 00 00 00 00 - score : 0.0 token 2 0x000039-0x00003c 00 00 00 00 - tok len : 0 0x00003d-0x00003c - token : '' 0x00003d-0x000040 00 00 00 00 - score : 0.0 token 3 0x000041-0x000044 01 00 00 00 - tok len : 1 0x000045-0x000045 00 - token : '\x00' 0x000046-0x000049 00 00 00 00 - score : 0.0 token 4 0x00004a-0x00004d 01 00 00 00 - tok len : 1 0x00004e-0x00004e 01 - token : '\x01' 0x00004f-0x000052 00 00 00 00 - score : 0.0 ... 31992 more tokens, not shown ... token 31997 0x0699c1-0x0699c4 03 00 00 00 - tok len : 3 0x0699c5-0x0699c7 e6 94 b6 - token : '收' 0x0699c8-0x0699cb 00 f4 f7 c6 - score : -31738.0 token 31998 0x0699cc-0x0699cf 03 00 00 00 - tok len : 3 0x0699d0-0x0699d2 e5 bc 98 - token : '弘' 0x0699d3-0x0699d6 00 f6 f7 c6 - score : -31739.0 token 31999 0x0699d7-0x0699da 03 00 00 00 - tok len : 3 0x0699db-0x0699dd e7 bb 99 - token : '给' 0x0699de-0x0699e1 00 f8 f7 c6 - score : -31740.0 After processing all 32,000 entries, the parser reaches the first tensor. Example tensor entries from the file: ==========================================================================================TENSORS offset, raw bytes, decoded field ========================================================================================== tensor 0 0x0699e2-0x0699e5 02 00 00 00 - n dims : 2 0x0699e6-0x0699e9 15 00 00 00 - name len: 21 0x0699ea-0x0699ed 02 00 00 00 - dtype : 2 0x0699ee-0x0699f5 00 10 00 00 00 7d 00 00 - dims : 4096, 32000 0x0699f6-0x069a0a 74 6f 6b 5f 65 6d 62 65 64 64 69 6e 67 73 2e 77 ... - name : 'tok embeddings.weight' 0x069a0b-0x069a1f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... - padding : 21 bytes aligns next field to 32-byte offset 0x069a20-0x069a31 1b 00 89 77 95 9d a9 e5 aa 69 04 67 b5 75 68 63 89 9e - data : first 18 of 73,728,000 raw bytes Q4 0, 4,096,000 block s of 32 elements ... 291 more tensors, not shown ... The first tensor begins at 0x0699e2. Its descriptor reads: n dims = 2, name len = 21 bytes, dtype = 2 Q4 0 . The two dimensions are 4096, 32000 , giving: n elems = 4096 × 32000 = 131,072,000 The parser then encounters 21 bytes of padding, moving the tensor data to the next aligned address, 0x069a20. The actual tensor data begins there. Verifying the tensor size against the quantization scheme Q4 0 : Total elements = 4096 × 32000 = 131,072,000 individual weightsTotal blocks = 131,072,000 / 32 = 4,096,000Total size = 4,096,000 × 18 = 73,728,000 bytes So every tensor in Q4 0 follows the same pattern: 2 bytes of scale, followed by 16 bytes of quantized weights, repeated per block. For this first tensor, block 0 breaks down as: I’m deliberately not decoding these specific bytes into actual float values here — that depends on endianness and exactly how the scale and 4-bit values combine, which is really a Part 3 topic once quantization itself is properly explained. The key structural point: metadata ↓padding ↓raw Q4 0 blocks The Q4 0 scales live inside those raw blocks — the GGML parser never creates a separate “scale section.” Every other tensor in the file is decoded the same way. The full structure of this file, condensed: GGJT v3 Llama-2-7B│├── Header│ ├── magic│ ├── version│ ├── n vocab = 32000│ ├── n embd = 4096│ ├── n mult = 256│ ├── n head = 32│ ├── n layer = 32│ ├── n rot = 128│ └── ftype = 2 → Q4 0│├── Vocabulary│ ├── token 0│ ├── token 1│ ├── ...│ └── token 31999│└── Tensors ├── tok embeddings.weight │ ├── descriptor │ ├── padding │ └── Q4 0 data │ ├── norm.weight │ ├── descriptor │ ├── padding │ └── F32 data │ ├── output.weight │ ├── descriptor │ ├── padding │ └── Q4 0 data │ ├── layers.0.attention.wq.weight │ ├── descriptor │ ├── padding │ └── Q4 0 data │ ├── layers.0.attention.wk.weight │ ├── descriptor │ ├── padding │ └── Q4 0 data │ └── ... 286 more tensors The important idea here is that the file is, fundamentally, just a sequential binary stream. The parser maintains a running offset and repeats: read field ↓advance offset ↓read next field ↓advance offset ↓... GGJT’s header is a fixed list of untyped hyperparameters : seven fields, in a fixed order, each an opaque 4-byte integer. That’s fine as long as every model is a LLaMA, but by mid-2023 llama.cpp needed to support Mistral, Falcon, and a growing list of architectures, each with different config needs: rotary embedding bases, grouped-query attention parameters, sliding window sizes, and more. None of that fits into “seven fixed integers.” Every time the project needed to store one more piece of information, the choices were: break the format, or bolt on a hack. Multiplied across dozens of contributors and architectures, that became unsustainable. GGJT had no concept of a typed, extensible metadata dictionary — everything had to be known and hardcoded by the loader in advance. That’s the exact problem GGUF was designed to solve. GGUF keeps GGJT’s core idea — a single, self-contained, mmap-friendly file — but replaces the fixed hyperparameter list with a typed key-value metadata store . Any number of arbitrary, named, typed fields can be added without breaking older readers. Version history each version is a small, deliberate change, not a rewrite : struct gguf file t { gguf header t header; gguf tensor info t tensor infos header.tensor count ; uint8 t padding ; // pad to nearest multiple of ALIGNMENT uint8 t tensor data ; // raw weights}; Header — 24 bytes, fixed size: php struct gguf header t { uint32 t magic; // "GGUF" - 0x47 0x47 0x55 0x46 uint32 t version; uint64 t tensor count; uint64 t metadata kv count; gguf metadata kv t metadata kv metadata kv count ;}; Each metadata entry — a typed key/value pair: struct gguf metadata kv t { gguf string t key; // hierarchical, e.g. "general.architecture" gguf metadata value type value type; gguf metadata value t value;}; Each tensor’s descriptor — separate from its data: struct gguf tensor info t { gguf string t name; // e.g. "blk.0.attn q.weight", max 64 bytes uint32 t n dimensions; // currently at most 4 uint64 t dimensions n dimensions ; ggml type type; uint64 t offset; // byte offset into tensor data, must be ALIGNMENT-aligned}; The alignment itself is just another metadata key general.alignment, default 32 — even that low-level detail is extensible rather than hardcoded. Unlike the older GGML/GGJT model-file formats, GGUF was designed to be extensible and self-describing . Instead of storing a fixed list of hyperparameters in a predetermined order, GGUF stores model metadata as typed key-value KV pairs . At a high level, a GGUF file looks like: GGUF file│├── Header│ ├── magic│ ├── version│ ├── tensor count│ └── metadata KV count│├── Metadata│ └── key-value pairs│├── Tensor Information│ └── descriptor for every tensor│├── Alignment padding│└── Tensor Data └── actual F32/F16/quantized weight bytes The GGUF header is 24 bytes : 4 bytes → magic4 bytes → version8 bytes → tensor count8 bytes → metadata KV count The four fields tell the loader how to interpret the rest of the file: Unlike the old GGML format, the header itself doesn’t contain a fixed list like n vocab, n embd, n layer. Those values now live in the metadata section as named KV pairs. The main improvement in GGUF is its typed metadata system . Each metadata entry conceptually contains: ┌─────────────────────────────┐│ Key length │├─────────────────────────────┤│ Key │├─────────────────────────────┤│ Value type │├─────────────────────────────┤│ Value │└─────────────────────────────┘ The important difference from the old GGML format is that metadata is now named and typed . Instead of the reader assuming: first integer → n vocabsecond integer → n embdthird integer → n mult... GGUF explicitly tells the reader: "llama.embedding length" → UINT32 → 4096 KEY VALUE TYPE VALUE This makes the format much easier to extend. New architecture-specific parameters can be added as new KV entries without redesigning the header at all. Unlike the fixed-size header, the metadata section’s size is dynamic — it depends entirely on how many KV pairs there are and how large their keys and values are. This section doesn’t contain the actual model weights — it contains a descriptor for every tensor, telling the loader how to find and interpret that tensor’s data. Conceptually, each tensor descriptor contains: name, n dimensions, dimensions, type, offset. The important field here is offset. It tells the loader where the tensor's data begins, relative to the start of the tensor-data blob — not relative to the start of the file. This separation matters: the tensor-information section works like a directory or index, while the tensor-data section holds the actual weights. The final major section holds the actual numerical tensor data. For a quantized tensor such as Q4 0, the raw bytes contain the complete quantized representation, including whatever the quantization format needs like scales . The tensor descriptor tells the runtime: What is this data? ↓name + shape + type + offset ↓Where is it? ↓offset into tensor-data blob ↓Read the corresponding raw bytes Before the tensor-data blob begins, GGUF can insert alignment padding so the data region starts aligned, typically to a 32-byte boundary. The GGUF counterpart of the same model, llama-2-7b.Q4 0.gguf downloaded from — TheBloke/Llama-2-7B-GGUF : Size : 3,825,807,040 bytes 3.56 GiB The script used to decode the following info from the file is linked here .. The first 24 bytes are: ==========================================================================================HEADER offset, raw bytes, decoded field ========================================================================================== 0x000000-0x000003 47 47 55 46 → magic : GGUF 0x000004-0x000007 02 00 00 00 → version : 2 0x000008-0x00000f 23 01 00 00 00 00 00 00 → tensor count : 291 0x000010-0x000017 13 00 00 00 00 00 00 00 → metadata kv count : 19 So immediately, before reading anything else, the loader already knows: This is a GGUF file ↓GGUF version = 2 ↓291 tensors are described later ↓19 metadata KV pairs must be read The metadata starts immediately after the 24-byte header: ==========================================================================================METADATA 19 key-value pairs offset, raw bytes, decoded field ========================================================================================== kv 0 0x000018-0x00001f 14 00 00 00 00 00 00 00 - key len : 20 0x000020-0x000033 67 65 6e 65 72 61 6c 2e 61 72 63 68 69 74 65 63 ... - key : 'general.architecture' 0x000034-0x000037 08 00 00 00 - value type : 8 value type 8 = STRING 0x000038-0x00003f 05 00 00 00 00 00 00 00 - value len : 5 0x000040-0x000044 6c 6c 61 6d 61 - value : 'llama' kv 1 0x000045-0x00004c 0c 00 00 00 00 00 00 00 - key len : 12 0x00004d-0x000058 67 65 6e 65 72 61 6c 2e 6e 61 6d 65 - key : 'general.name' 0x000059-0x00005c 08 00 00 00 - value type : 8 value type 8 = STRING 0x00005d-0x000064 08 00 00 00 00 00 00 00 - value len : 8 0x000065-0x00006c 4c 4c 61 4d 41 20 76 32 - value : 'LLaMA v2' kv 2 0x00006d-0x000074 14 00 00 00 00 00 00 00 - key len : 20 0x000075-0x000088 6c 6c 61 6d 61 2e 63 6f 6e 74 65 78 74 5f 6c 65 ... - key : 'llama.context length' 0x000089-0x00008c 04 00 00 00 - value type : 4 value type 4 = UINT32 0x00008d-0x000090 00 10 00 00 - value : 4096 kv 3 0x000091-0x000098 16 00 00 00 00 00 00 00 - key len : 22 0x000099-0x0000ae 6c 6c 61 6d 61 2e 65 6d 62 65 64 64 69 6e 67 5f ... - key : 'llama.embedding length' 0x0000af-0x0000b2 04 00 00 00 - value type : 4 value type 4 = UINT32 0x0000b3-0x0000b6 00 10 00 00 - value : 4096 kv 4 0x0000b7-0x0000be 11 00 00 00 00 00 00 00 - key len : 17 0x0000bf-0x0000cf 6c 6c 61 6d 61 2e 62 6c 6f 63 6b 5f 63 6f 75 6e ... - key : 'llama.block count' 0x0000d0-0x0000d3 04 00 00 00 - value type : 4 value type 4 = UINT32 0x0000d4-0x0000d7 20 00 00 00 - value : 32 kv 5 0x0000d8-0x0000df 19 00 00 00 00 00 00 00 - key len : 25 0x0000e0-0x0000f8 6c 6c 61 6d 61 2e 66 65 65 64 5f 66 6f 72 77 61 ... - key : 'llama.feed forward length' 0x0000f9-0x0000fc 04 00 00 00 - value type : 4 value type 4 = UINT32 0x0000fd-0x000100 00 2b 00 00 - value : 11008 kv 6 0x000101-0x000108 1a 00 00 00 00 00 00 00 - key len : 26 0x000109-0x000122 6c 6c 61 6d 61 2e 72 6f 70 65 2e 64 69 6d 65 6e ... - key : 'llama.rope.dimension count' 0x000123-0x000126 04 00 00 00 - value type : 4 value type 4 = UINT32 0x000127-0x00012a 80 00 00 00 - value : 128 kv 7 0x00012b-0x000132 1a 00 00 00 00 00 00 00 - key len : 26 0x000133-0x00014c 6c 6c 61 6d 61 2e 61 74 74 65 6e 74 69 6f 6e 2e ... - key : 'llama.attention.head count' 0x00014d-0x000150 04 00 00 00 - value type : 4 value type 4 = UINT32 0x000151-0x000154 20 00 00 00 - value : 32 ... 11 more KV pairs, not shown ... There are 19 KV pairs of metadata total. Taking the first one apart field by field: 0x000018-0x00001f → key length : 20 0x000020-0x000033 → key : 'general.architecture' 0x000034-0x000037 → value type : 8 STRING 0x000038-0x00003f → value length: 5 0x000040-0x000044 → value : 'llama' So the raw bytes decode to: general.architecture → STRING → "llama". The next KV pair is general.name → STRING → "LLaMA v2", then llama.context length → UINT32 → 4096, and so on through the remaining pairs. ==========================================================================================TENSOR INFO 291 tensors offset, raw bytes, decoded field ========================================================================================== Note: this is just the DESCRIPTOR list -- name, shape, type, and offset into the tensor data blob. The raw weights themselves live later in the file, all together see next section . tensor 0 0x0b0b29-0x0b0b30 11 00 00 00 00 00 00 00 - name len : 17 0x0b0b31-0x0b0b41 74 6f 6b 65 6e 5f 65 6d 62 64 2e 77 65 69 67 68 ... - name : 'token embd.weight' 0x0b0b42-0x0b0b45 02 00 00 00 - n dims : 2 0x0b0b46-0x0b0b55 00 10 00 00 00 00 00 00 00 7d 00 00 00 00 00 00 - dims : 4096, 32000 0x0b0b56-0x0b0b59 02 00 00 00 - type : 2 0x0b0b5a-0x0b0b61 00 00 00 00 00 00 00 00 - offset : 0 type 2 = Q4 0, 131,072,000 elements, ~73,728,000 bytes of tensor data at data-blob offset 0 tensor 1 0x0b0b62-0x0b0b69 16 00 00 00 00 00 00 00 - name len : 22 0x0b0b6a-0x0b0b7f 62 6c 6b 2e 30 2e 61 74 74 6e 5f 6e 6f 72 6d 2e ... - name : 'blk.0.attn norm.weight' 0x0b0b80-0x0b0b83 01 00 00 00 - n dims : 1 0x0b0b84-0x0b0b8b 00 10 00 00 00 00 00 00 - dims : 4096, 0x0b0b8c-0x0b0b8f 00 00 00 00 - type : 0 0x0b0b90-0x0b0b97 00 00 65 04 00 00 00 00 - offset : 73728000 type 0 = F32, 4,096 elements, ~16,384 bytes of tensor data at data-blob offset 73,728,000 tensor 2 0x0b0b98-0x0b0b9f 15 00 00 00 00 00 00 00 - name len : 21 0x0b0ba0-0x0b0bb4 62 6c 6b 2e 30 2e 66 66 6e 5f 64 6f 77 6e 2e 77 ... - name : 'blk.0.ffn down.weight' 0x0b0bb5-0x0b0bb8 02 00 00 00 - n dims : 2 0x0b0bb9-0x0b0bc8 00 2b 00 00 00 00 00 00 00 10 00 00 00 00 00 00 - dims : 11008, 4096 0x0b0bc9-0x0b0bcc 02 00 00 00 - type : 2 0x0b0bcd-0x0b0bd4 00 40 65 04 00 00 00 00 - offset : 73744384 type 2 = Q4 0, 45,088,768 elements, ~25,362,432 bytes of tensor data at data-blob offset 73,744,384 tensor 3 0x0b0bd5-0x0b0bdc 15 00 00 00 00 00 00 00 - name len : 21 0x0b0bdd-0x0b0bf1 62 6c 6b 2e 30 2e 66 66 6e 5f 67 61 74 65 2e 77 ... - name : 'blk.0.ffn gate.weight' 0x0b0bf2-0x0b0bf5 02 00 00 00 - n dims : 2 0x0b0bf6-0x0b0c05 00 10 00 00 00 00 00 00 00 2b 00 00 00 00 00 00 - dims : 4096, 11008 0x0b0c06-0x0b0c09 02 00 00 00 - type : 2 0x0b0c0a-0x0b0c11 00 40 e8 05 00 00 00 00 - offset : 99106816 type 2 = Q4 0, 45,088,768 elements, ~25,362,432 bytes of tensor data at data-blob offset 99,106,816 tensor 4 0x0b0c12-0x0b0c19 13 00 00 00 00 00 00 00 - name len : 19 0x0b0c1a-0x0b0c2c 62 6c 6b 2e 30 2e 66 66 6e 5f 75 70 2e 77 65 69 ... - name : 'blk.0.ffn up.weight' 0x0b0c2d-0x0b0c30 02 00 00 00 - n dims : 2 0x0b0c31-0x0b0c40 00 10 00 00 00 00 00 00 00 2b 00 00 00 00 00 00 - dims : 4096, 11008 0x0b0c41-0x0b0c44 02 00 00 00 - type : 2 0x0b0c45-0x0b0c4c 00 40 6b 07 00 00 00 00 - offset : 124469248 type 2 = Q4 0, 45,088,768 elements, ~25,362,432 bytes of tensor data at data-blob offset 124,469,248 tensor 5 0x0b0c4d-0x0b0c54 15 00 00 00 00 00 00 00 - name len : 21 0x0b0c55-0x0b0c69 62 6c 6b 2e 30 2e 66 66 6e 5f 6e 6f 72 6d 2e 77 ... - name : 'blk.0.ffn norm.weight' 0x0b0c6a-0x0b0c6d 01 00 00 00 - n dims : 1 0x0b0c6e-0x0b0c75 00 10 00 00 00 00 00 00 - dims : 4096, 0x0b0c76-0x0b0c79 00 00 00 00 - type : 0 0x0b0c7a-0x0b0c81 00 40 ee 08 00 00 00 00 - offset : 149831680 type 0 = F32, 4,096 elements, ~16,384 bytes of tensor data at data-blob offset 149,831,680 tensor 6 0x0b0c82-0x0b0c89 13 00 00 00 00 00 00 00 - name len : 19 0x0b0c8a-0x0b0c9c 62 6c 6b 2e 30 2e 61 74 74 6e 5f 6b 2e 77 65 69 ... - name : 'blk.0.attn k.weight' 0x0b0c9d-0x0b0ca0 02 00 00 00 - n dims : 2 0x0b0ca1-0x0b0cb0 00 10 00 00 00 00 00 00 00 10 00 00 00 00 00 00 - dims : 4096, 4096 0x0b0cb1-0x0b0cb4 02 00 00 00 - type : 2 0x0b0cb5-0x0b0cbc 00 80 ee 08 00 00 00 00 - offset : 149848064 type 2 = Q4 0, 16,777,216 elements, ~9,437,184 bytes of tensor data at data-blob offset 149,848,064 tensor 7 0x0b0cbd-0x0b0cc4 18 00 00 00 00 00 00 00 - name len : 24 0x0b0cc5-0x0b0cdc 62 6c 6b 2e 30 2e 61 74 74 6e 5f 6f 75 74 70 75 ... - name : 'blk.0.attn output.weight' 0x0b0cdd-0x0b0ce0 02 00 00 00 - n dims : 2 0x0b0ce1-0x0b0cf0 00 10 00 00 00 00 00 00 00 10 00 00 00 00 00 00 - dims : 4096, 4096 0x0b0cf1-0x0b0cf4 02 00 00 00 - type : 2 0x0b0cf5-0x0b0cfc 00 80 7e 09 00 00 00 00 - offset : 159285248 type 2 = Q4 0, 16,777,216 elements, ~9,437,184 bytes of tensor data at data-blob offset 159,285,248 ... 283 more tensor descriptors, not shown ... The tensor-information section describes all 291 tensors. Taking the first one apart: 0x0b0b29-0x0b0b30 → name length : 17 0x0b0b31-0x0b0b41 → name : 'token embd.weight' 0x0b0b42-0x0b0b45 → n dims : 2 0x0b0b46-0x0b0b55 → dimensions : 4096, 32000 0x0b0b56-0x0b0b59 → type : 2 = Q4 0 0x0b0b5a-0x0b0b61 → offset : 0 This tensor holds 4096 × 32000 = 131,072,000 elements, and since it's Q4 0, its data occupies roughly 73,728,000 bytes. The next descriptor, tensor 1 , reads: name : blk.0.attn norm.weightn dims : 1dimensions : 4096, type : 0 = F32offset : 73,728,000 This tells the runtime that blk.0.attn norm.weight's data begins 73,728,000 bytes into the tensor-data blob, and since it's F32, it needs 4096 × 4 = 16,384 bytes. So the tensor-information section is, in effect, a directory: it tells the runtime what every tensor is and exactly where to find its data. ==========================================================================================TENSOR DATA offset, raw bytes, decoded field ========================================================================================== 0x0b4eaf-0x0b4ebf 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ... - padding : 17 bytes aligns tensor data blob to 32-byte offset 0x0b4ec0-0x0b4ecf 1b 00 89 77 95 9d a9 e5 aa 69 04 67 b5 75 68 63 - token embd.weight: first 16 of 73,728,000 bytes Q4 0 0x4704ec0-0x4704ecf 00 00 f3 3c 00 00 5f 3c 00 00 01 3b 00 00 47 3c - blk.0.attn norm.weight: first 16 of 16,384 bytes F32 0x4708ec0-0x4708ecf d0 1d 58 65 d9 d6 6c cb b8 4a 6e b9 85 79 96 07 - blk.0.ffn down.weight: first 16 of 25,362,432 bytes Q4 0 After all 291 tensor descriptors, the file reaches the actual tensor-data region. There’s first some alignment padding at 0x0b4eaf-0x0b4ebf 17 bytes , and the tensor-data blob itself starts at absolute file offset 0x0b4ec0. This distinction matters: the offsets stored in the tensor descriptors are relative to the start of this blob , not absolute file offsets. For token embd.weight descriptor offset = 0 , the absolute file location is: tensor data start + tensor offset = 0x0b4ec0 + 0 = 0x0b4ec0 Its first bytes, 1b 00 89 77 95 9d a9 e5 aa 69 04 67 b5 75 68 63, are the beginning of the raw Q4 0 representation of token embd.weight. For blk.0.attn norm.weight, the descriptor says offset = 73,728,000, so its data begins at tensor data start + 73,728,000, and the first bytes there — 00 00 f3 3c 00 00 5f 3c 00 00 01 3b 00 00 47 3c — are ordinary 32-bit floats, since this tensor is F32. For blk.0.ffn down.weight, offset = 73,744,384, so its data begins at tensor data start + 73,744,384, and the bytes there are Q4 0 data, laid out exactly as described earlier: 2 bytes of scale followed by the packed quantized weights. GGUF Llama-2 file│├── HEADER│ ││ ├── magic = "GGUF"│ ├── version = 2│ ├── tensor count = 291│ └── metadata count = 19│├── METADATA│ ││ ├── general.architecture → "llama"│ ├── general.name → "LLaMA v2"│ ├── llama.context length → 4096│ ├── llama.embedding length → 4096│ ├── llama.block count → 32│ ├── llama.feed forward length → 11008│ ├── llama.rope.dimension count → 128│ ├── llama.attention.head count → 32│ └── ... 11 more KV pairs│├── TENSOR INFORMATION│ ││ ├── token embd.weight│ │ shape = 4096, 32000 │ │ type = Q4 0│ │ offset = 0│ ││ ├── blk.0.attn norm.weight│ │ shape = 4096, │ │ type = F32│ │ offset = 73,728,000│ ││ ├── blk.0.ffn down.weight│ │ shape = 11008, 4096 │ │ type = Q4 0│ │ offset = 73,744,384│ ││ ├── blk.0.ffn gate.weight│ ├── blk.0.ffn up.weight│ ├── blk.0.ffn norm.weight│ ├── blk.0.attn k.weight│ ├── blk.0.attn output.weight│ └── ... 283 more tensor descriptors│├── ALIGNMENT PADDING│ └── 17 bytes│└── TENSOR DATA │ ├── token embd.weight │ Q4 0 raw bytes │ ├── blk.0.attn norm.weight │ F32 raw bytes │ ├── blk.0.ffn down.weight │ Q4 0 raw bytes │ ├── blk.0.ffn gate.weight │ Q4 0 raw bytes │ └── ... remaining tensor data Same model, same q4 0 quantization, yet the GGUF file is about 34 MB bigger 3,825,807,040 vs. 3,791,725,184 bytes . The quantized tensor data itself is unchanged; what grew is the metadata . GGJT's header held seven raw integers and a bare list of token, score pairs. GGUF's key-value store carries the same information plus a lot more that GGJT had no room for: explicit architecture identifiers, human-readable names and licensing fields, full tokenizer metadata merges, token types, special-token IDs , and per-model fields specific to the architecture. That richness is precisely the extensibility GGJT couldn't offer, and it costs a few extra megabytes to store. Everything discussed above, GGML the tensor library, and GGML/GGMF/GGJT/GGUF the file formats, is the engine and the fuel tank . None of it has any idea what a “chat template” is, what temperature or top-p sampling mean, or how to turn a stream of token IDs into readable text. That’s not GGML’s job. llama.cpp is the layer that turns raw tensor math into an actual LLM you can talk to. It’s the orchestration code sitting on top of GGML: it knows the shape of a transformer forward pass, handles tokenization, applies prompt templates, manages the KV cache and context window, and runs the sampling logic temperature, top-p, repetition penalties that turns raw logits into the next token. None of that is GGML’s concern, GGML just multiplies matrices fast and hands back numbers. Put plainly: This is also why GGUF isn’t a “llama.cpp format,” it’s the GGML project’s format. That’s why other GGML-based tools, like whisper.cpp, can read GGUF files too: the format belongs to the engine, not to any one application built on top of it. That’s the full arc: GGML the library, GGML the format and its GGMF/GGJT detour , how GGUF fixed what GGJT couldn’t extend, and where llama.cpp actually sits relative to all of it, a wrapper, not the engine. Part 2 goes one level deeper into the ecosystem around the engine: what .safetensors is the format Hugging Face exposes most LLMs in today and how llama.cpp converts that, along with LoRA adapters, into .gguf. Dissecting llama.cpp, Part 1: From GGML to GGUF, and Why llama.cpp Is Just the Wrapper https://pub.towardsai.net/dissecting-llama-cpp-part-1-from-ggml-to-gguf-and-why-llama-cpp-is-just-the-wrapper-f8ab5f5a2c0b was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.