GGUF: The File Format That Made Local LLMs Practical

If you have ever downloaded a model from Hugging Face and run it locally with llama.cpp, Ollama, LM Studio, or GPT4All, you used a GGUF file. It is the format that turned local LLM inference from a research curiosity into something anyone with a laptop can do.

But most people treat GGUF as a black box: download the Q4_K_M variant, load it, get tokens. This post opens the box and walks through the actual binary layout — header, metadata key-value pairs, tensor descriptors, alignment padding, and the weight data itself. The interactive diagram below lets you step through each section.

GGUF was created by Georgi Gerganov, the developer of llama.cpp and GGML. It replaced the earlier GGML/GGMF/GGJT formats, which had ambiguous layouts and required external files for tokenizer and chat template metadata. GGUF fixed all of that with one design rule: the file must contain everything needed to load the model. No sidecar files. No guessing.

// GGUF header (24 bytes)
uint32_t magic;      // 0x46554747 ("GGUF")
uint32_t version;    // 3
uint64_t n_tensors;  // 291
uint64_t n_kv;       // 26

Why GGUF Won

Before GGUF, the situation was messy. The original GGML format had no metadata — the loader had to guess the architecture from the file size. GGMF added a version field but still required external tokenizer files. GGJT improved alignment but the metadata was still a fixed-order list of untyped values. Adding a new field meant breaking every loader.

GGUF fixed this with one decision: use string-keyed, typed key-value pairs for all metadata. Adding a new field (say, llama.rope_freq_base) does not break old loaders — they simply skip keys they do not recognize. This is why GGUF has survived through Llama 1, Llama 2, Llama 3, Mistral, Mixtral, Qwen, Gemma, Phi, Command-R, and dozens of other architectures without format changes.

GGUF vs SafeTensors

SafeTensors is not a competitor to GGUF — they solve different problems. SafeTensors is a safe, fast tensor serialization format designed for PyTorch/JAX/TensorFlow interop. It stores tensors and a small JSON header. It does not store tokenizer vocabularies, chat templates, model architecture metadata, or quantization format details.

GGUF is a deployment format. It is designed for the last step: you have a quantized model and you want to run it on a CPU or GPU with llama.cpp, Ollama, or CKE. The file must be self-contained because the user should not need to download three more files to make it work.

The Quantization Zoo

GGUF supports over 20 quantization types, from FP32 down to 1-bit ternary. The most commonly used formats for local inference:

The _M suffix in Q4_K_M means "mixed quantization": attention layers use Q4_K (4.5 bpw) but FFN layers use Q6_K (6.5 bpw) because FFN weights are more sensitive to quantization error. This per-layer format mixing is declared in the tensor info section — each tensor descriptor carries its own quantization type.

// Q4_K super-block header
typedef struct {
  ggml_half d;     // super-block scale
  ggml_half dmin;  // super-block minimum
  // ... sub-block scales, mins, nibbles
} block_q4_K;

How CKE Reads GGUF

C-Kernel-Engine uses GGUF as its primary weight format, but CKE does not treat GGUF as "the model" by itself. GGUF is the deployment artifact that carries the components: metadata, tokenizer rules, tensor descriptors, quantization types, and raw weight blocks. CKE still needs a template, a tensor binding map, a dimension map, and a kernel map before those components become an executable runtime.

That distinction matters. A GGUF file can say general.architecture = "qwen2", list a tensor named blk.0.attn_q.weight, and store it as Q4_K. But CKE still has to answer: which template slot does that tensor bind to, what dimensions should it have, which operation consumes it, which quantized kernel can execute it, and which generated C pointer should point at it?

The loading path:

1. Read the 24-byte header. Validate magic bytes and version.
2. Parse all metadata KV pairs. Extract architecture, tokenizer, hyperparameters.
3. Read all tensor info descriptors. Build a tensor name → offset map.
4. mmap the data section. Set up tensor pointers using offsets from step 3.
5. Match general.architecture and family-specific metadata to a CKE template.
6. Bind GGUF tensor names to template slots such as q_proj, k_proj, v_proj, gate_proj, up_proj, and down_proj.
7. Validate dimensions against the architecture contract: d_model, head_dim, n_heads, n_kv_heads, n_layers, vocab_size, and sequence/runtime limits.
8. Match each tensor's quantization type to legal CKE kernel-map candidates. A Q4_K tensor cannot accidentally run through a Q8-only path.
9. Generate IR, lower it into concrete buffer/kernel calls, and emit C that points directly into the mmap'd GGUF tensor data.

The important detail: CKE does not convert GGUF weights into a different private format before doing useful work. The goal is to operate on the quantized blocks directly. The mmap'd data region becomes the source of truth for weight pointers, and CKE's job is to bind those pointers into a template-defined circuit cleanly enough that generated C can execute it and parity/profiling can verify it.

GGUF Is The Component Box; CKE Builds The Circuit

The best mental model is electronics. GGUF is the box of labeled components. The CKE template is the circuit diagram. The dimension map is the pinout. The graph slots are the wires. The kernel map is the part catalog that says which implementation is legal for each operation. LoweredIR is the physical wiring plan. Generated C is the board that gets powered on.

This is why GGUF support is not just a file parser. A parser can read bytes. A runtime has to preserve meaning. If blk.0.attn_q.weight binds to the wrong projection slot, the file still loads. If rope_freq_base is ignored, the model may still generate tokens. If a Q4_K tensor is routed through a mismatched dot-product path, the program may still compile. The output can be wrong long before the loader crashes.

Where GGUF Enters The CKE Compiler Path

In CKE, GGUF enters before template lowering, not after. The file provides the physical weights and metadata. The template provides the architecture circuit. The binding layer joins them. From that point forward, the compiler path can become explicit:

GGUF header + metadata
  -> architecture family + tokenizer contract + hyperparameters

GGUF tensor descriptors
  -> tensor name map + quant type map + mmap offsets

CKE template
  -> header/body/footer + operation circuit + graph slots

Binding layer
  -> template slot -> GGUF tensor pointer -> checked dimensions

GraphIR / LoweredIR
  -> symbolic dataflow -> concrete buffers, dtype views, kernel ids

Generated C
  -> pointer expressions + kernel calls + runtime execution

This is also where the CKE IR visualizer becomes useful. A GGUF file can tell CKE what tensors exist. The IR visualizer can show whether those tensors were wired into the expected graph. That is the difference between "the file loaded" and "the model circuit is actually inspectable."

For the full CKE quantization details, see the CKE Quantization Deep Dive and the CKE Quantization Format Infographic. For the runtime/compiler side, the useful next links are the v7 runbook, the v8 runbook, and my note on CKE's v8 IR pipeline and codegen.

mmap Weights And The CKE Bump Arena

There are two different memory stories here, and they should not be mixed up. GGUF weight data lives in the file. When CKE opens the GGUF with mmap, tensor pointers can point directly into the mapped file region. The runtime does not need to deserialize every weight into a new heap allocation before it can reason about the model.

The bump arena is the other side of the runtime: activations, scratch buffers, temporary Q/K/V views, attention workspaces, recurrent state views, and generated-kernel outputs. Those buffers are not the GGUF file. They are runtime memory that CKE can size, reuse, and inspect while executing the template-defined circuit.

// Step 1: open the GGUF file
int fd = open("model-Q4_K_M.gguf",
              O_RDONLY);
// Read 24-byte header
// Parse metadata KV pairs
// Read tensor info descriptors

Weights are mmap-backed file pointers. Activations and scratch buffers live in a separate CKE arena. The two memory worlds never mix.

The CKE Sidecar: What Travels Alongside the Weights

But the bump arena with weight pointers is only half the CKE runtime story. A GGUF file gives CKE the raw components — tensor data, metadata, tokenizer. CKE still needs to answer: which template defines the layer circuit? Which kernel map is legal for this quantization type on this CPU? Where are the precomputed RoPE frequency tables? What is the chat template for conversation formatting? Where is the compiled shared library?

These answers come from the sidecar — the set of runtime artifacts that CKE generates alongside the BUMP weight blob during the conversion step (convert_gguf_to_bump). The sidecar is not a single file. It is a directory of artifacts that together form the complete runtime contract: template, dimension map, kernel map, tokenizer, RoPE tables, generated C, compiled libmodel.so, and canary metadata. Without the sidecar, the weights are just bytes. With it, they become an executable circuit.

# CKE conversion command
python convert_gguf_to_bump_v8.py \
  --gguf model-Q4_K_M.gguf \
  --run $RUN \
  --template qwen3 \
  --force-convert

This is also the natural handoff point for the next post. The sidecar's template — the architecture circuit definition that tells CKE which ops to wire, in what order, with which attention variant — deserves its own deep dive. That is where CKE's design diverges most from other inference engines: the template is not a Python class, not a config file, not an ONNX graph. It is a declarative circuit contract that the compiler can lower, validate, and generate C from. More on that next time.

Making Your Own GGUF

The easiest path from a Hugging Face model to GGUF:

# Using llama.cpp's convert script
python convert_hf_to_gguf.py \
  --outfile model-f16.gguf \
  --outtype f16 \
  ./path-to-hf-model/

# Quantize to Q4_K_M
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Or use Hugging Face's web tool:
# https://huggingface.co/spaces/ggml-org/gguf-my-repo

The conversion script reads the PyTorch/SafeTensors weights, maps tensor names to GGUF conventions (e.g. model.layers.0.self_attn.q_proj.weight → blk.0.attn_q.weight), embeds the tokenizer, and writes the GGUF file. The quantization step then reads the F16 GGUF and produces a quantized variant.

What Makes a Good GGUF File

Not all GGUF files are created equal. A well-made GGUF file:

• Includes the full tokenizer — vocabulary, merge rules, special tokens, and added tokens. Without these, the model generates garbage.
• Includes the chat template — the Jinja2 template that formats user/assistant turns. Without it, the model may work but conversations will be formatted wrong.
• Uses appropriate quantization per layer — Q4_K_M mixes Q4_K for attention and Q6_K for FFN. A naive all-Q4_0 file will be smaller but noticeably dumber.
• Follows the naming convention — BaseName-SizeLabel-Version-Encoding.gguf (e.g. Qwen3-0.6B-v1.0-Q4_K_M.gguf).
• Includes RoPE parameters — rope_freq_base, rope_scaling_type. Missing these causes silent degradation on long contexts.