Quantization and weight formats

Quantization

Models are trained with 16-bit or 32-bit numbers. Quantization rewrites those weights with 8, 4 or even fewer bits. A 4-bit file is roughly a quarter to a third of the 16-bit size, which is what makes many models fit on consumer hardware. The loss in quality is small at 8 bits and grows as bits drop, and it differs by model and by task, so a general rule is a poor guide for any one model.

Sources: Hugging Face: GGUF

GGUF

GGUF stores a model's tensors together with a standard set of metadata, so a runner can load it without extra configuration. Most GGUF files are quantized, and the quantization level is part of the file name, such as Q4_K_M or Q8_0. Large models are often split into several numbered parts. GGUF files run through llama.cpp and the tools built on it, on CPUs and on GPUs.

Sources: Hugging Face: GGUF · llama.cpp

Q4_K_M

In GGUF names, the number is the approximate bit width, K marks llama.cpp's k-quant method, and the last letter (S, M or L) says how many tensors are kept at higher precision. Q4_K_M sits in the middle: much smaller than 8-bit, usually with a modest quality drop. In llama.cpp's own example an 8B model goes from 32.1 GB at 32-bit to 4.9 GB at Q4_K_M. How much quality is lost depends on the model and the task.

Sources: llama.cpp quantize tool · Hugging Face GGUF quantization types

Q8_0

Q8_0 stores each weight in 8 bits, rounded to the nearest value. It is the level to pick when you have the memory and want output as close as possible to the original model while still using a GGUF file.

Sources: Hugging Face GGUF quantization types

MLX

MLX is an array framework from Apple's machine learning research team, designed around the shared memory of Apple silicon. MLX model files are a separate conversion from GGUF, with their own quantization scheme, so an MLX 4-bit file and a GGUF Q4_K_M file of the same model are different files and can behave slightly differently. Results measured on one do not carry over to the other. MLX also has Linux builds, but this site counts MLX files only for Apple silicon.

Sources: MLX

BF16

BF16 and FP16 both use 16 bits per weight and differ in how those bits are split between range and precision. A model's size in gigabytes at 16-bit is about twice its parameter count in billions.