Q4_K_M vs Q8_0: 2.1% Worse, 65% Faster on a 16GB Mac

Q4_K_M is what you get by default. Ollama's llama3.1:latest tag carries the same digest, 46e0c10c039e, as llama3.1:8b-instruct-q4_K_M, so ollama run llama3.1 gives you this file. The bigger files promise quality back: bartowski's model card, where my files come from, calls Q6_K "very high quality, near perfect" and Q8_0 "extremely high quality, generally unneeded." On a 16 GB machine the question is what that quality costs, so I downloaded the Q6_K and Q8_0 files of Llama 3.1 8B (15.1 GB, 72 minutes at this connection's 3.5 MB/s) and ran them against the Q4_K_M from my earlier posts.

The first Q8_0 run with a 32K context failed: Metal refused it after 3.4 seconds, as the arithmetic said it would. Across the rest, Q4_K_M vs Q8_0 comes down to three numbers. Q8_0 decodes 39% slower, 12.9 against 21.3 tokens per second. Q4_K_M's perplexity is 2.1% worse than Q8_0's. And when llama.cpp sizes the context itself, it gives Q8_0 half the room, 25,088 tokens against 50,176. Q6_K sits between the two on all three and gets most of the quality back, 0.31% from Q8_0 at a 23% speed cost.

What the three names mean, read from the files

A GGUF file lists every tensor with its encoding in the header, so I wrote a short parser that reads only the header and counts them. All three files hold the same 8,030,261,312 weights; they differ in how many bits each weight costs. Q8_0 stores every matrix in blocks of 32 one-byte integers plus a two-byte scale, 34 bytes per 32 weights. Q6_K uses 256-weight super-blocks of 210 bytes. Q4_K_M is the one whose name misleads: it is not a uniform 4-bit file.

The "M" in Q4_K_M is those 33 tensors: the output layer, plus the ffn_down and attn_v matrices in 16 of the 32 layers (the first four, the last four and every third one between), all kept at 6 bits. That is why the average is 4.894 bits and not the 4.5 a pure Q4_K file would cost. My counts match the bits-per-weight column in llama.cpp's quantize documentation for Llama 3.1 8B to the fourth decimal. The last column is the arithmetic from my first 8B post: generating one token reads every weight once, Apple rates the base M4 at 120 GB/s, so file size sets a hard ceiling on decode speed before anything else is measured.

Speed: Q8_0 decodes 39% slower and prompts slightly faster

I ran llama-bench on each file with flash attention on and the default f16 cache: a 512-token prompt and 128 generated tokens, three repetitions each, then 64 tokens generated with 16,384 tokens already in the cache.

Decode tracks file size almost exactly, because it is a memory-bandwidth problem: Q6_K is 23% slower than Q4_K_M and Q8_0 is 39% slower, and the bigger files get slightly closer to their own ceilings. Prompt processing does not track size at all. It is compute-bound, and the largest file processed prompts 3% faster than the smallest, most likely because Q8_0's plain 32-weight blocks take the least work to unpack. llama.cpp's own table shows the same shape on faster hardware, but the penalty there is milder: its Q8_0 generates at 71% of its Q4_K_M speed, against 61% on this Mac mini.

Quality: how far each file drifts from Q8_0

To measure what the smaller files lose I used the method in llama.cpp's perplexity documentation, the same one I used for the KV cache test earlier this evening. Run 8,192 tokens of WikiText-2 through a reference model and save the probability it assigns every possible next token, then run the same text through each smaller file and measure how far those distributions moved (mean KL divergence) and how often the single most likely token changed. The proper reference is the unquantized F16 model, but that file is 16 GB and does not fit on a 16 GB machine, so my reference is Q8_0. The documentation's own Llama 3 8B scoreboard puts Q8_0's divergence from F16 at 0.001355, small enough that it works as a stand-in, but every number below is drift measured from Q8_0, not from the original weights.

Q6_K is close to the reference. Perplexity rose 0.31%, and the most likely next token changed about once every 36 tokens. Q4_K_M is a real step down: perplexity rose 2.10%, its mean divergence is 6.7 times Q6_K's, and the top token changed about once every 17 tokens. The tail grows with it. On the worst 0.1% of tokens Q4_K_M drifted 0.473, more than five times Q6_K's worst, and those rare tokens are where a wrong word lands in the middle of an otherwise fine answer. The perplexity error bars look wide, ±0.15, but every file scored the same tokens, so the paired ratio is much tighter: 2.10% ±0.24%.

Two cross-checks. The Q4_K_M file scored 5.8654 in the earlier KV cache test, under the same settings, and 5.8654 again tonight, so the setup repeats to four decimals. And the documentation's scoreboard, measured against F16 on an older Llama 3 with files made without an importance matrix, lists 0.0055 for Q6_K and 0.0313 for Q4_K_M. My figures are smaller, as they should be with imatrix files and a reference that is itself 0.0014 from F16, but the gap between the two files comes out the same size, roughly six times.

Update, September 15: that scoreboard's no-imatrix rows left one variable this post did not test, so I made the same Q4_K_M from one Q8_0 file with and without bartowski's importance matrix. On WikiText-2 the static file drifted 25% further from Q8_0, 0.0216 against 0.0173, and it turned out requantized files can carry imatrix keys they never used. The details are in imatrix vs static quants, measured.

Memory: which file still leaves room for context

File size also decides how much context is left. Metal on this machine lets one process use recommendedMaxWorkingSetSize = 12713.12 MB, about 11.8 GiB, the ceiling I first hit when running a local LLM on the 16 GB Mac mini, and the default f16 KV cache costs 128 KiB per token for this model, so a 32K context is 4 GiB on top of the weights. On paper that is 8.6 GiB for Q4_K_M, 10.1 GiB for Q6_K and 11.95 GiB for Q8_0, which is over the line. I ran llama-completion with each file at explicit context sizes, then once more with -c left unset. In my build -c defaults to 0, which --help describes as "loaded from model", and for Llama 3.1 that is a 131,072-token window, so the unset runs show what the --fit option (on by default, keeping a 1,024 MiB margin) does with each file.

Q8_0 with a 32K f16 cache failed the way the arithmetic said it would. The process reached 12.14 GiB and Metal refused the first batch:

E error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
E llama_decode: failed to decode, ret = -3

It gave up in 3.4 seconds instead of grinding through swap, but it still pushed 0.82 GiB of other processes out to swap on the way. The same file at the same 32K ran cleanly once the cache was q8_0, at 10.22 GiB and 12.89 tokens per second, which is the fix from the KV cache post doing its job.

The unset runs are the more useful result. The log only prints the outcome, a line like generate: n_ctx = 25088, but the pattern is plain: instead of the model's 131,072 tokens, --fit gave Q4_K_M 50,176, Q6_K 38,656 and Q8_0 25,088. All three landed at 10.7 to 10.9 GiB of process memory, so the budget is the same every time and the file size decides how much of it is left for context, which leaves Q8_0 half of what Q4_K_M gets. My own arithmetic, Metal's grant minus the 1 GiB margin minus the weights, divided by 128 KiB per token, predicted 51K, 38K and 24K, within 6% of what llama.cpp chose.

What owners report, and why it disagrees

The advice in public threads points two ways. One Hacker News commenter wrote that "a Q6_K has minimal performance loss from the Q8, so in most cases if you can fit a Q6_K it's recommended to just use that" (comment). Another, discussing a model whose BF16 file is 55.65 GB, put the loss from BF16 to Q4_K_M at "~1-3% relative PPL increase" and called it "close to imperceptible" (comment). The reports on the other side are single tasks: one owner "tried the 4 bit quants (Q4_K_M) and was a bit unimpressed. Switching to Q6_K made a huge difference" (comment), and another had a first test fail at Q4_K_M and succeed at Q8_0 on a different 20B model (comment).

The closest report to my hardware is the bleakest. On a base M4 Mac mini, one commenter found that DeepSeek-R1-Distill-Llama-8B, built on the same Llama 3.1 8B architecture, at Q8_0 "gives me about 3t/s and destroys my system performance," and said they would try Q4_K_M next (comment). The comment does not say which runtime or context size was in use. On this Mac mini, a Q8_0 file of the same architecture decoded at 12.9 tokens per second and started cleanly with llama.cpp's defaults. My guess, which the comment cannot confirm, is that 3 tokens per second was memory pressure, a context too large for the file spilling into swap, rather than the file's own speed.

Which one I would run on 16 GB

Q4_K_M for anything that needs context or speed, which on this machine is most things. It decodes at 21 tokens per second, leaves room for 50K tokens with the default cache, and its 2.1% perplexity cost is real but small. Q6_K when the answers matter more than their length: its top token matched Q8_0's 97% of the time, it still gets 38K of context, and it decodes at 16 tokens per second. Q8_0 is hard to justify here. It buys 0.3% over Q6_K, gives up another 21% of decode speed and fits 25K tokens. If you want it anyway, pair it with a q8_0 cache, which ran 32K in 10.2 GiB at full speed. With 24 GB the budget moves, which is part of what I weighed in choosing between the 16 GB and 24 GB Mac mini.

Two limits on all of this. Perplexity on encyclopedia text is an average, and the Hacker News reports of Q4_K_M failing one specific task describe the kind of failure an average can hide, in the rare tokens where Q4_K_M's divergence reached 0.47. And my reference is Q8_0, not F16, so each file's true distance from the original weights is slightly larger than the table shows.

FAQ

Is Q8_0 slower than Q4_K_M?

For generating text, yes, roughly in proportion to file size. On a base M4 Mac mini with 16 GB, Llama 3.1 8B decoded at 21.27 tokens per second as Q4_K_M, 16.31 as Q6_K and 12.93 as Q8_0 in llama.cpp, because each generated token reads every weight from memory once and the Q8_0 file is 74% larger. Prompt processing is different: Q8_0 processed a 512-token prompt at 237 tokens per second, slightly faster than Q4_K_M's 230.

What does the M in Q4_K_M mean?

It marks a mixed file. In bartowski's Llama 3.1 8B Q4_K_M, 193 weight tensors (80.9% of the weights) use the 4-bit Q4_K encoding and 33 tensors (19.1%) use 6-bit Q6_K: the output layer plus the ffn_down and attn_v matrices in 16 of the 32 layers. That is why the file averages 4.894 bits per weight rather than 4.5. Q6_K and Q8_0 files use one encoding throughout.

How much quality does Q4_K_M lose compared with Q8_0?

On Llama 3.1 8B over 8,192 tokens of WikiText-2, with Q8_0 as the reference, Q4_K_M raised perplexity 2.1% (5.746 to 5.865), had a mean KL divergence of 0.0187 and picked a different most-likely token about 6% of the time. Q6_K raised perplexity 0.31%, with a mean KL divergence of 0.0028 and a different top token 2.7% of the time. Perplexity on encyclopedia text is not a task score, so the gap on a specific coding or reasoning task can be larger or smaller.

Sources and verification: every speed, memory, swap and divergence figure is my own reading on this Mac16,10 (base M4, 10-core GPU, 16 GB, macOS 26.4.1) on 2026-09-15 between 19:33 and 19:50 KST, using Homebrew llama.cpp 0.4.0 build 10809 and bartowski's imatrix GGUFs of Llama 3.1 8B Instruct: Q4_K_M at 4,920,739,232 bytes, Q6_K at 6,596,011,424 and Q8_0 at 8,540,775,840, the last two checked against the SHA-256 Hugging Face publishes. Tensor encodings come from reading each file's GGUF header with my own parser, and they match the bits-per-weight column in llama.cpp's quantize table. Quality runs used llama-perplexity with --kl-divergence-base on the WikiText-2 test file, 8 chunks of 2,048 tokens, with Q8_0 as the reference because F16 does not fit. Memory is /usr/bin/time -l maximum resident set size plus sysctl vm.swapusage read before and after each run, with 6.1 GiB of other work resident and 2.5 GiB already in swap when I started. Ollama's tag digests were read from its library page, and Hacker News comments through the Algolia API; I could not access Reddit. Not tested: F16 as the reference, any model other than Llama 3.1 8B, the same quants without an importance matrix, and long agentic tasks.