llama.cpp Parallel Requests: 9 Beat 8 on a Base M4 Mac mini

I have Llama 3.1 8B on this 16 GB Mac mini, and the obvious next step for an operation that runs several agents is to put llama-server in front of it and let more than one of them call it at once. llama.cpp parallel requests are one flag, -np, plus continuous batching, which is on by default. So I fired 1 to 12 simultaneous requests at it and timed them. Eight requests of 128 tokens each took 39.0 seconds. Nine took 22.4. Adding a ninth request made the whole batch finish 43% sooner.

The cause is one line in the Metal backend that picks a kernel by batch size. I rebuilt llama.cpp with that line switched off and the slow range went away. Below are the numbers, the line, and what I would set -np to on a base M4.

What llama.cpp parallel requests look like from the client

The server was Homebrew llama.cpp 0.4.0 (build 10809, commit 5266f24da), started with -np 12 -c 12288 and the Q4_K_M file from my Llama 3.1 8B Mac mini M4 test. An 18-line Python client opened N threads, and each sent a short prompt to /completion with n_predict: 128, ignore_eos: true and prompt caching off, so every request generated exactly 128 tokens.

Two things are wrong with that table. Going from 3 requests to 4 lowered total throughput, from 32.4 to 26.1 tokens per second, so the fourth caller made everyone slower and added nothing. Then 9 requests ran at twice the throughput of 8, and each of the nine callers got 5.9 tokens per second against 3.3 for each of the eight.

The server log shows why this matters even if you never pass -np. Started with only -c 12288, it printed n_slots = 4, n_ctx_slot = 12288, kv_unified = 'true': the automatic default is 4 slots, which is the first bad size in the table. Four simultaneous requests on that default server took 20.0 seconds (25.6 tokens/s total). On a server started with -np 2, the same four requests queued two at a time and finished in 16.3 seconds, and eight finished in 32.6 seconds against 39.0 on the 12-slot server.

The same cliff without the server

llama-batched-bench decodes B sequences in lockstep and reports generation speed with no HTTP or scheduling in the way. I ran it three times on the Q4_K_M file and once each on the Q6_K and Q8_0 files of the same model. The column below is total generation tokens per second (S_TG) with 64-token prompts and 64 generated tokens per sequence.

A first Q4_K_M run with 256-token prompts and 128 generated tokens gave the same shape: 32.7 at B=2, 26.0 at B=4 and at B=8, 88.1 at B=16. The two K-quant files drop at B=4 and recover at B=9. The Q8_0 file does not drop at 4; it doubles at 2, reaches 34 at 3, and then stays flat until 9. All three jump at 9.

The line that picks the kernel

A matrix multiply on Metal goes to one of three kernels, chosen by ne11, which during decoding is the number of sequences in the batch. The source at my build's commit (ggml-metal-ops.cpp, lines 2381 to 2411) reads, trimmed:

// first try to use small-batch mat-mv kernels
// these should be efficient for BS [2, ~8]
(Q8_0, Q4_0, F16, IQ4_NL, ...)      && (ne11 >= 2 && ne11 <= 8)
(Q4_K, Q5_K, Q6_K, Q2_K, Q3_K)      && (ne11 >= 4 && ne11 <= 8)
  -> mul_mv_ext
else if (... ne11 > 8)  -> mul_mm     // ggml-metal-device.m
else                    -> mul_mv

Those bounds match the measurements exactly. K-quants enter the small-batch kernel at 4, which is where Q4_K_M and Q6_K fall. Everything leaves it at 9 for the matrix-matrix kernel, which is where all three files jump. Q8_0 enters it at 2 and does not fall, so on this chip the small-batch kernel is not slow for every type; it is slower than the plain kernel for K-quants.

That was still an inference from matching numbers, so I tested it. I built llama.cpp from source at the same commit twice with no other changes: once as-is, and once with one edit that makes the K-quant condition always false, so B=4 to 8 falls through to the plain mul_mv kernel.

With the patch, B=4 went from 26.6 to 33.2 tokens per second and B=5 from 24.1 to 32.9. The Q6_K file went from 20.2 to 24.8 at B=4 and from 20.3 to 24.9 at B=8. B=1 to 3 and B=9 did not change beyond noise. The patch removes the dip, but it does not make 4 to 8 sequences faster than 2 or 3: the plain kernel stays flat at about 33 until the matrix kernel takes over at 9.

I am not the first to see this. A pull request opened on 27 August, #27776, reports that on an M3 Pro "the K-quant mul_mv_ext path … ended up running slower than the generic mat_vec path", disables it for M3 only, and asks whether "M4/M5 chips have the same issue". As of today it is open with no review. The kernel came from #10581 in December 2024, whose M2 Ultra table shows the opposite result: Q4_K at batch 5 went from 79 to 178 tokens per second. So this is a chip-dependent choice, and on the base M4 it currently goes the wrong way for K-quants. I have not tested an M4 Pro or Max.

The other trap: -np splits the context

Setting -np yourself changes a second default. The server I used above, -np 12 -c 12288, logged n_ctx_slot = 1024, kv_unified = 'false'. The server README says the unified KV buffer is "enabled if number of slots is auto", so an explicit -np turns it off and each slot gets -c divided by -np. A 1,500-word prompt then failed on both endpoints:

400 {"error":{"code":400,"message":"request (1502 tokens) exceeds the available
context size (1024 tokens), try increasing it","type":"exceed_context_size_error"}}

On the auto-slot server with the same -c 12288, a 9,002-token prompt went through, because the four slots share one pool. If you set -np, either add -kvu or multiply -c by the slot count. Multiplying costs memory: at f16 the cache for this model is 128 KiB per token, as I measured in the llama.cpp KV cache quantization post.

What I would set on a base M4

• Two or three agents calling one K-quant model: -np 2 or -np 3. Total throughput peaks at 2 to 3 sequences below the matrix kernel, and a fourth slot costs about 20%. Requests beyond that wait in the queue, which here finished sooner than running them together.
• Nine or more requests really in flight at once: -np 16 or more. From B=9 upward the matrix kernel scales to 170 tokens/s at 32. A high -np only helps while at least 9 slots are busy; with 16 slots and 5 active callers you are back in the slow range.
• Do not leave the default at 4 for a K-quant model on this chip, and say -kvu when you set -np.
• Q8_0 does not have the dip, but it starts at 12.6 tokens/s for a single request instead of 20.8, and the file is 8.5 GB on a 16 GB machine. The trade-offs between quant files are in imatrix vs static quants, and the memory ceiling in Mac mini 16GB vs 24GB for an agent server.

Community advice points the same way for other reasons. In a llama.cpp discussion about llama-server running slower than llama-cli, the first reply was to add --parallel 1. That thread is on an NVIDIA laptop GPU, so it is a different cause; the shared point is that the slot count is not a free setting.

FAQ

Does llama.cpp handle parallel requests?

Yes. llama-server serves parallel requests through slots set by -np (--parallel) and batches them with continuous batching, which is on by default. The default -np is -1, meaning automatic; in build 10809 that chose 4 slots sharing one KV cache.

Why is llama.cpp slower with 4 parallel slots than with 2 on a Mac?

On a base M4 with a K-quant model such as Q4_K_M, 4 to 8 sequences decoding together use the Metal mul_mv_ext kernel, which measured slower than the plain kernel used for 2 or 3. Total throughput fell from 33 tokens/s at 2 or 3 sequences to about 26 at 4 to 8, then rose to 53 at 9, when the matrix-matrix kernel takes over.

Does -np divide the context size in llama-server?

If you set -np explicitly, the unified KV cache is off by default and each slot gets the context size divided by the slot count: -np 12 -c 12288 gave 1,024 tokens per slot and rejected a 1,502-token prompt with exceed_context_size_error. Add -kvu to share one pool, or raise -c.

Sources and verification: every timing is my own measurement on this Mac16,10 (base M4, 10-core GPU, 16 GB, macOS 26.4.1) on 2026-09-18 between 19:33 and 20:05 KST, with other work resident and 1.7 GB already in swap. Server and batched-bench runs used Homebrew llama.cpp 0.4.0 build 10809 (commit 5266f24da) and bartowski's Llama 3.1 8B Instruct GGUFs; the stock-vs-patched comparison used two source builds at that same commit, with the one-line patch saved alongside the raw logs. Each configuration was run once except Q4_K_M, which has three batched-bench runs and the server run, all with the same shape. The kernel selection is quoted from the source at that commit and was unchanged on master the same day. Not tested: M4 Pro or Max, other model sizes, long prompts arriving while others decode, and flash attention off.