Qwen3.6-35B-A3B hits 63.8 tokens/s on a 32 GB Mac mini

Qwen3.6-35B-A3B decodes at 56.7 to 63.8 tokens/s on a base M4 Mac mini with 32 GB of unified memory. That is native MLX, with the model's own MTP head drafting tokens, and it is up to 2.5x the 25.3 tokens/s llama.cpp managed on Metal on the same box. The whole model runs on the GPU and peaks at 21.7 GB.

I have served this model from that machine since June, and most of that time went into llama.cpp and the Metal memory budget. The sections below follow that path, from the 262K-context sweep through the draft-model out-of-memory failures, to the MLX setup that produced the headline number.

The machine

The base M4 has a 10-core CPU (four performance cores, six efficiency cores), a 10-core GPU, and 32 GB of unified memory at roughly 120 GB/s. llama-server --list-devices reports a default Metal budget of 25,559 MiB. The model files live on an external SSD.

Nothing else about the box matters as much as that budget. Every failure in this post is something trying to allocate past it.

No expert offload on unified memory

On my RTX 3090 the main tuning lever for this MoE model was --n-cpu-moe, which moves expert FFN weights out of 24 GB of VRAM into separate system RAM. A Mac has one pool of DRAM. Offloading experts to the CPU there frees no memory at all; it only moves the expert matmuls onto the slower CPU path. So the launcher sets --n-gpu-layers 999 and has no expert-offload knob.

mmap stays on, which is the opposite of my 3090 launcher. With no CPU expert math there are no page-fault stalls to avoid, and on a slower external SSD lazily faulted, file-backed pages can be reclaimed under pressure instead of pushing the machine into swap.

The limit is the Metal compute buffer

With q8_0 KV at 262,144 context, the all-GPU footprint is about 24 GB. That fits the default Metal budget without raising the wired-memory limit, so the weights are not the constraint. The compute buffer is. It grows with ubatch and with the KV element size, and it has to fit in whatever is left over.

Two configurations load, start listening, report n_ctx = 262144, and then fail on the first real decode with kIOGPUCommandBufferCallbackErrorOutOfMemory and llama_decode: failed to decode, ret = -3:

• ubatch 4096, because its prefill compute buffer is too large
• f16 KV, which is about twice the size of q8_0 (turning flash attention off forces f16 KV, so that fails the same way)

A health check is a load check on this hardware. Prove inference with a real request.

The 262K sweep

These numbers come from June, on a Homebrew llama.cpp b9430 build, with a community reasoning-distill fine-tune of Qwen3.6-35B-A3B at Q4_K_M (20,182 MiB). Every row used the full 262,144 context, all layers on Metal, batch 2048, and the same fixed prompt of about 18,600 tokens. Prefill and decode rates come from the server's own per-request log.

Decode barely moves across the whole table. It sits around 23.5 tokens/s because it is bound by memory bandwidth, and no flag changes the bandwidth. Prefill is the number that responds to tuning: ubatch 2048 with six threads came out 11–13% ahead of the ubatch-512, four-thread starting point.

Threads affect prefill only, and not by much. Six (four P-cores plus two E-cores) was marginally best, four landed within about 2%, and eight was slightly worse, which I attribute to oversubscribing the E-cores. The GPU does the matmuls, so the CPU thread count is a small lever.

q4_0 KV matched q8_0 on prefill but lost decode speed (22.4 against 23.5–23.7) and adds quality risk at long context depth. I kept q8_0.

I did not re-run a needle-in-a-haystack test on the Mac. The full context loads and gave coherent output on the 18.6K-token prompt, but depth recall for this GGUF was only tested on the 3090 with identical weights.

Fitting a draft model

By August, decode on my real workload measured about 10–11 tokens/s and dominated wall time. That workload sends prompts of 10K to 100K tokens, far from the sweep's single 18.6K prompt. A 35B MoE target at batch size one looked like a good candidate for speculative decoding, so I tried an external draft: Qwen3.5-0.8B at Q4_K_S, a 508 MB file. I confirmed the vocabularies matched by hashing the GGUF token tables of draft and target. On a mismatch, that llama.cpp build logs draft model vocab type must match target model to use speculation at error level and then serves normally with speculation off. The server starts either way, so read the startup log.

The draft inherits the target's full context size, and its KV cache defaults to f16 regardless of --cache-type-k. I pinned it to q8_0 with --spec-draft-type-k and --spec-draft-type-v. Even so, at 131,072 context and ubatch 2048, every real request failed with the same Metal OOM while /health kept returning 200. Cutting context to 98,304 did not help either. The draft's reported footprint only went from 3,777.71 MiB to 3,445.71 MiB, so a 25% context cut bought 332 MiB.

Ubatch was the lever that worked. Holding context at 98,304 and every draft flag constant, ubatch 2048 to 1024 moved the draft's footprint from 3,445.71 MiB to 2,275.68 MiB, a 1,170 MiB saving. The next real completion returned 200 with zero new OOM lines in the log.

With the draft resident, that completion decoded at 28.08 tokens/s, with 41 of 44 draft tokens accepted (about 93%), and prefilled at 237.66 tokens/s. The prompt was only 231 tokens with 64 output tokens, so it does not compare directly with the 10–11 tokens/s production figure, and I have not measured a 10K–100K-token prompt with the draft loaded.

The ubatch cut also has a prefill cost. With no draft resident, ubatch 1024 ran 277.3 tokens/s prefill against 288.0 at ubatch 2048 in the June sweep, about 3.7% slower (measured at four threads, while the server runs six).

The context cut cost more than throughput. My workload's prompts measured a p99 of about 103K tokens and a maximum of about 151K, both above 98,304. The long tail of requests fails on context exhaustion. I accepted that trade, since a bounded set of failed requests beats a server that fails every one.

Quant choice turned out to be measured in hundreds of MiB. When I moved from the distill to the stock model, the unsloth UD-Q4_K_M (21,109 MiB) was 927 MiB heavier than the distill's Q4_K_M and overran the roughly 870 MiB of remaining Metal headroom on its first inference. UD-Q4_K_S (19,925 MiB) is 257 MiB lighter than the distill and fit. On this box, choose a quant by its size in MiB, not by its name.

Speculative decoding on b10502

Later I pinned the upstream llama.cpp b10502 macOS release and switched to another community fine-tune of Qwen3.6-35B-A3B at Q4_K_M (JoyFox). That build supports the model's native MTP head through --spec-type draft-mtp, so I ran a controlled comparison at 98,304 context, ubatch 1024, q8_0 KV, and one slot. Each row is three deterministic 256-token completions.

No speculative row beat plain serving. Native MTP at depth 2 came within 0.65% of plain decode, while median prefill fell about 8% and resident memory rose about 1.17 GiB. The best external-draft row lost about a fifth of decode throughput.

The sample is three short completions per mode; long prompts were not part of it. After the sweep the server went back to its previous external-draft profile (depth 4, p_min 0.00). That profile is not one of the measured rows, and changing it was left as a separate decision.

Do not compare the 28.08 tokens/s from the August proof request with the 25.3 here. The builds, the fine-tunes, and the prompt lengths all differ.

The llama.cpp flags

This is the launcher's argument list with the credential and listener flags removed.

llama-server \
  --model /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf \
  --alias Qwen3.6 \
  --ctx-size 98304 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --threads 6 \
  --threads-batch 6 \
  --batch-size 2048 \
  --ubatch-size 1024 \
  --parallel 1 \
  --kv-unified \
  --metrics

The launcher accepts at most one speculative mode. Native MTP adds:

  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --spec-draft-p-min 0.00

An external draft adds:

  --model-draft /path/to/Qwen3.5-0.8B-Q4_K_S.gguf \
  --spec-type draft-simple \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 1 \
  --spec-draft-p-min 0.00 \
  --n-gpu-layers-draft 999 \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0

How --kv-unified interacts with drafting was never verified. If you go back to the full 262,144 context with no draft, ubatch 2048 is the measured ceiling.

Moving to native MLX

On August 23 I moved serving to native MLX through mlx-vlm, pinned to a 0.6.15 release commit on Python 3.13. The reasons were the native Apple Silicon path and the telemetry it exposes for evaluating speculative decoding and prefix reuse. The target is an MLX 4-bit community fine-tune of Qwen3.6-35B-A3B (froggeric's Heretic build), and the drafter is mlx-community/Qwen3.6-35B-A3B-MTP-4bit.

The serving settings:

The block size started at 5 and dropped to 3 after a tuning pass on the live model against three long prompts.

Prefix caching is off because it breaks requests. The pinned runtime's warm-cache path restores a hybrid cache representation that Qwen's MTP verifier cannot consume. A prefix-cache hit is followed by an MTP verification failure, so a valid request that repeats an earlier prefix comes back as a server error. Prefix reuse was half the reason to evaluate MLX, and it stays off until a pinned runtime ships a compatible warm-cache path.

The server admits one sequence at a time. It reports in-flight work but no verified queue depth, so I do not alert on saturation for it: with one admitted sequence, in-flight proves work is running, not that anything is waiting. MTP acceptance comes straight from the runtime's lifetime accepted and drafted token counters.

The MLX numbers

I restarted the server fresh with thinking off, then ran three fixed prompts, each with one excluded warm-up and five measured runs, at temperature 0, seed 42, 512 max tokens, and no streaming. The rates are the server's own per-request figures.

Every run produced the full 512 output tokens. The llama.cpp rows used a different fine-tune and different prompts, so the 2.5x compares the two setups as I run them.

Real traffic is slower but still well past llama.cpp. Across 33 production requests from September 29 to October 25, the server's request history shows a median decode of about 42 to 43 tokens/s.