Qwen3.8 hits 81.4 tokens/s on an RTX 3090

I got Qwen3.8-27B to generate code at 81.4 tokens/s on my 24 GB RTX 3090. I used the 17.9 GB UD-Q4_K_XL quant, a 144K context window, Q8 KV cache, full model offload, one server slot, and the model's built-in MTP drafter at depth four. Let's dive in.

What fits in 24 GB

My plan was to pair deepseek harness with a local model that fits a consumer GPU, and I had high hopes for Qwen 3.8 as Muse Glimmer was a flop. And since we have only 24G to work with, both BF16 (55.6 GB) and FP8 (30.9 GB) were out of the question.

It's also an Ampere card, so no reason to reach for NVFP4 family models.

Previous Qwen model (3.6) performed quite well with Q4 quants locally, so I went for Unsloth UD-Q4_K_XL, which is just about right - 17.9 GB. Now, for my setup I am using Q8 KV, and after some mathing and subsequent trial and errors I got this model to work with 144K Q8 KV. Qwen3.8 has a 262,144-token native context window, and Qwen documents a YaRN configuration that extends it to 1M. 144K works for my use case, but if your preference is different, check other levers - Q8/Q4 quants for KV, offloading some of the layers to the CPU or even going with a smaller quants, though a quality impact is likely.

Qwen3.8 packages its MTP head with the model, so speculative decoding does not require a separate draft model to download and squeeze onto the card. llama.cpp can drive that head directly with --spec-type draft-mtp. The drafter keeps its own F16 KV, roughly 4 KB per token on top of the main cache. Quantizing a one-layer drafter can consume more VRAM because its metadata has very little payload to amortize; an upstream llama.cpp explanation covers that edge case. Basically, each additional 1,024 context tokens costs about 35 MB for the main cache and another 4 MB for the drafter.

One more thing, model has image and video capabilities. Since my primary use case is coding, I moved those off to the CPU. --mmproj <path> loads the multimodal projector that lets the model inspect images supplied with a prompt; --no-mmproj-offload keeps that projector on the CPU, so vision stays available without taking GPU memory away from the language model, Q8 KV cache, and MTP drafter.

Observed perf metrics

I see around 1.2K tokens/s prefill and around 60 tokens/s generation, with code peaks at 81.4 tokens/s and draft acceptance around 75%-80%. The rows compare complete configurations because context changed with depth.

Model reasoning

Qwen 3.8 supports Low, Medium and X-High reasoning. Anecdotally, low seems to be making more mistakes and has to correct them and x-high is overthinking, whereas most of my tasks require execution. With medium reasoning, model appears to use less tokens and not waste cycles correcting itself over and over.

Here's how you enable it in llama.cpp:

--chat-template-kwargs '{"reasoning_effort":"medium","preserve_thinking":true}'

My full llama.cpp template

llama-server \
  --model /path/to/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --mmproj /path/to/mmproj-Qwen3.8-27B-Q8_0.gguf \
  --no-mmproj-offload \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --alias Qwen/Qwen3.8-27B \
  --jinja \
  --ctx-size 147456 \
  --n-gpu-layers 99 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --threads 16 \
  --threads-batch 16 \
  --batch-size 2048 \
  --ubatch-size 256 \
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --chat-template-kwargs '{"reasoning_effort":"medium","preserve_thinking":true}' \
  --reasoning-budget 4096 \
  --reasoning-budget-message "Wait, I'm overthinking this. Let's answer now." \
  --metrics

Conclusions

Qwen 3.8 is a solid release, very capable model with a small footprint. I urge you to try it for your own setup.