Muse Glimmer 30B with DFlash hits 133.7 tokens/s on an RTX 3090

Muse Glimmer 30B decodes at up to 133.7 tokens/s on one 24 GB RTX 3090 once its DFlash drafter is switched on, a 2.29x mean speedup over target-only decoding. Without the drafter, the same model, binary, and weights hold at about 43 tokens/s; with it, three fixed prompts at a 32K context land between 65.83 and 133.68 tokens/s. The drafter costs about 1.8 GiB of VRAM, and the target, its vision projector, and the drafter all fit on the card together. Stretching the context from 32K to 98,304 tokens later added only 442 MiB of KV cache.

Meta's official GGUF card makes an unusually specific promise: the 17 GB K-Quant build targets a 24 GB GPU, including room for the vision encoder, KV cache, and its DFlash speculative decoder. I had a 24 GB RTX 3090 available, so I tried the exact release set instead of interpreting “targets” generously.

It fits. With a 32K context, quantized KV cache, one server slot, and all model and draft layers on the GPU, the loaded process used 19,376 MiB of VRAM. Text and image requests completed, DFlash proposed and accepted tokens, and the server retained about 5 GiB of headroom.

How the model behaves once it is running is a separate question, answered in Muse Glimmer is a flop. This post covers getting it onto the card and keeping it there.

The three-file memory budget

The official GGUF repository contains three files needed for this setup:

target model       16.76 GB
vision projector    1.40 GB
DFlash drafter      1.63 GB
                   --------
published files    19.79 GB

That total explains most of the VRAM result before considering CUDA buffers and KV cache. The “17gb” in the target filename only describes the main quantized model. Treating it as the complete memory budget is a good way to schedule an evening with nvidia-smi.

The larger dynamic K-Quant is 19.7 GB by itself and targets 32 GB. The 16.8 GB build is the relevant one for a 3090.

DFlash in llama.cpp

DFlash stands for Block Diffusion for Flash Speculative Decoding. A small draft model proposes a block of tokens in one pass. The target model verifies the block in parallel and accepts the valid prefix. The paper describes the method as exactness-preserving: the target distribution stays unchanged.

Muse Glimmer's drafter proposes a 16-token block. llama.cpp uses a maximum draft length of 15 because one position is consumed by conditioning context. Generic DFlash support arrived in llama.cpp PR #22105; Muse Glimmer support followed in the model's release-day merge.

The official GGUF card requires llama.cpp b10353 or newer. I used b10355 with CUDA enabled for compute capability 8.6:

cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --parallel \
  --target llama-server llama-cli llama-mtmd-cli

The server invocation uses explicit local files rather than downloading whatever happens to be current at startup:

llama-server \
  --model /models/muse-glimmer-30B-kquant-17gb.gguf \
  --mmproj /models/mmproj-kquant.gguf \
  --spec-draft-model /models/dflash-kquant.gguf \
  --spec-draft-ngl 99 \
  --spec-type draft-dflash \
  --spec-draft-n-max 15 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --threads 16 \
  --threads-batch 16 \
  --batch-size 2048 \
  --ubatch-size 256 \
  --parallel 1 \
  --jinja \
  --metrics

The Hugging Face launch guide uses the explicit --spec-type draft-dflash selector. I keep that selector explicit in the service definition.

Measured DFlash speedup

My first build used the initial Muse support merge and omitted --jinja. It produced 56.57 tokens/s on a fixed 256-token request. Meta's GGUF card specifies b10353 or newer and Jinja template processing. Moving to b10355 with the required template path raised the same request to 113.43 tokens/s.

I then measured DFlash against target-only decoding with three fixed prompts. Every request used the same b10355 binary, target weights, 32K context, one slot, Q8 target KV cache, default F16 draft KV cache, high-reasoning system message, greedy sampler, seed, 512-token cap, warm-up procedure, and prompt order.

The mean per-prompt speedup was 2.29x. Aggregate generation time fell from 32.79 seconds to 15.27 seconds, a 2.15x wall-time improvement. Raw candidate acceptance averaged 23.2% across prompts, or 21.7% weighted by candidate count. Mean tau was 4.46 output tokens per target verification cycle.

tau is 1 + accepted tokens / verification blocks, so it includes the target model's bonus token and estimates the output produced per target verification cycle. The raw candidate percentage divides by all 15 proposed positions and answers a different question. Target-only decoding held at about 43 tokens/s on all three prompts, so the speedup tracks tau: the arithmetic prompt gets roughly twice the multiple that prose does.

DFlash raised loaded VRAM from roughly 17,582 MiB to 19,422 MiB. The 1.8 GiB cost bought speedups ranging from 1.52x to 3.11x in this matrix.

Five-repetition DFlash matrix

To check how stable the DFlash-on numbers were, I reran the three prompts at a logical batch size of 2,048 and a physical ubatch of 256, with temperature 0, top-p 1, top-k 1, seed 42, a 512-token cap, and non-streaming responses. Each workload had five measured repetitions after one excluded warm-up. All 15 retained responses reported 26 cached prompt tokens and reproduced the same output inside each workload.

The table reports medians across the five measured repetitions. Decode lands within 0.6 tokens/s of the comparison run on every prompt, and tau matches it exactly. Peak DFlash-on VRAM was 19,416–19,422 MiB across the retained matrix. No request failed and the process recorded no out-of-memory event.

Greedy output equivalence

The canonical assistant-output hashes differed between target-only and DFlash runs on all three prompts. Repeated DFlash passes reproduced the DFlash hashes exactly, so DFlash output is deterministic within that mode. llama.cpp issue #25618 tracks greedy divergence for quantized targets under parallel draft verification; upstream has not yet confirmed that Muse Glimmer with DFlash hits the same bug.

The image request also exercised the separate projector and completed in 0.276 seconds. Muse Glimmer accepts text and images and emits text. The release describes video support as sampled frames, with no audio input.

Meta's batch-one RTX 5090 result provides a larger-hardware reference: the model card reports 74.9 tokens/s for target-only decoding and 233.4 tokens/s with DFlash, a 3.1x increase. On the 3090, only the arithmetic prompt reached that multiple.

The useful 24 GB configuration

The conservative choices are doing real work here. One slot prevents a second KV cache from consuming the remaining margin. A 32K context avoids betting the process on the advertised maximum. Q8 KV cache saves memory while keeping the first test simple. A physical ubatch of 256 limits temporary allocation during prompt processing.

Context was the next tuning step. Serving at 32K left 4,701 MiB free on the card.

Raising the context to 98,304 tokens

My other local model server already offered a 98,304-token window, so a prompt it accepted would be refused here. I raised this server to match. The working assumption was that tripling the context would triple the KV cache, and on this model it does not. Only 13 of the target's 52 layers use full attention. The other 39 use sliding-window attention, and llama.cpp caps their cache at 2,304 cells no matter how large the context gets. All five of the DFlash drafter's layers are sliding-window too.

The allocator reported these KV cache sizes with target, projector, and drafter loaded at both values:

Tripling the context added 442 MiB, under 10% of the free memory. I derived the figures twice by independent routes, once from the allocator's own output and once from closed-form arithmetic on llama.cpp's cache-sizing code (GGML_PAD(min(size_base, n_swa + n_ubatch), 256), which gives the 2,304 cells), and the two agree exactly.

The drafter inherits the target's context length, but its KV cache never grows with it, because every one of its layers lands at 2,304 cells. Its compute buffer is the largest single term in the drafter's budget, and the ubatch size drives that buffer, not the context. That is one reason batch, ubatch, and slot count stayed where they were. Two switches would invalidate the whole budget: --swa-full, which would take target KV to 2.59 GiB, and any --parallel above 1.

The KV figures came from a CPU-side load. Tensor shapes do not depend on the backend, so they carry over to the GPU. The compute-buffer sizes printed by that run belong to a CPU-offload graph, though, so I left them out of the arithmetic.

If the budget is wrong, neither failure is graceful. --fit adjusts the context only when --ctx-size is unset; with an explicit size it logs context size set by user -> no change, so a budget that does not fit is a hard out-of-memory error at load and then a restart loop. llama.cpp can also load at a large context and then run out of compute memory on the first real request. I have seen that twice on a different machine, and the process passed a /metrics health check the whole time. A 200 from /metrics does not prove that generation works, so a large-prompt completion has to follow any context change.

I never pushed past 98,304 tokens on this card, so the 131K ceiling the release advertises remains unmeasured here. I also never measured how long a prefill of tens of thousands of tokens takes to return its first response byte at this context.

Running it as a service

The server runs a llama.cpp build pinned to one commit, so an upstream change cannot swap the binary behind these numbers. The three model files are checked against their published checksums before the server starts, and an alert fires when the server goes down.

What replaced it

Muse Glimmer served a code-review workload on this card for less than a day. Early on August 12 that workload moved back to Qwen 3.6; the first-impressions post records how Muse Glimmer behaved in that loop.

On August 14 the service definition switched to Qwen3-Coder-30B-A3B-Instruct at Q4_K_M, paired with a DFlash drafter trained for that target. The target file is 18.56 GB and the Q8_0 drafter is 0.51 GB, with no vision projector. Everything else carried over: the same b10355 build, 98,304-token context, Q8 KV cache, batch 2,048, ubatch 256, and one slot. The draft length went from 15 to 16. A health check no longer counted as proof that the model worked; a real code generation with accepted draft tokens and a complete tool call did.

The same evening, the service definition moved again to Qwen3.8-27B with its built-in MTP drafter, with the Qwen3-Coder and DFlash pair kept as the fallback if Qwen3.8 fails its checks. Muse Glimmer was the only configuration retired outright.