Testing Nvidia's new FP4-local quant of Qwen 3.8 Flash Next
An incredible model specifically designed for single DGX Spark or RTX 6000 Pro setups.
10/16/2026
Nvidia released what they've called an "FP4-local" quant of my current favorite Qwen 3.8 Flash Next. Unlike their previous official NVFP4 quant this one is designed for a single DGX Spark or a single RTX 6000 Pro. With a single Spark you'd have a key/value store of around 850-900k tokens, and on a single RTX 6000 Pro at least 650k-750k tokens.
I decided to modify their recipe and benchmark it on a TP=2 dual Spark setup as part of my continuing LLM tuning saga. Two Sparks allows for a K/V store of >5M tokens and lots of headroom to spare on each node. As usual full results in GitHub.
The good if not incredible news, FP4-local was essentially as smart than nvidia's official NVFP4 quant, and smarter than the faster quants I had been using. Nvidia did quantization-aware distillation on it. The result is probably the smartest model that can run on 96GB of RAM.
| Test | n | A nvidia NVFP4 | B RadixArk BF16-dense | C RadixArk FP8-dense (prod) | FP4-Local | C-only / F4-only (p) |
|---|---|---|---|---|---|---|
| HumanEval+ base / plus, thinking off | 164 | 97.0 / 95.1 | 95.7 / 93.9 | 95.1 / 93.3 | 97.6 / 94.5 | 3 / 5 (p=0.73) |
| MBPP+ base / plus, thinking off | 378 | 93.9 / 79.9 | 92.3 / 79.4 | 93.1 / 79.9 | 93.4 / 79.9 | 5 / 5 (p=1.0) |
| GSM8K, thinking off | 1319 | 96.5 | 96.4 | 96.4 | 96.4 | 14 / 15 (p=1.0) |
| HumanEval+ base / plus, thinking on, 16K budget | 164 | — | 95.1 / 92.7 | 95.1 / 92.7 | 95.7 / 92.7 | 4 / 4 (p=1.0) |
| LiveCodeBench (80 problems), thinking on, 16K budget | 80 | — | 43.8 | 41.2 | 46.2 | 0 / 4 (p=0.125) |
| Long context 64K + 128K (retrieve / two-hop / reverse) | 144 | — | 96.5% (two-hop 43/48) | 95.1% (41/48) | 97.2% (44/48) | 0 / 3 (p=0.25) |
| Corpus NLL, nats/token (190,191 identical tokens) | 0.1315 | 0.1310 | 0.1340 | 0.1333 |
Now the downside, its a bit slower than the other quants. FP4-local is as the name describes designed for single GPU setups. Running it with TP=2 works but we do give up a little bit of speed based on the old quant and configuration: the new DGX Spark and RTX 6000 focused b12x kernel does not support expert parallelization. We're using a nightly vLLM build, so maybe this will come eventually. I benchmarked it across all levels of speculative coding (my previous config "C" has been running on K=2)
| ctx | task | config C | FP4-Local K=4 | K=3 | K=2 |
|---|---|---|---|---|---|
| thinking off | |||||
| 1K | code | 66.3 | 69.0 (3.39, 20.3) | 66.2 (3.01, 22.0) | 64.0 (2.60, 24.6) |
| 1K | code_ts | 68.0 | 72.0 (3.51, 20.5) | 69.9 (3.18, 22.0) | 64.0 (2.59, 24.8) |
| 1K | edit | 70.1 | 77.4 (3.96, 19.6) | 73.1 (3.42, 21.4) | 66.8 (2.73, 24.5) |
| 128K | code | 65.2 | 61.4 (3.14, 19.6) | 66.0 (3.08, 21.4) | 61.7 (2.56, 24.1) |
| 128K | code_ts | 65.1 | 63.5 (3.22, 19.7) | 64.8 (2.99, 21.7) | 62.9 (2.61, 24.1) |
| 128K | edit | 67.6 | 68.5 (3.51, 19.5) | 62.0 (2.92, 21.3) | 60.4 (2.56, 23.6) |
| mean | 67.0 | 68.7 | 67.0 | 63.3 | |
| thinking on (template default) | |||||
| 1K | code | 56.4 | 47.7 (2.39, 20.0) | 43.8 (2.00, 21.9) | 50.0 (2.05, 24.3) |
| 1K | code_ts | 49.2 | 44.3 (2.23, 19.9) | 47.0 (2.16, 21.7) | 46.5 (1.90, 24.4) |
| 1K | edit | 54.8 | 49.7 (2.51, 19.8) | 54.3 (2.47, 22.0) | 57.4 (2.33, 24.6) |
| 128K | code | 51.3 | 42.9 (2.19, 19.6) | 48.3 (2.27, 21.3) | 48.5 (2.03, 23.9) |
| 128K | code_ts | 48.2 | 41.5 (2.13, 19.5) | 43.4 (2.05, 21.2) | 43.6 (1.82, 23.9) |
| 128K | edit | 52.5 | 48.8 (2.51, 19.4) | 52.0 (2.45, 21.3) | 51.4 (2.18, 23.6) |
| mean | 52.1 | 45.8 | 48.1 | 49.5 |
Thinking-on speeds drop across the board, and it gets worse with concurrency, about 20% slower at full capacity. I think its acceptable loss given some of the other benefits, like the increased headroom and slightly improved quality, so this will be the new quant and configuration I run locally. The recipe nvidia shipped with the NVFP4-local model also greatly improves the configuration for warm-cache prefix hits. Agent turnarounds are a lot faster with pi.dev across the 4 agents I typically have working.
This is THE option for a single Spark setup. On an RTX 6000 Pro the model will absolutely scream as long as you don't need a larger K/V store. Once EP works with the b12x kernel its going to be the no brainer 2x Spark model too.