Qwen 3.8 Flash Next and the on-disk n-gram table
The on-disk n-gram table is truly a magic wand for performance
10/21/2026
Qwen 3.8 Flash Next has a 51B parameter n-gram embedding table in addition to its traditional 125B parameter weights. The little magic trick is that the n-gram table can sit on disk (realistically a fast NVME drive) and not require GPU RAM to serve it. With Nvidia's new tiny FP4-local quant of 3.8 Flash Next you now get full 3.8 Flash Next intelligence on a single DGX Spark with almost a million tokens of K/V.
On a pair of Sparks, I thought I'd try and see what kind of performance uplift you get by moving the n-gram table into RAM, which the b12x kernel supports. We can still have a 4.5+ million token K/V with the full n-gram in memory. The results? Consistent but not massive:
The n-gram table is an amazing unlock to allow such a small model to perform so "intelligently". Running it on disk barely costs you 1.5% of decode performance and 4% prefill speed on this setup with the Spark's relatively fast NVME disk. It is an amazing method for creating more RAM efficient models that perform way outside of their size class, game changer for local AI. Even serving models at scale in the data center, keeping the n-gram on disk is potentially worth it to just eek out more K/V size for concurrency.
I am going to keep the n-gram in memory going forward and take the small performance gains, but its definitely worth evaluating for your use case and again with future models. As always, my full tuning journey is documented here.