Benchmarking pruned GLM 5.3
Testing autotrusts new pruned GLM 5.3 to see if it can become my local darling
October 13, 2026
The folks at autotrust have been building pruned versions of GLM 5.3 with 256, 224, and now 192 experts. 192 even fits on a single B200 with a good chunk of space leftover for a nice K/V store... or of course a pair of DGX Sparks. So I asked Opus 5.5 to run through my usual benchmarks overnight with the results uploaded to this repo.
Pruned GLM still does quite well!
| Test | n | Qwen C | MiMo | GLM E192 | GLM vs Qwen: Qwen-only / GLM-only (p) | GLM vs MiMo: MiMo-only / GLM-only (p) |
|---|---|---|---|---|---|---|
| HumanEval+ (base), greedy, thinking off | 164 | 95.1 | 90.9 | 95.1 | ||
| HumanEval+ (plus), greedy, thinking off | 164 | 93.3 | 88.4 | 92.1 | 6 / 4 (p=0.75) | 6 / 12 (p=0.24) |
| MBPP+ (base) | 378 | 93.1 | 88.9 | 94.7 | ||
| MBPP+ (plus), greedy, thinking off | 378 | 79.9 | 74.3 | 79.9 | 16 / 16 (p=1.0) | 16 / 37 (p=0.005) |
| GSM8K, greedy, thinking off | 1319 | 96.4 | 96.4 | 96.3 | 26 / 25 (p=1.0) | 22 / 21 (p=1.0) |
| HumanEval+ (base), thinking | 164 | 95.1 | 95.1 | 97.0 | ||
| HumanEval+ (plus), thinking, 16K budget | 164 | 92.7 | 89.0 | 92.7 | 5 / 5 (p=1.0) | 7 / 13 (p=0.26) |
| LiveCodeBench (80 recent problems), thinking, 16K budget | 80 | 41.2 | 46.2 | 47.5 | 5 / 10 (p=0.30) | 6 / 7 (p=1.0) |
| Long context: retrieve / two-hop / reverse at 64K + 128K | 144 | 95.1% | 90.3% | 97.9% | 3 / 7 (p=0.34) | 2 / 13 (p=0.007) |
| ↳ two-hop only | 48 | 41/48 | 34/48 | 45/48 |
It's really smart. Really really smart. E224 was even smarter and still has plenty of room for 2M token key/value store:
| Test | n | Qwen C | E192 | E224 | E192-only / E224-only (p) | Qwen-only / E224-only (p) |
|---|---|---|---|---|---|---|
| HumanEval+ (base / plus), greedy, effort low | 164 | 95.1 / 93.3 | 95.1 / 92.1 | 95.1 / 92.1 | 7 / 7 (p=1.0) | 6 / 4 (p=0.75) |
| MBPP+ (base / plus), greedy, effort low | 378 | 93.1 / 79.9 | 94.7 / 79.9 | 94.4 / 82.0 | 11 / 19 (p=0.20) | 14 / 22 (p=0.24) |
| Long context 64K + 128K (retrieve / two-hop / reverse), effort low | 144 | 95.1% | 97.9% | 100% (two-hop 48/48) | 0 / 3 (p=0.25) | 0 / 7 (p=0.016) |
| Corpus NLL, bits/byte (side note) | 0.048 | 0.196 | 0.141 | top-1 agreement 87.6% |
If accuracy at difficult problems is your primary goal Autotrust's E224 is probably the best current model for a 2X DGX Spark setup. Now the bad: GLM is still quite slow. Unfortunately pruning experts doesn't speed the model up because the same number of experts are active at any time. We did see some weird (and unexplainable by thermals/clocks) results with GLM E192 that we didn't see with E224 where it stalled, but otherwise they both performed identically from a speed perspective.
| Qwen C | MiMo | GLM E192 | Verdict | |
|---|---|---|---|---|
| Single-user code decode, 1K, thinking off (tok/s, median of 3) | 66–70 | 37–52 | 20–32 | Qwen ×2.2–3.6 |
| Single-user code decode, 128K, thinking off | 65–68 | 25–30 | 20–31 | Qwen ×2.1–3.2; GLM ≈ MiMo |
| Decode at template default (Qwen thinking on / GLM effort max), 1K–128K | 48–56 | — | 27–29 | Qwen ×1.8–2.1 |
| Engine steps/s (flat 1K→128K?) | ~25.5 (yes) | 10.6 → 7.1 (no) | ~11.5 (yes), some runs 7.5 | |
| Recipe bench aggregate C1 / C6 (tok/s) | 58.3 / 228.2 | 42.8 / 139.6 | 21.0 / 88.0 | Qwen ×2.6–2.8 |
| Cold prefill 32K / 64K (tok/s) | 3,031 / 2,989 | 1,486 / 1,271 | 1,954 / 2,009 | Qwen ×1.5 |
For now I am going to stick with Qwen 3.8 Flash Next. The 200+ tok/s aggregates are pretty helpful when I have a full agent load going. But I am excited that Autotrust has made a GLM setup that works quite well on a pair of Sparks.