ShapeLearn-Lite Held Up. ShapeLearn Did Higher: Qwen 3.8 27B

TL;DR

• Full ShapeLearn strikes the measured quality-speed frontier past Lite. All 5 fashions within the new launch sit on the frontier in every of our six GPU comparisons.
• GPU-5 is our default suggestion wherever it suits, reaching 99.63% of BF16’s mixture benchmark rating. If it doesn’t match with the context you want, GPU-4 remains to be very aggressive: it reaches 98.72% of BF16 at a a lot smaller dimension (11.0 GB as a substitute of 13.1 GB), and it’s sooner.
• ShapeLearn-Lite additionally carried out higher than its KLD rating steered: three of its six fashions sit on the frontier within the Lite-versus-Unsloth Dynamic v3 comparability.
• Speculative Decoding with MTP or DFlash2 will increase throughput throughout each ShapeLearn mannequin and GPU examined. DFlash2 is normally sooner however requires extra reminiscence and doesn’t help picture inputs with llama.cpp. Choose DFlash2 for max text-only throughput when reminiscence permits, and MTP when VRAM or multimodal help issues extra.

Full ShapeLearn strikes the frontier

We are releasing the total ShapeLearn run for Qwen 3.8 27B.

Within this launch, bigger fashions yield larger mixture scores, whereas smaller fashions ship larger throughput. That ordering holds throughout all six GPUs examined. Because it is a dense mannequin and reminiscence transfers are the bottleneck, decrease BPW interprets extra immediately into larger TPS than it does for MoEs.

The per-GPU comparisons additionally embrace AtomicChat, Bartowski, ISTA-DASLab, and Unsloth Dynamic v3. Bartowski’s newest fashions had been launched after our testing and usually are not included. Full ShapeLearn is labelled ByteShape within the figures.

All 5 ShapeLearn fashions stay on the measured frontier, with GPU-5 attaining the very best mixture rating among the many plotted quants. Other groups additionally contribute aggressive factors. Notably, ISTA-DASLab’s glorious mannequin (the yellow “d” on the graph under) additionally sits on the frontier.

By “frontier,” we imply that no different plotted mannequin is each sooner and extra correct.

96 GB: RTX Pro 6000

RTX PRO 6000, the GPU with essentially the most reminiscence, lets us present the total vary of fashions examined.

GPU-5 is our default wherever you possibly can match it: it reaches 90.4 tok/s at 99.63% of the BF16 baseline.

32 GB: RTX 5090

The RTX 5090 tells an analogous story, resulting in the identical suggestions.

Once once more GPU-5 is our default selection, reaching 93.7 tok/s. Choose GPU-4 for barely extra context size or barely higher TPS.

24 GB: RTX 4090 and RTX 3090

Both 24 GB playing cards match all 5 ShapeLearn fashions. We plot them individually as a result of their throughput differs, however the ordering is similar on each.

RTX 4090

The RTX 4090 retains the identical sample: GPU-5 is the default, reaching 59.2 tok/s.

RTX 3090

Older, however nonetheless quick in these measurements.

GPU-4 reaches 49.5 tok/s, in contrast with 45.8 tok/s for GPU-5. Moving to the bigger mannequin prices about 7.5% in throughput, whereas the mixture rating rises from 98.72% to 99.63% of BF16. That makes GPU-5 the default right here as nicely.

16 GB: RTX 4080 and RTX 5060 Ti

With a tighter VRAM price range, the pragmatic selection is to go away room for the context you want, not simply the mannequin weights. These plots include fewer competing configurations, however all 5 ShapeLearn fashions are represented.

RTX 4080

On the RTX 4080, GPU-4 reaches 52.4 tok/s, whereas GPU-5 reaches 45.7 tok/s.

RTX 5060 Ti

On the RTX 5060 Ti, the corresponding figures are 33.1 tok/s and 29.1 tok/s.

GPU-5 stays the default on each playing cards when the mannequin, KV cache, and runtime buffers match inside your reminiscence price range. When they don’t, GPU-4 remains to be very aggressive: virtually 99% of BF16 at a a lot smaller dimension, and sooner. A mannequin showing in these measurements doesn’t set up that each context size or serving configuration will match.

ShapeLearn-Lite, looking back

ShapeLearn-Lite makes use of a smaller optimization price range than full ShapeLearn. It allow us to get Qwen 3.8 27B onto 12 GB to 24 GB GPUs inside a couple of days.

We launched after focused sanity checks and began the total analysis afterwards. The full ShapeLearn fashions had been prepared earlier than the benchmarking was completed. Evaluating each units, together with the competing fashions, is what took more often than not.

Then Unsloth launched its Dynamic v3 fashions. At comparable sizes, a number of had decrease KLD than Lite in our measurements. On KLD alone, Lite seemed much less aggressive.

KLD seemed decisive

KLD measures divergence between a quantized mannequin’s predicted token distributions and the BF16 reference underneath a selected analysis setup. It is helpful for diagnosing substantial modifications, however decrease divergence doesn’t robotically imply higher job efficiency.

We measure KLD on a dataset of about 5 million tokens of immediate and response pairs, drawn from a number of benchmarks, together with long-context and agentic duties. We additionally modified how KLD is computed, in order that it’s nearer to what we count on KLD to measure:

• KLD is measured on response tokens solely, not on immediate tokens. We don’t wish to measure how nicely a mannequin can generate prompts.
• KLD solely considers the tokens which have an opportunity of being sampled throughout era, the top-20, top-40, or top-60 tokens at every place. The tail tokens by no means get sampled, so they don’t contribute.
• Requests have clear boundaries. Each immediate and response pair is scored as its personal request, not as a part of one lengthy concatenated stream.

For instance, Unsloth’s UD-IQ3_S (vii) has about 20% decrease KLD than the equally sized smallest Lite mannequin (Lite-1): 0.028759 versus 0.035875. Yet its mixture benchmark rating is decrease: 95.55% versus 97.33% of BF16.

If decrease KLD had been adequate to rank these fashions by job efficiency, the benchmark ordering ought to have adopted it.

It didn’t.

The level is just not that KLD is ineffective. It is {that a} constancy rating is just not a task-performance rating. This is the excellence explored in our KLD evaluation blog. Our related paper on KLD and quantization fidelity metrics was additionally not too long ago accepted to the EMNLP Industry Track.

Lite held up

Naturally, we made extra plots.

Here, we present the RTX Pro 6000 as a result of it may possibly accommodate the total comparability. Each mannequin’s benchmark rating is reused throughout the GPU plots; the measured throughput and the set of displayed fashions change.

Leaving the total ShapeLearn fashions apart for a second, three of the six ShapeLearn-Lite fashions sit on the Lite-versus-Unsloth frontier: the three smallest Lite fashions, the lighter orange bubbles labelled 1-3.

Of the twelve Unsloth v3 fashions proven, three additionally sit on that frontier: UD-IQ2_S (A), UD-Q2_K_XL (B), and UD-IQ4_XS (F). UD-IQ4_XS (F) is a robust higher-quality level, whereas Lite earns its locations in the midst of the vary.

Add the 5 full ShapeLearn fashions again in (the darker orange bubbles), and so they take over the complete frontier.

Lite was by no means meant to be the ultimate consequence. It nonetheless held its personal the place it mattered.

Speculative Decoding

We additionally evaluated MTP and DFlash2 with the brand new fashions, utilizing 3 draft tokens for MTP and seven draft tokens for DFlash2. Both strategies elevated throughput for all 5 ShapeLearn fashions on all six GPUs examined.

DFlash2 was sooner than MTP in virtually all circumstances. Across the total lineup, DFlash2 reached 1.34-2.10x the baseline next-token prediction (NTP) throughput, whereas MTP reached 1.28-1.66x.

We measured with the sampling parameters Qwen recommends for pondering mode, over a various set of agentic coding, arithmetic, and general-knowledge requests. The speedups would possible be bigger underneath grasping decoding, however temperature-based sampling higher displays actual utilization.

The determine under exhibits NTP, MTP, and DFlash2 throughput for every GPU. The high quality axis is the target-model benchmark rating reported above. These plots don’t independently set up high quality equivalence between decoding strategies.

There can also be a reminiscence tradeoff between the 2 approaches. The embedded quantized MTP weights add solely about 250 MB to the mannequin, and if MTP is just not used, these weights usually are not loaded into GPU reminiscence. In comparability, the 4-bit DFlash2 draft mannequin is about 1.1 GB, so enabling DFlash2 requires roughly 1.1 GB of extra GPU reminiscence.

Packaging MTP as a separate GGUF file would largely get rid of this benefit. The standalone mannequin would want its personal MTP embedding and output layers, that are by far its largest tensors, bringing its reminiscence footprint to roughly 1 GB as nicely.

In addition, DFlash2 in llama.cpp presently doesn’t help picture inputs, which is a crucial consideration for multimodal use circumstances.

Benchmarking Methodology

We consider all reported fashions throughout a set of instruct and pondering benchmarks.

Instruct benchmarks:

• GSM8K for math
• IFEval for instruction following
• MMLU for normal data
• LiveCodeBench V6* for coding
• Multi-IF for multi-turn and multilingual instruction following
• ACEBench for device use and agentic duties

Thinking benchmarks:

• ACEBench for device use and agentic duties
• Multiple HumanEval for coding
• BFCL V4* for device calling and agentic duties

For the pondering benchmarks, we used Qwen 3.8’s medium pondering setting.

For every benchmark, the rating of a quantized mannequin is normalized by the rating of the corresponding BF16 mannequin. The general reported rating is the common of those normalized benchmark scores.

Our LiveCodeBench V6* analysis contains issues from January 1, 2024 onward, excluding the 2023 issues. We discovered the 2023 issues to be comparatively straightforward for present fashions, with most fashions attaining very excessive scores on them. As a consequence, they supply restricted discrimination between fashions whereas including substantial analysis time.

For BFCL V4*, we consider the next eight subsets:

• live_simple
• live_parallel
• live_parallel_multiple
• live_relevance
• multi_turn_base
• multi_turn_miss_func
• multi_turn_miss_param
• multi_turn_long_context

All evaluations had been run with llama.cpp b10430. For each instruct and pondering experiments, we use the sampling parameters advisable by Qwen for the corresponding mode.

Conclusion

ShapeLearn-Lite did what it was designed to do. It obtained helpful Qwen 3.8 27B quants onto 12 to 24 GB GPUs rapidly, and it held up higher than its KLD rating steered.

Full ShapeLearn goes additional. It improves the measured quality-speed trade-offs over Lite and contributes 5 frontier fashions throughout all six examined GPUs.

GPU-5 is our default suggestion wherever it suits, reaching 99.63% of BF16’s mixture benchmark rating. When reminiscence is tight, GPU-4 remains to be very aggressive: virtually 99% of BF16 at a a lot smaller dimension, and sooner.

KLD stays helpful, however it isn’t a task-performance leaderboard. Fidelity metrics inform us how a lot the mannequin’s distributions modified underneath a selected measurement. Benchmarks inform us whether or not these modifications matter on the duties we examined.

We had been impatient. This time, it labored out fairly nicely.