Close to-Lossless Compression in a 9x Smaller Footprint
Two months in the past, we launched our first Bonsai 27B fashions and confirmed {that a} 27B-class multimodal mannequin could possibly be compressed sufficient to run effectively on a neighborhood machine. Today, we’re releasing Ternary Bonsai 2 27B, our most succesful mannequin but.
Based on Qwen3.8 27B, Ternary Bonsai 2 27B brings stronger reasoning, coding, imaginative and prescient, and agentic functionality to the Bonsai collection whereas preserving the deployment profile that defines it: a dramatically smaller reminiscence footprint, excessive native throughput, and higher vitality effectivity.
Ternary Bonsai 2 27B makes use of ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 efficient bits per weight and a complete mannequin footprint of 5.9GB. The low-bit illustration is utilized finish to finish throughout the language mannequin. It helps a 262K-token context window, multimodal text-and-image enter, and is launched beneath the Apache 2.0 license.
Against its full-precision counterpart, Ternary Bonsai 2 27B is greater than 9x smaller whereas retaining 98.2% of mixture benchmark efficiency. At this degree of retention, compression turns into a deployment unlock: practically the identical functionality, in a footprint that may run in much more locations.
What modified from the primary Bonsai 27B launch
Our first Bonsai 27B launch was an vital milestone, providing a sensible approach to run 27B-class intelligence on native gadgets. Bonsai 2 27B focuses on the following step: enhancing the mannequin high quality and runtime efficiency wanted for real-world native purposes. Compared with the earlier Bonsai 27B technology, Bonsai 2 27B brings:
- a stronger base mannequin, Qwen3.8 27B
- increased mixture functionality retention of 98.2% in opposition to the full-precision mannequin
- improved reasoning, coding, imaginative and prescient, and long-horizon agentic efficiency
Higher functionality on the similar deployment level
Across a benchmark suite spanning reasoning, math, coding, instruction following, imaginative and prescient, and agentic software use, Ternary Bonsai 2 27B scores 83.9, retaining 98.2% of Qwen3.8 27B’s mixture efficiency.
Figure I: Benchmark scores of Ternary Bonsai 2 27B (pondering mode) in contrast with the full-precision Qwen3.8 27B and Qwen3.6 27B baselines. Full per-benchmark outcomes are within the whitepaper.
The key outcome shouldn’t be solely the combination rating, however the place the aptitude is retained. Coding brokers, tool-use programs, multimodal workflows, and long-horizon duties are significantly delicate to mannequin degradation as a result of small errors can compound over many steps. Bonsai 2 27B preserves a lot of the full-precision mannequin’s efficiency in precisely these areas whereas working at a fraction of the reminiscence footprint.
Compared with the full-precision mannequin and different low-bit options, Bonsai 2 27B stands out as an outlier on intelligence density. Many low-bit options grow to be deployable solely by giving up significant functionality in coding, imaginative and prescient, or agentic software use. Bonsai 2 27B pushes the frontier towards each increased functionality and decrease reminiscence utilization.
With Bonsai 2 27B, native fashions can begin to tackle actual information work: coding-agent loops, computer-use workflows, non-public doc evaluation, multimodal debugging, and hybrid orchestration the place native fashions deal with delicate or high-frequency duties whereas escalating selectively to the cloud.
Throughput and vitality effectivity
Ternary Bonsai 2 27B reaches as much as 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes simply 0.714 mWh/token, making it 40% extra energy-efficient than an 8B mannequin operating in full-precision.
For coding assistants, increased throughput means sooner edit-debug loops. For multimodal brokers, it means faster iterations over screenshots, paperwork, and gear calls. For non-public native workflows, higher vitality effectivity means extra helpful inference on the identical machine, longer battery life, and a extra life like path to assistants that may keep obtainable within the background with out continuously calling the cloud.
Why this launch issues
Compared to Ternary Bonsai 27B, the brand new Ternary Bonsai 2 27B has closed the retention hole between the complete precision mannequin from 95% to over 98%. This is a major enchancment that makes the present launch virtually “lossless”. It additional cements the notion that low-bit fashions might be one of the best ways to deploy AI.
That has implications properly past native inference. Low-bit fashions can change the economics and structure of AI programs throughout gadgets, workstations, and datacenters: becoming bigger fashions into the identical reminiscence envelope, serving extra customers on the identical {hardware}, lowering vitality per inference, and enabling hybrid programs that dynamically resolve what ought to run domestically and what ought to run within the cloud.
The query will more and more be not simply how succesful a mannequin is, however how a lot helpful intelligence might be delivered inside a given reminiscence, compute, and energy funds. If functionality can proceed to scale whereas these necessities fall dramatically, the deployment envelope for future fashions expands throughout the stack: from private gadgets to large-scale datacenters.
Platform Coverage
Bonsai 2 27B runs on NVIDIA GPUs by way of CUDA and on Apple gadgets (Mac, iPhone, iPad) by way of MLX, by way of customized low-bit kernels. Model weights can be found immediately beneath the Apache 2.0 License.
Full technical particulars of our compression, analysis, and benchmarking processes can be found in our whitepaper.
Work with Us
We work with groups to tailor Bonsai fashions to their purposes, from post-training on domain-specific knowledge to optimizing inference for goal {hardware}. If you’re constructing AI merchandise with tight reminiscence, latency, or energy necessities, we’d like to discover how Bonsai will help. Reach out at [email protected].
Join Us
PrismML emerged from a staff of Caltech researchers and was based with help from Khosla Ventures, Cerberus, and Google, with persevering with help from Samsung. We’ve spent years tackling one of many subject’s hardest issues: compressing neural networks with out sacrificing their reasoning capability.
If you need to assist construct the following technology of state-of-the-art AI, we would love to listen to from you. Check out our careers page.
