A Sovereign Open-Weight Model — Aleph Alpha

On the Day of German Reunification, we’re releasing our new mannequin: Kolibri.

Kolibri is an English-German Mixture-of-Experts Transformer with 78B whole parameters, 3B
energetic. It helps context lengths of as much as 1M tokens. The mannequin could be downloaded with the
full weights on Hugging Face and used beneath the Apache 2.0 license phrases.

Kolibri is the results of steady iteration of our mannequin coaching effort. We first constructed a model training pipeline and validated it by constructing Kolibri Origin, a 30B whole, 3B energetic mannequin with a a lot shorter
65k token context window. Kolibri ran via the identical pipeline: from information ingestion and curation,
via ablations, pre-training, and post-training, to the ultimate evals. It enabled operating a whole lot
of ablation experiments and secure pre-training that ran with out a individual having to step in when
{hardware} failed or a knowledge connection dropped. We repeatedly monitored coaching metrics and standardized
monitoring for customized benchmarks. The time we put into constructing and iterating on this pipeline
was a useful asset placement. We see it in how a lot better Kolibri is than Kolibri Origin, and in
how little time separates their releases.

Kolibri is a specialised language mannequin constructed for sovereign mission-critical work in
regulated areas together with public administration, industrials and aerospace. We specialised
Kolibri for German, reasoning, math, agentic conduct, and additional capabilities our
prospects want in manufacturing. The intention of this specialization was to optimize efficiency in
our prospects’ particular use circumstances. Through specialization, prospects obtain contextualized
efficiency of their AI operations and so they can monitor its financial impression, in order that ROI
stays measurable and grows over time.

Specialization alone shouldn’t be sufficient. Sovereignty is simply as necessary. Sovereignty, for us,
combines two dimensions: how we constructed the mannequin, and the way it transfers to our prospects. We
supply full supply-chain integrity and account for each resolution, from information ingestion,
via pre- and post-training, to the ultimate evaluations. We present transparency. Customers
have full freedom of deployment and intellectual-property security, so compliance comes as an
inherited property of the mannequin.

Read our tech report for full particulars.

What Kolibri Delivers

We optimized Kolibri for efficiency throughout a variety of sectors contemplating their
explicit domain-specific language, regulatory, and procedural realities. Its small and
environment friendly measurement offers our prospects with flexibility to run it effectively on-premise,
with out sending inside information to third-party inference companies. The highlight on this
part introduces the mannequin’s capabilities, earlier than we describe them in part How we built Kolibri at high velocity.

Foundational capabilities for enterprise and authorities

With Kolibri we optimize the trade-off between mannequin functionality and deployment prices, utilizing
3B energetic parameters out of 78B whole. Kolibri sits on the Pareto frontier for high quality
versus serving price, for each English and German. The Pareto frontier is an idea from
economics, marking one of the best achievable combos of two targets, the place bettering on
one means giving up a few of the different. None of the in contrast fashions delivers extra high quality at
the identical serving price, or the identical high quality at decrease price.

Across math, coding, grounding, and long-context duties, Kolibri matches fashions with as much as
4 occasions its energetic parameter rely, resembling Nemotron 3 Super.

Contextualized efficiency for real-world purposes

Public benchmarks fail to seize specialised sector wants, so we developed our personal inside
analysis suites for the verticals that matter to our prospects, such because the German public
sector, aviation, manufacturing and the automotive {industry}. Each suite mirrors the talents,
workflows, and edge circumstances required in these sectors, and paired artificial coaching
environments allow us to enhance Kolibri towards these evaluations with out ever coaching on
buyer information. Read extra in part Contextualized performance.

Answers which might be grounded in your paperwork. We educated Kolibri with abstention
information and with our Merlin-Arthur protocol. As a consequence, it’s educated to say “I do not know” when
the reply is not within the context. Our prospects worth and request this function, so we repeatedly
observe and validate abstention accuracy. We present extra particulars within the Grounding part.

Native German and English mannequin. We developed a bilingual German/English tokenizer
and targeted on including organic German data all through the coaching means of the mannequin, in order that 21.3% of the pre-training tokens are German.
We used translation sparingly (6% total), since translated textual content tends to hold the cultural
fingerprint of its supply language. The result’s a mannequin that’s bilingual by design, not an
English mannequin that has learn some German. Read extra in sections German pre-training data and Specialized tokenizers.

Control and compliance by design: upholding our prospects’ sovereignty

We constructed Kolibri with the EU AI Act, the General-Purpose AI Code of Practice and the GDPR in thoughts from the bottom up, with copyright regulation being a spotlight of our work to reliable
expertise.

We are clear about our model weights and about curation of our training data, in order that choices behind growth are seen. Through its reasoning traces, it
turns into explainable how the mannequin got here to a selected reply. With our Merlin-Arthur protocol the mannequin grounds trustworthiness: Kolibri refrains from answering when
the context does not help a solution.

Our groups constructed the mannequin in Germany, educated it on transport systems in Germany and Finland,
beneath European and German regulation, with no international management. We personal all the
pipeline, from information curation, via pre- and post-training, to optimization in our Model
Factory. This management of our finish to finish pipeline underpins the mannequin’s sovereignty. Control
additionally passes on to our prospects and Kolibri’s small measurement offers them full freedom of
deployment and helps controllable reasoning effort to commerce price and latency towards
reply high quality.

How We Built Kolibri at High Velocity

Our Model Factory: two fashions, three months aside

Our Model Factory is our reply to iteration pace, minimizing the time it takes to go from
figuring out what a mannequin will get mistaken to coaching one which does higher. We carried out the
coaching pipeline as code, in order that the learnings of our crew landed in a single versioned
coaching recipe as a substitute of scattered scripts and notes. Here is what that appeared like
between Kolibri Origin and Kolibri.

Work on the pipeline started in January. Five months and a whole lot of ablation runs later,
Kolibri Origin completed pre-training at goal scale on 11 June. Kolibri completed on 11
September. In the three months between these two dates we went from 30B parameters to 78B,
from a 65k-token context window to as much as 1M, and from 7.5T coaching tokens to 20T. To get
these 20T, the pipeline processed over 200T tokens of uncooked information, filtering, deduplicating,
and curating it right down to what we truly educated on. We modified the eye design,
tripled the variety of consultants, elevated sparsity, changed the routing algorithm, improved
our post-training information, greater than doubled the variety of surroundings duties, and taught the
mannequin to motive at 4 totally different effort ranges. Both fashions saved an analogous variety of
parameters energetic per token, but Kolibri educated sooner per token than Kolibri Origin thanks
to the work of our effectivity crew.

What made this potential is our coaching pipeline, an effort to transform research-grade mannequin
growth into a totally automated production-ready transport systems to design and practice giant
language fashions. Our pipeline codebase was shared by all. Every proposed code change triggers a small end-to-end mannequin run; coaching, analysis,
to seek out out if one thing broke inside minutes. Our runs are GitHub Actions workflows, and
reproducing one means testing a commit. We used the pipeline for the principle coaching and
the a whole lot of ablations that ran to make our architectures and data-mix choices.

Training checkpoints land roughly each hour and the pipeline robotically evaluates them
on English and German data, maths, code, instruction following, software use, lengthy context,
security, abstention to hallucination and grounding. We can watch capabilities seem each
day quite than discovering out how the mannequin turned out on the finish. The run itself held up
higher than we anticipated. Over 21 days of pre-training of Kolibri, we hit 38 unplanned
interruptions, roughly one per 10,000 GPU-hours, attributable to {hardware} faults or a connection
timing out. Those had been robotically dealt with by the pipeline with out handbook intervention.
The cluster robotically restarted the coaching job(s) on a unique set of nodes, and
coaching picked up from a checkpoint at most 250 steps again.

Having entry to the entire pipeline, with checkpoints out there to everybody, signifies that any
crew can personal a functionality finish to finish quite than a stage of an meeting line. One crew
educated and evaluated the grounding functionality of Kolibri as a single piece of labor,
seamlessly integrating into the general mannequin.

However, getting right here was not a linear stroll. We stumbled. We stopped the Kolibri Origin
pre-training after just a few trillion tokens and restarted it from scratch, as we recognized a
data-shuffling bug that escaped our exams. We ran ablations, took choices, solely to discover a
bug or an error within the configuration afterwards, forcing us to rerun some experiments.
Alongside our personal expertise, we tracked state-of-the-art architectures, finest practices, and
the most recent analysis advances in LLMs. Those amassed learnings led to enhancements in our
processes and guardrails in our pipeline.

Looking on the delta between Kolibri Origin and Kolibri, the development is notable, however
that is additionally the straightforward path: extra parameters, extra information, a well-understood structure
household, and nonetheless at small scale. As we’re considering scaling up, we’re glad to face
challenges on our basis. But probably the most sturdy factor we constructed this yr shouldn’t be the
pipeline, it’s a crew with the confirmed functionality to construct, post-train, and ship LLMs from
uncooked information at excessive velocity.

Architecture and pre-training

Kolibri has 78B whole parameters, which is 2.5 occasions greater than Kolibri Origin, with ~3B
energetic parameters. In our experiments, growing the dimensions of our mannequin from 32B to 123B led
to ever-improved efficiency. Yet the bigger measurement got here with bigger coaching and serving
prices. The latter drove the choice: 123B can deal with solely 3 long-context 256k-token person
queries on two H100s, whereas 78B handles 18 concurrent requests and decodes 28% sooner. We
used 384 smaller consultants quite than fewer huge ones, as they carried out higher in our exams.
We utilized the identical efficiency-first reasoning to consideration. Out of fifty whole layers, solely 10
course of full context, whereas the remaining 40 use a decent 512-token targeted window. This
retains decode computation and reminiscence bounded in these layers no matter context size.
The effectivity good points profit each serving the educated mannequin and coaching throughout
post-training RL, which depends on large quantities of inference.

We educated Kolibri on 768 B200 GPUs in three phases: 20T tokens of pre-training at a 16k sequence size over 21 days, 3.44T tokens
of mid-training at 64k, and 200B tokens of long-context adaptation at 256k. This is sort of 24T
tokens in whole, roughly thrice what Kolibri Origin consumed. German accounts for greater than
a fifth of the pre-training combine, about 4.3T tokens, towards roughly 62% English and 14% code.
Compared to pre-training, mid-training information is a way more selective pool of curated datasets
weighted towards reasoning, problem-solving, code and agentic information. For long-context, quite than
practice on lengthy paperwork alone, which tends to erode the talents acquired earlier, we interleaved
lengthy paperwork with the high-quality mid-training information of the earlier stage. For the long-context
combine, we eliminated artificial lengthy paperwork to keep away from artificially inflating benchmarks resembling RULER.

We optimized with Muon, as we did for Kolibri Origin. We put particular give attention to coaching
stability, leading to a sturdy coaching run with none loss spikes for both mannequin. With
Kolibri, for routing between consultants throughout coaching, we introduce precise quantile balancing.
Quantile balancing was launched in Kimi K3, the place the worldwide quantile is estimated from
histograms as a result of an actual computation was thought-about too costly to speak; we present
that it may be computed precisely at fastened price unbiased of batch measurement, and that the
exactness improves each load stability and mannequin high quality.

Post-training at scale

Post-training occurred in two phases; step one is supervised fine-tuning to show the
mannequin core reasoning means and learn how to work together in a chat, adopted by large-scale
reinforcement studying to coach the mannequin to motive throughout a various suite of long-horizon
duties. We constructed a pipeline for each of those steps, which allowed us to train
fine-grained management over the conduct of the mannequin.

For SFT we generated a complete of 174B tokens price of artificial information, which we filtered for
high quality and mixed with filtered variations of permissively licensed open-source datasets to
get hold of a high-quality coaching mixture of 268B tokens. For reinforcement studying we educated on
a broad set of environments that contained greater than 1.2 million curated duties throughout
various domains resembling code & math reasoning, agentic duties, instruction following,
query answering, software calling, and extra. We optimized our in-house coaching codebase for
high-performance and use asynchronous coaching, the place we generate coaching information on our
environments utilizing the present mannequin in parallel to coaching the mannequin on already generated
information.

Across each phases, we taught the mannequin to motive at totally different effort ranges (none, low,
medium, excessive), which signifies that the person can train management over how a lot compute the
mannequin ought to make investments to discover a resolution to the duty at hand. This permits our prospects to
trade-off price and inference pace towards the standard of the ultimate reply.

German pre-training information

From our experiments on small proxy fashions, we discovered that coaching with round 20% German
information results in optimum outcomes, which meant we wanted to seek out 4T German tokens to coach our
mannequin at a 20T horizon. Open German datasets assist, however are removed from sufficient: after
deduplication and filtering, we had been left with 390B German tokens, nicely wanting our aim.
As we argued in Sauerkraut, Not Burgers, German functionality has to return from high-quality German texts; relying closely on
machine-translations would result in poor outcomes on account of refined translation errors and a scarcity
of genuine German cultural context. We closed the token hole in three other ways.

The first was to curate German from Common Crawl ourselves. We constructed a pipeline specialised
for German information. German shouldn’t be English, and German information can’t be filtered like English
information: we needed to retune filtering parameters for the German language. One typical filter in a
language information pipeline is to take away paperwork with too many lengthy phrases, however German
administrative prose routinely exceeds the English certain on imply phrase size, so the
normal settings quietly take away the register that public administration writes in. After
retuning, our German pipeline gave us 1.3T distinctive tokens of natural German internet.

The second was to rephrase German paperwork we already had. An LLM rewrites an natural
German doc within the fashion of an encyclopedia entry, a Q&A dialogue or a textual content passage,
preserving its content material. This teaches the mannequin the identical details in a number of floor types and
multiplies the knowledge contained in scarce information, and it’s a totally different operation from
translation: the supply is German, so the subject material and the cultural affinity keep
German: chancellor, not president. It doesn’t add a lot new data, quite new phrasings
of data that was already within the corpus. Rephrasing gave us about 1T distinctive tokens,
making it the only largest supply of German within the mannequin.

The third was translation, and this we solely utilized in Kolibri Origin. Translating English into
German works when the mannequin, the prompts and the chunking are chosen rigorously, but it surely
carries two issues. Output can nonetheless present translationese, the literal rendering of idioms:
“Drive secure!” turns into “Fahre sicher!” quite than “Komm intestine an!”. More importantly, cultural
context doesn’t translate. A corpus translated from English inherits the geographic,
demographic and institutional distribution of the English internet, so a mannequin educated on it
speaks German a couple of globe that appears American.

In the top German entered Kolibri as a 2.4T-token distinctive pool, 80% of it curated or
generated by us and 20% from open datasets, and the mannequin noticed it at 21.3% of pre-training
tokens, roughly 4.3T over the 20T run via upsampling. Each German token was seen 1.8
occasions on common, nicely contained in the four-epoch restrict previous which repetition stops paying off.
Around 85% of the German the mannequin learn is internet textual content, both natural or rephrased from
natural. The relaxation is curated paperwork – parliamentary proceedings, authorized texts and different
information within the public area – and that small translated share.

Specialized tokenizers for English-German

We constructed a bilingual English-German tokenizer, educated and specialised on every mannequin’s
pre-training dataset. Because of the 21% German share in our information, it compresses German
language higher than different SOTA fashions. This led to extra environment friendly inference (fewer tokens)
minimizing prices and shortening response occasions. We introduce a brand new method to practice tokenizers,
UniBPE, that respects the morphology of languages higher than present approaches,
particularly the compound construction of German, with out sacrificing English token effectivity.

The two main approaches to coach tokenizers are BPE and Unigram. We mix these two, holding
the bottom-up strategy of BPE and utilizing the Unigram coaching goal for choosing which
merge so as to add to the vocabulary. On a 128k vocabulary educated on our English/German dataset,
this considerably improves tokenization.

Grounding: lowering hallucinations

Common LLM coaching and analysis rewards guessing: a guess has some likelihood of touchdown the
right reply, whereas abstention has none. Therefore, fashions study to reply with no matter
they’ve, even when their enter doesn’t present sufficient info to warrant a response.
Asking a mannequin to offer citations typically backfires for a similar motive: fashions can merely
hallucinate plausible-looking sources to justify an ungrounded reply.

For a regulated buyer, a mannequin that is aware of to abstain is the distinction between a pilot
and a deployment. We deal with saying “I do not know” as an necessary mannequin functionality and (1)
develop and observe devoted grounding and anti-hallucination measures and (2) use and
develop devoted coaching procedures to enhance this abstention functionality.

During coaching, we use coaching information samples the place the right reply is “I do not know”.
While fine-tuning for abstention is gaining traction industry-wide, high-quality unfavourable
examples stay onerous to return by, particularly the place our prospects want them, in slender
verticals the place all information is scarce. Therefore, we additionally practice with our Merlin-Arthur process, developed in-house and defined intimately in our blog post, which solves information shortage by robotically exploiting the mannequin’s weaknesses at every
coaching step and generates artificial unfavourable examples from out there paperwork. This
coaching turns into a recreation with three gamers. Arthur is the mannequin we practice and ship. Merlin
takes an present doc context, generates a brand new one which will increase Arthur’s likelihood
of answering accurately. Morgana generates a brand new datapoint by stripping out the related
proof from the doc and tries to lure Arthur right into a hallucination. Arthur doesn’t
know which one he is going through, so his legitimate technique turns into to rigorously learn the query and
the context, to have the ability to reply on Merlin’s context, whereas recognizing that abstention is
the strictly required reply for Morgana’s redacted context – making any guess, even a fortunate
one, incorrect.

For measuring hallucination abstention, we use established benchmarks alongside our personal
proxies derived from buyer use circumstances. We’ve additionally developed our personal “M/A grounding rating”
which falls out of our Merlin-Arthur setup representing a decrease certain on how a lot of the
reply provably got here from the doc.

Kolibri hallucinates far lower than Kolibri Origin: it abstains as a substitute of answering mistaken on
44% of AA-Omniscience objects (Origin: 15%), and on RGB it holds again
extra typically (86% vs 74%) and invents fewer falsehoods (87% vs 76%). On our personal M/A grounding score, which certifies quite than estimates how a lot of a solution got here from the doc,
Kolibri reaches 0.23 the place Kolibri Origin, alongside another fashions, attain 0.

Contextualized efficiency

Most public benchmarks fail to seize specialised sector wants. To measure contextualized
efficiency – that means how nicely a mannequin handles the distinctive workflows and area data of
particular industries – we’d like measurements of efficiency in specialised verticals that are
necessary to our prospects, such because the German public sector, authorized, {hardware}, client
electronics, and automotive.

For this, we first constructed domain-specific analysis suites that mirror the talents,
tool-calls, workflows, and edge circumstances required in key sectors and thus seize
contextualized efficiency. Once this was achieved, we created coaching environments by
producing high-quality specialised artificial information capturing these required abilities. To
enhance the real-world robustness of the mannequin on these duties, we moreover randomized
the environments round concrete particulars (instruments, configurations, harnesses).

Together, these contributed to an iterative engine that allowed us to sharpen the mannequin’s
agentic RAG capabilities, domain-specific logic and capabilities, and to hill-climb the
contextualized evaluations with out ever coaching on buyer information.

These analysis suites run robotically as a part of our pipeline: each checkpoint is
scored on them because it lands. Kolibri compares nicely towards competing open-weight fashions. On
the agentic-RAG benchmark Honeypot, Kolibri outperforms all in contrast fashions, together with the
a lot bigger Nemotron 3 Super and Mistral Small 4. On a cleaned model of the agentic-RAG
benchmark MuSiQue, it’s second solely to the extra large Nemotron 3 Super and nicely forward of
the remaining. On the 5 buyer purposes, Kolibri leads or is on par with one of the best mannequin
in 4 of 5. These findings communicate to our means to deal with a variety of information and harness
setups, throughout use-cases, whereas the in contrast fashions are delicate to those particulars.

These evaluations for measuring contextualized efficiency can exist as a result of prospects inform
us how their purposes fall brief. Each one encodes what we discovered from these
conversations: what sort of paperwork matter, which instruments the mannequin ought to get, which
questions are onerous, the place the deployed system frustrates its customers at this time. That makes the
alternate concrete in each instructions. A buyer who exhibits us a failure case will get it turned
right into a benchmark that each future checkpoint is measured towards, and we will hill-climb the
goal with out ever coaching on buyer information or risking over-fitting.

Benchmarks

Benchmarks had been run utilizing our own harnesses and, the place relevant, all fashions used the best respective reasoning effort.

Get Started

Our mannequin is accessible overtly on Hugging Face beneath an Apache 2.0 License.

Kolibri requires the aleph-alpha-inference bundle that gives the Kolibri
vLLM plugin. You can both use the supplied container picture ghcr.io/aleph-alpha/aleph-alpha-inference, or set up the bundle from Aleph-Alpha/aleph-alpha-inference, which additionally installs the vLLM model it helps:

pip set up "aleph-alpha-inference>=1.0"

Serve the mannequin with reasoning and tool-calling enabled.

vllm serve Aleph-Alpha/Kolibri-1 --kv-cache-dtype fp8 
  --reasoning-parser kolibri1 
  --tool-call-parser kolibri1 
  --enable-auto-tool-choice

To serve contexts past 262,144 tokens, add --max-model-len 1048576 --hf-overrides '{"max_position_embeddings": 1048576}'. The really useful sampling parameters for the mannequin are temperature=1.0, top_p=0.97 and top_k=128.

Contact for Deployment and Specialization

Contact our crew who can be glad to help you thru our enterprise deployment and
specialization choices: contact sales.