Flip GLM-5.3-Flash right into a Jev-like System One mannequin

TL;DR: In this put up, we present how an off-the-shelf LLM could make typed choices in a single ahead move. This strategy makes it doable to show an LLM right into a Jev-like resolution mannequin.

We consider the strategy utilizing GLM-5.3-Flash operating on Privatemode. Using a benchmark constructed from public knowledge units, we present that this setup delivers outcomes which might be on par with TypeSafe’s Jev by way of resolution accuracy/correctness and pace.

As a bonus, the setup with GLM-5.3-Flash permits typed choices on photos, which isn’t doable with Jev.

Why typed choices

Much of what software program asks an LLM is a choice. “Which workforce ought to deal with this ticket?”, or “Does this contract clause belong within the legal responsibility part?”.

In such instances, software program usually requires that the LLM’s response follows a sure format like JSON and that it comes from a pre-defined set like “sure” and “no”.

Given the best directions, LLMs can usually already fulfill this reliably. However, within the fundamental strategy, pace and prices develop into a difficulty: For every resolution, the LLM wants to put in writing a complete JSON object, and a reasoning mannequin might imagine for lots of of tokens earlier than that. Further, you additionally do not study the arrogance of the mannequin (until you explicitly ask it). All these facets can matter so much in observe and have to this point prevented folks from using LLMs for resolution making in high-volume/high-throughput situations.

Specialized resolution fashions (or “System One” fashions) like Jev and Laya are designed to handle this. You move in a bit of state and a set of named choices, and also you get again the chosen possibility along with a confidence worth (i.e., chance) for each.

Turning an LLM into a choice mannequin

Initially, we requested ourselves if an LLM might be was a choice mannequin with Jev-like properties. The quick reply is: “sure”. In the next, we present the way it works.

To perceive our strategy, it is necessary to know how LLMs work:

An LLM by no means writes textual content instantly. Given a immediate, an LLM outputs a chance distribution over its whole vocabulary of tokens. In textual content technology, within the easiest case, the token with the best chance is chosen because the subsequent token. The chosen token then is appended to the immediate and the entire course of repeats. As described above, that is expensive and sluggish if you happen to simply need to set a number of fields in a JSON object.

Our core perception is that it is pointless to have the LLM predict the entire JSON object, as we already know its form. We’re solely within the LLM’s typed judgement for a given enter.

We realized that it is doable to craft prompts in order that we get the typed judgement in a single run of the LLM — with no fine-tuning, on the mannequin precisely because it ships. This is the distinction to Jev and Laya, that are fashions skilled for the aim. The fundamental steps are as follows:

1. Number the choices. The state, the query, and the output choices go into the immediate as JSON, with an index on each possibility. The instruction asks the mannequin to reply with choice_index: adopted by an index.

2. Prefill the reply. The immediate ends with choice_index:. Consequently, the primary token the mannequin produces will probably be an index into the pre-defined choices.

3. Evaluate the output. Rather than studying the token the mannequin emits, we learn the possibilities it assigned to all possibility indexes at that single place. Normalized over the choices, these give a chance for every reply, and we merely choose probably the most possible one.

End the immediate inside the reply

consumer

{“state”: “I used to be charged twice for my order.”, “query”: “Which workforce?”, “choices”: [

{“index”: 0, “name”: “payments“}{“index”: 1, “name”: “complaints“}{“index”: 2, “name”: “technical“}

]}

assistant

choice_index:

Read one row of logits

max_tokens: 1temperature: 0allowed_token_ids

Keep the choices, renormalize

• 0 funds62.1%

• 1 complaints37.7%

• 2 technical0.2%

Answer funds
confidence 0.39, the place 1 means sure and 0 means the choices are equally probably

We carried out the above steps for GLM-5.3-Flash operating on vLLM (in Privatemode). We use the /chat/completions endpoint with continue_final_message and add_generation_prompt: false, as a result of these let the mannequin proceed the prefilled assistant flip from step 2 as an alternative of beginning a brand new one. They additionally allow us to move photos subsequent to the textual content, which is what makes typed choices on photos doable.

For vLLM and GLM-5.3-Flash, we discovered the next particulars to matter:

1. vLLM’s allowed_token_ids can be utilized to restrict the LLM’s output vocabulary solely to allowed choices. It drops each different token to -inf. We set it, however it’s a guardrail reasonably than a requirement.
2. top_logprobs is just not sufficient for step 3. It reviews the distribution earlier than the restriction is utilized, so formatting tokens akin to a number one area take up the highest slots, and a few choices drop off the listing and seem to have a chance of zero. vLLM’s logprob_token_ids solves this: it returns the log chance of precisely the token ids you ask for.
3. The token ids of the indexes rely upon the mannequin’s tokenizer. Digits aren’t all the time single tokens. GLM-5.3-Flash, for instance, has a single token for 12. Rather than delivery a model-specific tokenizer, the library will get the token ids from the server, which retains it easy to make use of with any mannequin: sending a immediate to /completions with echo returns its actual tokenization by the mannequin that’s really serving.

You can discover our implementation within the under repository.

edgelesssys/privatemode-decisionsThe Python library: token oracle, prompt, masking and renormalization, against any vLLM-backed endpoint.

Try it

The playground under runs GLM-5.3-Flash queried with the above setup on Privatemode, instantly out of your browser.
Pick one of many examples, amongst them a scanned bill and a query that is determined by your native time, or write your personal questions and add photos.
Each reply comes again as a distribution over its choices, usually inside a number of hundred milliseconds.

The distribution is usually very helpful, e. g., to determine whether or not to incorporate a human-in-the-loop. The mannequin solves most basic trick questions, however not all of them.

Benchmark outcomes

We evaluated our strategy utilizing a customized benchmark, which is obtainable within the under repository.

edgelesssys/privatemode-decisions-benchmarkThe benchmark: methodology, frozen dataset specs, harness and aggregation. Every number in this post can be recomputed from it.

We in contrast three techniques on 29 public, labeled datasets: GLM-5.3-Flash hosted on Privatemode and queried with the method above, TypeSafe’s Jev, and Convai’s Laya.

The datasets have between 2 and 151 choices and canopy intent routing, sentiment, matter classification, moderation, entailment, query answering, authorized textual content, and scanned paperwork. Both English and German textual content is included within the corpus.
All three techniques obtain the identical state, the identical possibility names in the identical order, and the identical instruction.

We ran every dataset twice. Even at temperature 0, our GLM-5.3-Flash and Jev modified as much as 3.5% of their solutions between equivalent runs: temperature 0 removes the randomness from sampling, however batching and floating-point arithmetic nonetheless hold a ahead move on a busy server from being bit-reproducible. We due to this fact deal with smaller variations as noise.

We ran Jev and Laya with their default settings and didn’t tune our immediate on these datasets.

Accuracy

Compared throughout the 28 textual content datasets, GLM-5.3-Flash and Jev are on par.
Each is extra correct on 10 datasets; on the remaining 8, the 2 are inside one share level of one another.
The median hole is 0.7 share factors in Jev’s favor, which isn’t statistically vital (p = 0.64).
Laya, a mannequin with 421 million parameters that we ran regionally, scores decrease than each on most datasets. Its median hole is 13 to fifteen share factors, which is statistically vital (p < 0.001).

The variety of choices has a bigger impact on accuracy than the selection between Jev and GLM-5.3-Flash.

Which one is extra correct, dataset by dataset?

10 GLM-5.3-Flash extra correct8 about the identical10 Jev extra correct

The typical hole is 0.7 share factors, barely in Jev’s favor. Two equally correct techniques would present a spot at the least this huge in about 6 out of 10 comparisons, so it’s effectively inside probability.

Accuracy by variety of choices

GLM-5.3-FlashJevLaya

with reasoningembedding similarity

Across datasets, the variety of choices adjustments together with the whole lot else in regards to the activity.
Three datasets, nevertheless, label the identical questions twice, as soon as coarsely and as soon as finely, so the duty stays the identical and solely the variety of choices adjustments.
On TREC, going from 6 to 42 choices, Jev drops from 92.1% to 85.6%, GLM-5.3-Flash from 91.2% to 79.6%, and Laya from 88.4% to 51.2%.
On MASSIVE, going from 18 situations to 59 intents raises the scores of Jev and GLM-5.3-Flash in each languages, whereas Laya’s rating drops.
The variety of choices alone does not decide how onerous a activity is.

GLM-5.3-Flash on PrivatemodeJevLaya

TREC questions

MASSIVE, English

MASSIVE, German

Latency and value

We measured latency in separate runs with one request at a time, as a result of timings taken underneath load measure the queue reasonably than the mannequin.

As Privatemode is hosted within the EU and Jev is hosted within the US, we ran 4 of the datasets from Germany and from the US on the similar time.
From Germany, Privatemode answered in 180 ms and Jev in 264 ms.
From the US, the order reverses: 164 ms for Jev in opposition to 299 ms for Privatemode.

On price, Jev is cheaper. One million choices price about EUR 62 with GLM-5.3-Flash and about EUR 16 with Jev, at every service’s listing costs.

Time per resolution

From Germany

GLM-5.3-Flash on Privatemode180 ms (173–263)

From the US

GLM-5.3-Flash on Privatemode299 ms (271–402)

0 ms450 ms

Cost per million choices

GLM-5.3-Flash on Privatemode€62

Most of the distinction comes from the worth per enter token, and a few from how every system packages a query.
Jev provides roughly 270 tokens of fastened overhead and about 10 tokens per possibility.
The GLM-5.3-Flash immediate provides about 55 tokens of fastened overhead and about 20 per possibility.
Below about 21 choices, it sends fewer tokens than Jev; above that, it sends extra.

Extra enter tokens GLM-5.3-Flash sends per query, in contrast with Jev

Below zero GLM-5.3-Flash sends fewer tokens, above zero extra.

Laya runs regionally, so there isn’t a comparable latency or value per resolution for it.

Multimodal choices

The state does not should be textual content. GLM-5.3-Flash is a vision-capable mannequin, so a query can include photos, akin to a scanned bill, a photograph of a broken parcel, or a screenshot.
The picture goes into the identical immediate, and the reply continues to be a single token with a chance for each possibility. The playground’s Scanned doc instance reveals this; you may as well paste or drop in your personal picture.

According to its documentation, Jev works on textual content, and Laya is a textual content encoder, so neither takes photos as enter.
On RVL-CDIP, a set of 1,600 scanned trade paperwork in 16 courses, GLM-5.3-Flash reaches an accuracy of 70.2% and is the one one of many three that may reply.
A doc prices greater than a sentence: the picture provides about 1,350 enter tokens, so one million doc choices price about EUR 270.

Capabilities

With many choices, every system hits a restrict. Laya’s possibility names share a funds of 192 tokens, which is sufficient for the 77 intents of banking77 however not for the 151 intents of CLINC150.
Privatemode’s deployment of GLM-5.3-Flash reviews at most 128 entries in logprob_token_ids, whereas the masks in allowed_token_ids takes each possibility.
So the library sends a query with 151 choices twice, identically, and reads the possibilities of 128 choices from the primary response and of the opposite 23 from the second. Both requests run the identical ahead move, so the merged result’s the distribution a single request would return, as much as the run-to-run noise above.
On CLINC150, GLM-5.3-Flash reaches 87.5% this manner, in opposition to 78.4% for Jev. The second request and the lengthy possibility listing price time: a choice takes 719 ms, in opposition to 249 ms for Jev.
Past that, the ceiling is the 191 possibility indexes that GLM-5.3-Flash spells as a single token.

Some duties just one or two of the techniques can deal with in any respect.

	GLM-5.3-Flash on Privatemode	Jev	Laya
Scanned paperwork RVL-CDIP, 1,600 trade paperwork, 16 varieties	70.2%	textual content solely	textual content solely
Choice of mannequin Same code, one other mannequin	any mannequin on the API	fastened	fastened

Further findings

Some of the remaining errors are within the labels. When most techniques agree on a solution and the dataset’s label disagrees, the label is usually one in every of two defensible solutions.
banking77 has many such pairs, for instance get_physical_card and order_physical_card, or declined_transfer and failed_transfer. About 17% of banking77’s examples fall into this class, so the best rating any system might attain there may be about 85% reasonably than 100%.

Reasoning helps, at a value. As a management, we let the identical GLM-5.3-Flash cause earlier than it solutions, on all 29 datasets.
It is extra correct in each band of possibility counts, from 89.9% in opposition to 85.5% with two choices to 82.0% in opposition to 79.2% between 21 and 80.
It additionally writes lots of of tokens per resolution as an alternative of 1 and prices about EUR 350 per million choices, in opposition to EUR 62.
At the opposite finish, plain embedding similarity, which picks the choice closest to the textual content with no resolution mannequin in any respect, reaches between 45.9% and 72.8% relying on the band.

Renaming the choices impacts the techniques in a different way. We re-ran each dataset with every possibility changed by a synonym and nothing else modified.
On boolq, the place true and false grew to become appropriate and unsuitable, GLM-5.3-Flash misplaced 20 factors, whereas the opposite two misplaced lower than three.
Because this renaming additionally adjustments the that means of the query, it does not isolate memorization. The outcomes for each dataset are within the benchmark repository.

Build it your self

Our library is written in Python and works with any vLLM-backed endpoint; it depends on vLLM’s extensions to the OpenAI API, akin to allowed_token_ids and logprob_token_ids.
Its README explains how one can set it up with Privatemode, and an AGENTS.md file tells coding brokers what an implementation has to get proper.
The benchmark repository incorporates the methodology, the dataset specs, the check harness, the aggregation, and each uncooked run as a obtain, so each quantity on this put up will be reproduced with out operating something once more.
If you recognize a setting that serves any of the three techniques higher, discover a mistake, or need to add a dataset or one other system, we welcome pull requests.

With Privatemode, these choices are protected by confidential computing: your knowledge stays encrypted in reminiscence even throughout processing, and the consumer verifies the deployment’s attestation report earlier than sending something.
You can learn extra on our security page.

Build it on Privatemode

Create an account, then give this put up to Claude Code, Codex, or the coding device of your selection. Together with the GitHub repository, it has the whole lot wanted to construct typed choices into your personal code, protected by confidential computing.