jaredpalmer/kev: tiny Jev-like household of choice fashions constructed on prime of Qwen3.5 you’ll be able to prepare and run by yourself · GitHub

Training makes use of cross-entropy on the proper reply. The adapter and head are skilled collectively; the remainder of the bottom weights keep mounted. Training examples and API requests use the identical textual content format. No Jev outputs had been used for coaching.

Asking questions collectively or individually produces possibilities inside 4e-6 within the fp32 assessments. This does not imply choice order is irrelevant: choices inside a query can nonetheless have an effect on each other. See the model code and parity tests.

On CUDA, set up flash-linear-attention for the Qwen3.5 fashions (the Modal picture does this); a five-question request takes tens of milliseconds on an H100.

On Apple Silicon there are not any quick kernels for the DeltaNet layers, so PyTorch runs reference code. Median mannequin time in bf16 on an M5, 5 questions with three choices every on a ~230-token state:

If you serve on a Mac and wish low latency, use the Qwen3 fashions for now. An MLX backend for the Qwen3.5 fashions is the subsequent deliberate change.

For the attention-only fashions the server merges the LoRA weights in fp32 earlier than casting, makes use of SDPA consideration on Apple GPUs, pads MPS inputs to 64-token buckets, and caches the state prefix for repeated requests (4 states of at the least 384 tokens by default). With a repeated 772-token state, Kev-4B (Qwen3) solutions in 242 ms as an alternative of 861 ms.

You can disable these with KEV_MERGE=0, KEV_ATTN=keen, KEV_SHAPE_BUCKET=1, and KEV_PREFIX_CACHE=0. On 24 new-source information, bf16 possibilities differed from fp32 by at most 0.017, with no change within the highest-probability reply. That is a small verify, not a assure for each enter.

The launched fashions use decision-v7: 10,000 examples from ten public datasets, 896 generated coverage examples, and 1,680 examples from 60 generated rule constructions. All prepare for 2 epochs with LoRA rank 16 and cross-entropy. The studying fee is 1e-4 for 0.8B and 5e-5 for 4B/9B. For Qwen3.5 bases the adapter additionally covers the DeltaNet projections; kev.prepare picks the proper targets from the mannequin config.

# sanity run, ~1 minute
uv run python -m kev.prepare --n_per_source 40 --accum 4 --out runs/smoke

# Kev-0.8B (~20 min on one H100; the Mac path works however is gradual for Qwen3.5 bases)
uv run python -m kev.prepare --suite evals/v7/decision-v7 --base Qwen/Qwen3.5-0.8B-Base --base_revision dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68 
    --epochs 2 --lr 1e-4 --batch 8 --dtype bf16 --p_none_pair 0.25 --device cuda --out runs/kev-0.8b

# the Kev-4B recipe (one H100 by way of Modal, ~1 h; see under). Swap in Qwen/Qwen3-4B-Base for the earlier era.
uv run python -m kev.prepare --suite evals/v7/decision-v7 --base Qwen/Qwen3.5-4B-Base --base_revision 1001bb4d826a52d1f399e183466143f4da7b741b 
    --epochs 2 --lr 5e-5 --batch 4 --accum 2 --dtype bf16 --checkpointing 1 --p_none_pair 0.25 --device cuda --out runs/kev-4b

The launched fashions had been skilled on public datasets and generated coverage examples. If your questions look totally different — your individual routing classes, your individual escalation guidelines, one other language — a brief fine-tune on a number of hundred labelled examples often helps greater than any immediate change.

Put your examples in a JSONL file, one request per line. It’s the identical form as an API request, plus a label on each query:

{"state": {"topic": "Charged twice", "physique": "I see two costs for order #4411. Please refund one."},
 "questions": {
   "staff":     {"sort": "alternative", "directions": "Which staff ought to deal with this ticket?",
                "standards": {"billing": "Payments and refunds", "transport": "Delivery issues", "entry": "Login and account entry"}, "label": "billing"},
   "offended":    {"sort": "noul",   "directions": "Is the shopper offended?", "label": false},
   "precedence": {"sort": "rating",  "directions": "How pressing is that this ticket?", "standards": ["low", "normal", "high"], "label": 1}}}

For alternative the label is the choice identify, for noul it is true or false, and for rating it is the extent’s place beginning at 0. Keep 10–20% of the file apart for analysis.

Then begin from a launched checkpoint with --init_from:

uv run python -m kev.prepare --data prepare.jsonl --base Qwen/Qwen3.5-4B-Base --init_from jaredpalmer/kev-4b 
    --epochs 2 --lr 2e-5 --batch 1 --accum 8 --dtype bf16 --checkpointing 1 --device cuda --out runs/mine

uv run python -m kev.benchmark --run runs/mine --data heldout.jsonl --out runs/mine-eval
KEV_DTYPE=bf16 uv run --extra serve python -m kev.serve --run runs/mine --port 8009

--init_from hundreds the adapter and pointer head from the launched mannequin earlier than coaching, so you retain what Kev already is aware of and add your area on prime. Starting from the bottom mannequin as an alternative throws that away: in a single person’s take a look at on 836 support-tool selections, a fine-tune from the bottom scored 0.33 on Kev’s personal analysis set, towards 0.84 for the launched mannequin; the identical information with --init_from stored 0.83 there and reached 0.88 on the brand new area. Use a smaller studying fee than the from-scratch recipe (2e-5 is an effective begin), and decide --base to match the checkpoint you begin from; the coach checks that the bottom, revision, LoRA rank, and head measurement agree earlier than it hundreds something.

--batch 1 --accum 8 in bf16 suits the 0.8B mannequin on a 4 GB GPU. The benchmark reviews accuracy, Brier rating, and calibration per query sort, so you’ll be able to see which of your questions the fine-tune helped. The checkpoint you began from is recorded in runs/mine/training_config.json.

Use uv run python -m kev.prepare --help for all coaching choices. The launched fashions do not use the optionally available --perm_kl or --ord_w losses. The model cards have the coaching settings and dataset lists; PLAN.md information what was tried and what helped.

On a Mac, run one coaching job at a time. Two jobs on the identical Apple GPU are a lot slower. Use Modal for longer runs.

Each trial will get its personal H100. The examine retains operating in the event you disconnect, and you may obtain the outcomes when it finishes:

uv run modal token new                                    # as soon as; opens the browser
KEV_GPU=T4 uv run modal run modal_app.py::smoke           # end-to-end verify, ~1 minute of GPU

uv run modal deploy modal_app.py                          # as soon as; research run on the deployed app and survive disconnects
uv run modal run modal_app.py::examine 
    --suite evals/v7/decision-v7 --plan experiments/v7-final.json 
    --name my-study --transfer evals/v4/transfer-v4 --budget 30 --timeout 7200
uv run modal run modal_app.py::pull --name my-study       # outcomes -> runs/my-study, ranked

Study plans checklist coaching settings. Each trial saves the settings, code hashes, dataset hashes, and outcomes. Choose fashions utilizing the event outcomes, not the locked take a look at. After selecting a last candidate, you’ll be able to learn its take a look at outcomes as soon as:

uv run modal run modal_app.py::locked_test --trial my-study/00-trial-0 --name my-candidate   # one learn, ever

The analysis information below evals/ is frozen: dataset variations and file checksums are recorded in every manifest. Large coaching recordsdata are downloaded from the Hub mirror and checked towards these hashes.

uv run python -m kev.benchmark --run jaredpalmer/kev-4b --suite evals/v4/transfer-v4 --out runs/my-eval      # out of area
uv run python -m kev.benchmark --run jaredpalmer/kev-4b --suite evals/v9/transfer-v9 --out runs/my-eval-v9   # + MMLU-Pro, buried states, unknowable objects
uv run python -m kev.benchmark --run jaredpalmer/kev-4b --suite evals/v7/decision-v7 --out runs/my-eval-id   # in distribution
uv run python -m kev.benchmark --remote http://127.0.0.1:8009 --suite evals/v4/transfer-v4 --out runs/my-remote   # any System One endpoint

These instructions use improvement information. Test information requires --allow-test. The benchmark reviews accuracy, Brier rating, calibration error, the share of choices you would automate at a 5% error price range, option-order modifications, and query isolation. transfer-v9 provides 10-way MMLU-Pro, information buried amongst unrelated textual content, and “unknowable” information whose deciding proof was eliminated; for these it reviews how typically the mannequin nonetheless solutions with at the least 0.9 confidence (Kev-9B 5%, Jev 9%, Kev-8B 26%). Published accuracy numbers use fp32 analysis, not the bf16 serving path.

evals/exterior/ holds two different tasks’ take a look at units transformed to this format, with their revealed stay Jev outcomes: SemIf‘s 144 authored selections (Kev-9B 0.917, Jev 0.965) and scienthoon‘s 900 help tickets (Kev-9B 0.952 on routing and 0.911 on tone, Jev 0.897 and 0.914).

kev.jev runs the identical questions towards Jev via Vercel AI Gateway. kev.examine compares two saved runs with paired bootstrap confidence intervals. For the complete experiment historical past, see PLAN.md and the leaderboard.

• Probabilities aren’t nicely calibrated on new sources. On the new-source improvement set, Kev-4B assigns at the least 0.9 likelihood to a flawed reply on 8.2% of questions (Kev-9B: 7.5%). Test it by yourself information earlier than selecting a likelihood threshold.
• Fine-tuning could make the bottom mannequin worse at particular person duties. Date arithmetic is the clearest case: the untrained Qwen3.5-9B base will get 0.82 on the deadline coverage questions and Kev-9B will get 0.72, as a result of coaching erodes the talent (issue #8, PLAN_Qwen35.md). Knowledge questions (MMLU 0.74 vs Jev 0.90) are the opposite massive hole.
• The present fashions are gradual on Apple Silicon (see Serving Performance) and wish transformers >= 5.17.
• Changing choice order can change a solution. Question isolation would not forestall this.
• Training makes use of at most 384 state tokens and 1,024 tokens for the state plus one query. Serving permits 8,192 tokens for the state plus one query; longer context wasn’t lined by coaching.
• The server handles one request at a time. It caches repeated state textual content, however would not batch requests from totally different callers.

uv run --extra serve python -m pytest assessments/test_unit.py assessments/test_research.py -q  # no weights, no server; runs in CI
KEV_BASE_URL=http://127.0.0.1:8009 uv run --extra serve python -m pytest assessments/test_api.py -q   # towards a operating server
cd playground && npm run lint && npx subsequent typegen && npx tsc --noEmit -p .

The API assessments run TypeProtected’s instance requests and the official SDK towards your native server.

Built with Devin. Thanks to Archer Hume for the structure write-up, TypeSafe for the API design, Qwen for the bottom fashions, 3x3xX3N0N for displaying the place the date-arithmetic failure actually is, and Radexito for --init_from.

Related work: Hydragen, DeFT, FIRST.

Apache-2.0. The Qwen3 and Qwen3.5 base fashions are additionally Apache-2.0. Training datasets have their very own licenses; see the model cards.