Sparticle62ops/pssa: A customized AI structure being developed in rust · GitHub

PSSA is a small language mannequin that isn’t a transformer. It reads textual content one
token at a time via a recurrent state-space layer, retains a financial institution of episodic
reminiscences it could look issues up in, and rewrites a part of its personal weights whereas it
runs. It is written in Rust from scratch, with no PyTorch, no TensorFlow, and no
ML framework of any type beneath it.

At matched parameters and on the identical corpus, it learns sooner than a
transformer and generates textual content about twelve instances faster on the identical CPU.

A transformer scores each pair of tokens within the context, so its price per step
grows with the sq. of the sequence size and the entire context is re-read at
each step. PSSA carries one fixed-size state alongside the sequence in a single
left-to-right cross, and appears issues up in a reminiscence financial institution as a substitute of re-reading
the context, so price grows linearly with size.

Two fashions, similar corpus, similar tokenizer, similar optimizer schedule, similar seed,
similar variety of parameters. One is PSSA, one is a typical transformer. Over
12.7M tokens of cleaned WikiText-103:

PSSA completed at 3.98 coaching cross-entropy, the transformer at 4.43.
That is a spot of 0.45 nats, perplexity 53.7 towards 83.7. The transformer
spent its whole 12.7M-token price range to succeed in a loss PSSA had already handed
round 2M tokens in.

The two curves by no means cross, they usually by no means contact:

Training loss solely says a mannequin match the stream it was fed. So each checkpoints
have been scored on a 198,939-token slice reduce from part of the corpus neither run
ever touched:

Every checkpoint of each runs, 64 PSSA hyperlinks and 43 transformer hyperlinks, scored on
a bounded 9,934-token window of that unseen slice. The curves by no means cross: PSSA
is forward from the primary hyperlink and finishes 0.51 nats decrease. The desk under is the
ultimate checkpoint of every run on the total slice.

The held-out hole, 0.43 nats, is actually the coaching hole. PSSA shouldn’t be
memorizing more durable, it’s generalizing higher.

Generating 200 tokens on the identical CPU, similar immediate, similar sampler:

A recurrent mannequin carries a fixed-size state, so the price of every new token does
not develop with the size of what got here earlier than. A transformer re-reads its complete
context each step.

• A recurrent state-space core. Learned steady state matrices carry
  info ahead in a fixed-size state, as a substitute of consideration over the total
  context window.
• An episodic reminiscence financial institution. 512 slots with hyperbolic (Poincare-style)
  retrieval and bounded top-4 search, written to and skim from throughout the run.
• Plastic weights. Fast updates reinforce what works, novelty drives development,
  and a refractory gate rate-limits overwrites so repeated contradictory enter
  does much less harm.
• Closed-form consolidation. A ridge-regression step folds the quick plastic
  updates again into the bottom transition matrix, the best way sleep consolidates a
  day’s studying.
• No framework. Hand-written linear algebra in Rust, with a CUDA path for
  coaching and a scalar CPU reference that each gradient is checked towards
  (max gradient distinction 2.98e-8).

Being straight concerning the scale, as a result of the numbers above are simple to
over-read:

• These are 1.5M-parameter fashions on 12.7M tokens. That is a analysis
  prototype, not a competitor to something you may have heard of.
• Text high quality at this scale is poor for each fashions. PSSA emits “a barget of
  the Prian Academy”, the transformer “a fabric circulation of the United
  States”. The comparability is about studying effectivity, not fluency.
• The pace comparability is CPU-to-CPU, which is honest. The coaching throughput
  numbers additional down are not hardware-matched and shouldn’t be learn as an
  structure consequence.
• Two experiments are nonetheless unmeasured: retention of earlier abilities after a
  corpus change, and whether or not ablating the reminiscence financial institution modifications the loss.

git clone https://github.com/Sparticle62ops/pssa.git
cd pssa
cargo construct --release
./goal/launch/oxide_ai_pssa

Running it with no arguments provides you a house display screen itemizing each command plus
any checkpoint and corpus it finds within the working listing.

The complete consequence above was skilled on a free hosted pocket book with a single
entry-level GPU, in 200,000-token hyperlinks, as a result of a session will get reduce after just a few
hours. Every fascinating query left, whether or not the hole holds at 10x or 100x
these parameters, whether or not the reminiscence financial institution issues at scale, the way it does towards
a contemporary recurrent baseline, wants one factor: a GPU with actual VRAM and
allocations measured in days as a substitute of hours. Anything meaningfully above the
entry-level card this ran on modifications what will be requested.

If you may have compute to grant, otherwise you work someplace that does, that’s the
single highest-leverage factor anybody can provide this mission.

Sponsorship funds compute and nothing else. In return you get named right here and in
the write-up of any consequence your {hardware} made doable. Get in contact earlier than
sending something so the main points will be agreed.

Issues and pull requests are welcome. The elements most in want of arms: kernel
efficiency, a contemporary recurrent baseline to check towards, and analysis
past next-token loss. Validate any department with cargo take a look at --release earlier than
opening a PR.

[email protected]

Solana: 4XPZ9uAa2BMoth6msoHRxTWL4mUrMfq3LGrxbAGja96h

Everything under is for operating, coaching, and dealing on the mission.

• Rust toolchain with Edition 2024 help, together with Cargo.
• Network entry solely when utilizing an HTTP/HTTPS dataset or a Hugging Face dataset.
• Enough reminiscence and disk for bigger corpora and serialized fashions.
• Optional: a CUDA system for the GPU coaching path. The CPU path is the
  reference and all the time out there.

Direct runtime dependencies are ureq for
dataset downloads and tokenizers for
byte-level BPE.

Both chains ran 64 hyperlinks of 200,000 encoded tokens, every hyperlink resuming from the
earlier checkpoint, so the learning-rate schedule and optimizer state proceed
throughout the entire run as a substitute of restarting per hyperlink.

• Identical corpus: one clean-wikitext cross over WikiText-103, reused byte for byte.
• Identical token IDs: the baseline pins --tokenizer-from to the PSSA chain’s
  personal checkpoint, so neither mannequin sees a special vocabulary.
• Identical optimization: 30,000-update cosine horizon, no warm-up restart, 512
  supervised goal tokens per replace, seed 42.
• PSSA: latent 256, recurrent state 16, 512 reminiscence slots, key width 32, vocab 2,048.
• Baseline: 1,541,120 parameters, 1 layer, width 256, 4 heads, FFN 448, vocab 2,048.

End-of-link coaching cross-entropy:

The baseline’s first session was reduce at hyperlink 43 by the pocket book session restrict
and its loss CSV didn’t survive, so hyperlinks 1 to 43 are learn again from that
session’s personal run log as a substitute. The chain resumed from ck43 in a second session
and completed all 64 hyperlinks, and each curves above now cowl the total run.

PSSA skilled on a Kaggle T4 at roughly 900 tokens/second. The baseline is
CPU-only, as a result of train-transformer has no GPU path, and held 212
tokens/second. Those two numbers say nothing concerning the architectures. On the
similar CPU-only Kaggle {hardware} the batched PSSA path measures 375 tokens/second
towards the baseline’s 212, and the loss comparability above is unaffected both
means, since it’s matched on tokens and updates fairly than on time.

The losses are end-of-link coaching cross-entropy on the stream being match, not
held-out analysis. For a held-out comparability on an unseen slice, use the
evaluate command described in docs/COMPARISON.md.
Generation high quality at this scale is poor for each fashions: PSSA emits “a barget
of the Prian Academy”, the baseline “a fabric circulation of the United
States”.

Two experiments aren’t but measured: retention of earlier abilities after a
corpus change, and whether or not ablating the 512 reminiscence slots modifications loss.

bash kaggle/kaggle_continue.sh              # the PSSA chain
bash kaggle/kaggle_transformer_baseline.sh  # the parameter-matched baseline

Both learn TOTAL, WINDOW and FRESH from the atmosphere and write
--loss-csv, so the curve survives a reduce session.

General kind:

Commands:

Options:

Positional arguments and lengthy/quick choices will be blended:

cargo run --release -- prepare knowledge/downloaded.txt -e 2 -o knowledge/experiment.pssa
cargo run --release -- prepare --data knowledge/downloaded.txt --epochs 2 --out knowledge/experiment.pssa

Inside the REPL:

• /exit or stop exits the method.
• /data prints the loaded mannequin path, reminiscence slot depend, and adapter depend.
• /temp stories a temperature worth however doesn’t apply it to later turns. Pass --temp when launching chat as a substitute.

--skip-tokens, --max-tokens and --resume collectively let an extended corpus be skilled as a sequence of quick runs, so a single run by no means has to outlive a session restrict. If a window crosses EOF, choice wraps to the start of the corpus. Each hyperlink trains its personal window and arms its optimizer state to the subsequent:

cargo run --release -- prepare knowledge/downloaded.txt -e 1 
  --skip-tokens 0      --max-tokens 200000 -o chain/ck01.pssa
cargo run --release -- prepare knowledge/downloaded.txt -e 1 
  --skip-tokens 200000 --max-tokens 200000 --resume chain/ck01.pssa -o chain/ck02.pssa

kaggle/kaggle_continue.sh drives this sample finish to finish: it units a window dimension and a hyperlink depend, walks the corpus offset by offset, and resumes every hyperlink from the earlier checkpoint. standing then stories each checkpoint within the chain with its form and optimizer step depend.

DatasetSupervisor accepts a number of comma-separated sources:

cargo run --release -- prepare science                       # built-in reference corpus
cargo run --release -- prepare knowledge/downloaded.txt           # native textual content file
cargo run --release -- prepare knowledge/                         # each readable file in a listing
cargo run --release -- prepare https://example.org/corpus.txt
cargo run --release -- prepare hf:proprietor/dataset              # Hugging Face repository
cargo run --release -- prepare science,knowledge/downloaded.txt   # a number of sources

Local information and directories are learn immediately; HTTP(S) URLs and express hf:proprietor/dataset
sources are downloaded. Structured responses are diminished utilizing widespread fields comparable to
textual content, content material, article, story, instruction, output, sentence, and abstract;
structured responses with no supported textual content discipline are rejected.

Byte-level BPE retains actual UTF-8 case, whitespace, punctuation, and line endings, and has a whole 256-byte fallback alphabet, so legitimate UTF-8 by no means collapses to . The earlier lowercase phrase splitter, together with its 10,000-word cap and habits, is offered solely with --tokenizer phrase.

Download a Hugging Face dataset into an area textual content file:

cargo run --release -- obtain wikimedia/wikipedia --out knowledge/downloaded.txt

Network downloads aren’t validated or curated by Oxide AI. Review licensing, privateness, and content material earlier than coaching on an exterior corpus.

Clean extracted wikitext-103-raw textual content earlier than a recent coaching run:

./goal/launch/oxide_ai_pssa clean-wikitext wiki.prepare.uncooked --out knowledge/wikitext-clean.txt
./goal/launch/oxide_ai_pssa prepare knowledge/wikitext-clean.txt -o knowledge/mannequin.pssa
# Also out there: oxide_ai_pssa assist clean-wikitext

The similar command can be utilized in Kaggle after extracting textual content from Parquet; it
accepts an area UTF-8 textual content file, not Parquet itself. -o and --out are aliases.
The output path is required and should not exist already (together with the enter
path or a hyperlink to it). This protects the unique corpus; select a brand new output
title for one more run. Read, UTF-8, and write failures exit nonzero via the
regular CLI error path, with partial output eliminated when doable.

The cross:

• Joins @-@, @.@, and @,@ to adjoining textual content: visitor @-@ starring →
  guest-starring, 52 @.@ 9 → 52.9, 500 @,@ 000 → 500,000.
• Drops balanced heading strains comparable to = Title = and = = Section = =.
• Removes and collapses remaining inline whitespace to single areas.
• Removes areas earlier than ., ,, ) and after (; trims every line.
• Retains at most one consecutive clean line, together with at the beginning/finish.
  Removing a heading doesn’t introduce a clean line.
• Writes LF line endings, together with a newline on the final retained line.

oxide_ai_pssa::dataset::clean_wikitext(reader, author) is the reusable library
API (BufRead / Write, returning std::io::Result<()>). The CLI makes use of buffered
file I/O, and the cleaner retains solely its enter/output line buffers: reminiscence is
proportional to the longest line, not the corpus dimension. Library callers utilizing a
buffered author should flush it themselves; the CLI explicitly checks the flush.
No new dependencies are required.

Cleaning is opt-in: current loaders, tokenizers, coaching instructions, and
kaggle/kaggle_continue.sh are unchanged. Do not change an in-flight resume
chain to a cleaned corpus: cleansing modifications token IDs/counts and the which means
of --skip-tokens offsets. Prepare and constantly reuse one cleaned corpus
for a brand new chain as a substitute.

The prepare command performs two phases:

1. Continuous recurrent ingestion: token transitions are processed via the PSSA layer. The mannequin updates state, reminiscence, adapters, and routing habits with a cosine learning-rate schedule.
2. Adapter consolidation: after every epoch, the plastic adapter’s quick coefficients are folded into its consolidated coefficients with the configured EMA price.

Defaults are latent 256, recurrent state 16, memory-key 32, reminiscence capability 512, chunk size 64, studying price 1e-3, 8 chunks per replace, and seed 42. The ensuing binary holds weights, configuration, reminiscence, adapters, and optimizer state. It shouldn’t be an interchange format for different ML frameworks and needs to be loaded via PSSALayer::import_from_pssa_bytes.

New saves use V7: the total V6 coaching/resume payload plus a bounded, length-prefixed normal tokenizer JSON. A V7 BPE checkpoint is self-contained and restores its actual ordered vocabulary with out entry to the coaching or analysis corpus. generate and chat reject --data for V7 BPE as a result of retraining a tokenizer on exterior knowledge wouldn’t validate provenance. V7 phrase checkpoints and V6 checkpoints retain the legacy optionally available --data exact-vocabulary comparability. Checked V5 artifacts stay inference-only and require --data as a result of they by no means contained tokenizer provenance.

Generation is autoregressive and makes use of temperature 0.70, a top-24 candidate restrict adopted by top-p 0.85 filtering, a 1.25 repetition penalty over a latest 64-token window, instant self-transition suppression, suppression, and a default cap of 64 new tokens, ending early after two generated durations.

V7 BPE inference restores the precise embedded tokenizer and by no means rebuilds it from a specific dataset. Evaluation provides its knowledge solely as held-out textual content to the restored tokenizer.

cargo run --release -- benchmark

The suite workout routines artificial streams for contradictory details, MQAR-style distractors, burst repetition, mannequin serialization, and quick era prompts. It prints milestone outcomes, shouldn’t be wired into Cargo’s take a look at harness, and isn’t a top quality analysis on basic language duties.

cargo fmt --all -- --check
cargo clippy --release --all-targets
cargo take a look at --release

Integration checks stay in checks/: allocations.rs, bpe_repair.rs, checkpoint_repair.rs, core_repair.rs, linalg.rs, and runtime_repair.rs, with shared artifacts beneath checks/fixtures/. They cowl tokenizer spherical journeys, checkpoint import/export throughout variations, linear-algebra kernels, allocation habits, and CLI runtime output. Clippy is clear of errors; numerous fashion warnings within the numeric kernels are left in place intentionally, since rewriting listed loops there would churn code the gradient checks pin down.

• CPU-oriented prototype with hand-written linear algebra. gpu-probe verifies a WebGPU system and a GEMM towards the CPU reference, however coaching and inference nonetheless run the layer math on the CPU.
• The CLI parser is deliberately minimal: no shell-style quoting, and little validation past numeric parsing.
• A lacking or unreadable dataset silently falls again to the built-in science corpus in a number of loading paths.
• Model and tokenizer vocabularies should stay appropriate; a dimension warning doesn’t restore a mismatch.
• Model form can’t change throughout a resume chain: latent, state, key, reminiscence and vocabulary should match the checkpoint being resumed.
• Downloaded content material will be massive and will include JSON, malformed textual content, or knowledge unsuitable for coaching.
• The REPL temperature command acknowledges a price with out altering the lively configuration.
• Benchmark output is milestone-oriented and doesn’t measure perplexity, factuality, latency, or security.
• Serialized .pssa information are project-specific binary artifacts with out model migration tooling.

See LICENSE for the mission license.