Deser: Rethinking Rust Serialization | Armin Ronacher’s Ideas and Writings

written on September 29, 2026

Serde is a tremendous serialization library for Rust and it
has been an enormous cause why I felt productive with it for years. However already
whereas at Sentry I received fairly annoyed with among the limitations with it however
truly changing Serde is hard due to the would possibly that it has within the
ecosystem. Also as a result of it’s fairly exhausting to really do higher with out additionally
making some probably painful compromises.

Here are three examples of Serde nook instances that present poor interactions of
Serde options or surprising limitations:

#[derive(Deserialize)]
#[serde(tag = "type")]
enum Shape {
    Circle { radius: f64 },
}

serde_json::from_str::<Shape>(r#"{"sort": "Circle", "radius": 1.5}"#)
// error: invalid sort: map, anticipated f64

#[derive(Deserialize)]
struct Stats {
    scores: HashMap<u32, u32>,
}

#[derive(Deserialize)]
struct Report {
    identify: String,
    #[serde(flatten)]
    stats: Stats,
}

serde_json::from_str::<Report>(r#"{"identify": "x", "scores": {"42": 23}}"#)
// error: invalid sort: string "42", anticipated u32 at line 1 column 35

fn from_hex<'de, D: Deserializer<'de>>(d: D) -> Result<u32, D::Error> { ... }

#[derive(Deserialize)]
struct Theme {
    #[serde(deserialize_with = "from_hex")]
    main: u32,
    #[serde(deserialize_with = "from_hex")]
    accent: Option<u32>,
}

//error[E0308]: `?` operator has incompatible sorts
//  |
//  |     #[serde(deserialize_with = "from_hex")]
//  |                                ^^^^^^^^^^ anticipated `Option`, discovered `u32`
//  |
//assist: attempt wrapping the expression in `Some`
//  |
//  |     #[serde(deserialize_with = Some("from_hex"))]
//  |                                +++++          +

None of those are bugs which can be straightforward to repair in Serde. They fall out of its
design, and that design is protected by Serde’s stability ensures.

Back in 2022 I began an experiment referred to as
Deser. It’s a serialization library for
Rust that takes the consumer expertise of Serde and places it on
prime of a totally completely different structure impressed by
miniserde. I by no means actually completed it
and it sat round for a couple of years. I picked it again up, and it has now reached
some extent the place I feel it’s price taking a look at. Even simply to encourage others to
see in the event that they need to discover the area.

The Name And Idea

The identify is Serde with its two halves swapped. Deser is Serde however the different manner
round. In Serde, a sort drives the deserialization course of: a Deserialize
impl asks the deserializer for the type of worth it expects, the format calls
again right into a customer. Every nested worth is dealt with by recursion which makes
Serde deserialization inherently develop the stack with every degree of nesting.

Deser then again turns this round and the format tells the kind of the
subsequent worth and pushes occasions right into a sink. When a sink hits the beginning of a
nested worth, it doesn’t name into it however fingers again a brand new sink to a driver,
which retains all state on the heap (in reality, in an area). On the way in which out,
emitters return their nested values as an alternative of recursing into them.

That additionally implies that Deser can not help codecs like protobuf that aren’t
self describing. They are in reality fairly deliberately neglected of the design
totally. Which is one strategy to say: if you wish to “repair” Serde, you might want to
make another compromises.

Most of the explanations for Deser’s concepts return to Sentry
Relay, which processes monumental quantities of
untrusted JSON. Over the years once I was at Sentry we bumped into the identical set of
issues many times, and plenty of of them usually are not actually bugs in Serde however
penalties of its design. Serde’s stability ensures imply that a variety of
them can’t be mounted with out breaking each format and each hand written
implementation. Most of those issues come from three selections:

1. One set of traits for all codecs. Serde serves each self describing
   codecs (JSON, YAML, TOML, …) and codecs the place the reader has to know the
   sort upfront (postcard, bincode, protobuf, …). That is extremely helpful,
   nevertheless it implies that some options solely work with some codecs, and you discover out
   at runtime. In case of Serde it additionally has some odd wrinkles the place a derived
   struct quietly accepts an array instead of an object in JSON for example.

2. A hard and fast information mannequin that loses data when buffering. Internally
   tagged enums, untagged enums and flatten have to buffer values earlier than
   they know what to do with them. The buffer can’t maintain every part the format
   knew, errors lose their location and extensions to the ecosystem depend on
   in-band signalling to precise issues similar to arbitrary precision numbers.

3. Recursion on the decision stack. Every degree of nesting makes use of stack area.
   Formats defend towards this with a recursion restrict, however the second you go
   by a code path that doesn’t have one (writing, dynamic values), deeply
   nested information can take down your course of. It additionally implies that a
   deserialization can’t be paused whilst you watch for extra enter.

Many of the corresponding Serde points have been open for years, and I wrote
about abusing Serde earlier than. People have tried
completely different angles on this over time. Some went minimal and dropped most
options to get quick compiles and no recursion. dtolnay’s personal
miniserde is the very best instance of that,
and deser’s trait design was initially modelled after it. Other current
makes an attempt went for runtime reflection, or for a brand new information mannequin with a concentrate on
binary codecs.

If you need to learn up on the entire collected challenges with Serde’s design,
I preserve a lengthy list here.

Dethroning Serde

First of all I don’t suppose it’s doubtless that one can substitute Serde. The orphan
rule
entrenches Serde extremely properly within the ecosystem. But some issues are inside
the attain of a crate writer’s management. In case of Deser it’s completeness.

Deser right now implements all essential self describing codecs from YAML, JSON,
TOML, CBOR, JSON5 and the likes, but additionally XML and plist to actually shut the hole.
XML specifically is one thing Serde has declined to help, and it reveals
(extra on that under). At the very least format help shouldn’t be the
cause to not use Deser.

The second drawback often is that truly fixing Serde’s points comes at a
important price in compile time and/or runtime efficiency. Deser isn’t any
completely different. While Deser’s compile instances are a bit higher than Serde’s, the
binary bloat is kind of a bit worse and the runtime efficiency is blended. It’s
roughly comparable for those who have a look at the numbers however relying on the format
construction you might be shedding considerably from among the tradeoffs.

That mentioned, it’s now in a state the place it’s not less than in precept a drop-in
substitute the place the tradeoffs would possibly work properly for customers.

Deser’s Design

Deser doesn’t attempt to be considerably completely different than Serde on the floor
degree. For most makes use of you derive Serialize and Deserialize after which begin
utilizing it together with your format implementing crate of selection. Most attributes are
very related, although they’re taking Rust expressions as an alternative of strings.

use deser::{Serialize, Deserialize};

#[derive(Debug, Serialize, Deserialize)]
#[deser(rename_all = "camelCase")]
pub struct Account {
    id: u64,
    account_holder: String,
    #[deser(default)]
    is_deactivated: bool,
}

let account: Account = deser_json::from_str(json)?;

The distinction within the design would change into extra obvious for those who implement a
serializer or deserializer your self. Instead of holiday makers that decision into every
different recursively, deserializing a sort creates a
sink which receives occasions which can be immediately emitted by the parser, and
serializing produces emitters that hand out values. Nested sinks and emitters
are handed again to a driver, which retains them on the heap. This design, which is
totally stolen from miniserde, offers some attention-grabbing penalties:

• No stack overflows. You can arbitrarily nest constructions with out points.
  For untrusted enter you set limits with a layer, and also you decide the quantity
  that you’re snug with, which is unbiased of your stack area.
• Suspendable. Because the state lives within the driver, a deserialization
  could be fed enter because it arrives. It’s additionally Send, so it might transfer between
  threads whilst you wait on IO which makes it a lot nicer to make use of with tokio.
  Formats like JSON, CBOR and MessagePack could be parsed as a stream for those who
  so want.
• An extensible information mannequin. The core information mannequin is small and product of atoms,
  maps and sequences. For all else, there are extension values (DateTime,
  Uuid, and many others.) that additionally all carry a fallback for codecs that don’t perceive
  them. Unlike Serde this implies it doesn’t depend on in-band signalling of
  objects with magic keys to smuggle values by.
• Lossless buffering. When a worth must be buffered (for example as a result of
  the tag of an internally tagged enum comes final), Deser information the occasions
  along with every part the format knew about them. Protocol particular
  extension sorts or error places all survive.
• Layers are a middleware system that sit between the format and your sorts
  and might observe issues like paths, implement security limits, rename keys or redact
  values with out having to the touch particular code paths.
• Native flattening that doesn’t buffer in any respect.

On prime of which can be a variety of issues that I simply needed to have:

• Allow enum tags to be of any sort, not simply strings
• Adapters that compose (as = Option>)
• Enabling validation as an adapter
• derive attributes which can be actual Rust expressions as an alternative of strings
• bytes as a core performance within the information mannequin
• duplicate keys rejected by default and errors that time on the drawback

Here is a small configuration sort that reveals a couple of of those collectively:

use deser::adapters::DisplayFromStr;
use deser::de::Recording;
use deser::{Deserialize, Serialize};
use deser_encoding::Hex;
use deser_validate::{Check, NonEmpty, Range};
use ipnet::IpNet;

#[derive(Debug, Serialize, Deserialize)]
pub struct Config {
    // not less than one 256-bit key, every written as hex
    #[deser(as = Check>)]
    secret_keys: Vec<[u8; 32]>,
    // `IpNet` is aware of nothing about deser, however has `FromStr` and `Display`
    #[deser(as = Option>)]
    allowed_networks: Option<Vec<IpNet>>,
    listeners: Vec<Listener>,
}

#[derive(Debug, Serialize, Deserialize)]
#[deser(tag = "type", rename_all = "snake_case")]
pub enum Listener {
    Unix { path: PathBuf },
    Tcp {
        host: IpAddr,
        #[deser(as = Check>)]
        port: u16,
    },
    // sorts this model doesn't know are saved and written again
    #[deser(other)]
    Other(#[deser(tag)] String, Recording),
}

Adapters are sorts, so Hex can go inside a Vec, and DisplayFromStr inside
a Vec inside an Option. Validators are adapters too, so
Check> decodes the keys after which checks that there’s at
least one. The catch-all variant retains the tag and a recording of every part
else in case somebody desires to course of it later.

Errors are one thing I care quite a bit about, so here’s what occurs when a worth
is improper:

secret_keys = ["9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"]
allowed_networks = ["10.0.0.0/8", "fd00::/8"]

[[listeners]]
sort = "unix"
path = "/run/app.sock"

[[listeners]]
host = "127.0.0.1"
port = 0
sort = "tcp"

[[listeners]]
sort = "quic"
host = "::1"
alpn = ["h3"]

let config: Config = deser_toml::Deserializer::from_str(enter)
    .deserialize_with(|driver| driver.push_layer(PathLayer::new()))?;

Note that right here the tag of the internally tagged enum comes final which implies that
the values should be buffered till the tag is thought. In Serde that is difficult
and we’d lose the placement if we used some tips so as to add it. With Deser
nevertheless, with the trail layer enabled Deser you the place within the construction the
drawback is:

Unexpected: invalid worth: should be between 1 and 65535 at line 10 column 8 (path: listeners[1].port)

Deser Meta Data

Deser actually desires to be extensible, and XML is a extra excessive instance of the
variations between Deser and Serde. Here is an Atom entry that mixes in Dublin
Core for the authors:

use chrono::{DateTime, Utc};
use deser::Deserialize;
use deser_value::Value;
use deser_xml::DeserializerConfig;

deser_xml::namespace!(
    atom = "http://www.w3.org/2005/Atom",
    dc = "http://purl.org/dc/elements/1.1/",
);

#[derive(Debug, Deserialize)]
struct Entry {
    #[deser(rename = atom!("title"))]
    title: String,
    #[deser(rename = dc!("creator"))]
    creators: Vec<String>,
    #[deser(rename = atom!("updated"))]
    up to date: DateTime<Utc>,
}

// entries we perceive, and every part else is saved as it's
#[derive(Debug, Deserialize)]
#[deser(untagged)]
enum Item {
    Entry(Entry),
    Other(Value),
}

let merchandise: Item = DeserializerConfig::new()
    .resolve_namespaces(true)
    .from_str(r#"
        
               xmlns:d="http://purl.org/dc/elements/1.1/">
          
          John
          2026-09-29T21:00:00Z
          Jane
        
    "#)?;

XML makes use of namespaces which implies that names must be matched by their namespace,
not by the prefix the doc occurs to make use of. Here the doc says d: and
the kind says dc!. atom!("title") is simply the string
{http://www.w3.org/2005/Atom}title, which works as a result of attributes are
expressions. The two creators are collected into one Vec though there’s
one other aspect between them, and the textual content of up to date goes straight right into a
chrono datetime. Because the enum is untagged, the entry must be buffered
earlier than a variant is picked, and deser’s buffer retains each creators. So the
result’s an Entry with John and Jane.

quick-xml, the most well-liked XML crate for Serde, drops the prefixes and ignores
namespaces totally, so a