Stalwart v1.0: The Fastest Mail Server on the Planet

Five years in the past, on 3 October 2021, the first commit landed in a small repository referred to as mail-parser. It was an e-mail parsing library written in Rust, and nothing extra. Over the next 5 years it grew into an entire mail and collaboration server with JMAP, IMAP, POP3, SMTP, CalDAV, CardDAV and WebDAV, a spam filter, clustering, and assist for half a dozen storage backends. Today, after 5 years of steady improvement, we’re proud to announce Stalwart v1.0, the primary steady launch.

Version 1.0 was by no means meant to be one other characteristic launch. It had two targets. The first was efficiency: making each a part of the server as quick as it may be, and proving it with numbers. The second was stability of the information itself: finalising the database schema in order that, any longer, upgrading Stalwart means putting in a brand new model and nothing extra, with out the disruptive migrations some earlier releases required. Along the best way a number of new options discovered their approach in too, together with full-text search inside PDF and workplace paperwork.

Let’s stroll by it.

Most of the work on this launch occurred underneath a profiler. For months we measured the place Stalwart spends its CPU cycles, its reminiscence and its database spherical journeys, rewrote no matter stood out, and measured once more. The launch incorporates near 200 commits touching greater than 1,300 information, and the overwhelming majority of them exist to make one thing quicker or lighter.

A mail server spends a shocking share of its time in search of particular bytes: the top of a header line, a MIME boundary, an equals register a quoted-printable physique, the area that ends an IMAP tag. Done one byte at a time, that’s plenty of loop iterations for a ten MB message. Two methods let a CPU do that work many bytes at a time:

• SIMD (Single Instruction, Multiple Data) makes use of the vast vector registers of recent processors (SSE2, SSSE3 and AVX2 on x86-64, NEON on ARM) to match or remodel 16 or 32 bytes with a single instruction.
• SWAR (SIMD Within A Register) applies the identical thought with abnormal 64-bit integers: a number of fastidiously chosen arithmetic and bitwise operations take a look at all eight bytes of a phrase directly. It wants no particular directions, so it really works on each CPU.

Stalwart now makes use of each, in all places it issues. At the underside of the stack sit two libraries that attain 1.0 along with the server:

• mail-parser 1.0, the very library the place Stalwart started, has been rewritten from scratch. SIMD kernels find MIME boundaries and header fields, SWAR handles header names, the parsed message lives in a handful of flat vectors, and our bodies are solely decoded when somebody really reads them. The CPU-specific kernel is picked at runtime, and each one is fuzzed in opposition to a plain scalar reference. Against different parsers it’s 3 instances quicker than the favored Rust mailparse crate, 9 instances quicker than Dovecot’s personal parser, and 15 to 31 instances quicker than libEtPan!, VMime and GMime on a full parse.
• encodify is a brand new encoding library with SIMD kernels for Base64, Base32 and quoted-printable, the encodings that carry almost each attachment and non-ASCII header in e-mail. Decoding a MIME-wrapped Base64 attachment is 22 instances quicker than with the usual Rust base64 crate on an AMD EPYC server, and quoted-printable decoding is 9 instances quicker than with the quoted_printable crate. It now handles each Base64, quoted-printable and IMAP UTF-7 conversion within the server.

Above them, the protocol layer received the identical remedy. The IMAP command parser is now table-driven and scans entire runs of bytes as an alternative of stepping a state machine per character. The SMTP command parser, the JSON parser behind JMAP, the spam filter tokenizers, the HTML-to-text converter, the variable-length integer codec and the log serializers all use SWAR scanning. Even the expression language, which directors use for routing and filtering choices, now compiles to a compact bytecode with specialised directions for frequent comparisons and evaluates them in a recycled reminiscence enviornment.

Fast parsing is wasted if each request then waits on the database, so we spent simply as lengthy on how Stalwart talks to its storage. Identifiers are actually assigned inside the identical write transaction that makes use of them, eradicating entire spherical journeys from each message supply. Read-modify-write updates, resembling including a key phrase or appending to a search index, run as merge operations contained in the backend. Stored objects are compressed with zstd utilizing dictionaries educated particularly on e-mail, calendar and speak to knowledge.

Each backend was additionally tuned individually:

• RocksDB: each sort of knowledge now lives in a column household configured for the way it’s accessed, with bloom filters for level lookups, massive blocks for scans and compaction tuned for knowledge that churns. Hot counters, resembling queue quotas, are unfold throughout as much as 256 slots to keep away from write conflicts.
• PostgreSQL and MySQL: pre-generated and cached ready statements, chunked iteration with automated retry, and fewer spherical journeys per write.
• FoundationDB: reads {that a} transaction will want are prefetched in parallel home windows, and enormous values are assembled in chunks.
• SQLite: web page measurement, cache, reminiscence mapping and WAL settings tuned for a mail workload.

On prime of the storage layer sits a redesigned cache subsystem. Each account’s message index is held in reminiscence in a compact packed type, and IMAP sorting, threading, flag fetches, searches on message properties and resynchronisation are actually answered from that cache with few or no database reads. Very massive accounts get their very own cache tier with a reminiscence finances, and cache snapshots may be swapped out to native disk, Redis or the blob retailer as an alternative of being rebuilt.

The built-in full-text search engine was rewritten from the bottom up. The earlier design saved one bitmap per phrase and had no notion of phrase positions. The new engine shops postings in blocks of 65,536 paperwork and retains phrase positions for each message. That makes actual phrase searches attainable, and lets a search inside one mailbox learn solely the blocks that mailbox touches. It indexes phrase stems and prefixes, compresses its knowledge with zstd, and indexes new mail by a background queue with concurrency management, retries and rebuilds. The PostgreSQL and MySQL search backends had been reworked as effectively. On a mailbox of 250,000 messages, a full-text search of each message physique now returns in underneath 4 milliseconds.

Optimising code with out measuring it underneath lifelike load is guesswork, so we constructed a devoted benchmarking and stress-testing lab to go along with this launch. It runs on Microsoft Azure, throughout greater than 30 areas, with as much as 18 full take a look at labs operating in parallel and several other hundred vCPUs busy directly.

The take a look at catalogue covers 35 sorts of take a look at, unfold over greater than 550 particular person take a look at cells, they usually cowl:

• Every main protocol: IMAP, JMAP (together with push over EventSource and WebSocket), SMTP inbound, submission and relay, CalDAV, CardDAV and WebDAV.
• PostgreSQL, FoundationDB and RocksDB as knowledge shops, Redis because the in-memory retailer, S3 and the filesystem as blob shops, and a three-node cluster underneath combined load.
• Large datasets: 1.6 million messages and 750,000 calendar and speak to gadgets throughout 500 accounts, particular person mailboxes of 1, 2 and 5 million messages, 100,000 accounts on a single server, mail queues of 10 million messages, and DMARC and TLS report floods of 1,000,000 stories.
• Storms and abuse: 20,000 new connections per second, mass logins, 1000’s of IDLE and push subscribers per account, hostile and malformed messages, slow-loris purchasers and outsized literals.
• Six-hour soak checks that search for reminiscence development, cache drift and sluggish leaks, together with one which pushed over 10 million messages by the MTA.

Runs had been profiled all through. CPU time was sampled with perf and was flame graphs, off-CPU evaluation confirmed the place threads waited on locks and I/O, the heap was profiled with jemalloc and heaptrack, and a particular instrumented construct counted each storage spherical journey, per command, so we may see precisely which requests hit the database greater than they need to. Each repair was then measured in opposition to the earlier construct on matched machines earlier than it was accepted.

The first full spherical of this cycle turned up 64 issues, which had been fastened and retested. The second spherical re-ran all the things and stuck 33 extra. Some examples of what the lab caught:

• Reopening a mailbox with 5 million messages went from 61 ms to 0.21 ms.
• A JMAP question on an account with 5 million messages went from 3.3 seconds to 12 ms.
• Peak inbound mail throughput on RocksDB almost doubled, from 9,690 to 18,394 messages per second.
• A reminiscence problem the place the method held as much as 9.6 GB whereas the precise heap was underneath 0.4 GB was discovered and stuck. In six-hour soak checks reminiscence now stays flat.

This lab isn’t a one-off effort for the discharge. It is now a part of how Stalwart is examined internally: each future launch goes by the identical benchmark and stress-test rounds earlier than it ships, so efficiency good points keep gained.

Internal numbers solely let you know whether or not Stalwart received quicker than it was. We additionally needed to know the way it compares to what many of the international stage runs. So we put Stalwart v1.0 face to face with probably the most extensively deployed open-source mail stack: Postfix 3.11 for SMTP and Dovecot 2.4 for IMAP.

Both ran on an identical Azure servers (8 AMD EPYC Zen 5 vCPUs, 32 GiB of RAM, native NVMe storage), loaded with the identical 1,000 mailboxes plus three very massive ones of as much as 250,000 messages, and obtained precisely the identical work: receiving mail, delivering it and serving it to IMAP purchasers. Encryption, password hashing and full-text search had been configured the identical approach on each, and the 2 techniques had been measured alternately on the identical {hardware}.

Stalwart was quicker in 35 of the 38 headline comparisons.

Some of the outcomes:

• IMAP: Stalwart is quicker than Dovecot in 14 of 15 on a regular basis shopper operations, by 1.4 to 19 instances. A login with a full TLS handshake takes 5.1 ms as an alternative of 28 ms, fetching a 1 MB message is 6 instances quicker, threading the INBOX 5 instances quicker, and transferring or deleting 100 messages 18 to 19 instances quicker.
• SMTP: Stalwart accepts a message 3.6 to 12 instances quicker than Postfix, and sustains 5,445 incoming messages per second in opposition to Postfix’s 723.
• Delivery: a message is seen to an IMAP shopper 16 instances sooner, and a shopper ready in IDLE is notified in 3.3 ms as an alternative of half a second.
• Load: Stalwart serves 5.4 instances extra concurrent IMAP periods per second and relays 3.8 instances extra mail earlier than it begins to decelerate.

The hole grows with the scale of the mailbox. On a 250,000-message mailbox, a full-text search of each message physique takes 3.75 ms in Stalwart and 136 ms in Dovecot, and transferring or deleting 25,000 messages finishes in a few tenth of a second as an alternative of 24 to 34 seconds: greater than 200 instances quicker.

The three checks Dovecot gained are all the identical operation: marking each message in a mailbox as learn directly.

Only Postfix and Dovecot had been examined, however they aren’t simply any two merchandise. They are the engine underneath almost each fashionable self-hosted mail server: each open-source dockerized mail server with greater than 10,000 stars on GitHub, together with docker-mailserver and mailcow, runs Postfix and Dovecot underneath the hood, as does Mail-in-a-Box. Beating that mixture on nearly each benchmark means beating the stack behind many of the self-hosted mail on the web. We are actually comfy saying it plainly: Stalwart is the quickest mail server on the planet.

Don’t take our phrase for it. Every script, configuration, take a look at situation and uncooked result’s within the benchmarks repository, along with a strategy and a frank checklist of caveats, such because the variations in disk-sync behaviour between the 2 techniques. Anyone with an Azure account can provision the identical servers and run the entire suite. Benchmarks of Stalwart’s totally different storage backends and clustered setups are being ready for a similar repository.

Performance was the headline, however v1.0 additionally brings a set of latest options:

• Search inside paperwork: Stalwart now extracts and indexes the textual content of PDF, Microsoft Office (docx, xlsx, pptx), OpenDocument (odt, ods, odp), EPUB and RTF paperwork, each in e-mail attachments and in file storage. Searching for a phrase now finds the contract connected to final 12 months’s message, not simply the message that mentions it. The extractor is written from scratch in Rust, with no exterior doc libraries, and processes each doc underneath strict measurement and complexity limits, so a malicious or malformed file can’t exhaust the server.
• Full-text search of file storage, plus an index queue with per-partition standing, retries and rebuilds.
• Metadata: JMAP Object Metadata for e-mails, mailboxes, calendars, contacts, information and extra, and IMAP METADATA (RFC 5464).
• Post-quantum DKIM2: composite ML-DSA-44 and Ed25519 signatures for DKIM2.
• OAuth and OpenID Connect: OAuth Client ID Metadata Documents, per-client scopes for mail, contacts, calendars and information, an inventory of linked purposes that customers can revoke, and viewers matching, opaque token validation and unscoped token insurance policies for exterior id suppliers.
• Storage and operations: a built-in in-memory retailer for single-node deployments, extra authentication strategies for Azure Blob Storage, secondary LDAP servers for failover, a reminiscence finances for buffered knowledge, SSRF safety for each outbound HTTP request, and configurable shopper IP sources and trusted proxy hops.
• AI fashions and an AI spam classifier within the Enterprise edition, with assist for OpenAI-compatible, Anthropic, Gemini and Ollama fashions.

The full checklist is within the changelog.

Finalising the database schema has a worth that’s paid as soon as: Stalwart v1.0 makes use of a brand new database structure, and present knowledge needs to be migrated to it. A v1.0 server won’t open a retailer created by an earlier model.

We know that some previous upgrades had been bumpy, and we took that severely. This migration was designed and examined to be as clean as we may make it, and it’s the final of its form: the schema is now steady, and future releases will improve in place. There are two methods to get to v1.0.

For deployments operating Stalwart v0.16, stalwart-migrate is the simplest route. It requires v0.16.27, so earlier 0.16 releases needs to be up to date to it first. The instrument converts your settings to the v1.0 schema and copies all knowledge into a brand new, empty v1.0 retailer. It solely reads from the outdated retailer, so the unique deployment stays intact and may be restarted at any level earlier than the change.

It additionally migrates incrementally. The first copy runs whereas v0.16 retains serving customers, nevertheless lengthy it takes. Each later run copies solely what modified for the reason that earlier one, and the one downtime is the brief closing run, after v0.16 is stopped. Account and message identifiers, IMAP UIDs and alter historical past are preserved, so IMAP and JMAP purchasers stick with it with out downloading something once more. Users who register with OAuth might want to register once more, and the full-text index is rebuilt within the background after the change.

For anybody operating Stalwart v0.15 or earlier, or anybody who prefers to maneuver customers one after the other, the migration guide describes the second route. The migration proxy sits in entrance of each the outdated and the brand new server and sends every consumer to whichever one holds their account, and Vandelay copies accounts throughout one after the other, incrementally. Each account is switched by flipping one entry in a mapping desk, with no downtime for that consumer, and the outdated server retains a full copy till you resolve to retire it. The identical process additionally works for transferring from Dovecot and Postfix to Stalwart. The migration overview compares the 2 approaches.

The construction of your settings is essentially the identical. Around 100 keys had been renamed to comply with constant naming conventions, a number of objects (OIDC directories, AI fashions, Azure storage, forwarded headers) had been restructured, and stalwart-migrate converts all of it robotically. Two adjustments want consideration earlier than you improve:

• Clusters want Redis. A cluster coordinator now requires a Redis-based in-memory store (Redis, Redis Cluster or Redis Sentinel). Earlier variations may fall again to the information retailer for this, however that was sluggish underneath load and has been eliminated. Single-node deployments get a built-in native in-memory retailer as an alternative.
• OAuth now follows the open public shopper profile. Stalwart now totally enforces the OAuth Profile for Open Public Clients, the rising customary that lets any mail shopper register to any mail server with OAuth with out being registered prematurely. Stalwart’s personal authorization server helps it out of the field. Many exterior id suppliers don’t but assist all the things the profile requires, resembling Dynamic Client Registration, and a few purchasers don’t but request the required scopes.

If you utilize an exterior id supplier that isn’t conformant, Stalwart’s token checks may be relaxed by the compatibility settings on the OIDC listing, at your personal threat, since every one weakens a safety verify. These settings primarily maintain internet purchasers such because the WebUI working. What they can’t repair is a supplier that lacks the profile itself, and that’s going to matter quickly: fashionable e-mail purchasers, together with Apple Mail, are adopting the profile, and as soon as they do, their customers won’t be able to authenticate in opposition to a non-conformant supplier. If you depend upon an exterior id supplier, now is an effective time to ask them to assist it.

A associated change impacts the WebUI. Instead of a hard and fast shopper id, it now identifies itself to the id supplier with a Client ID Metadata Document, a URL that describes the shopper. Some id suppliers don’t assist this but. With these, directors can register a shopper for the WebUI with the supplier and set its id in oauthClientId, and the WebUI indicators in as earlier than. The particulars are in client id of the WebUI.

If you run into an issue or have a query throughout the migration, put up it on support.stalw.art. We can be answering questions there every day all through the migration interval.

Five years in the past this was one Rust library for parsing e-mail. Today it’s a full mail and collaboration server, and with v1.0 it’s quicker than anything we’ve got measured it in opposition to. None of it will have occurred with out the group: everybody who filed a problem, despatched a patch, answered a query on the discussion board, or trusted Stalwart with their mail. Thank you.

Download Stalwart v1.0 and tell us what you assume.