Twelve years of distributed SQLite – Philip O’Toole

In September 2014 I revealed a short post announcing rqlite. The thought was easy: take SQLite, put a Raft log in entrance of it, and replicate each write to a cluster in order that the lack of a node didn’t imply the lack of the info. The source went on GitHub that very same weekend. As far as I can inform, that was the primary open-source system to mix Raft and SQLite. Nobody else was doing it, and for a number of years no person else tried.

Today “distributed SQLite” is a class with a dozen tasks in it, just a few of them venture-funded, two of them acquired. Since the folks now writing concerning the class have a tendency to start out the story round 2021, I assumed I ought to write down the model that begins in 2014. I used to be there for all of it.

2014: rqlite

The want was concrete. A clustered system I worked on at jut.io wanted a small quantity of relational storage, and the query of which node ought to maintain it had no good reply. Replicating SQLite with a consensus protocol solved it. We went one other means at work, however I kept building rqlite as a result of the issue was basic and the answer was small. The first model used goraft, the identical library InfluxDB and etcd used on the time (I joined InfluxDB as a core developer across the identical time). In early 2016 I replaced it with Hashicorp’s Raft implementation, which rqlite nonetheless makes use of.

The design choice that mattered most was made then and by no means modified. rqlite is a CP system. Writes should undergo the chief, a majority should acknowledge, and a partitioned minority refuses writes. Everything else, the HTTP API, the read-consistency levels, the snapshotting, adopted from that.

Before 2014, and alongside it

Two issues deserve point out for honesty’s sake. Expensify had been replicating SQLite inside its personal utilities for years with a Paxos-style protocol; they launched it as Bedrock in 2016. And ActorDB, an Erlang system that gave every “actor” its personal SQLite database replicated by Raft, appeared around the same time as rqlite. Neither was broadly adopted outdoors its origin, however each present the thought was within the air. What rqlite did was make it a standalone, general-purpose, open-source factor anybody may run.

2017: dqlite

Canonical constructed dqlite to get LXD and later MicroK8s an embedded replicated retailer with out operating etcd. Mark Shuttleworth truly known as me as soon as, and we talked, hoping that rqlite could possibly be made to work, however it by no means fairly match. dqlite began in Go and was rewritten in C with its personal Raft library. It is the undertaking most just like rqlite in intent, and the primary distinction is of one in every of packaging: dqlite is a library you embed, rqlite is a server you run. Canonical’s want was a database inside their merchandise: mine was a database beside anybody’s product.

2021: Litestream, and a change of method

Ben Johnson (whom I labored with at InfluxDB) launched Litestream in early 2021. Litestream doesn’t use consensus in any respect. It watches the SQLite WAL and streams it to S3, providing you with catastrophe restoration for a single-node SQLite database at virtually no value. Ben’s argument was that the majority functions don’t want a cluster; they use one machine and wish a strategy to get better if it dies. He was proper about an incredible many functions, and the “SQLite is sufficient” motion of the following few years, with weblog posts and convention talks, owes loads to that argument.

Litestream and rqlite reply totally different questions. Litestream: what if the machine dies? rqlite: what if the machine dies and I can’t wait to revive? The first query is extra frequent, the second is the one folks have once they come to rqlite.

Fly.io employed Ben in 2022 and shipped LiteFS, which replicates SQLite at filesystem stage with leases from Consul. It was the primary try and make replicated SQLite a platform characteristic somewhat than a factor you deploy. And in a ultimate twist, Litestream inspired rqlite’s v8.0 method to dealing with the SQLite WAL.

2022: forks and extensions

Two tasks in 2022 took approaches that neither rqlite nor Litestream had. cr-sqlite added CRDTs to SQLite as an extension, in order that a number of writers may diverge and merge with out a chief. That is the AP nook of the design house, and it fits local-first functions the place rqlite’s CP ensures can be a hindrance.

libSQL forked SQLite itself. ChiselStrike, later Turso, needed to vary issues contained in the engine that the SQLite undertaking, which doesn’t settle for outdoors contributions, and wouldn’t change. On prime of the fork they constructed a hosted service with replicas on the edge. Turso raised enterprise cash on that, and in 2026 Supabase acquired them.

Around the identical time, SQLite Cloud appeared, providing hosted SQLite replicated with Raft. It is the undertaking closest to rqlite’s authentic design, wrapped in a cloud service.

2022 onward: the platforms

Cloudflare introduced D1 in 2022, SQLite as a serverless database inside their edge community, and later put SQLite storage inside Durable Objects. This is replicated SQLite as a managed product from an organization that doesn’t have to let you know the way it works. Turso’s service is identical form. The class had gone from “a factor you run” to “a factor you purchase.”

What the twelve years present

Three issues, I feel.

The core was proper. SQLite as a storage engine, with replication added on the surface, has been reinvented by at the very least eight groups with totally different wants and totally different protocols. Nobody has discovered a motive to cease doing it.

The design house has three corners, and so they appeared so as. Consensus first (rqlite, dqlite): one chief, robust consistency, a cluster. Streaming second (Litestream): one author, low cost restoration, no cluster. Merge third (cr-sqlite, and components of the local-first motion): many writers, eventual consistency. Each is right for a unique query, and a lot of the arguments are arguments between their customers about which query is being requested.

And the cash went to the platforms. rqlite, dqlite, and Litestream are open supply and free. The tasks that raised the capital or received acquired are those that turned replicated SQLite right into a service with a invoice. That is the standard destiny of utilities concepts, and I don’t thoughts it. But it’s price recording that the thought was open supply for eight years earlier than anybody offered it.

rqlite continues to be right here. It does the identical factor it did in 2014, deal higher, and it’s nonetheless the factor you attain for once you need a fault-tolerant relational database you possibly can run your self, perceive absolutely, and never pay anybody for. Twelve years in, I’m content material with that place within the story.