I’ve had this project in my mind for more than a decade, but only in 2026 were the stars properly aligned for me to scratch that itch and build Melin: a replicated state machine capable of handling a million events per second with extremely low tail latency over LAN, strong durability guarantees, and automatic failover.
Most systems don’t need all of that. Some may tolerate occasional latency spikes, others loosely consistent ordering, or even optimistic durability. For all of these, a myriad of solutions already exist.
But in other cases, like in the HFT domain, compromising on latency, consistency or durability costs too much money. Companies will go to great lengths building specialized software to get a competitive advantage and keep it for themselves.
This project is an attempt to make such a system available to more, and for more use cases.
Melin’s use case
Technically you could build a whole website with just Melin behind an HTTP gateway, but that would make it the hammer in the "if the only tool you have is a hammer, you tend to see every problem as a nail" maxim.
Where Melin particularly shines is in highly concurrent state machines. Problems that are hard to shard and can be solved deterministically.
For example a matching engine can be sharded by instrument, but an exchange core with balance checks and risk checks cannot without consistency or latency tradeoffs, since every order may affect others. A game world can be sharded by zone, but any player can interact with any other, so every zone border becomes a synchronization hazard and anything global, like a shared economy or a leaderboard, still needs a single authoritative order. And even where sharding works, each shard needs its own replication, failover and journal, and recovering all of them to the same consistent point is a problem of its own.
The range of applications includes the financial domain, where most of the techniques I used are common, but also ads, logistics, games, science, and more. Its durability guarantees, recovery capabilities, and the byte-identical journals with a BLAKE3 hash chain make it particularly suitable for domains subject to regulatory audits.
On the other hand, Melin isn't a good fit for any service that involves digging into historical data, because the application state must fit in memory (twice actually, more on that later), and I/O is heavily discouraged for the sake of determinism and latency. It’s also designed to handle small payloads: requests and responses in the low hundreds of bytes.
Performance
Here’s how Melin performs:
Under load. About 1M events/s over four connections, each a closed loop keeping a fixed number of requests in flight.
| Transport |
In flight |
p50 |
p99 |
p99.9 |
p99.99 |
p99.999 |
| Kernel TCP |
25 |
90 µs |
182 µs |
248 µs |
302 µs |
357 µs |
| DPDK kernel bypass |
10 |
38 µs |
50 µs |
63 µs |
78 µs |
195 µs |
Single event. One client, one request in flight.
| Transport |
Rate |
p50 |
p99 |
p99.9 |
p99.99 |
| Kernel TCP |
22K/s |
44 µs |
67 µs |
108 µs |
177 µs |
| DPDK kernel bypass |
33K/s |
30 µs |
49 µs |
53 µs |
55 µs |
Every figure is a full round trip as the client sees it, including persistence and replication under the default ack policy, on four bare-metal servers (one client, one primary, two replicas). See Melin's readme for more details on the hardware and benchmarking conditions.
Building a durable, low-latency distributed system
Anyone who has read about low latency programming has probably heard about mechanical sympathy and the related best practices: avoid allocations, locks & yielding, pointer traversal, syscalls, encourage contiguous memory access and cache locality, atomics, amortization.
Another maybe less well-known piece but foundational to Melin is the LMAX architecture. It provides safe and efficient concurrency: multiple threads revolving through common ring buffers in an orderly fashion, with one task per thread and one thread per isolated core. This implies a relatively small context per thread that is likely to fit well in the CPU cache. Overall: high mechanical sympathy, few context switches. The caveat though is that you need a thread per task, and this is the reason why Melin in a production setup currently recommends 12 cores.
Fast PLP NVMe SSDs are now common at most providers and can perform writes with extremely low latency, typically in the microseconds range. While PLP isn't strictly necessary for Melin to enforce its durability guarantees it makes fdatasync much faster. Still, Melin can’t dictate when the firmware or the filesystem will interrupt writes for housekeeping reasons and these can cause spikes in the milliseconds range. But because it is built to operate in a cluster the natural option is to just wait for the fastest journal disk(s) before acknowledging a record as durable, hiding the impact of the slowest one(s). This is a case where, counterintuitively, a cluster can have a lower tail than a standalone node. This ack policy is configurable. By default, a record needs to be persisted on disk by one node and present on another in memory before being acknowledged as durable, enabling RPO=0 through any single-node failure, with exposure to a double failure limited to the time the second node takes to persist the record.
DPDK-capable NICs are also fairly common today and can even be found on servers to be rented by the hour, allowing for “cheap” experimentation. If you don’t know about DPDK it’s a way for a program to bypass the kernel and take direct control of the hardware interface. The result of this direct line of communication from Melin to the NIC was already shown in the benchmark table: the whole profile shifts down and the tail stays flat much further out than it does with kernel TCP. This is far from a free win though: not only does DPDK bring non-negligible operational complexity, a good chunk of Melin's code was written to deal with DPDK. Bypassing the kernel also means bringing its own TCP stack. Thankfully we have TCP stacks written in Rust! UDP is also an option, and should make Melin even faster, but it would mean tens of thousands more lines of code for a custom protocol.
Storage kernel bypass in the form of SPDK was also considered but quickly discarded. The first reason is that preventing the kernel from accessing the place where the critical data is stored sounded like an operational nightmare. The second reason is that a huge majority of Melin's latency is network-related, not storage-related. As tempting as it was for me to try to squeeze out even more microseconds of latency, the benefits/effort ratio is just too low.
Another thing that helps with reducing latency while also providing fast recovery is performing snapshots on a separate shadow thread. Regular snapshots avoid the need for a new node to retrieve the whole history before being live, but Melin can’t afford to pause the application in order to copy its state (the snapshot) to disk, because that would result in a very noticeable latency spike. To solve that, the shadow thread executes a copy of the application. This is the reason why the application state must fit twice in memory: because there’s a copy of the application running in parallel.
LLMs were essential in this project because of its scale, and their involvement is quite obvious in the code. There is a lot of data to keep in mind in such a project and it touches so many aspects of software engineering. I knew before I even started that this was impossible to do alone, since I had already tried a decade ago. I don’t want to dive too deep into the topic of LLMs, but I can't avoid mentioning these either. And while I miss the pride I had in writing my own code, there definitely are things that LLMs are orders of magnitude faster at than any human is. They, and all the other hardware and software advancements made in the past decade, made this project finally affordable for me. LLMs make many mistakes, humans do too, and code reviews, automated tests, and regular benchmarks are as much needed as they were before, but also cheaper in some ways.
The Rust ecosystem is also so rich nowadays that I was very happy to discover great tools such as Miri and TSan. The former found 3 cases of undefined behaviour (UB) that native tests couldn't catch. TSan found an fd use-after-close that could have silently written into a journal segment. There also is a Rust implementation of Raft that enabled me to bring replica automatic promotion to primary. Its architecture isn't a good fit for Melin's data plane, but it's a reasonable solution for this specific use case in the control plane. This way, if the primary fails, the replica with the longest journal can take over in a matter of seconds. This recovery time could be greatly reduced in the future by moving this control plane to Melin's low latency transports.
Why not Aeron Cluster Premium instead?
There are a few similar systems that have already been commercialized for years, and anyone considering using Melin should take a look into these. The most famous one is Aeron Cluster Premium from Adaptive.
Unfortunately its license and price prevent me from being able to build a fair Aeron vs Melin benchmark myself, and the only figures I could find aren't comparable for multiple reasons: different workload, different durability guarantees, and the EBS storage that isn't a good fit for Melin (at least not today).
But of course, performance isn’t everything and Aeron Cluster Premium and Melin have notable differences in other aspects:
- Adaptive has been in this space for years and their product is much more mature.
- Aeron relies on UDP whereas Melin is TCP-based, and their ack policies differ. There are pros & cons to both aspects.
- Aeron Cluster requires a JVM, Melin is nearly pure Rust (DPDK involves a bit of C).
- Aeron Cluster is free but the Premium features (like kernel bypass support) come with a 6-figure license with SLA & support. Melin is free to evaluate and to run outside production. Commercial terms and support will be worked out with its first users.
- Company location and the nature of the license have regulatory repercussions. Melin's license is a BSL, and each release turns Apache 2.0 within four years.
So while one may solve the same problems with Aeron or Melin there are various reasons why one may prefer the former or the latter, whether these are technical, financial or regulatory.
The path ahead
Working solo on this so far with zero users allowed me to spend as little time as possible on communication and synchronization. I could hop from one topic to another without disrupting anyone else. While I'm quite happy with Melin's performance and feature set today there is room for improvement in determinism hardening, the DPDK and kernel TCP paths still differ in various places, and I'd love to spend some time on reducing the unsafe surface or making Melin work better on fewer threads. But I reached a point where the API, protocol and journal format are stabilizing and further work will greatly benefit from actual use.
Which is a big motivation behind this post. If you have a use for Melin, let’s get in touch!
contact@melin-engine.com
https://github.com/melin-engine/melin
(And if you don't have a use for Melin but enjoyed reading this, you may want to remotely attend the P99 conference, it starts today!)
