A Rust-native ZeroMQ-compatible messaging runtime, io_uring by default with optional tokio and smol backends
Monocoque is a ZeroMQ-compatible messaging library written in Rust. It implements ZMTP 3.1 from scratch over a small runtime facade: io_uring by default (via compio
), with optional tokio and smol backends for portability. Whichever you pick, it interoperates with any existing libzmq peer while staying entirely within Rust's memory model.
The name comes from Formula 1 engineering, where the monocoque chassis achieves structural strength through form rather than bolt-on reinforcement. Same idea here: performance through correct architecture, not unsafe shortcuts.
- All 11 ZeroMQ socket types: REQ, REP, DEALER, ROUTER, PUB, SUB, XPUB, XSUB, PUSH, PULL, PAIR
- PLAIN and CURVE (CurveZMQ/X25519) authentication, ZAP support
- TCP and IPC (Unix domain socket) transports
- Automatic reconnection with exponential backoff on all socket types
- ZMTP 3.1 heartbeating (PING/PONG) wired into all send/recv loops
- Socket monitoring via channel-based lifecycle events
- Explicit batching API for maximum throughput, plus
recv_batch()
to drain a burst of messages in one.await
- Allocation-free receive via
recv_into
/try_recv_into
: reuse one buffer across a hot recv loop instead of allocating aVec
per message - Vectored (
writev
) sends for large frames: the body skips the userspace copy - PUB fan-out coalesces queued broadcasts into one vectored write per subscriber
- PUSH/PULL worker pools via
PushFanOut
(round-robin ventilator) andPullFanIn
(fair-queued sink) - Zero-copy message passing via
Bytes
refcounting
Benchmarked against rust-zmq (FFI bindings to libzmq). Separate OS threads for sender and receiver, real loopback TCP, Intel Core i7-1355U (12 threads), Linux 6.17, release build. The three runtime backends run the identical suite, and the figures below were re-measured together on the same machine; the compio column reflects the compio 0.19 runtime (its throughput and latency stepped up noticeably over the earlier runtime). The rust-zmq column uses the same live-connection timer.
PUSH/PULL throughput with write coalescing (with_write_coalescing(true)
):
| Message size | compio | tokio | smol | rust-zmq |
|---|---|---|---|---|
| 64 B | 13.6 M msg/s | 17.1 M msg/s | ||
| 13.2 M msg/s | 4.58 M msg/s | |||
| 256 B | 8.2 M msg/s | 12.0 M msg/s | ||
| 8.5 M msg/s | 2.60 M msg/s | |||
| 1 KB | 3.5 M msg/s | 4.6 M msg/s | ||
| 3.3 M msg/s | 1.01 M msg/s | |||
| 4 KB | 1.19 M msg/s | 1.60 M msg/s | ||
| 1.10 M msg/s | 383 K msg/s | |||
| 16 KB | 370 K msg/s | 462 K msg/s | ||
| 331 K msg/s | 130 K msg/s |
All three backends beat libzmq once coalescing batches the writes: ~3.0x (compio), ~3.7x (tokio), ~2.9x (smol) at 64 B, and ~2-4x across the size range. On these single-flow loopback microbenchmarks the epoll backends (tokio, smol) are the faster: a one-connection ping-pong does not exercise io_uring's strengths (batched submission, registered buffers, many concurrent connections) and just pays its per-op submission overhead. compio (io_uring) is the default and is where the wins land for real network I/O and high connection counts. Measure on your own workload.
Default (eager) mode sends each message immediately, one syscall per send()
, and
is the mode for latency-sensitive work where you want each message on the wire now
rather than batched. On a bulk one-way firehose libzmq's internal batching leads
at small sizes; steady-state REQ/REP latency, though, is ~2.6-3.9x lower on every
monocoque backend (~9-14 µs vs libzmq's ~36 µs; compio is the lowest at ~9 µs).
Turn on coalescing for small-message throughput. For large frames eager mode
automatically uses a vectored write (writev
) so the body is never copied into
the send buffer; the threshold (vectored_write_threshold
, default 32 KB) is tunable per workload. IPC (Unix domain sockets) is ~3x faster than TCP loopback on every backend for same-host throughput.
PUB/SUB leads libzmq on both axes: single-subscriber fan-out runs ~3.0x (compio), ~3.5x (tokio), ~3.2x (smol) faster, and topic filtering at 10% match is a near tie. See docs/performance.md for the full breakdown including latency numbers, per-backend tables, the vectored-write crossover measurements, PUB/SUB pattern results, and tuning guidance.
[dependencies]
monocoque-rs = { version = "0.3", features = ["zmq"] }
compio = { version = "0.19", features = ["runtime", "macros"] }
js
use monocoque::zmq::{DealerSocket, RouterSocket};
// Connect a DEALER
let mut dealer = DealerSocket::connect("tcp://127.0.0.1:5555").await?;
dealer.send(vec![b"Hello".into()]).await?;
let reply = dealer.recv().await?;
// Bind a ROUTER
let mut router = RouterSocket::bind("tcp://127.0.0.1:5555").await?;
let msg = router.recv().await?; // msg[0] is the routing identity
js
// PUB/SUB
let mut publisher = PubSocket::bind("tcp://127.0.0.1:5556").await?;
publisher.send(vec![b"events".into(), b"payload".into()]).await?;
let mut subscriber = SubSocket::connect("tcp://127.0.0.1:5556").await?;
subscriber.subscribe(b"events").await?;
let msg = subscriber.recv().await?;
For high throughput, enable write coalescing or use the explicit batch API.
By default each send()
issues one kernel write per message. Write coalescing batches
those writes into a 64 KB buffer and flushes them in a single syscall, which is where
the large throughput gains in the table above come from. Because messages may sit in
userspace until flush()
is called, coalescing is opt-in: you decide exactly when the data goes out. See docs/performance.md for the full explanation and tuning guide.
// Write coalescing: opt-in, requires flush() after each burst (PUSH/PULL)
let mut push = PushSocket::connect_with_options(
"127.0.0.1:5555",
SocketOptions::default().with_write_coalescing(true),
).await?;
for msg in &batch {
push.send(vec![msg.clone()]).await?;
}
push.flush().await?; // flush bytes that did not fill the 64 KB threshold
// Explicit batch API: encode N messages then one write (DEALER/ROUTER)
for msg in &batch {
dealer.send_buffered(msg.clone())?;
}
dealer.flush().await?;
Monocoque runs on io_uring
through compio by default, but the socket stack is written against a small runtime facade, so it can drive the same code on tokio or smol instead. Pick one backend at compile time:
monocoque-rs = { version = "0.3", features = ["zmq"] }
monocoque-rs = { version = "0.3", default-features = false, features = ["runtime-tokio", "zmq"] }
monocoque-rs = { version = "0.3", default-features = false, features = ["runtime-smol", "zmq"] }
The three backends are mutually exclusive. The protocol layer, frame codec and
buffer model are identical across all of them: only the connect/spawn/timer
primitives differ. The tokio and smol backends follow compio's thread-per-core
model, so run tokio on a current-thread runtime inside a LocalSet
(smol uses a
single-threaded LocalExecutor
; the backend-agnostic LocalRuntime
below sets up the right one for you).
let rt = tokio::runtime::Builder::new_current_thread().enable_all().build()?;
let local = tokio::task::LocalSet::new();
local.block_on(&rt, async {
let mut push = PushSocket::connect("127.0.0.1:5555").await?;
push.send(vec![b"hello".into()]).await?;
Ok::<_, std::io::Error>(())
})?;
If you would rather not name a runtime in your own code, monocoque::rt::LocalRuntime
is a backend-agnostic entry point: it builds the right single-threaded runtime for whichever feature is enabled, so the same source runs on either.
let rt = monocoque::rt::LocalRuntime::new()?;
rt.block_on(async {
let mut push = PushSocket::connect("127.0.0.1:5555").await?;
push.send(vec![b"hello".into()]).await?;
Ok::<_, std::io::Error>(())
})?;
The runtime_backends
example is the same program run both ways:
cargo run --example runtime_backends --features zmq # compio
cargo run --example runtime_backends --no-default-features --features runtime-tokio,zmq # tokio
cargo run --example runtime_backends --no-default-features --features runtime-smol,zmq # smol
unsafe
is confined to a handful of small, well-contained spots, each behind a documented contract:
monocoque-core/src/io.rs
- the owned-buffer read helpers shared by every backend.
fill_read
owns the workspace's singleset_buf_init
call (declaring how many bytes a read initialized in a buffer's spare capacity), andtake_read_buffer
hands out read-sized slabs from a reusedBytesMut
. The socket read paths calltake_read_buffer
in documentedunsafe
blocks.monocoque-core/src/tcp.rs
(and a few socket-tuning call sites) - TCP socket tuning (nodelay, keepalive) through the raw socket handle.monocoque-zmtp/src/inproc_stream.rs
- the in-process stream adapter that fills an owned buffer.
Everything else is safe Rust.
Memory invariants:
- Buffers are never reused while referenced (tracked via
Bytes
refcounts) - A read slab is frozen to
Bytes
in a one-way transition; no mutation after freeze - The read slab is allocated lazily on the first read, so an idle socket holds none
- PUB fanout is refcount-based (
Bytes::clone()
), never copies payloads
cargo build --release --workspace
cargo test --workspace --features zmq
cargo bench --features zmq # runs the benchmark suite
cargo test --workspace --no-default-features --features runtime-tokio,zmq
cargo bench --no-default-features --features runtime-tokio,zmq
cargo test --workspace --no-default-features --features runtime-smol,zmq
cargo bench --no-default-features --features runtime-smol,zmq
Interop testing against libzmq: see docs/INTEROP_TESTING.md.
Core features are complete. Possible future work:
- io_uring fixed buffers (
IORING_OP_READ_FIXED
) - removes the last kernel-boundary copy per read; ~5-15% latency improvement at an already low baseline. (Largewritesalready use vectoredwritev
.) - Prefix trie for topic matching - the publisher-side prefilter and per-subscriber matching use a linear prefix scan, which is fast for the handful of distinct prefixes a PUB typically holds; a trie would only help when a single PUB accumulates 100+ distinctsubscription prefixes or deep hierarchies - Per-subscriber concurrent writes - PUB fan-out throughput now exceeds libzmq and is sharded across worker threads (each write has a fault-isolation timeout), but writes withina worker are sequential, so one slow subscriber can still delay the others on its worker
Long term: high-performance RPC, additional transports (QUIC, shared memory), custom protocol framework.
MIT - see LICENSE.
Built with: compio
(default backend), tokio
or smol
(optional backends), bytes
, flume
, smallvec