An outliner sync engine that is right for the use case

A developer building an offline-first outliner app has designed a sync engine using Durable Objects and SQLite, targeting a single-user workspace that can handle up to 1.125 million nodes over ten years of use. The architecture, inspired by celld, prioritizes structural integrity over text merging, allowing concurrent additions to coexist while field edits resolve by version agreement. The project emerged from the developer's daily use of outliners like Logseq and Roam Research, with the first day of testing revealing about 100 fixes and a peak of 450 nodes created in a busy workday.

I love outliner apps. Well, love might be a strong word, I don't know. There was always something special for me about having what I write belong to a graph instead of files. For the past few years, be it Logseq, Roam Research, or Obsidian with a pile of plugins, an outliner has been my go-to place for any writing that matters. It has also been the side project I kept not starting. An outliner that feels right is a graph editor pretending to be a text file. I wanted to build an app which is offline-first, but where sync isn't an afterthought. Plus I really want to keep using things that right now are inseparable from my daily workflows, like automated note entries from emails, meetings, work tasks, etc. With the recent jump in what coding agents can do, I finally tried. This is the first of what I expect to be several posts about the technical problems along the way. This one is about the sync engine, because that is the part I was most afraid of, and also the part where I think I found a sensible architecture. Building from constraints The app is single-player. One workspace belongs to one person, but they can use it across many devices. That means edits will very rarely happen at the same time. There might still be various CLI, MCP, or API integrations pushing data to the workspace independently. I use those a lot. But they rarely edit an existing node I'm working on, mostly they add new material. So I'm fine with not implementing text merging within a single node for now. I need to retain the structure and keep both nodes when two clients append something at the same time. If they do edit the same field, they need to agree on which version wins. I'd still like to be able to retrieve the other version. How much data are we talking about? Before working on the sync engine, I built an offline version that uses SQLite and started using the editor with the goal of keeping it as my main destination for all notes during the day. The first day was rough. There were probably around a hundred fixes I had to make in the background. I had to work late on my actual work tasks that day. As soon as I had the basic outliner workflow working, I checked how many nodes I create during a busy workday. That came down to 450 nodes. A node is a single line of text in an outline, or it can be a long document. What I was usually missing in Logseq was the ability to toggle a mode for a node so I could work on it like it was a document. In my case that just means switching the default behavior of the Return key and adding some subtle indicators in the UI. A sidenote here, I think writing documents in outliners should be an edge case. When I write an article draft, I do it as an outline. Document mode is useful for things like a polished, formatted email that is ready to copy and paste, or a Slack update I'm finalizing. In the future, I think a good workflow would be to have the full source in an outline, then a predefined command which uses AI and my skill config to write the content as a document node. One that is clearly connected to the source, so I can see when they go out of sync. Based on those 450 nodes, a very rough estimate of ten years of usage during workdays would come to around a million nodes in a workspace. At 250 working days a year, it's 1.125 million. That's the size I want the backend to handle without issues. It also needs room for the operation history and attachments, which can grow much faster than the node count. And although I'm building for a single user, they might have several machines, with one falling months out of sync. I want that machine to catch up normally when it comes back online. The build Recently I'd been reading a lot about Durable Objects and celld https://celld.dev/ , a self-hosted implementation of that model, so naturally that was top of mind as the solution. One account gets one Durable Object with its own SQLite database. Each account owns one personal workspace. Cloudflare explicitly describes documents and users as sensible units for separate objects https://blog.cloudflare.com/sqlite-in-durable-objects/ . This doesn't give every user separate hardware, and one busy account still has to fit within one object's throughput. It does mean that adding users doesn't require putting all their writes through my own shared sequencer. For a personal outliner, that is a useful place to draw the boundary. Storing files One limitation I ran into was size. A Durable Object has a 2 MB limit on a string, BLOB, or table row, and a 10 GB SQLite database limit https://developers.cloudflare.com/durable-objects/platform/limits/ . I want users to be able to store files as nodes. They might be large files, and I don't want the database's row limit to decide what fits. If I can store a file on my machine, I'd like to be able to store it in the outliner as well. Plus, I want sync to be cheap to maintain. So I decided to store file nodes as a hash and metadata pointing to a blob uploaded to Cloudflare's R2 object storage. Apart from getting the file out of the database row, it's much, much cheaper per gigabyte stored. Paid storage beyond the included allowances costs $0.20 per GB-month in a Durable Object https://developers.cloudflare.com/durable-objects/platform/pricing/ and $0.015 in R2 Standard https://developers.cloudflare.com/r2/pricing/ , about thirteen times less. That's storage only, requests and compute have their own costs. The Durable Object keeps the file registry, including references and transfer progress. R2 keeps the bytes. I'll still need an account quota policy, but attachments won't be competing with the edit log for that 10 GB database. I also needed somewhere for authentication data and the account directory, so D1 was a natural choice to avoid complicating the infrastructure. php flowchart TD API "<b API</b <br/ Local JSON-RPC" -- Agent "<b Local client</b <br/ Headless daemon" MCP "<b MCP</b " -- Agent CLI "<b CLI</b " -- Agent Desktop "<b Desktop app</b " -- DesktopClient "<b Local client</b <br/ Desktop process" Web "<b Web app</b " -- WebClient "<b Browser client</b " Agent --- AgentDB "<b SQLite</b <br/ Agent replica" DesktopClient --- DesktopDB "<b SQLite</b <br/ Desktop replica" WebClient --- WebDB "<b IndexedDB</b <br/ Browser replica" Agent --- |"Sync"| Worker "<b Cloudflare Worker</b <br/ Authenticate + route" DesktopClient --- |"Sync"| Worker WebClient --- |"Sync"| Worker Worker -- |"Look up account"| D1 "<b D1</b <br/ Accounts + sessions" Worker -- |"Accept edits"| Account "<b Account Durable Object</b <br/ One per workspace<br/ SQLite log + file registry" Worker -- |"Transfer file bytes"| R2 "<b R2</b <br/ Attachments + baselines" Account -.- |"Track references"| R2 classDef entry fill: f8fafc,stroke: 94a3b8,color: 334155 classDef client fill: eef2ff,stroke: a5b4fc,color: 312e81 classDef edge fill: e0f2fe,stroke: 7dd3fc,color: 0c4a6e classDef state fill: fef3c7,stroke: f5cd69,color: 713f12 classDef storage fill: dcfce7,stroke: 86d5a0,color: 14532d class API,MCP,CLI,Desktop,Web entry class Agent,DesktopClient,WebClient client class Worker edge class Account,D1 state class AgentDB,DesktopDB,WebDB,R2 storage The clients share the core document rules and sync code, but each keeps its own replica. The CLI and MCP use the headless daemon's local API, so they can work while the desktop app is closed. The browser uses IndexedDB for its local storage. Auth I decided to host Better Auth instead of using Clerk, WorkOS, or similar external auth services because I really don't see much sense in using those for this app. Six months ago I would have picked Clerk or WorkOS just to save those precious hours, but for this build, the Better Auth integration turned out to be a few prompts of my time. On the syncing logic itself I'd been reading about Linear's sync engine while working on this. Its delta read-path write-up https://linear.app/now/rebuilding-delta-sync-read-path describes clients returning with the ID of the last change they applied and asking for the changes since. At Linear's scale, filtering those changes by permissions and subscriptions became a substantial problem of its own. My devices all need the same account's document. I could use the log-and-cursor idea without taking on that read-path problem. A quick note from after the build. I only discovered how close Logseq's newer DB sync architecture was after building mine. Its Worker configuration https://github.com/logseq/logseq/blob/3de7c751/deps/db-sync/worker/wrangler.toml has SQLite Durable Objects, D1 for metadata, and R2 for assets. Its protocol https://github.com/logseq/logseq/blob/3de7c751/docs/agent-guide/db-sync/protocol.md routes sync through a Durable Object per graph. Honestly, this was a silly omission, as I'd been checking how legacy Logseq implemented sync. This could have saved me some time, but I mainly started working on this to build my own outliner and maybe publish it at some point in the future. I'm happy I ended up with a similar approach, it's reassuring. The sync design is different though. That version of Logseq's DB protocol rejects transaction batches based on a stale server position and has checksums and server-side block repair. I wanted to keep as many rules as possible in the client's shared core. It saves me from maintaining a server implementation of those rules too. It also leaves conflict handling and structural correctness on the client side. How it works Let's say I add a line on my laptop. The editor applies it immediately and saves the operation in local SQLite. The server doesn't have to be reachable for any of this. The local operation log looks like this: CREATE TABLE oplog seq INTEGER PRIMARY KEY AUTOINCREMENT, grp TEXT NOT NULL, version TEXT NOT NULL, op TEXT NOT NULL STRICT; seq is the position in this local database. grp ties related operations together, version records their ordering information, and op contains the operation. That local sequence number isn't an identity shared between devices. The editor groups related operations together. For typing, a group normally seals after a 900 ms pause. This gives undo a useful unit and avoids uploading each keystroke separately. Sealing finalizes the version stamps too, and only sealed groups go into the outgoing queue. Each writer has an identity and an increasing origin seq . The group has its own ID, that origin sequence, and hashes linking it to the writer's previous group. The client saves the queued group, so restarting or retrying sends the same group again. Sending it to the server On connection, the client first exchanges state with the server. It sends its writer identity and the last position it integrated from the account's log. The server returns its current position and the baseline available for a new device to download. Then the client pushes its pending groups. The Worker authenticates the request and forwards it to the account's Durable Object. The object checks whether it already accepted this group, whether the origin sequence is the next one for this writer, and whether the previous hash matches. Large groups can arrive in chunks. Until the whole group arrives, it stays staged. Once it's complete, a SQLite transaction records it, assigns a server seq , updates the writer's head, and records any attachment claims. The server replies after that transaction commits. These are the relevant parts of the server tables. I've left out payload storage and some metadata: CREATE TABLE groups account id TEXT NOT NULL, space id TEXT NOT NULL, server seq INTEGER NOT NULL, group id TEXT NOT NULL, replica id TEXT NOT NULL, origin seq INTEGER NOT NULL, prev head TEXT NOT NULL, head hash TEXT NOT NULL, PRIMARY KEY account id, space id, server seq , UNIQUE account id, space id, group id , UNIQUE account id, space id, replica id, origin seq STRICT; CREATE TABLE replicas account id TEXT NOT NULL, space id TEXT NOT NULL, replica id TEXT NOT NULL, head origin seq INTEGER NOT NULL DEFAULT 0, head hash TEXT NOT NULL, integrated seq INTEGER NOT NULL DEFAULT 0, PRIMARY KEY account id, space id, replica id STRICT; There are two sequences here because they answer different questions. server seq orders the account's accepted groups and gives clients a place to resume reading. origin seq lets the server check that a writer is extending its own history in order. If the server commits the edit but the response gets lost, the laptop retries with the same group ID. The server finds the existing receipt and returns the sequence it already assigned. It doesn't append the edit again. What the other devices see The editing device gets that receipt and saves it locally. Other connected devices get a small WebSocket notification with the new log position. The editing device can get the notification too. There's no edit body in it, a device that's behind pulls the missing groups through the regular HTTP endpoint. sequenceDiagram participant A as Editing device<br/ Local SQLite participant S as Account object<br/ Server log participant B as Other device<br/ Local SQLite Note over A: Apply edit locally<br/ Save and seal group A- S: Push group<br/ Writer sequence + hashes activate S Note over S: Validate writer<br/ Commit group<br/ and receipt S-- A: Receipt<br/ Accepted at sequence 42 S-- B: WebSocket notification<br/ Log is now at 42 deactivate S Note over B: Last integrated<br/ through sequence 40 B- S: Pull after 40 activate S S-- B: Missing groups<br/ through sequence 42 deactivate S Note over B: Apply the groups<br/ Save document<br/ and cursor together B- S: Report progress<br/ Integrated through 42 If a notification gets lost, the edit is still in the log. The client will find it when it reconnects or polls. I don't need a separate history for each WebSocket connection. Something I had to keep separate was the upload receipt and the download cursor. Say the laptop has read through 40. Another device's edit gets 41, then the laptop uploads its own edit and gets 42. It still needs to pull after 40. Using the receipt to jump its cursor to 42 would skip the other device's edit. So the client tracks integrated progress separately from upload receipts. It also caches the highest sequence assigned to its own work, in the same transaction as the receipts, to avoid calculating it from history on every state read. Applying incoming edits Incoming groups wait in a local inbox until all their chunks have arrived. The client then integrates a contiguous run of complete groups, saving the document changes and its progress through the log in one transaction. If the app crashes, the cursor can't get ahead of what's actually saved. For ordinary fields like a node's text, the version comes from the group's hybrid logical clock, the writer's ID, and the operation's index within the group. The clock combines physical time with a logical counter. Each client derives the same version for the same operation, so competing writes resolve the same way even if packets arrive in a different order. Moves and parent-child relationships have their own rules in the shared client core. Two devices adding siblings should keep both additions, and moving a node shouldn't duplicate it or lose an unrelated edit. The server doesn't interpret those operations, it orders the groups that contain them. File nodes take the same path for their metadata. A node can arrive with its name, size, and hash before the actual upload finishes. The bytes go separately through the Worker to R2, using multipart uploads. The Durable Object tracks completed parts so a transfer can retry, and tracks references so cleanup can check whether a file is still needed. Coming back online If a device has been offline without making edits, it reports its last integrated position and pulls the groups after it. Pulls are paginated, and each successful integration saves the new cursor, so it can work through a large backlog in batches. If it made edits while offline, those groups are already in its local queue. It pushes them under its existing writer identity, then pulls the remote groups it hasn't seen. As in the example above, receiving a later sequence for its own upload doesn't let it skip the earlier remote edits. The client applies incoming groups alongside the local work using the same rules. For a new device, the first device establishes a baseline archive in R2. The new one downloads it, loads its groups locally, then pulls changes after the baseline's sequence. The archive and its sequence have to describe the same point in history for that handoff to work. An unrelated local database joining an existing account is a separate case. That needs an import or recovery decision. It can't be treated as an existing device just coming back online. Summary Building it was fun. It was fun because my AI setup is solid enough to write good enough code from specs, and I could focus on system and product design. If I made any errors, please let me know by email mailto:chris@k22i.com . I'd be glad to learn.