{"slug": "an-outliner-sync-engine-that-is-right-for-the-use-case", "title": "An outliner sync engine that is right for the use case", "summary": "A developer building an offline-first outliner app has designed a sync engine using Durable Objects and SQLite, targeting a single-user workspace that can handle up to 1.125 million nodes over ten years of use. The architecture, inspired by celld, prioritizes structural integrity over text merging, allowing concurrent additions to coexist while field edits resolve by version agreement. The project emerged from the developer's daily use of outliners like Logseq and Roam Research, with the first day of testing revealing about 100 fixes and a peak of 450 nodes created in a busy workday.", "body_md": "I love outliner apps. Well, love might be a strong word, I don't know. There was always something special for me about having what I write belong to a graph instead of files. For the past few years, be it Logseq, Roam Research, or Obsidian with a pile of plugins, an outliner has been my go-to place for any writing that matters.\n\nIt has also been the side project I kept not starting. An outliner that feels right is a graph editor pretending to be a text file. I wanted to build an app which is offline-first, but where sync isn't an afterthought. Plus I really want to keep using things that right now are inseparable from my daily workflows, like automated note entries from emails, meetings, work tasks, etc.\n\nWith the recent jump in what coding agents can do, I finally tried. This is the first of what I expect to be several posts about the technical problems along the way. This one is about the sync engine, because that is the part I was most afraid of, and also the part where I think I found a sensible architecture.\n\n## Building from constraints\n\nThe app is single-player. One workspace belongs to one person, but they can use it across many devices. That means edits will very rarely happen at the same time.\n\nThere might still be various CLI, MCP, or API integrations pushing data to the workspace independently. I use those a lot. But they rarely edit an existing node I'm working on, mostly they add new material. So I'm fine with not implementing text merging within a single node for now. I need to retain the structure and keep both nodes when two clients append something at the same time. If they do edit the same field, they need to agree on which version wins. I'd still like to be able to retrieve the other version.\n\n### How much data are we talking about?\n\nBefore working on the sync engine, I built an offline version that uses SQLite and started using the editor with the goal of keeping it as my main destination for all notes during the day.\n\nThe first day was rough. There were probably around a hundred fixes I had to make in the background. I had to work late on my actual work tasks that day.\n\nAs soon as I had the basic outliner workflow working, I checked how many nodes I create during a busy workday. That came down to 450 nodes.\n\nA node is a single line of text in an outline, or it can be a long document. What I was usually missing in Logseq was the ability to toggle a mode for a node so I could work on it like it was a document. In my case that just means switching the default behavior of the Return key and adding some subtle indicators in the UI.\n\nA sidenote here, I think writing documents in outliners should be an edge case. When I write an article draft, I do it as an outline. Document mode is useful for things like a polished, formatted email that is ready to copy and paste, or a Slack update I'm finalizing. In the future, I think a good workflow would be to have the full source in an outline, then a predefined command which uses AI and my skill config to write the content as a document node. One that is clearly connected to the source, so I can see when they go out of sync.\n\nBased on those 450 nodes, a very rough estimate of ten years of usage during workdays would come to around a million nodes in a workspace. At 250 working days a year, it's 1.125 million. That's the size I want the backend to handle without issues. It also needs room for the operation history and attachments, which can grow much faster than the node count.\n\nAnd although I'm building for a single user, they might have several machines, with one falling months out of sync. I want that machine to catch up normally when it comes back online.\n\n## The build\n\nRecently I'd been reading a lot about Durable Objects and [celld](https://celld.dev/), a self-hosted implementation of that model, so naturally that was top of mind as the solution. One account gets one Durable Object with its own SQLite database. Each account owns one personal workspace.\n\nCloudflare explicitly describes [documents and users as sensible units for separate objects](https://blog.cloudflare.com/sqlite-in-durable-objects/). This doesn't give every user separate hardware, and one busy account still has to fit within one object's throughput. It does mean that adding users doesn't require putting all their writes through my own shared sequencer. For a personal outliner, that is a useful place to draw the boundary.\n\n### Storing files\n\nOne limitation I ran into was size. A Durable Object has a [2 MB limit on a string, BLOB, or table row, and a 10 GB SQLite database limit](https://developers.cloudflare.com/durable-objects/platform/limits/).\n\nI want users to be able to store files as nodes. They might be large files, and I don't want the database's row limit to decide what fits. If I can store a file on my machine, I'd like to be able to store it in the outliner as well. Plus, I want sync to be cheap to maintain.\n\nSo I decided to store file nodes as a hash and metadata pointing to a blob uploaded to Cloudflare's R2 object storage. Apart from getting the file out of the database row, it's much, much cheaper per gigabyte stored. Paid storage beyond the included allowances costs [$0.20 per GB-month in a Durable Object](https://developers.cloudflare.com/durable-objects/platform/pricing/) and [$0.015 in R2 Standard](https://developers.cloudflare.com/r2/pricing/), about thirteen times less. That's storage only, requests and compute have their own costs.\n\nThe Durable Object keeps the file registry, including references and transfer progress. R2 keeps the bytes. I'll still need an account quota policy, but attachments won't be competing with the edit log for that 10 GB database.\n\nI also needed somewhere for authentication data and the account directory, so D1 was a natural choice to avoid complicating the infrastructure.\n\n``` php\nflowchart TD\n API[\"API
Local JSON-RPC\"] --> Agent[\"Local client
Headless daemon\"]\n MCP[\"MCP\"] --> Agent\n CLI[\"CLI\"] --> Agent\n Desktop[\"Desktop app\"] --> DesktopClient[\"Local client
Desktop process\"]\n Web[\"Web app\"] --> WebClient[\"Browser client\"]\n\n Agent --- AgentDB[(\"SQLite
Agent replica\")]\n DesktopClient --- DesktopDB[(\"SQLite
Desktop replica\")]\n WebClient --- WebDB[(\"IndexedDB
Browser replica\")]\n\n Agent --->|\"Sync\"| Worker[\"Cloudflare Worker
Authenticate + route\"]\n DesktopClient --->|\"Sync\"| Worker\n WebClient --->|\"Sync\"| Worker\n Worker -->|\"Look up account\"| D1[(\"D1
Accounts + sessions\")]\n Worker -->|\"Accept edits\"| Account[\"Account Durable Object
One per workspace
SQLite log + file registry\"]\n Worker -->|\"Transfer file bytes\"| R2[(\"R2
Attachments + baselines\")]\n Account -.->|\"Track references\"| R2\n\n classDef entry fill:#f8fafc,stroke:#94a3b8,color:#334155\n classDef client fill:#eef2ff,stroke:#a5b4fc,color:#312e81\n classDef edge fill:#e0f2fe,stroke:#7dd3fc,color:#0c4a6e\n classDef state fill:#fef3c7,stroke:#f5cd69,color:#713f12\n classDef storage fill:#dcfce7,stroke:#86d5a0,color:#14532d\n class API,MCP,CLI,Desktop,Web entry\n class Agent,DesktopClient,WebClient client\n class Worker edge\n class Account,D1 state\n class AgentDB,DesktopDB,WebDB,R2 storage\n```\n\nThe clients share the core document rules and sync code, but each keeps its own replica. The CLI and MCP use the headless daemon's local API, so they can work while the desktop app is closed. The browser uses IndexedDB for its local storage.\n\n### Auth\n\nI decided to host Better Auth instead of using Clerk, WorkOS, or similar external auth services because I really don't see much sense in using those for this app. Six months ago I would have picked Clerk or WorkOS just to save those precious hours, but for this build, the Better Auth integration turned out to be a few prompts of my time.\n\n## On the syncing logic itself\n\nI'd been reading about Linear's sync engine while working on this. Its [delta read-path write-up](https://linear.app/now/rebuilding-delta-sync-read-path) describes clients returning with the ID of the last change they applied and asking for the changes since. At Linear's scale, filtering those changes by permissions and subscriptions became a substantial problem of its own. My devices all need the same account's document. I could use the log-and-cursor idea without taking on that read-path problem.\n\nA quick note from after the build. I only discovered how close Logseq's newer DB sync architecture was after building mine. Its [Worker configuration](https://github.com/logseq/logseq/blob/3de7c751/deps/db-sync/worker/wrangler.toml) has SQLite Durable Objects, D1 for metadata, and R2 for assets. Its [protocol](https://github.com/logseq/logseq/blob/3de7c751/docs/agent-guide/db-sync/protocol.md) routes sync through a Durable Object per graph.\n\nHonestly, this was a silly omission, as I'd been checking how legacy Logseq implemented sync. This could have saved me some time, but I mainly started working on this to build my own outliner and maybe publish it at some point in the future. I'm happy I ended up with a similar approach, it's reassuring.\n\nThe sync design is different though. That version of Logseq's DB protocol rejects transaction batches based on a stale server position and has checksums and server-side block repair. I wanted to keep as many rules as possible in the client's shared core. It saves me from maintaining a server implementation of those rules too. It also leaves conflict handling and structural correctness on the client side.\n\n## How it works\n\nLet's say I add a line on my laptop. The editor applies it immediately and saves the operation in local SQLite. The server doesn't have to be reachable for any of this. The local operation log looks like this:\n\n```\nCREATE TABLE oplog (\n seq INTEGER PRIMARY KEY AUTOINCREMENT,\n grp TEXT NOT NULL,\n version TEXT NOT NULL,\n op TEXT NOT NULL\n) STRICT;\n```\n\n`seq` is the position in this local database. `grp` ties related operations together, `version` records their ordering information, and `op` contains the operation. That local sequence number isn't an identity shared between devices.\n\nThe editor groups related operations together. For typing, a group normally seals after a 900 ms pause. This gives undo a useful unit and avoids uploading each keystroke separately. Sealing finalizes the version stamps too, and only sealed groups go into the outgoing queue.\n\nEach writer has an identity and an increasing `origin_seq`. The group has its own ID, that origin sequence, and hashes linking it to the writer's previous group. The client saves the queued group, so restarting or retrying sends the same group again.\n\n### Sending it to the server\n\nOn connection, the client first exchanges state with the server. It sends its writer identity and the last position it integrated from the account's log. The server returns its current position and the baseline available for a new device to download. Then the client pushes its pending groups.\n\nThe Worker authenticates the request and forwards it to the account's Durable Object. The object checks whether it already accepted this group, whether the origin sequence is the next one for this writer, and whether the previous hash matches.\n\nLarge groups can arrive in chunks. Until the whole group arrives, it stays staged. Once it's complete, a SQLite transaction records it, assigns a `server_seq`, updates the writer's head, and records any attachment claims. The server replies after that transaction commits.\n\nThese are the relevant parts of the server tables. I've left out payload storage and some metadata:\n\n```\nCREATE TABLE groups (\n account_id TEXT NOT NULL,\n space_id TEXT NOT NULL,\n server_seq INTEGER NOT NULL,\n group_id TEXT NOT NULL,\n replica_id TEXT NOT NULL,\n origin_seq INTEGER NOT NULL,\n prev_head TEXT NOT NULL,\n head_hash TEXT NOT NULL,\n PRIMARY KEY (account_id, space_id, server_seq),\n UNIQUE (account_id, space_id, group_id),\n UNIQUE (account_id, space_id, replica_id, origin_seq)\n) STRICT;\n\nCREATE TABLE replicas (\n account_id TEXT NOT NULL,\n space_id TEXT NOT NULL,\n replica_id TEXT NOT NULL,\n head_origin_seq INTEGER NOT NULL DEFAULT 0,\n head_hash TEXT NOT NULL,\n integrated_seq INTEGER NOT NULL DEFAULT 0,\n PRIMARY KEY (account_id, space_id, replica_id)\n) STRICT;\n```\n\nThere are two sequences here because they answer different questions. `server_seq` orders the account's accepted groups and gives clients a place to resume reading. `origin_seq` lets the server check that a writer is extending its own history in order.\n\nIf the server commits the edit but the response gets lost, the laptop retries with the same group ID. The server finds the existing receipt and returns the sequence it already assigned. It doesn't append the edit again.\n\n### What the other devices see\n\nThe editing device gets that receipt and saves it locally. Other connected devices get a small WebSocket notification with the new log position. The editing device can get the notification too. There's no edit body in it, a device that's behind pulls the missing groups through the regular HTTP endpoint.\n\n```\nsequenceDiagram\n participant A as Editing device
Local SQLite\n participant S as Account object
Server log\n participant B as Other device
Local SQLite\n Note over A: Apply edit locally
Save and seal group\n A->>S: Push group
Writer sequence + hashes\n activate S\n Note over S: Validate writer
Commit group
and receipt\n S-->>A: Receipt
Accepted at sequence 42\n S-->>B: WebSocket notification
Log is now at 42\n deactivate S\n Note over B: Last integrated
through sequence 40\n B->>S: Pull after 40\n activate S\n S-->>B: Missing groups
through sequence 42\n deactivate S\n Note over B: Apply the groups
Save document
and cursor together\n B->>S: Report progress
Integrated through 42\n```\n\nIf a notification gets lost, the edit is still in the log. The client will find it when it reconnects or polls. I don't need a separate history for each WebSocket connection.\n\nSomething I had to keep separate was the upload receipt and the download cursor. Say the laptop has read through 40. Another device's edit gets 41, then the laptop uploads its own edit and gets 42. It still needs to pull after 40. Using the receipt to jump its cursor to 42 would skip the other device's edit.\n\nSo the client tracks integrated progress separately from upload receipts. It also caches the highest sequence assigned to its own work, in the same transaction as the receipts, to avoid calculating it from history on every state read.\n\n### Applying incoming edits\n\nIncoming groups wait in a local inbox until all their chunks have arrived. The client then integrates a contiguous run of complete groups, saving the document changes and its progress through the log in one transaction. If the app crashes, the cursor can't get ahead of what's actually saved.\n\nFor ordinary fields like a node's text, the version comes from the group's hybrid logical clock, the writer's ID, and the operation's index within the group. The clock combines physical time with a logical counter. Each client derives the same version for the same operation, so competing writes resolve the same way even if packets arrive in a different order.\n\nMoves and parent-child relationships have their own rules in the shared client core. Two devices adding siblings should keep both additions, and moving a node shouldn't duplicate it or lose an unrelated edit. The server doesn't interpret those operations, it orders the groups that contain them.\n\nFile nodes take the same path for their metadata. A node can arrive with its name, size, and hash before the actual upload finishes. The bytes go separately through the Worker to R2, using multipart uploads. The Durable Object tracks completed parts so a transfer can retry, and tracks references so cleanup can check whether a file is still needed.\n\n## Coming back online\n\nIf a device has been offline without making edits, it reports its last integrated position and pulls the groups after it. Pulls are paginated, and each successful integration saves the new cursor, so it can work through a large backlog in batches.\n\nIf it made edits while offline, those groups are already in its local queue. It pushes them under its existing writer identity, then pulls the remote groups it hasn't seen. As in the example above, receiving a later sequence for its own upload doesn't let it skip the earlier remote edits. The client applies incoming groups alongside the local work using the same rules.\n\nFor a new device, the first device establishes a baseline archive in R2. The new one downloads it, loads its groups locally, then pulls changes after the baseline's sequence. The archive and its sequence have to describe the same point in history for that handoff to work.\n\nAn unrelated local database joining an existing account is a separate case. That needs an import or recovery decision. It can't be treated as an existing device just coming back online.\n\n## Summary\n\nBuilding it was fun. It was fun because my AI setup is solid enough to write good enough code from specs, and I could focus on system and product design. If I made any errors, [please let me know by email](mailto:chris@k22i.com). I'd be glad to learn.", "url": "https://wpnews.pro/news/an-outliner-sync-engine-that-is-right-for-the-use-case", "canonical_source": "https://k22i.com/journal/outliner-sync-engine/", "published_at": "2026-09-07 15:22:06+00:00", "updated_at": "2026-09-07 15:56:40.696517+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["Logseq", "Roam Research", "Obsidian", "Cloudflare", "celld"], "alternates": {"html": "https://wpnews.pro/news/an-outliner-sync-engine-that-is-right-for-the-use-case", "markdown": "https://wpnews.pro/news/an-outliner-sync-engine-that-is-right-for-the-use-case.md", "text": "https://wpnews.pro/news/an-outliner-sync-engine-that-is-right-for-the-use-case.txt", "jsonld": "https://wpnews.pro/news/an-outliner-sync-engine-that-is-right-for-the-use-case.jsonld"}}