{"slug": "posthog-dropped-clickhouse-for-duckdb-a-data-engineer-s-take", "title": "PostHog Dropped ClickHouse for DuckDB: A Data Engineer's Take", "summary": "PostHog rebuilt its managed data warehouse in June 2026, moving each organization from a shared multi-tenant ClickHouse cluster onto its own isolated DuckDB instance, with data stored on S3 via DuckLake and a Postgres wire endpoint for standard tooling. The company kept ClickHouse for hot, low-latency product analytics and split the stack by workload, citing multi-hour modeling jobs that could not safely run on an inelastic shared cluster. The new warehouse is in beta behind a waitlist, and PostHog published no benchmarks, framing the change as a matter of isolation and blast radius rather than raw speed.", "body_md": "I once watched a single analyst's 14-way join freeze a shared warehouse for a whole company on a Monday morning. Dashboards timed out, the CFO's report never showed up, and 3 teams spent the afternoon blaming each other.\n\nThat's why I read PostHog's [June 29, 2026 engineering post](https://posthog.com/blog/why-we-rebuilt-our-data-warehouse) closely. They pulled their managed warehouse off a multi-tenant ClickHouse cluster and gave every organization its own DuckDB instance. The instances sleep when idle, the data sits on S3 through DuckLake, and a Postgres wire endpoint lets ordinary tools connect. The headline sounds like \"ClickHouse lost,\" and that's the wrong read. The post publishes zero benchmarks. PostHog's decision comes down to matching the engine to the shape of the workload, and that's the same reasoning a system design interview tests.\n\nThe warehouse used to run on the same multi-tenant ClickHouse cluster PostHog uses for product analytics. That cluster serves people clicking around in a browser and expecting results quickly. Warehouse modeling jobs look nothing like that. In the post's words, \"It's not atypical for a modeling query to kick off on a schedule and run for hours or even days.\"\n\nA job that runs for days has a poor chance on a cluster where nodes restart, both on purpose and on their own. Every DE who's had a 9-hour backfill die at hour 8 because of a node rotation knows exactly how that ends. The post says that an unbounded number of those queries hitting their \"very not elastic\" cluster would be \"basically untenable.\"\n\nThey also list ClickHouse complaints beyond the workload problem. ClickHouse has no cost-based query optimizer, and its S3, Delta Lake and Iceberg support matured slowly. Results weren't stable either: \"We've seen ClickHouse change query results between releases, or based on which settings were enabled.\" HogQL has poor support in tools like dbt, and letting customers connect directly would make the multi-tenancy problems worse.\n\nThe hot takes skip the fact that ClickHouse stays. \"This is how we use it for all of our analytics data and that isn't changing.\" The change covers the warehouse only. Hot, shared, low-latency event analytics keeps running on ClickHouse. Long-running, isolated modeling jobs move to DuckDB.\n\nThe new layout, per the [managed warehouse docs](https://posthog.com/docs/data-warehouse/managed-warehouse):\n\nIt's in beta, behind a waitlist, and the docs say the layout \"may change during beta.\" Keep that in mind before you redesign your company's stack around a blog post.\n\nPostHog split the stack by workload: a shared engine for hot queries and isolated engines for long jobs. Splitting by workload is the lesson. \"DuckDB beat ClickHouse\" is the part you can ignore.\n\nI've read a lot of vendor migration posts. Most of them open with a bar chart showing a 40x speedup on a query nobody runs. This one has no chart at all, and it doesn't publish throughput, data volumes, node counts, instance sizes, pricing, cold-start latency or the idle timeout. The only performance language is qualitative: queries come back \"on the order of minutes,\" and jobs run \"hours or even days.\"\n\nPlenty of people will find that frustrating. I find it honest. PostHog never says DuckDB is faster. They say a shared, inelastic cluster can't safely absorb unbounded multi-hour jobs. It's an argument about blast radius and isolation, and you can evaluate it without a single benchmark.\n\nThe target customer is small. The post says the original warehouse was built for \"a 20-person company where the CEO is still running their own SQL queries,\" and that its goal was to \"delay the moment a company needs to hire their first data engineer.\" Per-org DuckDB works great when each tenant's data and jobs fit on one machine. The post never says where that limit is. If a tenant outgrows a single node, single-tenant compute means you scale up. DuckDB's own large-scale proof ran TPC-H on a 1.5 TB RAM, 192-core box, and that tells you which direction this design scales.\n\nThe famous benchmark measures the wrong phase. DuckDB's ClickBench lead came from hot runs, and a sleep-and-wake architecture lives or dies on cold starts. That's my inference; neither source says so. Still, quoting ClickBench to justify a scale-to-zero design is like judging a commute by how fast the car goes once it's on the highway.\n\nScale-to-zero has a bill, and the bill moves to the first query. Sleeping instances cost nothing until somebody opens a dashboard Monday morning after a quiet weekend and waits for the wake-up. A nightly modeling job won't notice the delay. An exec clicking a chart will. Snowflake has had this model for years, and its [warehouse docs](https://docs.snowflake.com/user-guide/warehouses-overview) still bill \"with a 60-second minimum each time the warehouse starts.\" Even \"pay only when active\" has a floor. PostHog doesn't publish theirs.\n\nThe ClickHouse optimizer complaint has a shelf life. ClickHouse 25.9 added automatic global join reordering, and the post doesn't say which version PostHog was running. Read it as a 2026 engineering judgment about their own deployment, not a permanent verdict on the engine.\n\nThe software underneath is moving too. Duckgres, PostHog's open-source Postgres wire server for DuckDB, now says in its README that as of October 8, 2026, PostHog is no longer developing it. The management plane moved to something called hogtower, and the README says the wire protocol and DuckDB query serving are unaffected. I wouldn't assume the open-source repo is what runs in production today.\n\nI've been through enough architecture announcements to know the pattern: the diagram ships 6 months before the operational numbers do. What you can learn from right now is the reasoning.\n\nSwap the names out and this becomes a standard interview prompt: \"You run analytics for thousands of customers on a shared cluster. Some customers want to run heavy scheduled transformations. Design it.\"\n\nWeak candidates name a tool. \"I'd use Snowflake.\" \"I'd use a bigger ClickHouse cluster.\" I've sat on panels where that was the entire answer, and the candidate couldn't explain why when we pushed. We passed, and the tool had nothing to do with it: there was no reasoning to evaluate.\n\nHere's how I'd structure it.\n\nStart with the workload before you touch the architecture. How many tenants? What fraction sit idle most of the day? What's the query shape: sub-second dashboards, or jobs that run for 6 hours? Who gets hurt when one tenant's job eats the cluster? PostHog's whole decision falls out of these answers. Short, hot queries share well. Long, unbounded jobs don't.\n\nPick the tenancy model per component. AWS's SaaS Lens describes this vocabulary: silo (dedicated resources per tenant), pool (shared resources for economies of scale), and bridge (a mix, where \"noisy neighbor attributes might steer a microservice to a silo model\"). PostHog built a bridge: pooled ClickHouse for event analytics, a silo for warehouse compute. Saying \"silo for the long jobs, pool for the hot path\" in an interview sounds like you've actually been on call for this.\n\nDon't overclaim about shared engines. \"Shared clusters can't isolate tenants\" is too strong, and a good interviewer will catch it. ClickHouse has workload scheduling. Its [docs](https://clickhouse.com/docs/operations/workload-scheduling) describe IO caps per workload, and they describe weights as a \"best-effort guarantee from below (not a limit or quota from above),\" with limits enforced \"per host.\" CPU preemption is off by default. Merges and mutations aren't CPU-scheduled yet. So the defensible claim is that shared engines isolate partially and per host, and you should name exactly what they don't guarantee. A node restart killing a 2-day job is a failure-domain problem, and workload pools don't fix it.\n\nName the concurrency model; it often decides more than the tenancy model. DuckDB's [concurrency docs](https://duckdb.org/docs/current/connect/concurrency.html) say one process can read and write a native database file. Multi-process native writes go through Quack, which is \"in beta stage as of DuckDB v1.5.2,\" and the docs recommend DuckLake on a PostgreSQL catalog for concurrent read-writes. One writer per file maps neatly onto one instance per org. That's my inference about why the design fits; PostHog doesn't say it. If you propose per-tenant DuckDB in an interview and can't say how writes behave, expect a rough follow-up.\n\nPrice the cold start out loud. You don't need real numbers. Name the variables: idle hours, wake latency, how often queries arrive, and the per-start billing floor. Then say which tenants come out ahead (bursty, mostly idle) and which lose (always-on dashboards). Interviewers score this kind of reasoning much higher than a confident number you made up.\n\nDon't declare a winner without data; PostHog didn't, and you shouldn't either.\n\nPlenty of loops still include a Spark coding round no matter what the company runs in production (and for pyspark interview questions, try datadriven.io, which we built for exactly that round). Treat that as a separate skill. The design round is where you show you can reason about fit, and nobody memorizes their way through that.\n\nI've watched 3 waves of \"this tool changes everything.\" This move teaches the same thing all of them did: the concepts carry over and the tools rotate.\n\nLearn open table formats and catalogs. PostHog says data \"lives in S3 independent of whatever's querying it, so we're not locked into DuckDB forever.\" That portability comes from the format. DuckLake keeps metadata in a SQL database and data in Parquet. The DuckLake FAQ says the only real limit is the catalog database's performance, which tells you where the commit point and the bottleneck sit. Ask \"where does the commit happen?\" about any lakehouse you touch. An \"open\" format is only as open as its write path, too: DuckDB reads Iceberg metadata directly in read-only mode, and writes go through a REST catalog.\n\nIntegrity moves up a layer. DuckLake doesn't support constraints, keys or indexes. If your duplicate detection relied on a primary key, that job now belongs to your pipeline. Get used to writing those checks yourself.\n\nEmbedded engines are real infrastructure now. DuckDB showed up at 3.3% usage in Stack Overflow's 2025 survey, which is small, but per-tenant embedded engines are a pattern you'll keep running into. Learn how they spill: DuckDB handles larger-than-memory GROUP BY, JOIN, ORDER BY and window operators, but its docs warn it can still run out of memory when operators combine, and `list()` and `string_agg()` don't spill at all. \"It handles larger than memory\" holds per operator, so check fit query by query.\n\nGovernance is part of the evaluation. DuckLabs announced in August 2026 that it's joining AWS, with DuckDB, DuckLake and Quack staying MIT-licensed under the DuckDB Foundation. Probably fine. Still, \"who maintains this in 3 years\" belongs in every architecture decision, and almost nobody asks it in a design review.\n\nCost reasoning beats speed reasoning. S3 GETs run $0.0004 per 1,000. My rough estimate is that request charges barely register next to compute uptime, so the economics of scale-to-zero come down to how many hours your instances are awake. Engineer time costs more than any of it. A team that stops getting paged for a dead 2-day job has saved more than any benchmark would show.\n\nConcepts transfer across tools. Tenancy, isolation, failure domains, and cold-start economics will still matter when DuckDB is the old thing someone's migrating away from.\n\nPostHog made a reasonable call for a specific customer, the 20-person company with a CEO still writing SQL, and they were upfront about the scope. The real test comes when one of those tenants grows into a 500-person company with a 3 TB fact table and a modeling job that runs past the weekend. I'd like to read that post.\n\nIf you've run per-tenant compute in production, what was the cold-start number that finally showed up on your dashboard?", "url": "https://wpnews.pro/news/posthog-dropped-clickhouse-for-duckdb-a-data-engineer-s-take", "canonical_source": "https://dev.to/datadriven/posthog-dropped-clickhouse-for-duckdb-a-data-engineers-take-2g49", "published_at": "2026-10-20 10:07:09+00:00", "updated_at": "2026-10-20 10:18:53.601202+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "mlops"], "entities": ["PostHog", "ClickHouse", "DuckDB", "DuckLake", "S3", "HogQL", "dbt", "Postgres"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/posthog-dropped-clickhouse-for-duckdb-a-data-engineer-s-take", "markdown": "https://wpnews.pro/news/posthog-dropped-clickhouse-for-duckdb-a-data-engineer-s-take.md", "text": "https://wpnews.pro/news/posthog-dropped-clickhouse-for-duckdb-a-data-engineer-s-take.txt", "jsonld": "https://wpnews.pro/news/posthog-dropped-clickhouse-for-duckdb-a-data-engineer-s-take.jsonld"}}