{"slug": "5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse", "title": "5 gotchas I hit moving LLM logs from Postgres to ClickHouse", "summary": "The developer behind Spanlens, an open-source LLM observability platform, migrated the platform's `requests` table from Supabase Postgres to ClickHouse after recognizing that the insert-heavy, append-only workload would cause Postgres performance to degrade at scale. During the migration, the developer encountered five key pitfalls, including ClickHouse's requirement for space-separated timestamps instead of ISO 8601 format, and its silent conversion of numeric columns to JSON strings in `JSONEachRow` output. To prevent data loss during the transition, the developer implemented a fallback design that writes failed ClickHouse inserts to a Supabase `requests_fallback` table, with a cron job replaying the data back into ClickHouse every five minutes.", "body_md": "I am building Spanlens, an open-source LLM observability platform. Every call to OpenAI, Anthropic, or Gemini gets recorded with its model, latency, tokens, cost, and full request and response body. At low traffic on Supabase Postgres this was fine, but I could already see a few signs that this specific table would not stay fine for long.\n\n`requests`\n\ntable will dominate the DB at any meaningful scale. Every other table is bounded by org or project counts, but `requests`\n\ngrows with every API call.`created_at`\n\nare the dashboard's primary query pattern, and Postgres cannot compress the JSON body columns well, so these queries would get slower as the table grew.So I migrated to ClickHouse early, before it became a fire. This post is what I wish I had known before the migration, with 5 gotchas that bit me and the fallback design I built so I would not lose data while finding them.\n\nIf you want to see the full implementation in context, [Spanlens is open source on GitHub](https://github.com/spanlens/Spanlens) under MIT.\n\nHere is the shortlist I evaluated.\n\nThe decision driver was the workload shape. It is insert-heavy, append-only, and almost all reads are time-range scans with one or two equality filters like `organization_id`\n\nand `model`\n\n. ClickHouse is built for exactly this.\n\nI kept Postgres (Supabase) for everything relational with RLS, including orgs, projects, members, API keys, prompts, alerts, and billing. ClickHouse holds only one table called `requests`\n\nand it is the one that will grow.\n\n``` php\nWeb/Server  -> ClickHouse  (write path, fire-and-forget INSERT)\n            |  on failure\n            v\n            -> Supabase requests_fallback  (durable queue)\n            |  every 5 minutes (cron)\n            v\n            -> replay back into ClickHouse\n```\n\nThis is the outbox-ish pattern. I will come back to why.\n\nReads always go through a single helper that injects scope and retention.\n\n```\n// apps/server/src/lib/requests-query.ts\nexport async function requestsScope(\n  orgId: string,\n  opts: { ignoreRetention?: boolean } = {},\n) {\n  const plan = await getOrgPlan(orgId)            // 'free' | 'pro' | 'team'\n  const retentionDays = LOG_RETENTION_DAYS[plan]  // 14 | 90 | 365\n  const whereScope = opts.ignoreRetention\n    ? 'organization_id = {orgId:UUID}'\n    : 'organization_id = {orgId:UUID} ' +\n      'AND created_at >= now() - INTERVAL {retentionDays:UInt32} DAY'\n  return { whereScope, scopeParams: { orgId, retentionDays }, plan }\n}\n```\n\nDirect `getClickhouse().query()`\n\ncalls outside this helper are something I avoid. Multi-tenant data leaks are the worst kind of bug, and ClickHouse has no row-level security, so the discipline has to live in the query layer.\n\nJavaScript's `new Date().toISOString()`\n\nreturns `2026-05-16T11:49:23.749Z`\n\n. ClickHouse expects `2026-05-16 11:49:23.749`\n\nwith a space instead of T and no trailing Z. Insert with the JS default and you get this.\n\n```\nCode: 27. DB::Exception: Cannot parse input: expected \" \" but got \"T\"\n```\n\nI added a tiny helper and banned `.toISOString()`\n\nin any code path that writes to ClickHouse.\n\n```\nexport function toClickhouseTimestamp(d: Date): string {\n  return d.toISOString().replace('T', ' ').replace('Z', '');\n}\n```\n\nReading is the reverse. If you parse a DateTime64 back into a JS `Date`\n\n, you need to put T and Z back.\n\n``` js\nconst date = new Date(row.created_at.replace(' ', 'T') + 'Z');\n```\n\nThis one is sneaky because the bug is silent.\n\nClickHouse's `JSONEachRow`\n\nformat returns all numeric columns including `Decimal(18, 8)`\n\n, `UInt64`\n\n, and `Int32`\n\nas JSON strings, not numbers.\n\n```\n{ \"cost_usd\": \"0.00012345\", \"tokens\": \"421\" }\n```\n\nThen your innocent `r.cost_usd + 1`\n\ndoes string concatenation, so `\"0.00012345\" + 1 === \"0.000123451\"`\n\n. No error. Just wrong.\n\nThe fix is mechanical but you have to do it everywhere.\n\n``` js\nconst rows = (await ch.query(...)).map(r => ({\n  ...r,\n  cost_usd: Number(r.cost_usd ?? 0),\n  tokens:   Number(r.tokens ?? 0),\n}));\n```\n\nI now treat this as a strict boundary. The helper that wraps `ch.query()`\n\ndoes the coercion before anything else touches the rows.\n\nClickHouse does not have Postgres's `ILIKE`\n\n. If your previous code looked like `.ilike('model', '%gpt%')`\n\n, the direct rewrite is this.\n\n```\nWHERE positionCaseInsensitive(model, 'gpt') > 0\n```\n\n`NULLS LAST`\n\nordering is a similar story. It has to be explicit instead of implicit.\n\n```\nORDER BY cost_usd DESC NULLS LAST\n```\n\nBoth are easy fixes once you know about them, but they are easy to miss if you rewrite queries in a hurry because they are syntactic differences, not semantic ones. The previous query keeps \"working\" with no SQL error but silently changes behavior.\n\nClickHouse rejects unknown fields in `JSONEachRow`\n\ninserts by default.\n\n```\nCode: 117. Unknown field 'truncated'\n```\n\nIf you add a new column in your INSERT code, ship that code, and your production ClickHouse cluster has not run the migration yet, every insert from that pod fails until the migration lands. Streaming and non-streaming, all of it.\n\nI addressed this with two patterns.\n\nFirst, run the migration before deploying the code. `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`\n\nlands first, then the code that writes the column ships.\n\nSecond, I added a belt-and-suspenders setting.\n\n``` js\n// lib/clickhouse.ts\nconst ch = createClient({\n  url,\n  clickhouse_settings: { input_format_skip_unknown_fields: 1 },\n});\n```\n\nThis silently skips unknown columns instead of failing the insert. It rescues you from deployment-order mistakes, but it also hides typos. Pair it with a smoke-test that asserts new fields actually make it in.\n\nClickHouse has good uptime, but \"good\" isn't 100%. And LLM logs are write-heavy in a way that means every dropped insert is a dollar of cost data you will never get back.\n\nI added a Postgres-backed fallback queue.\n\n```\nexport async function logRequestAsync(data: RequestLogData) {\n  try {\n    await getClickhouse().insert({\n      table: 'requests',\n      format: 'JSONEachRow',\n      values: [data],\n    })\n  } catch (err) {\n    // ClickHouse rejecting or unreachable, queue to durable Postgres backup\n    const message = err instanceof Error ? err.message : String(err)\n    await supabaseAdmin.from('requests_fallback').insert({\n      payload: data,\n      organization_id: data.organization_id,\n      last_error: message.slice(0, 500),\n    })\n  }\n}\n```\n\nA cron job at `/cron/replay-fallback`\n\nruns every 5 minutes and drains the queue.\n\n```\nexport async function replayFallbackQueue() {\n  // 1. Drop poisoned rows first (7+ days old or 100+ retries) in one DELETE\n  //    so the limited batch budget goes to fresh entries.\n  const expiry = new Date(Date.now() - 7 * 86400_000).toISOString()\n  await supabaseAdmin.from('requests_fallback')\n    .delete()\n    .or(`created_at.lt.${expiry},retry_count.gte.100`)\n\n  // 2. Pull the next 50 in FIFO order so a long outage drains in arrival order.\n  const { data: rows } = await supabaseAdmin.from('requests_fallback')\n    .select('id, payload')\n    .order('created_at', { ascending: true })\n    .limit(50)\n\n  if (!rows?.length) return\n\n  // 3. One bulk INSERT for the whole batch instead of N round trips.\n  //    ClickHouse JSONEachRow accepts arrays trivially.\n  try {\n    await getClickhouse().insert({\n      table: 'requests',\n      format: 'JSONEachRow',\n      values: rows.map(r => r.payload),\n    })\n    // 4. Success ??delete the entire batch in one query.\n    await supabaseAdmin.from('requests_fallback')\n      .delete()\n      .in('id', rows.map(r => r.id))\n  } catch {\n    // Leave them in the queue. Next cron run picks them up and the\n    // expiry step above eventually drops them if they stay stuck.\n  }\n}\n```\n\nTwo design choices worth calling out.\n\nI use Postgres for the queue instead of Redis. I already had Postgres for transactional state. Adding Redis just for a recovery queue would be a separate failure domain. Postgres going down is so much worse than ClickHouse going down that pairing them is fine, because if Postgres is also down then the whole product is down and that is a more obvious incident than missing logs.\n\nI do not deduplicate. The `requests`\n\ntable has no unique constraint, so a race could insert a row twice. I accepted this trade-off because duplicate logs are a UI cosmetic problem, not a billing problem. If I ever sell on \"exactly-once logging\" I will redesign this.\n\nIt is still early days at Spanlens, so I do not have dramatic before-and-after benchmarks to share. What I can say qualitatively is this.\n\n`requests`\n\ntable dominating my Postgres backups or query times as Spanlens grows.The single biggest lesson is to invest in the safety net before you need it.\n\nA few things in hindsight.\n\nSet `input_format_skip_unknown_fields: 1`\n\nfrom day one. It is a small change that buys real resilience against deploy-order mistakes.\n\nAdd a synthetic write and read smoke test on every deploy that confirms a known row makes it through. This catches typos that the unknown-field setting now hides.\n\nMake helpers the API boundary loudly. Direct `ch.query()`\n\ncalls are the easiest way to introduce a multi-tenant leak. I plan to enforce this with CI lint rules.\n\nIf you are building anything write-heavy with time-range queries on top, whether that is observability, audit logs, event streams, or IoT telemetry, ClickHouse is worth considering early instead of waiting until Postgres becomes a fire. The footguns are real but they all show up early, and the runtime characteristics after that are very pleasant.\n\nSpanlens, the open-source LLM observability platform I built this for, is on GitHub at [github.com/spanlens/Spanlens](https://github.com/spanlens/Spanlens) under MIT. The migration helpers quoted here are all under `apps/server/src/lib/`\n\n. If you have done a similar migration or are staring at one, I would love to hear what gotchas hit you in the comments.", "url": "https://wpnews.pro/news/5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse", "canonical_source": "https://dev.to/spanlens/5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse-2458", "published_at": "2026-05-27 13:00:47+00:00", "updated_at": "2026-05-27 13:10:39.583446+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["Spanlens", "OpenAI", "Anthropic", "Gemini", "Supabase", "Postgres", "ClickHouse", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse", "markdown": "https://wpnews.pro/news/5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse.md", "text": "https://wpnews.pro/news/5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse.txt", "jsonld": "https://wpnews.pro/news/5-gotchas-i-hit-moving-llm-logs-from-postgres-to-clickhouse.jsonld"}}