{"slug": "your-coding-agent-keeps-a-diary", "title": "Your Coding Agent Keeps a Diary", "summary": "OpenCode, a terminal coding agent, stores session metadata in a local SQLite database, and a dlt pipeline moves that data into DuckDB for analytics. The pipeline, built by Alena Astrakhantseva and Alexey Grigorev in a DataTalksClub session, uses four resources for the session, message, part, and todo tables with write_disposition='replace'. A marimo notebook reads the DuckDB file read-only to display KPIs such as total cost, session count, and cache-read share, helping users understand their AI usage costs.", "body_md": "[Infrastructure](https://www.sfrt.io/tag/infrastructure/)\n\n# Your coding agent keeps a diary\n\nEvery opencode session leaves a trail in a local SQLite file. dlt turns it into analytics you actually read.\n\nTL/DR: OpenCode writes every session, message, and token count to a local SQLite file. A small dlt pipeline moves that into DuckDB, and a marimo notebook tells me what my AI habit actually costs.\n\nOn Monday, 2026-07-06, Alena Astrakhantseva and Alexey Grigorev did a [DataTalksClub](https://datatalks.club/?ref=sfrt.io) session on ingesting agent traces with dlt: pulling the structured logs an AI agent emits into a local DuckDB or cloud lakehouse so you can query them. They pulled from local Claude JSON files and a hosted traces API. My coding agent doesn't have a usage API, but it also keeps a diary on disk. Why not point dlt at that?\n\n## Where OpenCode hides its traces\n\nOpenCode is my terminal coding agent.\n\nIt stores metadata in a local SQLite database, on Windows at `%LOCALAPPDATA%/opencode/opencode.db`\n\n. Four tables carry the interesting stuff:\n\n`session`\n\n: one row per session: title, model, agent, tokens, cost, timestamps`message`\n\n: one row per message, with a`data`\n\nJSON blob`part`\n\n: one row per message part (tool calls, text, reasoning)`todo`\n\n: the agent's own todo items per session\n\nThat looks like a trace: Every prompt, every tool call, every token billed. It just sits there in a format nobody wants to query by hand 😅\n\n## The pipeline\n\nThe whole thing is a dlt [source](https://dlthub.com/docs/general-usage/source?ref=sfrt.io) with four resources, one per table, using a small factory `_t`\n\nbecause `sql_table`\n\nsources don't accept source-level defaults without a `@dlt.resource()`\n\n[wrapper](https://dlthub.com/docs/general-usage/resource?ref=sfrt.io) (which I don't really need for this small pipeline):\n\n``` python\n@dlt.source(name=\"opencode_logs\")\ndef opencode_logs_source(db_path: str = DB_PATH):\n    credentials = f\"sqlite:///{db_path}\"\n\n    def _t(name: str, primary_key):\n        return sql_table(\n            credentials=credentials,\n            table=name,\n            write_disposition=\"replace\",\n            primary_key=primary_key,\n        )\n\n    yield _t(\"session\", \"id\")\n    yield _t(\"message\", \"id\")\n    yield _t(\"part\", \"id\")\n    yield _t(\"todo\", (\"session_id\", \"position\"))\n```\n\nEach table becomes a dlt resource with `write_disposition=\"replace\"`\n\n. Every run is a full refresh: no incremental bookkeeping, no state to corrupt. For a local log I regenerate on demand, replace is the honest choice.\n\nThen point the pipeline at a DuckDB file and run:\n\n```\npipeline = dlt.pipeline(\n    pipeline_name=\"opencode_logs\",\n    destination=dlt.destinations.duckdb(DUCKDB_PATH),\n    dataset_name=\"logs\",\n)\n\nload_info = pipeline.run(opencode_logs_source())\n```\n\nWhy dlt instead of the DuckDB extension `sqlite`\n\n? Because dlt handles schema inference, type coercion, and the SQLite-to-DuckDB hop for free. I describe four tables, dlt deals with the plumbing. When opencode adds a column in the next release, the pipeline picks it up without me touching the code.\n\n## What the traces say\n\nWith the data in DuckDB, a [marimo](https://marimo.io/?ref=sfrt.io) notebook reads it directly. marimo is a reactive Python notebook: change a filter, every dependent cell recomputes. The connection is read-only, so the dashboard can never corrupt the load:\n\n```\ncon = duckdb.connect(db_path, read_only=True)\nraw_session = con.execute(\n    \"\"\"\n    SELECT id, title, agent, model,\n           time_created, cost,\n           tokens_input, tokens_output,\n           tokens_cache_read, tokens_cache_write\n    FROM logs.session\n    WHERE time_created IS NOT NULL\n    \"\"\"\n).df()\n```\n\nFrom there the KPIs write themselves: total cost, session count, input vs output tokens, and cache-read share. Cached tokens are far cheaper than fresh input, so the higher that share, the less each session costs me:\n\n```\ncache_pct = (\n    100.0 * total_cache_r / (total_input + total_cache_r)\n    if (total_input + total_cache_r) > 0\n    else 0.0\n)\n```\n\nThe charts cover daily cost, a stacked daily token mix (input, cache read, cache write, output), sessions-and-tokens on a dual axis, top models by cost, and the top 15 sessions by cost. That last one is the guilty-pleasure table: which single conversation burned the most money? And was it worth it?\n\n## Why bother tracing my own agent\n\nTwo reasons. The obvious one is cost: an AI coding agent bills per token, and without a dashboard I have no idea whether last week cost five dollars or fifty. The second is behavioural. The `part`\n\nand `todo`\n\ntables record how the agent actually worked: which tools it reached for, how it broke tasks down, where it looped: you cannot improve what you cannot see.\n\nThe difference to Alena's and Alexey's session on Monday is scale: They built for a hosted, multi-user traces API, I built for one developer (me) and one SQLite file. The dlt pipeline barely changes between the two. Swap the source, keep the resources, pick a destination. That's the point of dlt 😜\n\nThe full dlt pipeline:\n\n```\n# → ad-hoc: load the local OpenCode SQLite log into DuckDB for analysis\n\"\"\"\ndlt pipeline: OpenCode logs (SQLite at $LOCALAPPDATA/opencode.db) → DuckDB\n\nLoads the four user-relevant tables from the OpenCode local metadata DB into a\nlocal DuckDB file for offline analysis (token usage, session history, todos).\n\nTables loaded:\n- session  — one row per session (title, model, agent, tokens, cost, time_created)\n- message  — one row per message (session_id, time_created, data JSON)\n- part     — one row per message part (message_id, session_id, data JSON)\n- todo     — one row per todo item (session_id, content, status, position)\n\nData flow:\n    $LOCALAPPDATA/opencode.db (SQLite) → dlt sql_database (sqlalchemy) → ./opencode_logs.duckdb\n\nLives in `dlt/local/` because the OpenCode SQLite DB is only on the developer's\nmachine, so this pipeline cannot run on dltHub Runtime. Run from any venv that\nhas `dlt[sql-database,duckdb]` installed (the dltHub venv at `dlt/dltHub/.venv`\nworks).\n\"\"\"\nimport os\n\nimport dlt\nfrom dlt.sources.sql_database import sql_table\n\n# ---------------------------------------------------------------------------\n# Config\n# ---------------------------------------------------------------------------\n# OpenCode stores its DB under %LOCALAPPDATA% on Windows.\n# Override via env var OPENCODE_DB if you sync between machines.\nDEFAULT_DB = os.path.join(\n    os.environ.get(\"LOCALAPPDATA\", os.path.expanduser(\"~\")),\n    \"opencode\",\n    \"opencode.db\",\n)\nDB_PATH = os.environ.get(\"OPENCODE_DB\", DEFAULT_DB)\nDUCKDB_PATH = os.environ.get(\n    \"OPENCODE_DUCKDB_PATH\",\n    os.path.join(os.path.dirname(os.path.abspath(__file__)), \"opencode_logs.duckdb\"),\n)\n\n# ---------------------------------------------------------------------------\n# dlt source\n# ---------------------------------------------------------------------------\n@dlt.source(name=\"opencode_logs\")\ndef opencode_logs_source(db_path: str = DB_PATH):\n    if not os.path.exists(db_path):\n        raise FileNotFoundError(\n            f\"opencode SQLite DB not found at {db_path}. \"\n            f\"Set OPENCODE_DB env var to override.\"\n        )\n\n    credentials = f\"sqlite:///{db_path}\"\n\n    # sql_table streams via SQLAlchemy fetchmany in chunks of `chunk_size` (50k\n    # by default), so memory stays flat regardless of source size.\n    def _t(name: str, primary_key):\n        return sql_table(\n            credentials=credentials,\n            table=name,\n            write_disposition=\"replace\",\n            primary_key=primary_key,\n        )\n\n    yield _t(\"session\", \"id\")\n    yield _t(\"message\", \"id\")\n    yield _t(\"part\", \"id\")\n    yield _t(\"todo\", (\"session_id\", \"position\"))\n\n# ---------------------------------------------------------------------------\n# Main\n# ---------------------------------------------------------------------------\ndef load_opencode_logs() -> None:\n    pipeline = dlt.pipeline(\n        pipeline_name=\"opencode_logs\",\n        destination=dlt.destinations.duckdb(DUCKDB_PATH),\n        dataset_name=\"logs\",\n    )\n    print(f\"Source: {DB_PATH}\", flush=True)\n    print(f\"Destination: {DUCKDB_PATH}\", flush=True)\n\n    load_info = pipeline.run(opencode_logs_source())\n    print(load_info)\n\n    with pipeline.sql_client() as client:\n        print(\"\\nRow counts:\", flush=True)\n        for t in (\"session\", \"message\", \"part\", \"todo\"):\n            try:\n                with client.execute_query(f'SELECT count(*) FROM \"{t}\"') as cur:\n                    n = cur.fetchone()[0]\n                    print(f\"  {t}: {n:,}\")\n            except Exception as e:\n                print(f\"  {t}: ERROR {e}\")\n    print(\"Done.\", flush=True)\n\nif __name__ == \"__main__\":\n    load_opencode_logs()\n```\n\nFor the marimo dashboard, ask you agent to build it for you 😎", "url": "https://wpnews.pro/news/your-coding-agent-keeps-a-diary", "canonical_source": "https://www.sfrt.io/your-coding-agent-keeps-a-diary/", "published_at": "2026-08-13 17:10:18+00:00", "updated_at": "2026-08-13 17:13:31.937718+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops"], "entities": ["OpenCode", "dlt", "DuckDB", "marimo", "Alena Astrakhantseva", "Alexey Grigorev", "DataTalksClub"], "alternates": {"html": "https://wpnews.pro/news/your-coding-agent-keeps-a-diary", "markdown": "https://wpnews.pro/news/your-coding-agent-keeps-a-diary.md", "text": "https://wpnews.pro/news/your-coding-agent-keeps-a-diary.txt", "jsonld": "https://wpnews.pro/news/your-coding-agent-keeps-a-diary.jsonld"}}