{"slug": "ghostlog-a-local-seance-for-dead-data-pipelines", "title": "Ghostlog: a local séance for dead data pipelines", "summary": "A developer built Ghostlog, an open-source Python tool that reconstructs the lineage of retired data pipelines from old cron logs, Airflow logs, or deprecated SQL files and generates a browser-viewable \"tombstone\" page describing what the job did. The tool uses sqlglot to compute table lineage and a locally run Gemma model via Ollama to narrate the job in first person, with a grounding check that rejects model output naming numbers or jobs not present in the extracted facts and falls back to a deterministic template. Ghostlog is MIT licensed, runs fully offline, and ships with 36 offline tests and CI on Python 3.10, 3.12, and 3.13.", "body_md": "*This is a submission for the [Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine](https://dev.to/challenges/hacktoberfest-week4-2026-10-26)*\n\nEvery data team has a graveyard. It holds the nightly job nobody remembers writing, the `.sql` file in `legacy/`, and the cron log that just stops one night in March. Someone always asks what that job did, and the person who knew left two reorgs ago.\n\n**Ghostlog** holds a séance for that job. You point it at the remains, either an old cron or Airflow log or a deprecated SQL file, and it:\n\n`raw.orders -> stg.orders -> mart.daily_revenue`);\nIt fits Día de los Muertos: the job gets remembered properly, and the next person who inherits the warehouse gets the context it took with it.\n\nSummoning a nightly ETL job that died on 2025-03-14 when an upstream column disappeared. This is real Gemma output (`gemma3:4b` on a CPU-only Linux machine, about 8 seconds):\n\nThe tombstone it wrote:\n\nA 60-second narrated walkthrough of a real run, summoning the job with Gemma and then opening its tombstone:\n\n**Live tombstone:** [nightmoose.github.io/ghostlog/sample/tombstone.html](https://nightmoose.github.io/ghostlog/sample/tombstone.html). All the samples are at [nightmoose.github.io/ghostlog](https://nightmoose.github.io/ghostlog/).\n\nThe raw transcripts and generated pages are committed in [`docs/sample/`](https://github.com/nightmoose/ghostlog/tree/main/docs/sample), including one run against a deprecated SQL file with a CTE and one run with `--offline`.\n\n**A local séance for dead data pipelines.**\n\nPoint Ghostlog at the remains of a retired job (an old cron or Airflow log, or a\ndeprecated `.sql` file) and it summons the job's ghost. Ghostlog reads the\nremains, rebuilds the job's table lineage with [sqlglot](https://github.com/tobymao/sqlglot)\nand asks [Gemma](https://ai.google.dev/gemma), running locally through\n[Ollama](https://ollama.com), to let the job speak for itself in the first person\nThen it carves an ofrenda-style tombstone page you can open in any browser.\n\nEverything runs on your own machine. Your logs and SQL stay on your laptop, and it keeps working with the Wi-Fi off.\n\nBuilt for the DEV Hacktoberfest Open-Source AI Challenge, Week 4: *Ghost in the Machine*.\n\n```\ngit clone https://github.com/nightmoose/ghostlog\ncd ghostlog\npython -m venv .venv && . .venv/bin/activate\npip install -e .\n# optional, for the ghost's own voice:\nollama pull gemma3:4b\n\nghostlog summon examples/legacy/nightly_orders_etl.log\n# then open\n```\n\n…\nMIT licensed. It has one runtime dependency (sqlglot, MIT), 36 offline tests, and CI on Python 3.10, 3.12, and 3.13.\n\n```\npip install -e .\nollama pull gemma3:4b\nghostlog summon examples/legacy/nightly_orders_etl.log\n```\n\nGhostlog is Python, split into four small modules.\n\n`logparse`` column \"cust_id\" does not exist` becomes `lineage`` INSERT`, `CREATE TABLE AS`, `MERGE`, or `UPDATE` to sqlglot and records a source-to-target edge for each table it reads. CTE names are skipped, so `WITH active AS (...)` doesn't show up as a phantom table. SQL that fails to parse is reported rather than crashing the séance. The tombstone shows the longest source-to-sink chain.`medium`` format` forces a reply with exactly three lines of testimony and three lines of epitaph.`tombstone`\n**What I learned from the model.** My first runs used `gemma3:1b`. It had plenty of atmosphere but a loose grip on facts: it once said the job ran \"for fourteen days each night.\" So I split the work. **Code computes the facts, and the model only supplies the voice.** On top of that, a grounding check rejects any reply where the ghost doesn't name its own job or says a number that isn't in the facts. When a reply fails the check, Ghostlog says why and falls back to a deterministic template. `gemma3:4b` passed the check on the sample log, so it became the default. The fallback also covers the case where Ollama isn't installed at all, so the séance always finishes.\n\n**Tests.** The suite runs fully offline. A fixture fails any test that tries a real network call, and the model is exercised through fakes for valid, malformed, ungrounded, and unreachable replies. CI also smoke-tests the CLI on a runner with no model installed.\n\nPipeline logs and SQL are some of the most sensitive text a company has. They hold table names, customer columns, and the shape of the business. With an open-weight model running locally, **nothing leaves the laptop**: no API key, no upload, no usage bill. It works on a plane with the Wi-Fi off.\n\nOpen weights also let me **swap models with one flag** (`--model gemma3:1b`, `--model gemma3:4b`, or anything else Ollama serves). That's how I found the right size for the job: I tried both locally and kept the one that stayed grounded. sqlglot, also open source, does the precise work of reading SQL across dialects, so the model never has to guess at lineage.\n\nAnd because the whole stack is open, a tool for remembering abandoned software can't itself be abandoned by a vendor. If Ghostlog dies one day, anyone can dig it up and run it again.\n\nThe first time the 4b model answered *\"My existence ceased on 2025-03-14 due to the loss of cust_id,\"* the job I'd made up felt like a real colleague. Every warehouse has a few jobs like that, and they deserve a marigold.\n\n`gemma3:4b`, with `gemma3:1b` compared) runs locally through Ollama and gives each dead job its voice, constrained by a JSON schema and a grounding check.", "url": "https://wpnews.pro/news/ghostlog-a-local-seance-for-dead-data-pipelines", "canonical_source": "https://dev.to/nightmoose/ghostlog-a-local-seance-for-dead-data-pipelines-4f8f", "published_at": "2026-10-31 17:18:05+00:00", "updated_at": "2026-10-31 17:22:12.719444+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Ghostlog", "Gemma", "Ollama", "sqlglot", "DEV Hacktoberfest", "nightmoose"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ghostlog-a-local-seance-for-dead-data-pipelines", "markdown": "https://wpnews.pro/news/ghostlog-a-local-seance-for-dead-data-pipelines.md", "text": "https://wpnews.pro/news/ghostlog-a-local-seance-for-dead-data-pipelines.txt", "jsonld": "https://wpnews.pro/news/ghostlog-a-local-seance-for-dead-data-pipelines.jsonld"}}