The Grammar of Data: From Definition to Execution

A second-part technical walkthrough demonstrates the "grammar of data" approach by building a data engineering digest that ingests RSS feeds, a live Bluesky firehose, raw PyPI downloads, and GitHub Archive events with dlt, then transforms them using xorq expressions across DataFusion, DuckDB, and Snowflake engines. The project registers reusable metric definitions such as star_velocity_30d and download_trend_90d as content-addressed, git-versioned catalog entries, with DataFusion as the default engine, and includes a small ML job. The example code is published in the de-ecosystem-digest GitHub repository.

In Part 1 https://www.ssp.sh/blog/grammar-data-engineering/ , we discovered the grammar for data: a way to define a data project with its complex requirements and how we define it declaratively as a grammar in one sentence with nouns sources , transformations verbs , templates, and modifiers, essentially being able to define it once and run it anywhere with different execution engines. This Part 2 will demonstrate how this looks in a data engineering digest project where we process data from RSS feeds, a live Bluesky firehose, and GitHub datasets, and find trends with an all-integrated horizontal data architecture running xorq based on the grammar described. We use dlt for ingestion outside the grammar , then Ibis, DataFusion/DuckDB/Snowflake for the engine, a cataloging feature to compress and discover metrics, and a small ML job. This article will guide you through that project and explain why xorq and the grammar of data are helpful to you. Note Want to jump right into the code: GitHub Repo Then follow along, the repository is at de-ecosystem-digest https://github.com/ssp-data/de-ecosystem-digest , the showcase we will go through as an example for the grammar of data below. As a reminder, the grammar dedicated to data consists of these parts and constructs a full sentence as our data project: In our data engineering digest example project https://github.com/ssp-data/de-ecosystem-digest we create a data engineering digest based on my DE RSS Feeds https://www.ssp.sh/brain/rss-feeds-for-data-engineering/ I collected over the years, Bluesky posts, raw PyPI downloads and GitHub Archive events as source data that we ingest with dlt https://github.com/dlt-hub/dlt . Here’s an overview of the project: Model once, represent everywhere - the transformation never changes, only the engine binding does | Read left to right: every part of speech maps to a xorq call We transform the data with xorq expressions mutate, filter, group by, aggregate, order by , use templates to bind the sentence to any repo dbt-core, polars, … and modifiers to bind the engine or a fitted ML model , and manifest it as a unique hash. Each named expression can be registered as a content-addressed, git-versioned catalog entry with the catalog https://docs.xorq.dev/api reference/cli/index.html catalog being the shelf of all of them such as star velocity 30d , download trend 90d , each reproducible on its own because xorq bundles the source read at build time. Then we run those reusable metric definitions on any engine with pre-existing pipelines to make it easier to run with make preview , which executes every named expression, while make catalog registers the curated ones as versioned entries. The default engine is DataFusion, but I added DuckDB and Snowflake, using xorq’s multi-compute engine capabilities . To illustrate the grammar and expression of the grammar in plain Python, here is how github.py https://github.com/ssp-data/de-ecosystem-digest/blob/main/src/de ecosystem/catalog/github.py could look, in six lines: | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | python def star velocity 30d con, repo : TEMPLATE: bind this sentence to any repo cutoff = datetime.now - timedelta days=30 t = con.table "raw github events" NOUN: a lazy pointer, no computation yet return t.filter VERB t.repo name == repo, t.type == "WatchEvent", t.created at cutoff, .mutate week=t.created at.truncate "W" VERB .group by "week" VERB .agg stars=t.id.count VERB .order by "week" VERB | Every one of these functions and the Makefile https://github.com/ssp-data/de-ecosystem-digest/blob/main/Makefile lets us run the grammar as steps of the grammatical grammar, building a sentence like this. I added this for illustration, but as an overview, if we map the commands to a xorq call , we can see the connection from the grammar of data to the xorq function: | make target | xorq / Python call | grammar part | |---|---|---| | make noun | con.table ... | noun source | | make verb | .filter/.mutate/.agg deferred expr | verbs transform | | make template | star velocity 30d con, repo | template bind by arg | | make modifier | settings.backend engine | modifier engine/fit | | make lineage | expr.op / ibis.to sql / expr.ls | lineage what xorq sees | | make manifest | xorq build expr.py -e star velocity | model once → expr.yaml | | make catalog | xorq catalog add … → ./catalog | versioned entry store | | make run-sentence | digest.main → .execute | execute the sentence | | make engines | settings.backend x + expr.execute | represent everywhere | note Ingestion and installation are excluded on purpose here To initialize, we also need make install to install dependencies and make ingest to load data with dlt locally. make run-sentence or make full-pipeline runs the full grammar of data. Additional commands preview catalog catalog-run summary digest ml test clean are added separately. If we run the demo project https://github.com/ssp-data/de-ecosystem-digest with the 90-day windows PyPI max provides this window without storing data ourselves , we get a couple of interesting insights that this demo project produces from digesting the full Data Engineering ecosystem . The digest ranks tools and terms of data engineering by their momentum with PyPI download growth and GitHub data, and enriches each tool with its Bluesky chatter on socials<sup 1 fn:1 </sup . This is how it looks with make digest : | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | package growth pct recent daily total downloads buzz xorq 130.9 968 125,104 10 ibis-framework 49.5 90,384 13,655,147 15 sqlglot 44.7 2,484,432 379,638,408 17 duckdb 42.5 1,712,837 263,396,944 3,200 dagster 39.7 283,339 43,951,132 275 polars 35.8 2,160,110 339,334,602 61 pydantic 30.0 35,567,354 5,688,585,163 90 sqlmesh 28.6 17,964 2,885,574 13 pyiceberg 24.4 1,300,103 211,933,200 640 prefect 19.7 441,434 73,236,617 38 dbt-core 9.9 3,527,875 609,587,211 369 apache-airflow 8.6 702,582 122,123,125 113 dlt -13.8 237,792 46,500,226 178 | Interesting to see that we get a rise of the dataframe & query engines : the fastest-growing DE packages over the last 90 days are all query/dataframe engines: ibis-framework +49%, sqlglot +45%, DuckDB +42%, Polars +36%. One caveat: I included xorq. It has the biggest growth, but it’s also the smallest package overall and still early, so the growth can have more spikes we went from ~400 to ~968 . And it’s worth mentioning that xorq, the tool we use for the grammar series, uses and is built on ibis , its great expression layer, as xorq builds on a rising dataframe for a composable, in-process engine. We also see that sqlmesh keeps climbing even after the Fivetran acquisition and dbt Labs joining Fivetran: sqlmesh grew +28.6% over 90 days, while dbt-core grew the slowest of the pack +9.9% . Not surprising, DuckDB wins social media attention. On Bluesky, its buzz score is 3200, 5× more than the next tool pyiceberg 640, dbt 369 . DuckDB is the tool that is both growing fast and the “loudest”. Number 1 by raw downloads is pydantic . By absolute volume, pydantic tops everything at 5.64B downloads ~9× dbt-core’s 610M : | 1 2 3 4 5 6 7 | ── Naive leaderboard: raw PyPI downloads all-time ── package downloads pydantic 5,639,574,035 dbt-core 603,522,561 sqlglot 376,710,633 polars 335,901,439 duckdb 261,307,529 | This is probably because Pydantic https://github.com/pydantic/pydantic is powering half of PyData while it doesn’t really have a lot of hype, but is a Data Engineering Toolkit https://www.ssp.sh/brain/data-engineering-toolkit/ for data validation and settings management using Python type annotations, used by any data engineer. It’s a part of the grammar that makes sure re-runs run deterministically. note What the blogs RSS say My RSS feeds were the noisiest signal: titles skew to whoever writes the most, e.g. Mr. Robin Moffatt rmoff’s random ramblings alone are ~690 of ~1,600 articles 😉. General sentiment clusters around the incumbents dbt, dagster, Spark, Snowflake , while the surging engines polars, SQLMesh are barely mentioned. Blog coverage lags the download signal, which is precisely why the digest triangulates four sources instead of trusting just one. stack.yaml and the Exchangeable Engine The key is really that model and execution are separated. Once the stack is defined in our demo project I used stack.yaml as a declarative config , we can just change the engine by editing a YAML file, and everything else stays the same: | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | engine: datafusion duckdb | datafusion | snowflake <- swap the engine here db path: de ecosystem.duckdb sources: nouns — dlt ingests these outside the grammar bluesky: max pages: 40 github: slice: "data/raw/gharchive/ .json.gz" momentum: the digest metric window days: 90 90 = laptop pulse; 365 = warehouse year-in-review tools: dbt-core, dagster, dlt, ibis-framework, xorq, apache-airflow, polars, duckdb, great-expectations, pyiceberg, prefect, mage-ai, sqlmesh, soda-core, pydantic, sqlglot | Imagine in your deploy scripts for dev you’d use DataFusion and on prod you’d just specify Snowflake as the variable. No implementation code is touched as in a typical imperative workflow. With make manifest we can compile the full data stack’s expressions into a deferred execution file builds/<hash /expr.yaml , which builds deterministically and is diffable. The demo shows that point well: if we make the above engine change to engine=duckdb from datafusion and rebuild, the semantic change shows up as a reviewable git diff . Let’s run manifest with engine: datafusion : | 1 2 3 4 | make manifest .... Written 'star velocity' to builds/10671a1c33cf builds/10671a1c33cf | Now changing engine: duckdb and re-running: | 1 2 3 4 | make manifest .... Written 'star velocity' to builds/f02f2c4dca81 builds/f02f2c4dca81 | The diff shows a couple of interesting bits, e.g. xorq changed the scale for timestamps for duckdb see both files datafusion https://github.com/ssp-data/de-ecosystem-digest/blob/dde2fbfc19948a3147178bd36781247a8d78e98a/builds/10671a1c33cf/expr.yaml L16 and DuckDB https://github.com/ssp-data/de-ecosystem-digest/blob/dde2fbfc19948a3147178bd36781247a8d78e98a/builds/f02f2c4dca81/expr.yaml L16 : | 1 2 | - scale: 9 datafusion → nanosecond timestamps + scale: ~ duckdb → microsecond timestamps | The profile.yaml shows the literal change we did: | 1 2 3 4 5 | - con name: xorq datafusion - config: ~ + con name: duckdb + database: ":memory:" + read only: false | This shows that swapping the engine by configuration has a real impact: DataFusion carries Timestamp scale=9 nanoseconds and DuckDB defaults to microseconds. The manifest captures that difference explicitly even before we run anything, reviewable instead of a silent runtime error through the built expression graphs https://docs.xorq.dev/getting started/your first expression.html before executing them with one expression, many engines . Apart from the grammar we look at, the project comes with catalog and ML functions to showcase the full capabilities of xorq and what you typically want to do in a data engineering project. The full data lineage can also be tracked and shown. The project has added a catalog to retrieve versioned entries of our metric and created artifacts. With make catalog , this project with xorq-catalog registers a curated set of expressions as versioned, content-addressed entries in a local, git-backed catalog at ./catalog . The catalog.yaml manifest is the shelf where entries are addressed by content hash, and aliases are the human-readable handles: | 1 2 3 4 5 6 7 8 9 10 11 12 13 | entries: content hashes a new hash = a new version - 5adcf6bccba9 dbt-star-velocity - a4f38c87079a dbt-download-trend - a636a49497fb dbt-momentum … aliases: - dbt-star-velocity - dbt-download-trend - dbt-momentum - duckdb-buzz - dbt-mentions - dbt-health - rising-tools | Each entry holds its own metadata sidecar with the expression + metadata + cached result, addressed by hash . Here is dbt-download-trend , the whole metric captured declaratively kind, output schema, the compiled SQL over the source table, and the cache key : | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | md5sum: 5a7d75db1e0cf46dc59a713a6ffa9573 backends: xorq datafusion expr metadata: kind: expr schema out: date: timestamp 9 downloads: int64 package: string cache keys: key: xorq cache-snapshot-4788a41e541c0526df207922813c0377 relative path: parquet sql queries: - - main - xorq datafusion - |- SELECT "t0"."date", "t0"."downloads", "t0"."package" FROM "raw pypi downloads" AS "t0" WHERE "t0"."package" = 'dbt-core' AND "t0"."category" = 'without mirrors' AND "t0"."date" DATE TRUNC 'DAY', '2026-05-14' ORDER BY "t0"."date" ASC | Because xorq bundles the source read at build time, an entry is self-contained: make catalog-run ALIAS=dbt-momentum re-executes it with no re-ingest . Edit an expression say days=30 → 90 and its content hash changes, so it registers as a new version while the old one stays retrievable. The project also ships a small ML task build feature matrix, split, fit a sklearn LogisticRegression wrapped in a xorq Pipeline , predict an adoption label to mimic prediction and training of a real-life project. It could be interesting to add more sophisticated logic once more data is downloaded, and potentially even more sources are added. With that example, we use the grammar of data with xorq to get essentially needed steps as part of the Data Engineering Lifecycle https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/ch02.html the data engineering lifecycle-id000095 . We can have full lineage, we get deterministic reruns, we get the metrics in the catalog. Plus, we get extensibility at each stage if we need it. E.g. extend the metrics from ‘catalog - Boring Semantic Layer https://github.com/boringdata/boring-semantic-layer ’, or use dlt for ingestion as I did in this project to load data incrementally into a staging area. For instance, make lineage shows everything xorq knows about a sentence before it touches a single row: its source nouns, output schema, bound engine, and the verbs compiled to SQL. This is how it looks: | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | LINEAGE — what xorq knows before any data is read: expr.op .find DatabaseTable — the source nouns this expression reads: 'raw github events' expr.schema — the output columns resolved at build time : ibis.Schema { week: timestamp; stars: int64 } expr.ls.backends — the engine s bound to it MODIFIER : 'Backend' ibis.to sql expr — the VERB chain, compiled to SQL: SELECT "t1"."week", COUNT "t1"."id" AS "stars" FROM SELECT DATE TRUNC 'WEEK', "created at" AS "week", "id" FROM "raw github events" WHERE "repo name" = 'dbt-labs/dbt-core' AND "type" = 'WatchEvent' GROUP BY "t1"."week" ORDER BY "t1"."week" | If you use the xorq Desktop app, this is integrated into a nice UI like this: The tool-adoption model reuses the same four parts of speech: fit ... attaches a modifier - the fitted model rides along as metadata xorq tracks a training hash without changing what the expression computes, which is precisely Part 1 https://www.ssp.sh/blog/grammar-data-engineering/ ’s definition of a modifier. predict ... is just a verb returning an Ibis table expression. And because predict is an expression, xorq build can manifest the inference pipeline too. Deploying a model collapses into the same write → manifest → execute cycle as deploying a metric. example Add Semantic Layer Here we use the inbuilt catalog and metrics are defined as expressions in Ibis. If you like, you could lift the catalog metrics into the Boring Semantic Layer for dimensions/measures. BSL is built by Hussain, the creator of xorq, and is tightly integrated. Check boring-semantic-layer https://github.com/boringdata/boring-semantic-layer if that is of interest, or check a recent article I wrote, Why Semantic Layers Matter https://motherduck.com/blog/semantic-layer-duckdb-tutorial/ , with a practical example. Part 1 https://www.ssp.sh/blog/grammar-data-engineering/ introduced the concept of grammar for data. With the data engineering digest project, we applied it to a demo project to capture the momentum of a tool in the data ecosystem. We mapped xorq’s features to the grammar to illustrate it better, and we’ve built a deterministic and versionable data stack with a single stack.yaml , providing the end-to-end data capabilities a data engineering project needs. Everything supports the case for a grammar for data. xorq gives us the bottom-up approach, working with our data, mapping all parts of the DE lifecycle, running it locally on multiple engines and discovering errors at pre-run time. Ultimately, shifting left https://www.rilldata.com/blog/what-shifting-left-means-and-why-it-matters-for-data-stacks once more. The grammar buys us two guarantees: answers that are faithful to the expression that produced them, and reproducible whenever we rerun it. But that doesn’t always mean correct. For example, the expression can still encode the wrong interpretation of the question. That third guarantee comes from reviewed definitions the catalog entries and semantic models we built above . You can learn more about it in a follow-up article Faithful, Reproducible, Wrong https://xorq.dev/blog/faithful-reproducible-wrong/ , where a checker plus a reviewed semantic model takes an agent from 4/100 to 100/100 correct answers on the same question. Another related question is: if the grammar defines and the manifest records, who verifies it? That’s what we look at next in this series. You can get a sneak peek with a checker inside an agent harness pi https://pi.dev/ , where every quantitative claim must be discharged by rerunning a content-addressed expression with its lineage intact. You can watch the loop in action in this recording https://asciinema.org/a/1263221 and reproduce it from the pi-xorq-verification-example https://github.com/xorq-labs/pi-xorq-verification-example repo. More on verification in Part 3. If you got interested and want to know more about how xorq works, check out the docs https://docs.xorq.dev/ , or the open source repo on GitHub https://github.com/xorq-labs/xorq . Also check out the upcoming xorq Desktop app https://xorq.dev/ join the waitlist , which targets data analysts from the top down. It’s a desktop app on macOS and a trusted harness. It has additional features that do a verification check and more. Additional project information for running the GitHub project, if you are interested in running it yourself and having a closer look. All four sources use idempotent upsert, so re-running only adds new rows / updates existing ones never duplicates : write disposition="merge" on primary key="id" article URL / post URI INSERT OR REPLACE on package, date, category INSERT OR IGNORE on event id To run a full year or more, just use Snowflake, for example by installing xorq snowflake and ingesting the data into Snowflake. Re-running is always safe idempotent merge , so just pull more and re-run. Each source has its own ceiling: | 1 2 3 4 5 6 7 8 9 10 11 | 1. More Bluesky history — edit src/de ecosystem/ingest/bluesky.py MAX PAGES = 200 40 → 200, pages much further back in time 2. More GitHub events — download extra hours into data/raw/gharchive/ gitignored for h in $ seq 0 23 ; do wget -nc -P data/raw/gharchive "https://data.gharchive.org/2026-08-05-$h.json.gz" done 3. Re-ingest safe, merges and re-run the whole sentence make ingest make run-sentence | The one limit is that the live pypistats.org API only serves ~180 days of downloads, so PyPI momentum caps at a 90-day-vs-prior-90-day window. A full-year digest needs a source that actually stores that history, and that is where Snowflake or BigQuery’s public bigquery-public-data.pypi.file downloads comes in, both holding years of daily download stats. Because the grammar separates what from where , you don’t rewrite the metric. You just point the noun at the warehouse and run the same sentence: | 1 2 3 4 | uv sync the snowflake driver ships as a base dependency put SNOWFLAKE ACCOUNT / USER / PASSWORD / ROLE / DATABASE / WAREHOUSE / SCHEMA in .env or export them in your shell make engines runs star velocity on DuckDB, DataFusion AND Snowflake | And with the same verb and only the engine binding changed: | 1 2 | con = settings.backend "snowflake" xo.snowflake.connect env momentum = download momentum con, "duckdb", window days=365 a full year | The catalog code itself has no engine awareness. settings.backend "duckdb" | "datafusion" | "snowflake" is the only thing that changes. Define once, represent it everywhere: your laptop for a 90-day pulse, a warehouse for the year-in-review. The one-time grants de digest must exist and DLT LOADER ROLE needs CREATE TABLE and CREATE STAGE , since the loader stages the tables before copying them in : | 1 2 3 4 5 6 7 8 9 10 11 12 13 | -- run once as ACCOUNTADMIN; the demo materialises the 4 raw tables into de digest.PUBLIC USE ROLE ACCOUNTADMIN; CREATE DATABASE IF NOT EXISTS de digest; -- PUBLIC schema is created automatically GRANT USAGE ON DATABASE de digest TO ROLE DLT LOADER ROLE; GRANT USAGE ON SCHEMA de digest.PUBLIC TO ROLE DLT LOADER ROLE; GRANT CREATE TABLE ON SCHEMA de digest.PUBLIC TO ROLE DLT LOADER ROLE; GRANT CREATE STAGE ON SCHEMA de digest.PUBLIC TO ROLE DLT LOADER ROLE; -- adbc/pandas bulk load GRANT USAGE, OPERATE ON WAREHOUSE COMPUTE WH TO ROLE DLT LOADER ROLE; GRANT ROLE DLT LOADER ROLE TO USER loader; -- if not already | And the tables it created in Snowflake if everything works correctly: Full article published at xorq.dev - written as part of my services Bluesky has a firehose and can simply be queried with DuckDB, e.g. see Querying Bluesky with DuckDB and SQL https://www.ssp.sh/brain/querying-bluesky-with-duckdb-and-sql/ ↩︎ fnref:1