{"slug": "the-grammar-of-data-from-definition-to-execution", "title": "The Grammar of Data: From Definition to Execution", "summary": "A second-part technical walkthrough demonstrates the \"grammar of data\" approach by building a data engineering digest that ingests RSS feeds, a live Bluesky firehose, raw PyPI downloads, and GitHub Archive events with dlt, then transforms them using xorq expressions across DataFusion, DuckDB, and Snowflake engines. The project registers reusable metric definitions such as star_velocity_30d and download_trend_90d as content-addressed, git-versioned catalog entries, with DataFusion as the default engine, and includes a small ML job. The example code is published in the de-ecosystem-digest GitHub repository.", "body_md": "In [Part 1](https://www.ssp.sh/blog/grammar-data-engineering/), we discovered the grammar for data: a way to define a data project with its complex requirements and how we define it declaratively as a grammar in one sentence with nouns (sources), transformations (verbs), templates, and modifiers, essentially being able to define it once and run it anywhere with different execution engines.\n\nThis Part 2 will demonstrate how this looks in a data engineering digest project where we process data from RSS feeds, a live Bluesky firehose, and GitHub datasets, and find trends with an all-integrated horizontal data architecture running xorq based on the grammar described.\n\nWe use dlt for ingestion (outside the grammar), then Ibis, DataFusion/DuckDB/Snowflake for the engine, a cataloging feature to compress and discover metrics, and a small ML job. This article will guide you through that project and explain why xorq and the grammar of data are helpful to you.\n\n[!Note] Want to jump right into the code: GitHub Repo\n\nThen follow along, the repository is at [de-ecosystem-digest](https://github.com/ssp-data/de-ecosystem-digest), the showcase we will go through as an example for the grammar of data below.\n\nAs a reminder, the grammar dedicated to data consists of these parts and constructs a full sentence as our data project:\n\nIn our [data engineering digest example project](https://github.com/ssp-data/de-ecosystem-digest) we create a data engineering digest based on my [DE RSS Feeds](https://www.ssp.sh/brain/rss-feeds-for-data-engineering/) I collected over the years, Bluesky posts, raw PyPI downloads and GitHub Archive events as source data that we ingest with [dlt](https://github.com/dlt-hub/dlt). Here’s an overview of the project:\n\n*Model once, represent everywhere - the transformation never changes, only the engine binding does | Read left to right: every part of speech maps to a xorq call*\n\nWe transform the data with xorq expressions (mutate, filter, group by, aggregate, order by), use templates to bind the sentence to any repo (dbt-core, polars, …) and modifiers to bind the engine (or a fitted ML model), and manifest it as a unique hash. Each named expression can be registered as a **content-addressed, git-versioned catalog entry** with the [catalog](https://docs.xorq.dev/api_reference/cli/index.html#catalog) being the shelf of all of them (such as `star_velocity_30d`, `download_trend_90d`), each reproducible on its own because xorq bundles the source read at build time.\n\nThen we run those reusable metric definitions on any engine with pre-existing pipelines to make it easier to run with `make preview`, which executes every named expression, while `make catalog` registers the curated ones as versioned entries. The default engine is DataFusion, but I added DuckDB and Snowflake, using xorq’s **multi-compute engine capabilities**.\n\nTo illustrate the grammar and expression of the grammar in plain Python, here is how [github.py](https://github.com/ssp-data/de-ecosystem-digest/blob/main/src/de_ecosystem/catalog/github.py) could look, in six lines:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n```\n\n | \n\n``` python\ndef star_velocity_30d(con, repo):          # TEMPLATE: bind this sentence to any repo\n    cutoff = datetime.now() - timedelta(days=30)\n    t = con.table(\"raw_github_events\")      # NOUN: a lazy pointer, no computation yet\n    return (\n        t.filter([                          # VERB\n            t.repo_name == repo,\n            t.type == \"WatchEvent\",\n            t.created_at > cutoff,\n        ])\n        .mutate(week=t.created_at.truncate(\"W\"))   # VERB\n        .group_by(\"week\")                          # VERB\n        .agg(stars=t.id.count())                   # VERB\n        .order_by(\"week\")                          # VERB\n    )\n```\n\n | \n\nEvery one of these functions and the [Makefile](https://github.com/ssp-data/de-ecosystem-digest/blob/main/Makefile) lets us run the grammar as steps of the grammatical grammar, building a sentence like this. I added this for illustration, but as an overview, if we map the commands to **a xorq call**, we can see the connection from the grammar of data to the xorq function:\n\n| `make` target | xorq / Python call | grammar part | \n|---|---|---|\n| `make noun` | `con.table(...)` | noun (source) | \n| `make verb` | `.filter/.mutate/.agg` (deferred expr) | verbs (transform) | \n| `make template` | `star_velocity_30d(con, repo)` | template (bind by arg) | \n| `make modifier` | `settings.backend(engine)` | modifier (engine/fit) | \n| `make lineage` | `expr.op()` /`ibis.to_sql` /`expr.ls` | lineage (what xorq sees) | \n| `make manifest` | `xorq build expr.py -e star_velocity` | model once → expr.yaml | \n| `make catalog` | `xorq catalog add … → ./catalog` | versioned entry store | \n| `make run-sentence` | `digest.main()` →`.execute()` | execute the sentence | \n| `make engines` | `settings.backend(x)` +`expr.execute()` | represent everywhere | \n\n[!note] Ingestion and installation are excluded on purpose here\n\nTo initialize, we also need `make install` to install dependencies and `make ingest` to load data with dlt locally. `make run-sentence` or `make full-pipeline` runs the full grammar of data. Additional commands `preview` `catalog` `catalog-run` `summary` `digest` `ml` `test` `clean` are added separately.\n\nIf we run the [demo project](https://github.com/ssp-data/de-ecosystem-digest) with the 90-day windows (PyPI max provides this window without storing data ourselves), we get a couple of interesting insights that this demo project produces from **digesting the full Data Engineering ecosystem**. The digest ranks tools and terms of data engineering by their **momentum** with PyPI download growth and GitHub data, and enriches each tool with its Bluesky chatter on socials<sup>[1](#fn:1)</sup>.\n\nThis is how it looks with `make digest`:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n```\n\n | \n\n```\npackage          growth_pct  recent_daily  total_downloads   buzz\nxorq                  130.9           968          125,104     10\nibis-framework         49.5        90,384       13,655,147     15\nsqlglot                44.7     2,484,432      379,638,408     17\nduckdb                 42.5     1,712,837      263,396,944  3,200\ndagster                39.7       283,339       43,951,132    275\npolars                 35.8     2,160,110      339,334,602     61\npydantic               30.0    35,567,354    5,688,585,163     90\nsqlmesh                28.6        17,964        2,885,574     13\npyiceberg              24.4     1,300,103      211,933,200    640\nprefect                19.7       441,434       73,236,617     38\ndbt-core                9.9     3,527,875      609,587,211    369\napache-airflow          8.6       702,582      122,123,125    113\ndlt                   -13.8       237,792       46,500,226    178\n```\n\n | \n\nInteresting to see that we get a **rise of the dataframe & query engines**: the fastest-growing DE packages over the last 90 days are all query/dataframe engines:\n\n`ibis-framework` +49%, `sqlglot` +45%, `DuckDB` +42%, `Polars` +36%.\nOne caveat: I included xorq. It has the biggest growth, but it’s also the smallest package overall and still early, so the growth can have more spikes (we went from ~400 to ~968). And it’s worth mentioning that xorq, the tool we use for the grammar series, uses and is built on `ibis`, its great expression layer, as xorq builds on a rising dataframe for a composable, in-process engine.\n\nWe also see that **sqlmesh keeps climbing** even after the Fivetran acquisition and dbt Labs joining Fivetran:\n\n`sqlmesh` grew +28.6% over 90 days, while `dbt-core` grew the slowest of the pack (+9.9%).\nNot surprising, **DuckDB wins social media** attention. On Bluesky, its buzz score is 3200, 5× more than the next tool (pyiceberg 640, dbt 369). DuckDB is the tool that is both growing fast and the “loudest”.\n\n**Number 1** by raw downloads **is pydantic**. By absolute volume, pydantic tops everything at 5.64B downloads (~9× dbt-core’s 610M):\n\n| \n\n```\n1\n2\n3\n4\n5\n6\n7\n```\n\n | \n\n```\n── Naive leaderboard: raw PyPI downloads (all-time) ──\npackage            downloads\npydantic       5,639,574,035\ndbt-core         603,522,561\nsqlglot          376,710,633\npolars           335,901,439\nduckdb           261,307,529\n```\n\n | \n\nThis is probably because [Pydantic]https://github.com/pydantic/pydantic) is powering half of PyData while it doesn’t really have a lot of hype, but is a [Data Engineering Toolkit](https://www.ssp.sh/brain/data-engineering-toolkit/) for data validation and settings management using Python type annotations, used by any data engineer. It’s a part of the grammar that makes sure re-runs run deterministically.\n\n[!note] What the blogs (RSS) say\n\nMy RSS feeds were the noisiest signal: titles skew to whoever writes the most, e.g. Mr. Robin Moffatt (rmoff’s random ramblings alone are ~690 of ~1,600 articles) 😉. General sentiment clusters around the incumbents (dbt, dagster, Spark, Snowflake), while the surging engines (polars, SQLMesh) are barely mentioned. Blog coverage *lags* the download signal, which is precisely why the digest triangulates four sources instead of trusting just one.\n\n`stack.yaml` and the Exchangeable Engine\nThe key is really that model and execution are separated. Once the stack is defined (in our demo project I used `stack.yaml` as a declarative config), we can just change the engine by editing a YAML file, and everything else stays the same:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n```\n\n | \n\n```\nengine: datafusion              # duckdb | datafusion | snowflake  <- swap the engine here\ndb_path: de_ecosystem.duckdb\n\nsources:                        # nouns — dlt ingests these (outside the grammar)\n  bluesky:\n    max_pages: 40\n  github:\n    slice: \"data/raw/gharchive/*.json.gz\"\n\nmomentum:                       # the digest metric\n  window_days: 90               # 90 = laptop pulse; 365 = warehouse year-in-review\n  tools: [dbt-core, dagster, dlt, ibis-framework, xorq, apache-airflow, polars,\n          duckdb, great-expectations, pyiceberg, prefect, mage-ai, sqlmesh,\n          soda-core, pydantic, sqlglot]\n```\n\n | \n\nImagine in your deploy scripts for `dev` you’d use DataFusion and on `prod` you’d just specify Snowflake as the variable. No implementation code is touched as in a typical imperative workflow.\n\nWith `make manifest` we can compile the full data stack’s expressions into a deferred execution file `builds/<hash>/expr.yaml`, which builds deterministically and is diffable. The demo shows that point well: if we make the above engine change to `engine=duckdb` from `datafusion` and rebuild, the *semantic* change shows up as a reviewable `git diff`.\n\nLet’s run manifest with `engine: datafusion`:\n\n| \n\n```\n1\n2\n3\n4\n```\n\n | \n\n```\nmake manifest\n....\nWritten 'star_velocity' to builds/10671a1c33cf\nbuilds/10671a1c33cf\n```\n\n | \n\nNow changing `engine: duckdb` and re-running:\n\n| \n\n```\n1\n2\n3\n4\n```\n\n | \n\n```\nmake manifest\n....\nWritten 'star_velocity' to builds/f02f2c4dca81\nbuilds/f02f2c4dca81\n```\n\n | \n\nThe diff shows a couple of interesting bits, e.g. xorq changed the scale for timestamps for duckdb (see both files [datafusion](https://github.com/ssp-data/de-ecosystem-digest/blob/dde2fbfc19948a3147178bd36781247a8d78e98a/builds/10671a1c33cf/expr.yaml#L16) and [DuckDB](https://github.com/ssp-data/de-ecosystem-digest/blob/dde2fbfc19948a3147178bd36781247a8d78e98a/builds/f02f2c4dca81/expr.yaml#L16)):\n\n| \n\n```\n1\n2\n```\n\n | \n\n```\n  -      scale: 9      # datafusion → nanosecond timestamps\n  +      scale: ~      # duckdb → microsecond timestamps\n```\n\n | \n\nThe `profile.yaml` shows the literal change we did:\n\n| \n\n```\n1\n2\n3\n4\n5\n```\n\n | \n\n```\n  -  con_name: xorq_datafusion\n  -    config: ~\n  +  con_name: duckdb\n  +    database: \":memory:\"\n  +    read_only: false\n```\n\n | \n\nThis shows that swapping the engine by configuration has a real impact: DataFusion carries Timestamp(scale=9) (nanoseconds) and DuckDB defaults to microseconds. The **manifest captures that difference explicitly** even before we run anything, **reviewable instead of a silent runtime error** through the built [expression graphs](https://docs.xorq.dev/getting_started/your_first_expression.html) before executing them with **one expression, many engines**.\n\nApart from the grammar we look at, the project comes with catalog and ML functions to showcase the full capabilities of xorq and what you typically want to do in a data engineering project.\n\nThe full data lineage can also be tracked and shown.\n\nThe project has added a catalog to retrieve versioned entries of our metric and created artifacts. With `make catalog`, this project with xorq-catalog registers a curated set of expressions as **versioned, content-addressed entries** in a local, git-backed catalog at `./catalog`. The `catalog.yaml` manifest is the shelf where entries are addressed by content hash, and aliases are the human-readable handles:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n```\n\n | \n\n```\nentries:                     # content hashes (a new hash = a new version)\n  - 5adcf6bccba9             # dbt-star-velocity\n  - a4f38c87079a             # dbt-download-trend\n  - a636a49497fb             # dbt-momentum\n  # …\naliases:\n  - dbt-star-velocity\n  - dbt-download-trend\n  - dbt-momentum\n  - duckdb-buzz\n  - dbt-mentions\n  - dbt-health\n  - rising-tools\n```\n\n | \n\nEach entry holds its own metadata sidecar with the *expression + metadata + cached result, addressed by hash*. Here is `dbt-download-trend`, the whole metric captured declaratively (kind, output schema, the compiled SQL over the source table, and the cache key):\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n```\n\n | \n\n```\nmd5sum: 5a7d75db1e0cf46dc59a713a6ffa9573\nbackends: [xorq_datafusion]\nexpr_metadata:\n  kind: expr\n  schema_out:\n    date: timestamp(9)\n    downloads: int64\n    package: string\n  cache_keys:\n    key: xorq_cache-snapshot-4788a41e541c0526df207922813c0377\n    relative_path: parquet\n  sql_queries:\n    - - main\n      - xorq_datafusion\n      - |-\n        SELECT \"t0\".\"date\", \"t0\".\"downloads\", \"t0\".\"package\"\n        FROM \"raw_pypi_downloads\" AS \"t0\"\n        WHERE \"t0\".\"package\" = 'dbt-core'\n          AND \"t0\".\"category\" = 'without_mirrors'\n          AND \"t0\".\"date\" > DATE_TRUNC('DAY', '2026-05-14')\n        ORDER BY \"t0\".\"date\" ASC\n```\n\n | \n\nBecause xorq bundles the source read at build time, an entry is self-contained: `make catalog-run ALIAS=dbt-momentum` re-executes it with **no re-ingest**. Edit an expression (say `days=30 → 90`) and its content hash changes, so it registers as a *new* version while the old one stays retrievable.\n\nThe project also ships a small ML task (build feature matrix, split, fit a sklearn `LogisticRegression` wrapped in a xorq `Pipeline`, predict an adoption label) to mimic prediction and training of a real-life project. It could be interesting to add more sophisticated logic once more data is downloaded, and potentially even more sources are added.\n\nWith that example, we use the grammar of data with xorq to get essentially needed steps as part of the [Data Engineering Lifecycle](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/ch02.html#the_data_engineering_lifecycle-id000095). We can have full lineage, we get deterministic reruns, we get the metrics in the catalog. Plus, we get extensibility at each stage if we need it.\n\nE.g. extend the metrics from ‘catalog -> [Boring Semantic Layer](https://github.com/boringdata/boring-semantic-layer)’, or use dlt for ingestion as I did in this project to load data incrementally into a staging area.\n\nFor instance, `make lineage` shows everything xorq knows about a sentence *before* it touches a single row: its source nouns, output schema, bound engine, and the verbs compiled to SQL. This is how it looks:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n14\n15\n16\n17\n```\n\n | \n\n```\nLINEAGE — what xorq knows before any data is read:\n\n>> expr.op().find(DatabaseTable)   — the source nouns this expression reads:\n     ['raw_github_events']\n\n>> expr.schema()                   — the output columns (resolved at build time):\n     ibis.Schema { week: timestamp; stars: int64 }\n\n>> expr.ls.backends                — the engine(s) bound to it (MODIFIER):\n     ['Backend']\n\n>> ibis.to_sql(expr)               — the VERB chain, compiled to SQL:\nSELECT \"t1\".\"week\", COUNT(\"t1\".\"id\") AS \"stars\"\nFROM ( SELECT DATE_TRUNC('WEEK', \"created_at\") AS \"week\", \"id\"\n       FROM \"raw_github_events\"\n       WHERE \"repo_name\" = 'dbt-labs/dbt-core' AND \"type\" = 'WatchEvent' )\nGROUP BY \"t1\".\"week\" ORDER BY \"t1\".\"week\"\n```\n\n | \n\nIf you use the xorq Desktop app, this is integrated into a nice UI like this:\n\nThe tool-adoption model reuses the same four parts of speech: `fit(...)` attaches a **modifier** - the fitted model rides along as metadata (xorq tracks a `training_hash`) without changing what the expression computes, which is precisely [Part 1](https://www.ssp.sh/blog/grammar-data-engineering/)’s definition of a modifier. `predict(...)` is just a verb returning an Ibis table expression. And because predict is an expression, `xorq build` can manifest the *inference* pipeline too. Deploying a model collapses into the same `write → manifest → execute` cycle as deploying a metric.\n\n[!example] Add Semantic Layer\n\nHere we use the inbuilt catalog and metrics are defined as expressions in Ibis. If you like, you could lift the catalog metrics into the Boring Semantic Layer for dimensions/measures. BSL is built by Hussain, the creator of xorq, and is tightly integrated. Check [boring-semantic-layer](https://github.com/boringdata/boring-semantic-layer) if that is of interest, or check a recent article I wrote, [Why Semantic Layers Matter](https://motherduck.com/blog/semantic-layer-duckdb-tutorial/), with a practical example.\n\n[Part 1](https://www.ssp.sh/blog/grammar-data-engineering/) introduced the concept of grammar for data. With the data engineering digest project, we applied it to a demo project to capture the momentum of a tool in the data ecosystem. We mapped xorq’s features to the grammar to illustrate it better, and we’ve built a deterministic and versionable data stack with a single `stack.yaml`, providing the end-to-end data capabilities a data engineering project needs. Everything supports the case for a grammar for data.\n\nxorq gives us the bottom-up approach, working with our data, mapping all parts of the DE lifecycle, running it locally on multiple engines and discovering errors at pre-run time. Ultimately, [shifting left](https://www.rilldata.com/blog/what-shifting-left-means-and-why-it-matters-for-data-stacks) once more.\n\nThe grammar buys us two guarantees: answers that are *faithful* to the expression that produced them, and *reproducible* whenever we rerun it. But that doesn’t always mean correct. For example, the expression can still encode the wrong interpretation of the question. That third guarantee comes from reviewed definitions (the catalog entries and semantic models we built above). You can learn more about it in a follow-up article [Faithful, Reproducible, Wrong](https://xorq.dev/blog/faithful-reproducible-wrong/), where a checker plus a reviewed semantic model takes an agent from 4/100 to 100/100 correct answers on the same question.\n\nAnother related question is: if the grammar defines and the manifest records, who verifies it? That’s what we look at next in this series. You can get a sneak peek with a checker inside an agent harness ([pi](https://pi.dev/)), where every quantitative claim must be discharged by rerunning a content-addressed expression with its lineage intact. You can watch the loop in action in [this recording](https://asciinema.org/a/1263221) and reproduce it from the [pi-xorq-verification-example](https://github.com/xorq-labs/pi-xorq-verification-example) repo. More on verification in Part 3.\n\nIf you got interested and want to know more about how xorq works, check out the [docs](https://docs.xorq.dev/), or the open source repo on [GitHub](https://github.com/xorq-labs/xorq).\n\nAlso check out the [upcoming xorq Desktop app](https://xorq.dev/) (join the waitlist), which targets data analysts from the top down. It’s a desktop app on macOS and a trusted harness. It has additional features that do a verification check and more.\n\nAdditional project information for running the GitHub project, if you are interested in running it yourself and having a closer look.\n\nAll four sources use idempotent upsert, so re-running only adds new rows / updates existing ones (never duplicates):\n\n`write_disposition=\"merge\"` on `primary_key=\"id\"` (article URL / post URI)`INSERT OR REPLACE` on `(package, date, category)`` INSERT OR IGNORE` on event `id`\nTo run a full year or more, just use Snowflake, for example by installing `xorq[snowflake]` and ingesting the data into Snowflake. Re-running is always safe (idempotent merge), so just pull more and re-run. Each source has its own ceiling:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n```\n\n | \n\n```\n# 1. More Bluesky history — edit src/de_ecosystem/ingest/bluesky.py\nMAX_PAGES = 200            # 40 → 200, pages much further back in time\n\n# 2. More GitHub events — download extra hours into data/raw/gharchive/ (gitignored)\nfor h in $(seq 0 23); do\n  wget -nc -P data/raw/gharchive \"https://data.gharchive.org/2026-08-05-$h.json.gz\"\ndone\n\n# 3. Re-ingest (safe, merges) and re-run the whole sentence\nmake ingest\nmake run-sentence\n```\n\n | \n\nThe one limit is that the live `pypistats.org` API only serves ~180 days of downloads, so PyPI momentum caps at a 90-day-vs-prior-90-day window.\n\nA *full-year* digest needs a source that actually stores that history, and that is where Snowflake (or BigQuery’s public `bigquery-public-data.pypi.file_downloads`) comes in, both holding years of daily download stats.\n\nBecause the grammar separates *what* from *where*, you don’t rewrite the metric. You just point the **noun** at the warehouse and run the same sentence:\n\n| \n\n```\n1\n2\n3\n4\n```\n\n | \n\n```\nuv sync                       # the snowflake driver ships as a base dependency\n# put SNOWFLAKE_ACCOUNT / USER / PASSWORD / ROLE / DATABASE / WAREHOUSE / SCHEMA\n# in .env (or export them in your shell)\nmake engines                  # runs star_velocity on DuckDB, DataFusion AND Snowflake\n```\n\n | \n\nAnd with the same verb and only the engine binding changed:\n\n| \n\n```\n1\n2\n```\n\n | \n\n```\ncon = settings.backend(\"snowflake\")                             # xo.snowflake.connect_env()\nmomentum = download_momentum(con, \"duckdb\", window_days=365)    # a full year\n```\n\n | \n\nThe catalog code itself has no engine awareness. `settings.backend(\"duckdb\" | \"datafusion\" | \"snowflake\")` is the only thing that changes. Define once, represent it everywhere: your laptop for a 90-day pulse, a warehouse for the year-in-review.\n\nThe one-time grants (`de_digest` must exist and `DLT_LOADER_ROLE` needs `CREATE TABLE` **and** `CREATE STAGE`, since the loader stages the tables before copying them in):\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n```\n\n | \n\n```\n-- run once as ACCOUNTADMIN; the demo materialises the 4 raw tables into de_digest.PUBLIC\nUSE ROLE ACCOUNTADMIN;\n\nCREATE DATABASE IF NOT EXISTS de_digest;   -- PUBLIC schema is created automatically\n\nGRANT USAGE          ON DATABASE de_digest       TO ROLE DLT_LOADER_ROLE;\nGRANT USAGE          ON SCHEMA   de_digest.PUBLIC TO ROLE DLT_LOADER_ROLE;\nGRANT CREATE TABLE   ON SCHEMA   de_digest.PUBLIC TO ROLE DLT_LOADER_ROLE;\nGRANT CREATE STAGE   ON SCHEMA   de_digest.PUBLIC TO ROLE DLT_LOADER_ROLE;  -- adbc/pandas bulk\nload\nGRANT USAGE, OPERATE ON WAREHOUSE COMPUTE_WH      TO ROLE DLT_LOADER_ROLE;\n\nGRANT ROLE DLT_LOADER_ROLE TO USER loader;   -- if not already\n```\n\n | \n\nAnd the tables it created in Snowflake if everything works correctly:\n\n```\nFull article published at xorq.dev - written as part of my services\n```\n\nBluesky has a firehose and can simply be queried with DuckDB, e.g. see [Querying Bluesky with DuckDB and SQL](https://www.ssp.sh/brain/querying-bluesky-with-duckdb-and-sql/) [↩︎](#fnref:1)", "url": "https://wpnews.pro/news/the-grammar-of-data-from-definition-to-execution", "canonical_source": "https://www.ssp.sh/blog/from-definition-to-execution-grammar-of-data/", "published_at": "2026-09-14 06:00:08+00:00", "updated_at": "2026-09-14 23:07:25.433845+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "mlops"], "entities": ["xorq", "dlt", "Ibis", "DataFusion", "DuckDB", "Snowflake", "Bluesky", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/the-grammar-of-data-from-definition-to-execution", "markdown": "https://wpnews.pro/news/the-grammar-of-data-from-definition-to-execution.md", "text": "https://wpnews.pro/news/the-grammar-of-data-from-definition-to-execution.txt", "jsonld": "https://wpnews.pro/news/the-grammar-of-data-from-definition-to-execution.jsonld"}}