Querying the entire internet (100 billion rows!) with MotherDuck

MotherDuck published an analysis of Common Crawl's web archive, which has collected roughly 2.2 billion pages and about 150 TiB per monthly crawl since 2008, to measure how much personal, platform-hosted publishing has grown on services such as GitHub, Vercel and Lovable. The piece contrasts Common Crawl with the HTTP Archive, which covers about 16.2 million websites per crawl (15.4 million mobile, 12.2 million desktop) under its 2025 methodology, and the Internet Archive's Wayback Machine, which holds more than 1 trillion URL captures and about 99 PB unique at roughly 150 TB per day. MotherDuck argues the full multi-year Common Crawl corpus, which would appear to total about 15 petabytes, can be queried directly on S3 rather than fully downloaded.

DuckDB is small. The data it queries doesn't have to be. So let's try something big. Common Crawl has been crawling and publishing web pages since 2008, and every crawl lands on S3 for anyone to read. Coding agents have turned a lot of people into web developers, so here is the internet-sized question for today: how much has personal, platform-hosted publishing actually grown? GitHub, Vercel, Lovable and the rest all hand you a subdomain for whatever you just built, a dating app for ducks included. The obvious guess is that those subdomains are multiplying fast. Let's check. First some history before we get our hands on those 100 billion rows, so cue the old man yelling at the cloud. Common Crawl, HTTP Archive, Wayback Machine: same same, but different? If you've ever wanted to see the history of a page or find a site that has since been deleted, you may be familiar with the Wayback Machine, part of the Internet Archive. It is a great database of internet history, and while it is perfect for finding historical artefacts it doesn't lend itself well to analysis across sites. The HTTP Archive is a Google-backed project that cares less about the content than about the technology of the web, and it even loads JavaScript, partially, when it crawls. The downside is that it mostly captures the homepage of a site and nothing else. Our dataset of choice, Common Crawl, does not load JavaScript, so it might miss parts of a JavaScript-powered single-page application, but it does go wide and deep, following all the links it finds. That makes it perfect to find, for example, github.io pages, which usually have the structure <github-username .github.io/<repo-name . In short: Common Crawl is a broad net for all page content, HTTP Archive is a lab measurement of a fixed number of sites, and the Wayback Machine is a time series of individual URLs. Common Crawl HTTP Archive Wayback Machine Size / scope ~2.2B pages and ~150 TiB per monthly crawl since 2008 ~16.2M websites per crawl 15.4M mobile, 12.2M desktop , sourced from Chrome UX Report 2025 methodology , since 2010 1 trillion URL captures, ~99 PB unique, ~150 TB/day blog.archive.org/trillion . Internet Archive since 1996, Wayback Machine launched 2001 Home page vs. paths Link-following crawl, so /v2/category/... style URLs are there, though coverage per site is a sample, not exhaustive Home pages primarily, plus some secondary pages in recent years Whatever anyone or any crawler ever requested What's stored Raw WARC headers + HTML , plus WAT metadata and WET plain text , plus a columnar Parquet index and a hyperlink/PageRank graph HAR files, Lighthouse audits, technology detection Full WARC captures of the original bytes, replayable in a browser CDX API + web.archive.org/web/<timestamp /<url ; no bulk dump Best for Text corpora LLM training , link graphs, domain inventories Web-performance, Core Web Vitals, framework/tracker share over time Point lookups and history of a specific page How much of the Common Crawl dataset do you actually need? Common Crawl collects about 150 TiB per crawl. Every crawl is a sample, so I want several crawls per year and a few years of history. That sounds like querying 15 petabytes. It isn't. Most of the time "Big Data" just means a poorly designed dataset. This one is designed well, and it all sits on S3 for us to query and slice as we need. So let's look at what's in the dataset and what we actually need. Common Crawl looks roughly like this. The folder contains both the data in different formats and pointers to the data WARC contains the full payload WAT contains the metadata: HTTP headers, meta tags, outbound links, etc. WET only contains the plain text CDX URL index: the 300 or so files with just the index to point you to the right WARC file. You can try it out directly with e.g. https://index.commoncrawl.org/CC-MAIN-2026-34-index?url=motherduck.com&output=json Every 3 months crawls are combined into a web graph. This dataset records how domains connect to each other and ranks them with several algorithms. It produces a list for both domains e.g. github.io and hostnames duckfan67.github.io . For each of these there is: A list of all nodes, that is the host or domain name with their IDs. A list of all edges, that is which ID connects to which ID. A ranking based on harmonic centrality and PageRank. So we can take a shortcut and query the web graph's hostname list instead of the crawls it was built from. One release of that list is between 4.6GB and 11GB gzipped and holds every hostname the crawl saw across three months: 247 million of them in June 2026, spread over 121 million registered domains. Sixteen releases, one per quarter back to 2021, comes to 103GB of gzip. That drops us from 15 petabytes to 103 gigabytes without giving up a single hostname. And I do want the hostnames, because a count you can't drill into is a count you can't check. The hostname list isn't quite enough, though. Remember the shape of a github.io URL. The hostname tells you who published, and how many repositories they published lives on the path. So we'll read both: the hostname list for breadth, the columnar URL index for paths. Two ways to query Common Crawl with DuckDB First we need to know what to query. 1. Query Common Crawl on S3 All Common Crawl data sits in S3, free to read, and it is Hive partitioned, so querying several crawls at once costs you one glob. from read parquet 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026- /subset=warc/ .parquet', union by name = true where url surtkey like 'com,motherduck %' limit 10000 That returns every MotherDuck page in the 2026 crawls in about 7 seconds. url surtkey is the sort key, so DuckDB reads the Parquet row-group statistics and skips nearly every file. No cluster to start, no pool of servers, just a small DuckDB instance. The full scan back to 2008 takes three to four minutes. Note that you will need a free AWS account with an S3 access key to make this work. If you don't want to do that you can also query directly over HTTP. For the rest of the article I'll use S3. 2. Query Common Crawl over HTTP If you don't want to use S3, you can still query Common Crawl. DuckDB uses HTTP range headers to download only the parts of the Parquet file it needs. With HTTP you cannot glob using the as you would with S3, so you have to get all the actual paths of the Parquet files first. -- store all Parquet files as a list set variable index files = select list 'https://data.commoncrawl.org/' || path from read csv 'https://data.commoncrawl.org/crawl-data/CC-MAIN-2025-38/cc-index-table.paths.gz', 'https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-34/cc-index-table.paths.gz' , header = false, columns = {'path': 'STRING'} where path like '%subset=warc%' ; -- access all Parquet files in the list from read parquet getvariable 'index files' , union by name = true where url surtkey like 'com,motherduck %' limit 10000; The extraction: pulling hostnames out of the web graph First, the list of platforms to look for. Swap in any you like. I picked the ones I could name off the top of my head, plus a reference group I'll come back to in the analysis. create or replace table your db.commoncrawl.platforms as select from values 'base44.app', 'app.base44', 'Base44', 'platform' , 'bolt.host', 'host.bolt', 'Bolt', 'platform' , 'deno.dev', 'dev.deno', 'Deno Deploy', 'platform' , 'firebaseapp.com', 'com.firebaseapp', 'Google Firebase', 'platform' , 'web.app', 'app.web', 'Google Firebase', 'platform' , 'fly.dev', 'dev.fly', 'Fly.io', 'platform' , 'framer.app', 'app.framer', 'Framer', 'platform' , 'framer.website', 'website.framer', 'Framer', 'platform' , 'github.io', 'io.github', 'GitHub Pages', 'platform' , 'gitlab.io', 'io.gitlab', 'GitLab Pages', 'platform' , 'glitch.me', 'me.glitch', 'Glitch', 'platform' , 'herokuapp.com', 'com.herokuapp', 'Heroku', 'platform' , 'lovable.app', 'app.lovable', 'Lovable', 'platform' , 'neocities.org', 'org.neocities', 'Neocities', 'platform' , 'netlify.app', 'app.netlify', 'Netlify', 'platform' , 'onrender.com', 'com.onrender', 'Render', 'platform' , 'pages.dev', 'dev.pages', 'Cloudflare', 'platform' , 'workers.dev', 'dev.workers', 'Cloudflare', 'platform' , 'railway.app', 'app.railway', 'Railway', 'platform' , 'replit.app', 'app.replit', 'Replit', 'platform' , 'replit.dev', 'dev.replit', 'Replit', 'platform' , 'streamlit.app', 'app.streamlit', 'Streamlit', 'platform' , 'surge.sh', 'sh.surge', 'Surge', 'platform' , 'v0.app', 'app.v0', 'Vercel v0 ', 'platform' , 'v0.dev', 'dev.v0', 'Vercel v0 ', 'platform' , 'vercel.app', 'app.vercel', 'Vercel', 'platform' , 'webflow.io', 'io.webflow', 'Webflow', 'platform' , -- the old guard, as a sanity check that the crawl itself isn't just growing 'blogspot.com', 'com.blogspot', 'Blogspot', 'reference' , 'medium.com', 'com.medium', 'Medium', 'reference' , 'myshopify.com', 'com.myshopify', 'Shopify', 'reference' , 'shopify.com', 'com.shopify', 'Shopify', 'reference' , 'squarespace.com', 'com.squarespace', 'Squarespace', 'reference' , 'substack.com', 'com.substack', 'Substack', 'reference' , 'tumblr.com', 'com.tumblr', 'Tumblr', 'reference' , 'weebly.com', 'com.weebly', 'Weebly', 'reference' , 'wixsite.com', 'com.wixsite', 'Wix', 'reference' , 'wordpress.com', 'com.wordpress', 'WordPress.com', 'reference' as t domain, domain rev, platform, kind ; Note the middle column. The web graph stores every name reversed, so github.io becomes io.github and duckfan67.github.io becomes io.github.duckfan67. Everything under one platform therefore sorts together, and the platform a hostname belongs to is its first two labels. That is the join key we'll build in a moment. But first: the actual data. We'll create the table, then go through each release one by one and insert them into the created table. create or replace table your db.commoncrawl.platform hosts release varchar, release end date date, domain rev varchar, host rev varchar, harmonicc pos bigint, harmonicc val double, pr pos bigint, pr val double ; insert into your db.commoncrawl.platform hosts with hosts as select -- io.github.duckfan67 - io.github, the reversed registered domain split part host rev, '.', 1 || '.' || split part host rev, '.', 2 as domain rev, host rev, harmonicc pos, harmonicc val, pr pos, pr val from read csv 's3://commoncrawl/projects/hyperlinkgraph/cc-main-2026-apr-may-jun/host/cc-main-2026-apr-may-jun-host-ranks.txt.gz', delim = '\t', header = true, columns = { 'harmonicc pos' : 'bigint', 'harmonicc val' : 'double', 'pr pos' : 'bigint', 'pr val' : 'double', 'host rev' : 'varchar'} select 'cc-main-2026-apr-may-jun', date '2026-06-30', domain rev, host rev, harmonicc pos, harmonicc val, pr pos, pr val from hosts where domain rev in select domain rev from your db.commoncrawl.platforms ; The join key has to be derived, which is the one bit of thinking in that query. Every platform on the list is a two-label domain, so the first two labels of a reversed hostname are the reversed domain, and DuckDB can hash-join that against the platform list instead of testing 37 patterns per row. One release takes three to six minutes and lands 10 to 14 million rows, scaling with the size of the gzip. Sixteen of them, run one after another: 103GB read, 64 minutes wall clock, 190,685,102 rows kept. That is the entire compute bill for this post, because the actual analysis queries in MotherDuck are counted in milliseconds rather than minutes. Backfilling the web graph with a MotherDuck Flight Now that we have the model for backfilling the 3-month spanning web graphs, we won't be running 16 separate queries ourselves. Because the same job runs once per release, this is a natural fit for a Flight, a small Python job that runs on MotherDuck compute on a schedule. The where release not in select release from ... guard makes it idempotent, so you can re-run it and it only fetches what's missing. The full flight code is in the MotherDuck cookbook. WARNING: Resist the temptation to glob cc-main-202 seems like an easy one-liner to get all releases, but since 2024 Common Crawl publishes the web graph as a rolling three-month window, released monthly. cc-main-2025-jul-aug-sep, cc-main-2025-aug-sep-oct and cc-main-2025-sep-oct-nov share two thirds of their input. Glob them all and you download 40 files, 250GB of gzip, for maybe 13 independent data points. We'll pick one release per quarter and pin the names to make our backfill consistent. Some sets leap over into the next year cc-main-2022-23-sep-nov-jan, for example ; we'll take those as a good enough representation of the underlying quarter. cc-main-2021-feb-apr-may cc-main-2024-apr-may-jun cc-main-2025-jul-aug-sep cc-main-2021-jun-jul-sep cc-main-2024-jul-aug-sep cc-main-2025-oct-nov-dec cc-main-2021-22-oct-nov-jan cc-main-2024-oct-nov-dec cc-main-2026-jan-feb-mar cc-main-2022-may-jun-aug cc-main-2025-jan-feb-mar cc-main-2026-apr-may-jun cc-main-2022-23-sep-nov-jan cc-main-2025-apr-may-jun cc-main-2023-mar-may-oct cc-main-2023-24-sep-nov-feb INFO: Skip the download If you'd rather not fetch all sixteen releases yourself, attach the share I prepared: ATTACH 'md: share/commoncrawl platforms public/96fe0b63-45b0-4775-be4e-9bfcff677311' AS commoncrawl platforms; from commoncrawl platforms.main.platform hosts limit 10; The analysis: growth relative to the old free-hosting web We cannot assume that every crawl covers the exact same parts of the internet. Even though the web graph already covers 3 months of crawls, we'll have to account for the fact that there is still sampling. Instead of looking at absolute numbers, then, we can analyse the relative presence of the tools. That is what the reference rows in the platform list are for. WordPress, Blogspot, Wix and friends are the boring, established part of the free-hosting web. If the crawl gets bigger, they get bigger too. If a platform grows relative to them, something real is happening. The SQL for our analysis is relatively simple: take all the different platforms group by platform and crawl date count the number of hosts compare the reference platforms to the 'new' platforms with ranked as select p.platform, p.kind, r.release end date, count as hosts from your db.commoncrawl.platform hosts r join your db.commoncrawl.platforms p using domain rev group by all , baseline as select release end date, sum hosts as reference hosts from ranked where kind = 'reference' group by all select platform, release end date, hosts, round 1000.0 hosts / reference hosts, 2 as hosts per 1k reference, round 100.0 hosts / first value hosts over partition by platform order by release end date , 1 as indexed to first from ranked join baseline using release end date where kind = 'platform' order by platform, release end date; The first findings: how fast the vibe-coded web is growing To make our analysis a bit easier, I've built a MotherDuck Dive that compares the old free-hosting web WordPress.com, Blogspot, and the rest of the gang to the new platforms. I've grouped them into four categories. Visual site builders like Webflow or Framer The full AI experience: Lovable, Replit, v0, Base44, Bolt Frontend and edge hosting: Cloudflare, Vercel, Netlify Every category is up and to the right, and the visual site builders lead, at 55.6 hostnames per 1,000 reference hostnames in June 2026. Webflow is most of that. The more recent vibe-coding trend shows up in frontend and edge hosting, which went from 2.6 to 47.3 over the same five years, and the AI app builders are the newest line on the chart: flat at zero until 2025, then 4.7 by mid-2026. NOTE Common Crawl data has its downsides: it only counts crawled pages and pages with inbound links, not total sites built. It also doesn't crawl domains that deny access to bots, which more of them have done over the last few years. That explains the dip in crawled WordPress.com hostnames, for example. Even so, I think it's a reasonable proxy for how fast these platforms are growing. Deselect the categories and pick individual platforms, and the story sharpens. Webflow is the line at the top. GitHub Pages grows steadily as you'd expect from the default answer of the last decade. Vercel grows steadily too, but Cloudflare Pages is the one nobody wrote about: from roughly nothing in 2023 to a peak above Vercel in late 2025, before settling back. Lovable barely registers on this scale, which is the sober version of the AI-builder story: plenty of new hostnames, none of them linked to by much yet. Now switch the Dive to Absolute hostnames and add the established platforms back in, because in absolute numbers these platforms still pale compared to the big names. WordPress.com alone fell from 6.0M crawled hostnames to under 2.0M over the five years, which is a bigger move in absolute terms than everything the new platforms added put together. Most of the relative growth in the first chart is the old web shrinking. What do you make of the difference in growth and decline? You now have the numbers and tools to answer your own questions. One user, many projects: querying 100 billion rows of URL index We have forgotten one thing though. github.io pages are not one user, one project. You can have many different projects based on different repositories in your GitHub account. I've written before about what the growth of GitHub and package registries looks like during the AI boom. But we're not here to be lazy, we're here to query the internet. So let's query some paths. WARNING: Big data ahead We'll scan 100 billion rows in a couple of minutes, but did you know that for a long time, the fastest way to transfer 100 petabytes of data was with an actual truck? So let's drop the web graph dataset and go back to the columnar URL index, because that is the only place where the actual path of a page is captured. And let me tell you, this is what we call going out with a banger. We are querying: 5 years of data 2022-2026 12,300 Parquet files 109,431,785,857 rows. That is MORE THAN 100 BILLION ROWS Those two numbers come straight out of the Parquet footers, which is why counting 109 billion rows takes half a minute rather than half a day: Of course reading all of it would take forever, but url surtkey is the sort key of the index, so DuckDB reads the Parquet row-group statistics and skips almost every file without opening it. Parquet lets us cut the other way too: we only ask for ten of the index's columns, and a columnar format can skip the rest without reading a byte of them as opposed to, say, Postgres or MySQL . This is also the reason why friends don't let friends select . create or replace table your db.commoncrawl.crawl githubio as select url, url host name, url host name reversed, url path, fetch time, fetch status, content digest, content mime type, content languages, crawl from read parquet 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022- /subset=warc/ .parquet', 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2023- /subset=warc/ .parquet', 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2024- /subset=warc/ .parquet', 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2025- /subset=warc/ .parquet', 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026- /subset=warc/ .parquet' , hive partitioning = true, union by name = true where url surtkey like 'io,github,%'; That is 16,353,324 rows kept out of 109 billion, across 41 crawls, and it takes a couple of minutes. The first path segment is the repository, and everything under it is one project no matter how deep the crawler went: create or replace view your db.commoncrawl.githubio projects as select crawl, url host name, case -- /myrepo/docs/api.html - myrepo; /about.html - the user site itself when url path = '/' or url path not like '/%/%' then ' user site root ' else split part url path, '/', 2 end as project from your db.commoncrawl.crawl githubio where fetch status = 200; Absolute counts still can't be compared across crawls, because the crawls themselves differ in size. But we can take the total row count per crawl as a reference point, and that one is cheap for the same reason: it only reads the metadata in the footer of each Parquet file, not the file itself. create or replace table your db.commoncrawl.crawl sizes as select crawl, count as total pages from read parquet 's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2025- /subset=warc/ .parquet', hive partitioning = true group by all; One year of footers comes back in a few seconds. Run it five times, once per year, and you have the size of all 41 crawls. They sum to 109,431,785,857, which is the cheapest possible check that you globbed what you meant to glob. The crawls shrink as they go, from 2,986,392,030 pages in CC-MAIN-2022-05 to 2,139,617,681 in CC-MAIN-2026-34. Two crawls per quarter, and always two One crawl is one sample, and for this question it is a thin one. Common Crawl ran two crawls in the third quarter of 2026. Each of them found about 47,179 distinct user.github.io/repo pairs. Pool the two and you get 79,976, a factor of 1.70. Two crawls three weeks apart overlap far less than you would guess, so a second crawl buys most of a second crawl's worth of projects rather than a rounding error. That is the whole argument for pooling: a quarter sees more of the web than any crawl in it. The catch is the release schedule. Common Crawl ran six crawls in 2022, five in 2023 but twelve in 2025. A quarter pooled from one crawl and a quarter pooled from three is a different type of measurement. We're not doing double blind testing here, but we do want some semblance of a meaningful statistic. So to accommodate we take exactly the two earliest crawls of every quarter that has two. create or replace view your db.commoncrawl.crawl pairs as select crawl, total pages, -- CC-MAIN-2026-34 means ISO week 34 of 2026 year crawl start || '-Q' || quarter crawl start as quarter from select , date trunc 'week', make date cast split part crawl, '-', 3 as int , 1, 4 + to days cast split part crawl, '-', 4 as int - 1 7 as crawl start from your db.commoncrawl.crawl sizes qualify count over partition by quarter = 2 and row number over partition by quarter order by crawl <= 2; That leaves 14 quarters, 2022-Q3 to 2026-Q3. 2022-Q1, 2022-Q2, 2023-Q1 and 2024-Q1 drop out with a single crawl each, and Common Crawl ran nothing at all in the third quarter of 2023. Now pool: with per user as select c.quarter, p.url host name, count as urls, count distinct project filter where project < ' user site root ' as n projects from your db.commoncrawl.githubio projects p join your db.commoncrawl.crawl pairs c using crawl group by all , effort as select quarter, sum total pages as pages crawled from your db.commoncrawl.crawl pairs group by all select quarter, pages crawled, count as user sites, round 1e6 count / pages crawled, 2 as sites per million, round 1e6 sum n projects / pages crawled, 1 as project sites per million, round 1.0 sum n projects / count , 2 as projects per user site from per user join effort using quarter group by quarter, pages crawled -- not group by all : pages crawled sits inside an aggregate order by quarter; Weeks below are the two ISO weeks pooled into that quarter, so 2022-Q3 is CC-MAIN-2022-27 plus CC-MAIN-2022-33. Every row is two crawls, so every row is the same instrument, and only the per-million columns are comparable down the table. Quarter Weeks Pages crawled User sites User sites per million Project sites per million 2022-Q3 27 + 33 5,698,299,922 40,008 7.02 20.9 2022-Q4 40 + 49 6,558,375,936 42,559 6.49 16.7 2023-Q2 14 + 23 6,259,207,381 45,369 7.25 16.8 2023-Q4 40 + 50 6,799,057,161 52,894 7.78 17.3 2024-Q2 18 + 22 5,496,678,032 44,185 8.04 18.5 2024-Q3 30 + 33 4,862,556,811 43,336 8.91 16.2 2024-Q4 42 + 46 5,183,002,501 47,030 9.07 17.7 2025-Q1 05 + 08 5,710,984,393 44,117 7.72 14.7 2025-Q2 18 + 21 5,224,501,712 43,087 8.25 13.9 2025-Q3 30 + 33 4,865,073,985 38,163 7.84 12.1 2025-Q4 43 + 47 4,911,269,769 40,645 8.28 12.9 2026-Q1 04 + 08 4,496,622,477 41,410 9.21 14.2 2026-Q2 17 + 21 4,356,077,177 54,850 12.59 20.6 2026-Q3 30 + 34 4,288,619,137 52,573 12.26 18.6 Normalised for crawl size, distinct github.io user sites are up 75% over the four years, 7.02 per million crawled pages to 12.26, and about half of that arrives in the last two quarters. Distinct project sites per million barely move, 20.9 to 18.6. More people, the same amount of stuff. Which points straight at the number in the last column of the pooled query, and that one needs a much harder look. NOTE Note that the actual sites crawled on Github Pages is much lower than the total number of Github Pages sites in the web graph. The web graph also takes into account links, but not every link is crawled every time. Final results The platform web went from 28 hostnames per 1,000 of the old free-hosting web in mid-2021 to 177 per 1,000 in mid-2026. 6.3x in five years and it keeps rising quarter-over-quarter including the quarters where the absolute platform count fell, because the crawl shrank faster than the platforms did. Normalised for crawl size, distinct github.io user sites are up 75% in four years. The drop in projects per user is the crawler taking fewer URLs per host, not people publishing less: hold the crawler's effort constant and the number is flat. More people are putting things on the internet. Whether that is the AI boom or just what happens when publishing gets cheap enough is a question for a different post, but the trend itself is unambiguous and all it took was a few SQL queries. The part you should take away though is that you can build your own billion row analysis in no time. Common Crawl publishes roughly 15 petabytes across the crawls in this analysis, but the answer came out in gigabytes. The power of partitioned Parquet files and DuckDB means that you can do this analysis on your laptop instead of a cluster of servers. Reproduce it yourself Everything above is four tables, a few SQL queries and a lot of Parquet files. platforms, the hand-written list of 27 platforms and 10 reference domains. platform hosts, one insert per pinned web-graph release, three to six minutes each. The Flight in the cookbook runs all sixteen and skips the ones you already have. crawl githubio, one read parquet over 12,300 Parquet files of the columnar URL index. crawl sizes, one metadata-only count per year, seconds each, and the denominator for everything normalised. The charts are the Dive embedded above, so you can switch between relative and absolute, pick your own platforms, and watch the numbers change when a new web-graph release lands. If you want the numbers without the wait, attach the share. All four tables are in it, with crawl githubio published as githubio pages: ATTACH 'md: share/commoncrawl platforms public/96fe0b63-45b0-4775-be4e-9bfcff677311' AS commoncrawl platforms; from commoncrawl platforms.main.githubio pages limit 10; And if you want it to keep updating itself as new web-graph releases land, take the Flight from the cookbook and give it a monthly cron. Swap my platform list for yours, point the surtkey filter at a different domain, and the same two queries answer a different question about the internet. Classify text in SQL with prompt jev , powered by TypeSafe's Jev. 100,000 rows labeled in 40 seconds for $0.50 at frontier-LLM accuracy. Live on paid plans. Define your dashboards in YAML with dbt Charts, then the same dbt build that runs your pipeline deploys them to MotherDuck Dives. Just 2 config changes Your Dives will refresh live whenever your stakeholders take a look. See how we constructed the dbt package as well.