DuckDB is small. The data it queries doesn't have to be. So let's try something big. Common Crawl has been crawling and publishing web pages since 2008, and every crawl lands on S3 for anyone to read. Coding agents have turned a lot of people into web developers, so here is the internet-sized question for today: how much has personal, platform-hosted publishing actually grown?
GitHub, Vercel, Lovable and the rest all hand you a subdomain for whatever you just built, a dating app for ducks included. The obvious guess is that those subdomains are multiplying fast. Let's check. First some history before we get our hands on those 100 billion rows, so cue the old man yelling at the cloud.
Common Crawl, HTTP Archive, Wayback Machine: same same, but different?
If you've ever wanted to see the history of a page or find a site that has since been deleted, you may be familiar with the Wayback Machine, part of the Internet Archive. It is a great database of internet history, and while it is perfect for finding historical artefacts it doesn't lend itself well to analysis across sites. The HTTP Archive is a Google-backed project that cares less about the content than about the technology of the web, and it even loads JavaScript, partially, when it crawls. The downside is that it mostly captures the homepage of a site and nothing else.
Our dataset of choice, Common Crawl, does not load JavaScript, so it might miss parts of a JavaScript-powered single-page application, but it does go wide and deep, following all the links it finds. That makes it perfect to find, for example, github.io pages, which usually have the structure <github-username>.github.io/<repo-name>.
In short: Common Crawl is a broad net for all page content, HTTP Archive is a lab measurement of a fixed number of sites, and the Wayback Machine is a time series of individual URLs.
Common Crawl
HTTP Archive
Wayback Machine
Size / scope
~2.2B pages and ~150 TiB per monthly crawl since 2008
~16.2M websites per crawl (15.4M mobile, 12.2M desktop), sourced from Chrome UX Report (2025 methodology), since 2010
1 trillion URL captures, ~99 PB unique, ~150 TB/day (blog.archive.org/trillion). Internet Archive since 1996, Wayback Machine launched 2001
Home page vs. paths
Link-following crawl, so /v2/category/... style URLs are there, though coverage per site is a sample, not exhaustive
Home pages primarily, plus some secondary pages in recent years
Whatever anyone or any crawler ever requested
What's stored
Raw WARC (headers + HTML), plus WAT (metadata) and WET (plain text), plus a columnar Parquet index and a hyperlink/PageRank graph
HAR files, Lighthouse audits, technology detection
Full WARC captures of the original bytes, replayable in a browser
CDX API + web.archive.org/web/<timestamp>/<url>; no bulk dump
Best for
Text corpora (LLM training), link graphs, domain inventories
Web-performance, Core Web Vitals, framework/tracker share over time
Point lookups and history of a specific page
How much of the Common Crawl dataset do you actually need?
Common Crawl collects about 150 TiB per crawl. Every crawl is a sample, so I want several crawls per year and a few years of history. That sounds like querying 15 petabytes. It isn't. Most of the time "Big Data" just means a poorly designed dataset. This one is designed well, and it all sits on S3 for us to query and slice as we need.
So let's look at what's in the dataset and what we actually need. Common Crawl looks roughly like this.
The folder contains both the data in different formats and pointers to the data
WARC contains the full payload
WAT contains the metadata: HTTP headers, meta tags, outbound links, etc.
WET only contains the plain text
CDX URL index: the 300 or so files with just the index to point you to the right WARC file. You can try it out directly with e.g. https://index.commoncrawl.org/CC-MAIN-2026-34-index?url=motherduck.com&output=json
Every 3 months crawls are combined into a web graph. This dataset records how domains connect to each other and ranks them with several algorithms. It produces a list for both domains (e.g. github.io) and hostnames (duckfan67.github.io). For each of these there is:
A list of all nodes, that is the host or domain name with their IDs.
A list of all edges, that is which ID connects to which ID.
A ranking based on harmonic centrality and PageRank.
So we can take a shortcut and query the web graph's hostname list instead of the crawls it was built from. One release of that list is between 4.6GB and 11GB gzipped and holds every hostname the crawl saw across three months: 247 million of them in June 2026, spread over 121 million registered domains. Sixteen releases, one per quarter back to 2021, comes to 103GB of gzip. That drops us from 15 petabytes to 103 gigabytes without giving up a single hostname. And I do want the hostnames, because a count you can't drill into is a count you can't check.
The hostname list isn't quite enough, though. Remember the shape of a github.io URL. The hostname tells you who published, and how many repositories they published lives on the path. So we'll read both: the hostname list for breadth, the columnar URL index for paths.
Two ways to query Common Crawl with DuckDB
First we need to know what to query.
- Query Common Crawl on S3
All Common Crawl data sits in S3, free to read, and it is Hive partitioned, so querying several crawls at once costs you one glob.
from read_parquet('s3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-*/subset=warc/*.parquet', union_by_name = true)
where url_surtkey like 'com,motherduck)%'
limit 10000
That returns every MotherDuck page in the 2026 crawls in about 7 seconds. url_surtkey is the sort key, so DuckDB reads the Parquet row-group statistics and skips nearly every file. No cluster to start, no pool of servers, just a small DuckDB instance. The full scan back to 2008 takes three to four minutes.
Note that you will need a (free) AWS account with an S3 access key to make this work. If you don't want to do that you can also query directly over HTTP. For the rest of the article I'll use S3.
- Query Common Crawl over HTTP
If you don't want to use S3, you can still query Common Crawl. DuckDB uses HTTP range headers to download only the parts of the Parquet file it needs. With HTTP you cannot glob using the * as you would with S3, so you have to get all the actual paths of the Parquet files first.
-- store all Parquet files as a list
set variable index_files = (
select list('https://data.commoncrawl.org/' || path)
from read_csv([
'https://data.commoncrawl.org/crawl-data/CC-MAIN-2025-38/cc-index-table.paths.gz',
'https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-34/cc-index-table.paths.gz'
],
header = false,
columns = {'path': 'STRING'})
where path like '%subset=warc%'
);
-- access all Parquet files in the list
from read_parquet(getvariable('index_files'), union_by_name = true)
where url_surtkey like 'com,motherduck)%'
limit 10000;
The extraction: pulling hostnames out of the web graph
First, the list of platforms to look for. Swap in any you like. I picked the ones I could name off the top of my head, plus a reference group I'll come back to in the analysis.
create or replace table your_db.commoncrawl.platforms as
select * from (values
('base44.app', 'app.base44', 'Base44', 'platform'),
('bolt.host', 'host.bolt', 'Bolt', 'platform'),
('deno.dev', 'dev.deno', 'Deno Deploy', 'platform'),
('firebaseapp.com', 'com.firebaseapp', 'Google Firebase', 'platform'),
('web.app', 'app.web', 'Google Firebase', 'platform'),
('fly.dev', 'dev.fly', 'Fly.io', 'platform'),
('framer.app', 'app.framer', 'Framer', 'platform'),
('framer.website', 'website.framer', 'Framer', 'platform'),
('github.io', 'io.github', 'GitHub Pages', 'platform'),
('gitlab.io', 'io.gitlab', 'GitLab Pages', 'platform'),
('glitch.me', 'me.glitch', 'Glitch', 'platform'),
('herokuapp.com', 'com.herokuapp', 'Heroku', 'platform'),
('lovable.app', 'app.lovable', 'Lovable', 'platform'),
('neocities.org', 'org.neocities', 'Neocities', 'platform'),
('netlify.app', 'app.netlify', 'Netlify', 'platform'),
('onrender.com', 'com.onrender', 'Render', 'platform'),
('pages.dev', 'dev.pages', 'Cloudflare', 'platform'),
('workers.dev', 'dev.workers', 'Cloudflare', 'platform'),
('railway.app', 'app.railway', 'Railway', 'platform'),
('replit.app', 'app.replit', 'Replit', 'platform'),
('replit.dev', 'dev.replit', 'Replit', 'platform'),
('streamlit.app', 'app.streamlit', 'Streamlit', 'platform'),
('surge.sh', 'sh.surge', 'Surge', 'platform'),
('v0.app', 'app.v0', 'Vercel (v0)', 'platform'),
('v0.dev', 'dev.v0', 'Vercel (v0)', 'platform'),
('vercel.app', 'app.vercel', 'Vercel', 'platform'),
('webflow.io', 'io.webflow', 'Webflow', 'platform'),
-- the old guard, as a sanity check that the crawl itself isn't just growing
('blogspot.com', 'com.blogspot', 'Blogspot', 'reference'),
('medium.com', 'com.medium', 'Medium', 'reference'),
('myshopify.com', 'com.myshopify', 'Shopify', 'reference'),
('shopify.com', 'com.shopify', 'Shopify', 'reference'),
('squarespace.com', 'com.squarespace', 'Squarespace', 'reference'),
('substack.com', 'com.substack', 'Substack', 'reference'),
('tumblr.com', 'com.tumblr', 'Tumblr', 'reference'),
('weebly.com', 'com.weebly', 'Weebly', 'reference'),
('wixsite.com', 'com.wixsite', 'Wix', 'reference'),
('wordpress.com', 'com.wordpress', 'WordPress.com', 'reference')
) as t(domain, domain_rev, platform, kind);
Note the middle column. The web graph stores every name reversed, so github.io becomes io.github and duckfan67.github.io becomes io.github.duckfan67. Everything under one platform therefore sorts together, and the platform a hostname belongs to is its first two labels. That is the join key we'll build in a moment. But first: the actual data. We'll create the table, then go through each release one by one and insert them into the created table.
create or replace table your_db.commoncrawl.platform_hosts (
release varchar,
release_end_date date,
domain_rev varchar,
host_rev varchar,
harmonicc_pos bigint,
harmonicc_val double,
pr_pos bigint,
pr_val double
);
insert into your_db.commoncrawl.platform_hosts
with hosts as (
select
-- io.github.duckfan67 -> io.github, the reversed registered domain
split_part(host_rev, '.', 1) || '.' || split_part(host_rev, '.', 2) as domain_rev,
host_rev, harmonicc_pos, harmonicc_val, pr_pos, pr_val
from read_csv(
's3://commoncrawl/projects/hyperlinkgraph/cc-main-2026-apr-may-jun/host/cc-main-2026-apr-may-jun-host-ranks.txt.gz',
delim = '\t',
header = true,
columns = {
'harmonicc_pos' : 'bigint',
'harmonicc_val' : 'double',
'pr_pos' : 'bigint',
'pr_val' : 'double',
'host_rev' : 'varchar'})
)
select
'cc-main-2026-apr-may-jun', date '2026-06-30',
domain_rev, host_rev, harmonicc_pos, harmonicc_val, pr_pos, pr_val
from hosts
where domain_rev in (select domain_rev from your_db.commoncrawl.platforms);
The join key has to be derived, which is the one bit of thinking in that query. Every platform on the list is a two-label domain, so the first two labels of a reversed hostname are the reversed domain, and DuckDB can hash-join that against the platform list instead of testing 37 patterns per row.
One release takes three to six minutes and lands 10 to 14 million rows, scaling with the size of the gzip. Sixteen of them, run one after another: 103GB read, 64 minutes wall clock, 190,685,102 rows kept. That is the entire compute bill for this post, because the actual analysis queries in MotherDuck are counted in milliseconds rather than minutes.
Backfilling the web graph with a MotherDuck Flight
Now that we have the model for backfilling the 3-month spanning web graphs, we won't be running 16 separate queries ourselves. Because the same job runs once per release, this is a natural fit for a Flight, a small Python job that runs on MotherDuck compute on a schedule. The where release not in (select release from ...) guard makes it idempotent, so you can re-run it and it only fetches what's missing. The full flight code is in the MotherDuck cookbook.
WARNING: Resist the temptation to glob
cc-main-202* seems like an easy one-liner to get all releases, but since 2024 Common Crawl publishes the web graph as a rolling three-month window, released monthly. cc-main-2025-jul-aug-sep, cc-main-2025-aug-sep-oct and cc-main-2025-sep-oct-nov share two thirds of their input. Glob them all and you download 40 files, 250GB of gzip, for maybe 13 independent data points.
We'll pick one release per quarter and pin the names to make our backfill consistent. Some sets leap over into the next year (cc-main-2022-23-sep-nov-jan, for example); we'll take those as a good enough representation of the underlying quarter.
cc-main-2021-feb-apr-may cc-main-2024-apr-may-jun cc-main-2025-jul-aug-sep
cc-main-2021-jun-jul-sep cc-main-2024-jul-aug-sep cc-main-2025-oct-nov-dec
cc-main-2021-22-oct-nov-jan cc-main-2024-oct-nov-dec cc-main-2026-jan-feb-mar
cc-main-2022-may-jun-aug cc-main-2025-jan-feb-mar cc-main-2026-apr-may-jun
cc-main-2022-23-sep-nov-jan cc-main-2025-apr-may-jun
cc-main-2023-mar-may-oct
cc-main-2023-24-sep-nov-feb
INFO: Skip the download
If you'd rather not fetch all sixteen releases yourself, attach the share I prepared:
ATTACH 'md:_share/commoncrawl_platforms_public/96fe0b63-45b0-4775-be4e-9bfcff677311' AS commoncrawl_platforms;
from commoncrawl_platforms.main.platform_hosts limit 10;
The analysis: growth relative to the old free-hosting web
We cannot assume that every crawl covers the exact same parts of the internet. Even though the web graph already covers 3 months of crawls, we'll have to account for the fact that there is still sampling. Instead of looking at absolute numbers, then, we can analyse the relative presence of the tools. That is what the reference rows in the platform list are for. WordPress, Blogspot, Wix and friends are the boring, established part of the free-hosting web. If the crawl gets bigger, they get bigger too. If a platform grows relative to them, something real is happening.
The SQL for our analysis is relatively simple:
take all the different platforms
group by platform and crawl date
count the number of hosts
compare the reference platforms to the 'new' platforms
with ranked as (
select
p.platform,
p.kind,
r.release_end_date,
count(*) as hosts
from your_db.commoncrawl.platform_hosts r
join your_db.commoncrawl.platforms p using (domain_rev)
group by all
),
baseline as (
select release_end_date, sum(hosts) as reference_hosts
from ranked where kind = 'reference'
group by all
)
select
platform,
release_end_date,
hosts,
round(1000.0 * hosts / reference_hosts, 2) as hosts_per_1k_reference,
round(100.0 * hosts / first_value(hosts) over (
partition by platform order by release_end_date), 1) as indexed_to_first
from ranked
join baseline using (release_end_date)
where kind = 'platform'
order by platform, release_end_date;
The first findings: how fast the vibe-coded web is growing
To make our analysis a bit easier, I've built a MotherDuck Dive that compares the old free-hosting web (WordPress.com, Blogspot, and the rest of the gang) to the new platforms. I've grouped them into four categories.
Visual site builders like Webflow or Framer
The full AI experience: Lovable, Replit, v0, Base44, Bolt
Frontend and edge hosting: Cloudflare, Vercel, Netlify
Every category is up and to the right, and the visual site builders lead, at 55.6 hostnames per 1,000 reference hostnames in June 2026. Webflow is most of that. The more recent vibe-coding trend shows up in frontend and edge hosting, which went from 2.6 to 47.3 over the same five years, and the AI app builders are the newest line on the chart: flat at zero until 2025, then 4.7 by mid-2026.
NOTE
Common Crawl data has its downsides: it only counts crawled pages and pages with inbound links, not total sites built. It also doesn't crawl domains that deny access to bots, which more of them have done over the last few years. That explains the dip in crawled WordPress.com hostnames, for example. Even so, I think it's a reasonable proxy for how fast these platforms are growing.
Deselect the categories and pick individual platforms, and the story sharpens. Webflow is the line at the top. GitHub Pages grows steadily as you'd expect from the default answer of the last decade. Vercel grows steadily too, but Cloudflare Pages is the one nobody wrote about: from roughly nothing in 2023 to a peak above Vercel in late 2025, before settling back. Lovable barely registers on this scale, which is the sober version of the AI-builder story: plenty of new hostnames, none of them linked to by much yet.
Now switch the Dive to Absolute (hostnames) and add the established platforms back in, because in absolute numbers these platforms still pale compared to the big names.
WordPress.com alone fell from 6.0M crawled hostnames to under 2.0M over the five years, which is a bigger move in absolute terms than everything the new platforms added put together. Most of the relative growth in the first chart is the old web shrinking. What do you make of the difference in growth and decline? You now have the numbers and tools to answer your own questions.
One user, many projects: querying 100 billion rows of URL index
We have forgotten one thing though. github.io pages are not one user, one project. You can have many different projects based on different repositories in your GitHub account. I've written before about what the growth of GitHub and package registries looks like during the AI boom. But we're not here to be lazy, we're here to query the internet. So let's query some paths.
WARNING: Big data ahead
We'll scan 100 billion rows in a couple of minutes, but did you know that for a long time, the fastest way to transfer 100 petabytes of data was with an actual truck?
So let's drop the web graph dataset and go back to the columnar URL index, because that is the only place where the actual path of a page is captured. And let me tell you, this is what we call going out with a banger. We are querying:
5 years of data (2022-2026)
12,300 Parquet files
109,431,785,857 rows. That is MORE THAN 100 BILLION ROWS!
Those two numbers come straight out of the Parquet footers, which is why counting 109 billion rows takes half a minute rather than half a day:
Of course reading all of it would take forever, but url_surtkey is the sort key of the index, so DuckDB reads the Parquet row-group statistics and skips almost every file without opening it. Parquet lets us cut the other way too: we only ask for ten of the index's columns, and a columnar format can skip the rest without reading a byte of them (as opposed to, say, Postgres or MySQL). This is also the reason why friends don't let friends select *.
create or replace table your_db.commoncrawl.crawl_githubio as
select
url,
url_host_name,
url_host_name_reversed,
url_path,
fetch_time,
fetch_status,
content_digest,
content_mime_type,
content_languages,
crawl
from read_parquet([
's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2022-*/subset=warc/*.parquet',
's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2023-*/subset=warc/*.parquet',
's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2024-*/subset=warc/*.parquet',
's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2025-*/subset=warc/*.parquet',
's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-*/subset=warc/*.parquet'
], hive_partitioning = true, union_by_name = true)
where url_surtkey like 'io,github,%';
That is 16,353,324 rows kept out of 109 billion, across 41 crawls, and it takes a couple of minutes. The first path segment is the repository, and everything under it is one project no matter how deep the crawler went:
create or replace view your_db.commoncrawl.githubio_projects as
select
crawl,
url_host_name,
case
-- /myrepo/docs/api.html -> myrepo; /about.html -> the user site itself
when url_path = '/' or url_path not like '/%/%'
then '(user site root)'
else split_part(url_path, '/', 2)
end as project
from your_db.commoncrawl.crawl_githubio
where fetch_status = 200;
Absolute counts still can't be compared across crawls, because the crawls themselves differ in size. But we can take the total row count per crawl as a reference point, and that one is cheap for the same reason: it only reads the metadata in the footer of each Parquet file, not the file itself.
create or replace table your_db.commoncrawl.crawl_sizes as
select crawl, count(*) as total_pages
from read_parquet(
's3://commoncrawl/cc-index/table/cc-main/warc/crawl=CC-MAIN-2025-*/subset=warc/*.parquet',
hive_partitioning = true)
group by all;
One year of footers comes back in a few seconds. Run it five times, once per year, and you have the size of all 41 crawls. They sum to 109,431,785,857, which is the cheapest possible check that you globbed what you meant to glob. The crawls shrink as they go, from 2,986,392,030 pages in CC-MAIN-2022-05 to 2,139,617,681 in CC-MAIN-2026-34.
Two crawls per quarter, and always two
One crawl is one sample, and for this question it is a thin one. Common Crawl ran two crawls in the third quarter of 2026. Each of them found about 47,179 distinct user.github.io/repo pairs. Pool the two and you get 79,976, a factor of 1.70. Two crawls three weeks apart overlap far less than you would guess, so a second crawl buys most of a second crawl's worth of projects rather than a rounding error. That is the whole argument for pooling: a quarter sees more of the web than any crawl in it.
The catch is the release schedule. Common Crawl ran six crawls in 2022, five in 2023 but twelve in 2025. A quarter pooled from one crawl and a quarter pooled from three is a different type of measurement. We're not doing double blind testing here, but we do want some semblance of a meaningful statistic. So to accommodate we take exactly the two earliest crawls of every quarter that has two.
create or replace view your_db.commoncrawl.crawl_pairs as
select
crawl,
total_pages,
-- CC-MAIN-2026-34 means ISO week 34 of 2026
year(crawl_start) || '-Q' || quarter(crawl_start) as quarter
from (
select *,
date_trunc('week', make_date(cast(split_part(crawl, '-', 3) as int), 1, 4))
+ to_days((cast(split_part(crawl, '-', 4) as int) - 1) * 7) as crawl_start
from your_db.commoncrawl.crawl_sizes
)
qualify count(*) over (partition by quarter) >= 2
and row_number() over (partition by quarter order by crawl) <= 2;
That leaves 14 quarters, 2022-Q3 to 2026-Q3. 2022-Q1, 2022-Q2, 2023-Q1 and 2024-Q1 drop out with a single crawl each, and Common Crawl ran nothing at all in the third quarter of 2023. Now pool:
with per_user as (
select
c.quarter,
p.url_host_name,
count(*) as urls,
count(distinct project) filter (where project <> '(user site root)') as n_projects
from your_db.commoncrawl.githubio_projects p
join your_db.commoncrawl.crawl_pairs c using (crawl)
group by all
),
effort as (
select quarter, sum(total_pages) as pages_crawled
from your_db.commoncrawl.crawl_pairs
group by all
)
select
quarter,
pages_crawled,
count(*) as user_sites,
round(1e6 * count(*) / pages_crawled, 2) as sites_per_million,
round(1e6 * sum(n_projects) / pages_crawled, 1) as project_sites_per_million,
round(1.0 * sum(n_projects) / count(*), 2) as projects_per_user_site
from per_user join effort using (quarter)
group by quarter, pages_crawled -- not `group by all`: pages_crawled sits inside an aggregate
order by quarter;
Weeks below are the two ISO weeks pooled into that quarter, so 2022-Q3 is CC-MAIN-2022-27 plus CC-MAIN-2022-33. Every row is two crawls, so every row is the same instrument, and only the per-million columns are comparable down the table.
Quarter
Weeks
Pages crawled
User sites
User sites per million
Project sites per million
2022-Q3
27 + 33
5,698,299,922
40,008
7.02
20.9
2022-Q4
40 + 49
6,558,375,936
42,559
6.49
16.7
2023-Q2
14 + 23
6,259,207,381
45,369
7.25
16.8
2023-Q4
40 + 50
6,799,057,161
52,894
7.78
17.3
2024-Q2
18 + 22
5,496,678,032
44,185
8.04
18.5
2024-Q3
30 + 33
4,862,556,811
43,336
8.91
16.2
2024-Q4
42 + 46
5,183,002,501
47,030
9.07
17.7
2025-Q1
05 + 08
5,710,984,393
44,117
7.72
14.7
2025-Q2
18 + 21
5,224,501,712
43,087
8.25
13.9
2025-Q3
30 + 33
4,865,073,985
38,163
7.84
12.1
2025-Q4
43 + 47
4,911,269,769
40,645
8.28
12.9
2026-Q1
04 + 08
4,496,622,477
41,410
9.21
14.2
2026-Q2
17 + 21
4,356,077,177
54,850
12.59
20.6
2026-Q3
30 + 34
4,288,619,137
52,573
12.26
18.6
Normalised for crawl size, distinct github.io user sites are up 75% over the four years, 7.02 per million crawled pages to 12.26, and about half of that arrives in the last two quarters. Distinct project sites per million barely move, 20.9 to 18.6. More people, the same amount of stuff. Which points straight at the number in the last column of the pooled query, and that one needs a much harder look.
NOTE
Note that the actual sites crawled on Github Pages is much lower than the total number of Github Pages sites in the web graph. The web graph also takes into account links, but not every link is crawled every time.
Final results
The platform web went from 28 hostnames per 1,000 of the old free-hosting web in mid-2021 to 177 per 1,000 in mid-2026. 6.3x in five years and it keeps rising quarter-over-quarter including the quarters where the absolute platform count fell, because the crawl shrank faster than the platforms did. Normalised for crawl size, distinct github.io user sites are up 75% in four years. The drop in projects per user is the crawler taking fewer URLs per host, not people publishing less: hold the crawler's effort constant and the number is flat.
More people are putting things on the internet. Whether that is the AI boom or just what happens when publishing gets cheap enough is a question for a different post, but the trend itself is unambiguous and all it took was a few SQL queries.
The part you should take away though is that you can build your own billion row analysis in no time. Common Crawl publishes roughly 15 petabytes across the crawls in this analysis, but the answer came out in gigabytes. The power of partitioned Parquet files and DuckDB means that you can do this analysis on your laptop instead of a cluster of servers.
Reproduce it yourself
Everything above is four tables, a few SQL queries and a lot of Parquet files.
platforms, the hand-written list of 27 platforms and 10 reference domains.
platform_hosts, one insert per pinned web-graph release, three to six minutes each. The Flight in the cookbook runs all sixteen and skips the ones you already have.
crawl_githubio, one read_parquet over 12,300 Parquet files of the columnar URL index.
crawl_sizes, one metadata-only count(*) per year, seconds each, and the denominator for everything normalised.
The charts are the Dive embedded above, so you can switch between relative and absolute, pick your own platforms, and watch the numbers change when a new web-graph release lands.
If you want the numbers without the wait, attach the share. All four tables are in it, with crawl_githubio published as githubio_pages:
ATTACH 'md:_share/commoncrawl_platforms_public/96fe0b63-45b0-4775-be4e-9bfcff677311' AS commoncrawl_platforms;
from commoncrawl_platforms.main.githubio_pages limit 10;
And if you want it to keep updating itself as new web-graph releases land, take the Flight from the cookbook and give it a monthly cron. Swap my platform list for yours, point the surtkey filter at a different domain, and the same two queries answer a different question about the internet.
Classify text in SQL with prompt_jev(), powered by TypeSafe's Jev. 100,000 rows labeled in 40 seconds for $0.50 at frontier-LLM accuracy. Live on paid plans.
Define your dashboards in YAML with dbt Charts, then the same dbt build that runs your pipeline deploys them to MotherDuck Dives. Just 2 config changes! Your Dives will refresh live whenever your stakeholders take a look. See how we constructed the dbt package as well.