# Methodology > Source: > Published: 2026-08-20 15:55:49+00:00 ### Data collection To get a better picture of the prevalence of AI-authored content across the internet, we used webpage data sampled from [Common Crawl](https://commoncrawl.org/), a nonprofit organization that collects and maintains a large web archive stretching back to 2008. Roughly once a month, Common Crawl completes a “crawl” of the observable internet, creating a snapshot of the web at that point in time. To be included in a Common Crawl snapshot, pages must be publicly accessible. Sites that have paywalls or require a login to access content are likely underrepresented in these samples. We randomly sampled 10,000 English-language pages from each of the 49 crawls created between January 2021 and July 2026, for a total of 490,000 pages. For each sampled page, we collected Common Crawl’s WARC (Web ARChive) record containing the full HTML, as well as the WET (WARC Encapsulated Text) file containing just the body text of the page, excluding any HTML code, images or other media. About 10% to 15% of the webpages in a given crawl sample contain a publication date field in their HTML code. We recorded these dates where available and used them to determine which pages from each crawl had been published since the public release of ChatGPT on Nov. 30, 2022. The subset of pages with a detectable publication date is not a random subset of the web, so our post-ChatGPT estimate reflects the prevalence of AI authorship among dated content rather than the web as a whole. However, our findings in this analysis are broadly in line with related studies, including “[The Impact of AI-Generated Text on the Internet](https://ai-on-the-internet.github.io/)” (Dolezal et al., 2026), which used data from the Internet Archive to determine that 35% of newly published websites contained AI-generated or AI-assisted text. While derived by a different methodology, this finding can be compared to our finding that 35% of pages in the July 2026 crawl with post-ChatGPT publication dates show signs of AI authorship. ### Detecting AI authorship To detect signs of AI authorship in the sampled webpages, we input the body text from each page through [editlens_Llama-3.2-3B](https://huggingface.co/pangram/editlens_Llama-3.2-3B), an open-weight AI detection model developed by [Pangram](https://www.pangram.com/blog/introducing-open-pangram). This model accepts text as input and returns a numeric score ranging from 0 to 1, with 0 indicating fully human-written text and 1 indicating fully AI-generated text. This allows the model to more accurately describe texts that are not entirely AI-generated, but instead are likely to have a mix of human and AI authorship. For this analysis, any page with a score of 0.2 or greater was considered to contain “meaningful” signs of AI authorship or editing.[1](#fn-356393-1) Pangram’s open models are designed for research use, but their commercial offerings can be more accurate in some cases. To confirm that our findings were not dependent on the specific model used, we also ran 62,370 pages from seven crawls through their flagship commercial model Pangram 3.3 and compared these results to our analysis. Overall, the two models are relatively closely aligned: They agree in 96% of cases, with a Cohen’s kappa value of 0.61. The page-level disagreements we observed between the two models are a reminder that AI detection models are probabilistic tools, and their classifications of an individual page should not necessarily be taken as definitive verdicts about that page’s authorship. But applying this tool consistently across a large collection of pages allows us to systematically track how the aggregate share of AI-authored content has shifted over time. When looking at aggregate values for how much of a given sample shows signs of AI authorship, the overall percentages generated by Open Pangram and Pangram 3.3 were reasonably close to each other, indicating that the overall trend of increased AI authorship over time is consistent regardless of model. Some of the largest deviations between the two models were observed in early, pre-ChatGPT crawls: Estimates derived from Pangram 3.3 were very low, and estimates derived from Open Pangram were higher at around 1%. This suggests that Open Pangram has a higher false-positive rate on pages created before AI use was common. Reported shares from 2021 and 2022 crawls should be treated as approximate and likely include some human-written content that was misclassified by Open Pangram, rather than as evidence that meaningful amounts of AI authorship existed across the web prior to ChatGPT’s release.