I tried to clean Commoncrawl

Adhavan has released CC-FilteredCorpus, a dataset on Hugging Face that filters Common Crawl and converts web content into clean Markdown, preserving document structure such as headings, lists, links, code blocks, and tables, intended for LLM pretraining and NLP research.

I created a dataset called CC-FilteredCorpus where I’m trying to filter Common Crawl and extract the content as clean Markdown . The goal is to preserve useful document structure—headings, lists, links, code blocks, tables, etc.—instead of flattening everything into plain text. I’m hoping this can be useful for LLM pretraining and other NLP research where preserving the structure of web documents matters. The dataset is at helloadhavan/CC-FilteredCorpus https://huggingface.co/datasets/helloadhavan/CC-FilteredCorpus