I tried to clean Commoncrawl

I created a dataset called CC-FilteredCorpus where I’m trying to filter Common Crawl and extract the content as clean Markdown.

The goal is to preserve useful document structure—headings, lists, links, code blocks, tables, etc.—instead of flattening everything into plain text.

I’m hoping this can be useful for LLM pretraining and other NLP research where preserving the structure of web documents matters.

The dataset is at helloadhavan/CC-FilteredCorpus