{"slug": "a-bitter-lesson-for-data-filtering", "title": "A Bitter Lesson for Data Filtering", "summary": "A paper posted to arXiv on 19 May 2026 finds that for large-model pretraining in the high-compute, data-scarce regime, the best data filter is no data filter. The scaling studies report that sufficiently trained large-parameter models not only tolerate low-quality and distractor data but benefit from nominally \"poor\" data, contradicting the common belief that pretraining data must be filtered to high-quality information only.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 19 May 2026]\n\n# Title:A Bitter Lesson for Data Filtering\n\n[View PDF](https://arxiv.org/pdf/2605.19407)\n\n[HTML (experimental)](https://arxiv.org/html/2605.19407v1)\n\nAbstract:We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.\n    \n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/a-bitter-lesson-for-data-filtering", "canonical_source": "https://arxiv.org/abs/2605.19407", "published_at": "2026-09-18 04:01:53+00:00", "updated_at": "2026-09-18 04:27:53.568155+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/a-bitter-lesson-for-data-filtering", "markdown": "https://wpnews.pro/news/a-bitter-lesson-for-data-filtering.md", "text": "https://wpnews.pro/news/a-bitter-lesson-for-data-filtering.txt", "jsonld": "https://wpnews.pro/news/a-bitter-lesson-for-data-filtering.jsonld"}}