{"slug": "github-polymathicai-the-well-a-15tb-collection-of-physics-simulation-datasets", "title": "GitHub – PolymathicAI/The_well: A 15TB Collection of Physics Simulation Datasets", "summary": "PolymathicAI released The Well, a 15TB collection of 16 physics simulation datasets for machine learning, covering domains like fluid dynamics and supernova explosions. The open-source repository provides tools for downloading, streaming, and benchmarking surrogate models.", "body_md": "Welcome to the Well, a large-scale collection of machine learning datasets containing numerical simulations of a wide variety of spatiotemporal physical systems. The Well draws from domain scientists and numerical software developers to provide 15TB of data across 16 datasets covering diverse domains such as biological systems, fluid dynamics, acoustic scattering, as well as magneto-hydrodynamic simulations of extra-galactic fluids or supernova explosions. These datasets can be used individually or as part of a broader benchmark suite for accelerating research in machine learning and computational sciences.\n\nOnce the Well package installed and the data downloaded you can use them in your training pipeline.\n\n``` python\nfrom the_well.data import WellDataset\nfrom torch.utils.data import DataLoader\n\ntrainset = WellDataset(\n    well_base_path=\"path/to/base\",\n    well_dataset_name=\"name_of_the_dataset\",\n    well_split_name=\"train\"\n)\ntrain_loader = DataLoader(trainset)\n\nfor batch in train_loader:\n    ...\n```\n\nFor more information regarding the interface, please refer to the [API](https://github.com/PolymathicAI/the_well/tree/master/docs/api.md) and the [tutorials](https://github.com/PolymathicAI/the_well/blob/master/docs/tutorials/dataset.ipynb).\n\nIf you plan to use The Well datasets to train or evaluate deep learning models, we recommend to use a machine with enough computing resources.\nWe also recommend creating a new Python (>=3.10) environment to install the Well. For instance, with [venv](https://docs.python.org/3/library/venv.html):\n\n```\npython -m venv path/to/env\nsource path/to/env/activate/bin\n```\n\nThe Well package can be installed directly from PyPI.\n\n```\npip install the_well\n```\n\nIt can also be installed from source. For this, clone the [repository](https://github.com/PolymathicAI/the_well) and install the package with its dependencies.\n\n```\ngit clone https://github.com/PolymathicAI/the_well\ncd the_well\npip install .\n```\n\nDepending on your acceleration hardware, you can specify `--extra-index-url`\n\nto install the relevant PyTorch version. For example, use\n\n```\npip install . --extra-index-url https://download.pytorch.org/whl/cu121\n```\n\nto install the dependencies built for CUDA 12.1.\n\nIf you want to run the benchmarks, you should install additional dependencies.\n\n```\npip install the_well[benchmark]\n```\n\nThe Well datasets range between 6.9GB and 5.1TB of data each, for a total of 15TB for the full collection. Ensure that your system has enough free disk space to accomodate the datasets you wish to download.\n\nOnce `the_well`\n\nis installed, you can use the `the-well-download`\n\ncommand to download any dataset of The Well.\n\n```\nthe-well-download --base-path path/to/base --dataset active_matter --split train\n```\n\nIf `--dataset`\n\nand `--split`\n\nare omitted, all datasets and splits will be downloaded. This could take a while!\n\nMost of the Well datasets are also hosted on [Hugging Face](https://huggingface.co/collections/polymathic-ai/the-well-67e129f4ca23e0447395d74c). Data can be streamed directly from the hub using the following code.\n\n``` python\nfrom the_well.data import WellDataset\nfrom torch.utils.data import DataLoader\n\n# The following line may take a couple of minutes to instantiate the datamodule\ntrainset = WellDataset(\n    well_base_path=\"hf://datasets/polymathic-ai/\",  # access from HF hub\n    well_dataset_name=\"active_matter\",\n    well_split_name=\"train\",\n)\ntrain_loader = DataLoader(trainset)\n\nfor batch in train_loader:\n    ...\n```\n\nFor better performance in large training, we advise [downloading the data locally](#downloading-the-data) instead of streaming it over the network.\n\nThe repository allows benchmarking surrogate models on the different datasets that compose the Well. Some state-of-the-art models are already implemented in [ models](https://github.com/PolymathicAI/the_well/tree/master/the_well/benchmark/models), while\n\n[dataset classes](https://github.com/PolymathicAI/the_well/tree/master/the_well/data)handle the raw data of the Well. The benchmark relies on\n\n[a training script](https://github.com/PolymathicAI/the_well/blob/master/the_well/benchmark/train.py)that uses\n\n[hydra](https://hydra.cc/)to instantiate various classes (e.g. dataset, model, optimizer) from\n\n[configuration files](https://github.com/PolymathicAI/the_well/tree/master/the_well/benchmark/configs).\n\nFor instance, to run the training script of default FNO architecture on the active matter dataset, launch the following commands:\n\n```\ncd the_well/benchmark\npython train.py experiment=fno server=local data=active_matter\n```\n\nEach argument corresponds to a specific configuration file. In the command above `server=local`\n\nindicates the training script to use [ local.yaml](https://github.com/PolymathicAI/the_well/tree/master/the_well/benchmark/configs/server/local.yaml), which just declares the relative path to the data. The configuration can be overridden directly or edited with new YAML files. Please refer to\n\n[hydra documentation](https://hydra.cc/)for editing configuration.\n\nYou can use this command within a sbatch script to launch the training with Slurm.\n\nThe model benchmarked in the original paper of the Well have been designed as a a simple baseline. They should not be considered as state-of-the-art. We hope that the community will build upon these results to develop better architectures for PDE surrogate modeling.\n\nMost of the checkpoints of the models are available on [Hugging Face](https://huggingface.co/collections/polymathic-ai/the-well-benchmark-models-67e69bd7cd8e60229b5cd43e). To load a specific checkpoint follow the example below of the FNO model trained on the `active_matter`\n\ndataset.\n\n``` python\nfrom the_well.benchmark.models import FNO\n\nmodel = FNO.from_pretrained(\"polymathic-ai/FNO-active_matter\")\n```\n\n- The dataset\n`viscoelastic_instability`\n\nhas been deprecated due to processing errors in the data. It remains available for backwards comparisons. However`viscoelastic_instability_v2`\n\ncontains the same data without the processing error.\n\nThis project has been led by the [Polymathic AI](https://polymathic-ai.org/) organization, in collaboration with researchers from the Flatiron Institute, University of Colorado Boulder, University of Cambridge, New York University, Rutgers University, Cornell University, University of Tokyo, Los Alamos Natioinal Laboratory, University of California, Berkeley, Princeton University, CEA DAM, and University of Liège.\n\nIf you find this project useful for your research, please consider citing\n\n```\n@article{ohana2024well,\n  title={The well: a large-scale collection of diverse physics simulations for machine learning},\n  author={Ohana, Ruben and McCabe, Michael and Meyer, Lucas and Morel, Rudy and Agocs, Fruzsina and Beneitez, Miguel and Berger, Marsha and Burkhart, Blakesly and Dalziel, Stuart and Fielding, Drummond and others},\n  journal={Advances in Neural Information Processing Systems},\n  volume={37},\n  pages={44989--45037},\n  year={2024}\n}\n```\n\nFor questions regarding this project, please contact [Ruben Ohana](https://rubenohana.github.io/) and [Michael McCabe](https://mikemccabe210.github.io/) at {rohana,mmccabe}@flatironinstitute.org.\n\nTo report a bug (in the data or the code), request a feature or simply ask a question, you can [open an issue](https://github.com/PolymathicAI/the_well/issues) on the [repository](https://github.com/PolymathicAI/the_well).", "url": "https://wpnews.pro/news/github-polymathicai-the-well-a-15tb-collection-of-physics-simulation-datasets", "canonical_source": "https://github.com/PolymathicAI/the_well/", "published_at": "2026-07-11 14:28:56+00:00", "updated_at": "2026-07-11 14:34:52.233837+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research", "ai-tools", "ai-infrastructure"], "entities": ["PolymathicAI", "The Well", "Hugging Face", "PyTorch", "CUDA"], "alternates": {"html": "https://wpnews.pro/news/github-polymathicai-the-well-a-15tb-collection-of-physics-simulation-datasets", "markdown": "https://wpnews.pro/news/github-polymathicai-the-well-a-15tb-collection-of-physics-simulation-datasets.md", "text": "https://wpnews.pro/news/github-polymathicai-the-well-a-15tb-collection-of-physics-simulation-datasets.txt", "jsonld": "https://wpnews.pro/news/github-polymathicai-the-well-a-15tb-collection-of-physics-simulation-datasets.jsonld"}}