{"slug": "earthmover-and-hugging-face-tackle-the-slow-part-of-fast-weather-ai", "title": "Earthmover and Hugging Face tackle the slow part of fast weather AI", "summary": "Earthmover and Hugging Face published a joint tutorial on September 8th that pairs Microsoft's Aurora weather model with Earthmover's array-based data infrastructure, enabling a 24-hour forecast in under 30 seconds end to end. The workflow streams selected ERA5 variables instead of downloading the typical 1GB of initial conditions, addressing the data bottleneck that leaves GPUs idle during forecast preparation. The tutorial also includes an interactive demo for users to run models and inspect stage durations.", "body_md": "# Earthmover and Hugging Face tackle the slow part of fast weather AI\n\n**The tutorial pairs Microsoft's Aurora with Earthmover data and Hugging Face compute, turning a 1GB data-fetch problem into a runnable forecast.**\n\n        By [RuntimeWire Staff](/author/runtimewire-staff)\n        · Published \n\nPrimary source: [Hugging Face Newsroom](https://huggingface.co/blog/hugging-science/earthmover-hf)\n\n## Why it matters\n\nOpen weather models are becoming easier to obtain, shifting the engineering burden toward data access and validation. Earthmover is positioning its array infrastructure as the layer every model run still needs.\n\n[Ryan Abernathey (@rabernat)](https://x.com/rabernat?ref=runtimewire) and [Joe Hamman](https://joehamman.com/about/?ref=runtimewire), the climate scientists who founded [Earthmover](https://earthmover.io/?ref=runtimewire), published a workflow with [Hugging Face](https://huggingface.co/?ref=runtimewire) on September 8th that makes open-weight AI weather models easier to initialize, run and evaluate.\n\nThe [joint tutorial](https://huggingface.co/blog/hugging-science/earthmover-hf?ref=runtimewire) connects models hosted on Hugging Face with analysis-ready weather data from Earthmover. Its main example pulls ECMWF ERA5 data into Microsoft's Aurora, generates a four-step, 24-hour forecast and compares the output against ERA5 reanalysis. A separate [interactive demo](https://huggingface.co/spaces/hugging-science/ai-weather-models-with-earthmover-data?ref=runtimewire) lets users run available models and inspect how long each stage takes.\n\nThe collaboration puts Abernathey and Hamman back on the problem that led them to start Earthmover in 2022. Abernathey earned a PhD from MIT, spent about a decade teaching and researching ocean circulation at Columbia University, and resigned from a tenured position to build Earthmover. Hamman trained as a civil and environmental engineer at the University of Washington, worked as a scientist at the National Center for Atmospheric Research and co-founded CarbonPlan before joining Abernathey.\n\nBoth founders had worked on Pangeo and Xarray, open-source projects designed for multidimensional scientific data. Their founding argument was straightforward: mainstream cloud databases organize information around tables, while weather and climate systems produce arrays spanning time, altitude, latitude, longitude and dozens of atmospheric variables.\n\n### The GPU is waiting for the weather\n\nOpen weights have removed one barrier to experimenting with AI forecasting. ECMWF publishes AIFS weights, Microsoft distributes Aurora under an MIT license, and Google DeepMind has released WeatherNext 2. The surrounding data pipeline remains heavy.\n\n[Earthmover and Hugging Face estimate in the tutorial](https://huggingface.co/blog/hugging-science/earthmover-hf?ref=runtimewire) that a typical forecast requires about 1GB of initial conditions. Backtesting across a year requires roughly 360GB of disk space before saving any model output. Inference may finish within seconds on a compatible GPU, leaving expensive hardware idle while data is downloaded and prepared.\n\nThe new workflow streams selected ERA5 variables from Earthmover rather than downloading complete archives. Earthmover stores the data using Zarr, which divides multidimensional arrays into independently readable chunks, and [Icechunk](https://icechunk.io/en/stable/overview/?ref=runtimewire), which adds transactions, snapshots and concurrent writes. A user can request specific variables, pressure levels and timestamps without pulling every byte in the underlying dataset.\n\nThe tutorial says its demo can produce a 24-hour forecast in under about 30 seconds end to end. Running Aurora locally requires no GPU, although each six-hour step takes two to three minutes on a CPU versus seconds on a GPU. Users without suitable hardware can send the same script to [Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs-quickstart?ref=runtimewire) and store the result in a Hugging Face bucket or compatible cloud storage.\n\nHardware compatibility still matters. AIFS, for example, depends on FlashAttention and therefore on supported GPU architectures. Hugging Face handles that part of the workflow by providing selectable compute, while Earthmover supplies the model inputs and historical data needed for validation.\n\n### Earthmover finds its distribution layer\n\nAbernathey originally described Earthmover as a modern data stack for scientific arrays. The sharper commercial wedge emerged around weather, where forecasts and observations change constantly and customers cannot treat ingestion as a one-time job.\n\nEarthmover opened its [Data Marketplace](https://earthmover.io/blog/announcing-data-marketplace/?ref=runtimewire) in January 2026 with datasets from ECMWF, NOAA and independent weather providers. [Earthmover says the marketplace now contains more than 60 petabytes](https://huggingface.co/blog/hugging-science/earthmover-hf?ref=runtimewire) of analysis-ready weather, climate and Earth-observation data. Free ERA5 access in the tutorial covers records from 1940 through data that lag real time by several months.\n\nThat focus followed an earlier shift toward frequently updated weather and geospatial workloads. Earthmover had more than 10 paying customers when it [raised a $7.2M seed round](https://techcrunch.com/2025/09/22/earthmover-wants-to-become-the-snowflake-of-weather-and-geospatial-data/?ref=runtimewire) in September 2025, according to Abernathey. Lowercarbon Capital led the round, with Costanoa Ventures and Preston-Werner Ventures participating. Earthmover previously raised a $1.7M pre-seed led by Costanoa Ventures.\n\nThe Hugging Face collaboration gives Earthmover a direct route to researchers and developers already testing open models. A developer can encounter Aurora or AIFS on the Hub, run the demo, and create an Earthmover account when the model needs initial conditions. Hugging Face gets another workload for its Jobs and storage products. Earthmover gets a practical demonstration of why access to model weights does not eliminate the need for specialized data infrastructure.\n\n### A pipeline demo, rather than a forecast benchmark\n\nThe September 8th release concerns operations. Earthmover and Hugging Face introduced no new weather model, and the walkthrough does not establish that Aurora, AIFS or WeatherNext 2 is the most accurate or economical option for a production forecasting service.\n\nThe Aurora example does include a useful validation step. It compares forecast temperatures with ERA5 at matching timestamps and calculates a latitude-weighted root mean square error. That shows developers how to test an output instead of accepting a generated weather map at face value. A production decision would still require longer backtests, operational data and comparisons across models, regions and forecast horizons.\n\nHugging Face says it is working to integrate weather models, including WeatherNext 2, into its Transformers library. If that work reduces model-specific dependencies, data access becomes a larger share of the remaining engineering burden. That is the market Abernathey and Hamman have spent years preparing to serve.\n\nEarthmover's bet is that open weather AI will produce many model runners and model variants, each repeatedly consuming the same enormous archives. The models may take seconds. The durable business sits in keeping the data ready every time they run.", "url": "https://wpnews.pro/news/earthmover-and-hugging-face-tackle-the-slow-part-of-fast-weather-ai", "canonical_source": "https://runtimewire.com/article/earthmover-hugging-face-open-weather-ai-data-pipeline", "published_at": "2026-09-09 04:16:21+00:00", "updated_at": "2026-09-09 05:18:36.412629+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-infrastructure"], "entities": ["Earthmover", "Hugging Face", "Microsoft", "Aurora", "ECMWF ERA5", "Ryan Abernathey", "Joe Hamman", "Icechunk"], "alternates": {"html": "https://wpnews.pro/news/earthmover-and-hugging-face-tackle-the-slow-part-of-fast-weather-ai", "markdown": "https://wpnews.pro/news/earthmover-and-hugging-face-tackle-the-slow-part-of-fast-weather-ai.md", "text": "https://wpnews.pro/news/earthmover-and-hugging-face-tackle-the-slow-part-of-fast-weather-ai.txt", "jsonld": "https://wpnews.pro/news/earthmover-and-hugging-face-tackle-the-slow-part-of-fast-weather-ai.jsonld"}}