Stream the Universe from Your Laptop

Researchers converted the Multimodal Universe dataset, containing 80TB+ of astronomical data from over 30 surveys, into the HATS format, enabling crossmatching from a Macbook Air using LSDB and Hugging Face tools, eliminating the need for a cluster.

macOS • xterm-256color • zsh 64 views The Multimodal Universe MMU pools together 80TB++ of data from over 30 astronomical surveys into one place. Crossmatching linking observations of the same object across surveys is its killer feature, but until now it required downloading hefty chunks of data to local disk. We got tired of needing a cluster just to run a crossmatch, so we gathered in the UniverseTBD and Hugging Science Discord servers to fix that. We’ve converted the MMU to the parquet-based HATS format so that you can use the LSDB and Hugging Face ecosystems to crossmatch from a Macbook Air. Write up here: https://huggingface.co/blog/hugging-science/multimodal-universe-hats https://huggingface.co/blog/hugging-science/multimodal-universe-hats