Show HN: Icebug-format: immutable, interoperable graph standard

Icebug-format, a new open-source graph standard combining Apache Arrow and Compressed Sparse Rows (CSR) into an immutable, interoperable format, has been released under the MIT license. The format, which supports on-disk storage via Apache Parquet and in-memory via Arrow, includes bindings for Python, TypeScript, and Rust, and has converted the largest graph with 3 billion edges from the LDBC Graphalytics benchmark.

Most graph analytics packages have a mutable graph implementation that uses a heap allocated vector to store the graph. It works for toy graphs. But if you're loading a billion edge graph using G.add edge it's going to take a while. We don't need to invent new standards. Such interoperable, immutable memory standards already exist: Apache Arrow and Compressed Sparse Rows CSR . CSR is widely used in scipy, cugraph and columnar graph databases among others. Both on CPUs and GPUs. icebug-format combines both into a on-disk standard based on Apache Parquet and an in-memory format based on Apache Arrow. Bindings available in many popular languages including python, typescript and rust. The package ships with convenience scripts to convert flat tables such as vertex.parquet and edges.parquet to this format in RAM/disk constrained environments. Sample graphs: https://huggingface.co/datasets/ladybugdb/ldbc-csr/tree/main https://huggingface.co/datasets/ladybugdb/ldbc-csr/tree/main Converted from: https://ldbcouncil.org/benchmarks/graphalytics/datasets/ https://ldbcouncil.org/benchmarks/graphalytics/datasets/ Largest converted graph has 3B edges. License: MIT Quick Start: uv tool install icebug-format icebug-format --source-dir wiki-Talk lbug -i wiki-Talk-csr/schema.cypher Comments URL: https://news.ycombinator.com/item?id=49381909 https://news.ycombinator.com/item?id=49381909 Points: 1 Comments: 0