{"slug": "euclidrust-rust-fits-reader-for-euclid-10x-faster-than-astropy", "title": "EuclidRust – Rust FITS reader for Euclid, 10x faster than astropy", "summary": "A Rust-based FITS reader for Python, euclidrust, reports reading speeds 10x faster than astropy, according to the project's own benchmarks. The package, built with PyO3 and installable via pip, was developed as a testbed for ESA Euclid mission data-processing tools and handles 2-D FITS images, including Rice and GZIP tile-compressed files, with exact 64-bit integer handling. The project's code, benchmarks and documentation were written by Anthropic's Claude Code coding agent under the author's direction, and the author describes it as experimental, urging users to run the benchmark themselves.", "body_md": "High-performance FITS I/O for Python, written in Rust and exposed via\n[PyO3](https://pyo3.rs/) as a normal, `pip install`-able Python extension\nmodule. Built as a testbed for data-processing tools for the ESA **Euclid**\nmission, but useful for any FITS-based workflow. The Python package it contains, and the name you `import`, is `euclidrust`.\n\n``` python\nimport euclidrust\n\nstats = euclidrust.fits_stats(\"tile.fits\")\ncutout = euclidrust.extract_cutout(\"tile.fits\", ra_deg=83.82, dec_deg=-5.39, width=50, height=50)\neuclidrust.export_cutout_parquet(\"tile.fits\", 83.82, -5.39, 50, 50, \"cutout.parquet\")\n```\n\n**About this project.** The code, benchmarks and documentation here were\nwritten by [Claude Code](https://claude.com/claude-code) (Anthropic's coding\nagent). The author decided what to build, in what order, how to measure it\nand when to stop, and steered the agent throughout. Treat the project as\nexperimental: read the code, run the benchmark yourself, and judge the\nnumbers rather than the author.\n\n**Contents:**\n[What files it works with](#what-kind-of-files-does-it-work-with) ·\n[How Python and Rust fit together](#how-python-and-rust-fit-together-pyo3) ·\n[Install](#install) ·\n[Use it](#use-it) ·\n[Many images at once](#many-images-at-once) ·\n[NVIDIA GPU](#using-an-nvidia-gpu) ·\n[Tests](#run-the-tests-optional) ·\n[Run the benchmark](#run-the-benchmark-on-your-own-fits-file) ·\n[Troubleshooting](#troubleshooting) ·\n[Motivation](#motivation) ·\n[Benchmark results](#benchmark-results) ·\n[Limitations](#known-limitations-read-before-relying-on-this-for-science) ·\n[Roadmap](#roadmap)\n\n**Short answer: 2-D FITS images, raw or tile-compressed (Rice / GZIP), whether the\nimage sits in the primary HDU or in an extension.**\n\nFITS (Flexible Image Transport System) is the standard file format of\nastronomy. A FITS file is a sequence of blocks called HDUs (\"header/data\nunits\"): each has a text header and a payload, which can be an image or a\ntable. `euclidrust` reads **images**: a rectangular grid of pixels (a sky\nmosaic, a detector frame...), plus the header keywords that say how pixels\nmap to the sky (the WCS).\n\nWhere the image sits depends on the product. A Euclid **MER mosaic** keeps\nits image in the first HDU (the \"primary HDU\"), which is what `euclidrust`\nreads by default. **Calibrated frames** (VIS, NISP) and **raw frames** have\nan *empty* primary HDU and one image extension per detector, with names such\nas `DET11.SCI` or `1-1.E`. For those you pick the HDU with the `hdu=`\nargument (an index or a name), and `list_hdus(path)` shows what a file\ncontains.\n\n**Supported:**\n\n| Requirement | Details | \n|---|---|\n| An image HDU | 2-D ( `NAXIS = 2` ). The primary HDU by default, or any image extension through`hdu=` (0-based index or`EXTNAME` ) | \n| Compression | None, or FITS tile compression with `RICE_1` ,`GZIP_1` or`GZIP_2` (the`.fits.fz` format written by`fpack` and by astropy's`CompImageHDU` ), including quantized floating-point images (`NO_DITHER` ,`SUBTRACTIVE_DITHER_1` ,`SUBTRACTIVE_DITHER_2` ). A compressed image always lives in an extension, so pass`hdu=1` (or its name) | \n| Pixel type ( `BITPIX` ) | `8` ,`16` ,`32` ,`64` (integers),`-32` ,`-64` (floating point).`BZERO` /`BSCALE` are applied, so unsigned 16-bit raw data (`BZERO = 32768` ) works. 64-bit integers, signed or unsigned (FITS stores unsigned 64-bit as signed with`BZERO = 2^63` ), are handled**exactly** , in raw and in gzip-compressed images (cfitsio compresses 64-bit integers with gzip only), including a whole-number`BSCALE` /`BZERO` : the exact values are in`min_int` /`max_int` (`fits_stats` ) and`data_int` (`extract_cutout` ), as Python ints, and the Parquet`value` column is`Int64` /`UInt64` ; the ordinary float fields hold the nearest`float64` | \n| WCS, only needed for cutouts | `CTYPE1/2` =`RA---xxx` /`DEC--xxx` with`xxx` one of`TAN` ,`SIN` ,`ARC` ,`STG` ,`ZEA` ,`ZPN` (the zenithal projections),`TAN-SIP` (tangent plus the SIP polynomial distortion) or`TPV` (tangent plus the`PV1_m` /`PV2_m` polynomials used by the Euclid VIS pipeline), together with`CRVAL1/2` ,`CRPIX1/2` and a linear matrix written as`CD` (`CD1_1` ,`CD1_2` ,`CD2_1` ,`CD2_2` ), as`CDELT1/2` with an optional`PC` matrix, or as`CDELT1/2` with the legacy`CROTA2` .`LONPOLE` is honoured.`fits_stats` needs no WCS | \n\n**Not supported.** Anything else is rejected with a `ValueError` that says\nwhy, rather than being read wrongly (these cases are covered by\n`tests/test_input_validation.py`):\n\n| Kind of file | Error you get | \n|---|---|\n| Tile compression with another algorithm ( `HCOMPRESS_1` ,`PLIO_1` ) | `HDU 1 (SCI): tile-compressed image uses ZCMPTYPE = HCOMPRESS_1 which is not supported yet (supported: RICE_1, GZIP_1, GZIP_2)` | \n| An extension that is a table (catalogue, spectrum...) | `HDU 1 is a BINTABLE table, not an image` | \n| 3-D cubes ( `NAXIS = 3` ) | `NAXIS = 3 is not supported ...: only 2-D images are handled` | \n| Not a FITS file | `truncated file: no END card found` | \n| A projection that is not implemented (cylindrical or all-sky ones such as `CAR` ,`AIT` ,`MOL` ,`HPX` ...), a mismatched`CTYPE` pair, or a missing`CTYPE` | Cutouts raise `unsupported WCS projection: found CTYPE1/CTYPE2 = ...; supported are RA---TAN/DEC--TAN (also with -SIP distortion), RA---TPV/DEC--TPV and the zenithal projections SIN, ARC, STG, ZEA and ZPN ...` (`fits_stats` still works) | \n| Distortion stored in lookup tables (HST `CPDIS` /`D2IMDIS` keywords) | Cutouts raise `this header declares distortion corrections stored in lookup tables ... which euclidrust does not apply` (`fits_stats` still works) | \n| No WCS at all (no `CRVAL` /`CRPIX` / matrix keywords) | Cutouts raise `this file has no WCS header (CRVAL, CRPIX and either a CD matrix or CDELT/PC)` (`fits_stats` still works) | \n\n**Null pixels.** NaN pixels of floating-point images are left out of `min`, `max` and `mean` and counted in `n_nan` (infinities are values and stay in). The `BLANK` null value of integer images is **not** honoured: those pixels are counted as ordinary values.\n\nIf you forget `hdu=` on a file whose primary HDU is empty, the error lists\nthe image HDUs it found and tells you what to pass, for example\n`[1] DET11.SCI (2040x2040, BITPIX -32) ... Pass hdu=`.\n\n**What this means for Euclid Q1 data.** I read the headers of a handful of\nfiles in each product family of the public Q1 release (this is a sample, not\na full survey of the archive) and ran the code on real files from four of\nthem, comparing with astropy:\n\n| Product | Where the image is | Status | \n|---|---|---|\n| MER mosaics ( `EUC_MER_BGSUB-MOSAIC-...` , uncompressed) | primary HDU | **Works** (stats and cutouts). All benchmark numbers come from the NISP Y-band tile. The other bands and the`RMS` /`FLAG` mosaics have the same layout but were not run | \n| NISP calibrated frames ( `EUC_NIR_W-CAL-IMAGE-BKG...` ) | extensions `DETnn.SCI` ,`.RMS` ,`.DQ` (`TAN` WCS) | **Works** with`hdu=\"DET11.SCI\"` etc. Checked on one detector: stats and a cutout identical to astropy | \n| VIS calibrated frames ( `EUC_VIS_SWL-BKG...` ) | extensions `1-1.E` ... (`TPV` WCS) | **Works** with`hdu=\"1-1.E\"` etc. Checked on one detector: stats identical to astropy; over 300 random sky positions the cutout center agrees with astropy/wcslib to 3 × 10^-8 pixel, and all cutout pixels are identical | \n| RAW frames, NISP ( `EUC_LE1_NISP...` ) | extensions `DETnn.SCI` (unsigned 16-bit via`BZERO` ) | **Stats work** (checked on one detector). Cutouts are refused: the WCS is in detector-mosaic coordinates (`Y_MOSA` ), not sky coordinates | \n| RAW frames, VIS ( `EUC_LE1_VIS...` ) | extensions `1-1.E` ... | Same layout as the other frames, **not run** | \n| MER segmentation maps ( `EUC_MER_FINAL-SEGMAP...` ) | primary HDU, `BITPIX = 64` | **Stats work** (checked on the real 2.9 GB file: min and max identical to astropy, exact in`min_int` /`max_int` , mean agreeing to ~13 digits). No`CD` matrix in the header, so no cutouts | \n| Catalogues, PSF files, other tables | table extensions | Not images, out of scope | \n\nNone of the Q1 image files I looked at were tile-compressed, so there was no compressed Euclid file to test on. Compressed-image support was validated on copies of real Euclid data that I compressed with astropy/cfitsio (see [Compressed images](#compressed-images)).\n\nTo check a file yourself before running anything on it, list its HDUs:\n\n``` python\nimport euclidrust\nfor hdu in euclidrust.list_hdus(\"file.fits\"):\n print(hdu)\n# \n# \n```\n\nor look at the header keywords with astropy:\n\n``` python\nfrom astropy.io import fits\n\nh = fits.getheader(\"file.fits\", 1) # 0 = primary HDU, 1 = first extension...\nprint(\"NAXIS\", h[\"NAXIS\"], \"| BITPIX\", h[\"BITPIX\"])\nprint(\"CTYPE\", h.get(\"CTYPE1\"), h.get(\"CTYPE2\"))\nprint(\"linear matrix keywords:\", [k for k in (\"CD1_1\", \"CDELT1\", \"PC1_1\", \"CROTA2\") if k in h])\n```\n\nYou want `NAXIS 2`, a `BITPIX` from the table above and, for cutouts, a `CTYPE`\npair from the projection list above and some linear-matrix keywords (`CD1_1`,\nor `CDELT1`).\n\nIf you are an astronomer who knows Python but has never touched Rust, this\nis the one thing to understand: **you never write or run Rust yourself.**\nYou install the package and `import` it like `numpy` or `astropy`.\n\nUnder the hood there are three pieces:\n\n| Piece | Where | What it does | \n|---|---|---|\n| The Rust code | `euclidrust/src/lib.rs` | Does the actual work: memory-maps the FITS file, runs the parallel computation, converts sky coordinates to pixels | \n| **PyO3** | a dependency in `euclidrust/Cargo.toml` | A Rust library that turns Rust functions and structs into Python functions and objects, and converts arguments between the two (for example, `hdu=` accepts a Python`int` or`str` , which PyO3 maps to a Rust`enum` ) | \n| **maturin** | `build-backend` in`euclidrust/pyproject.toml` | Compiles the Rust into a shared library and installs it in your Python environment ( `pip` calls it for you) | \n\nPyO3 shows up in `lib.rs` as a handful of annotations. Here is the whole\nmapping, taken from the real code:\n\n``` php\n#[derive(FromPyObject)] // -> accepts a Python int OR str\nenum HduSelector { Index(usize), Name(String) }\n\n#[pyfunction] // -> a Python function\nfn fits_stats(path: &str, hdu: Option) -> PyResult { /* ... */ }\n\n#[pyclass] // -> a Python class\nstruct FitsStats {\n #[pyo3(get)] min: f64, // -> readable attribute: stats.min\n #[pyo3(get)] max: f64,\n #[pyo3(get)] mean: f64,\n /* ... */\n}\n\n#[pymodule] // -> `import euclidrust`\nfn euclidrust(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {\n m.add_function(wrap_pyfunction!(fits_stats, m)?)?;\n /* ... */\n Ok(())\n}\n```\n\nand on the Python side it is just:\n\n``` python\nimport euclidrust # loads the compiled Rust library\ns = euclidrust.fits_stats(\"tile.fits\") # calls the Rust function\nprint(s.min, s.max, s.mean) # reads the fields of the Rust struct\n```\n\nSo the build chain is: **Rust source -> (maturin, run by pip) -> compiled\nlibrary such as `euclidrust.cpython-313-x86_64-linux-gnu.so` (`.pyd` on\nWindows) -> installed into your virtual environment -> `import euclidrust`.**\nThe scripts in `scripts/` contain no Rust and no PyO3 code; they only\n`import euclidrust`. Errors raised in Rust arrive in Python as ordinary\n`ValueError` exceptions.\n\nYou need three things on your machine:\n\n- **Python 3.8 or newer** (developed and tested with 3.13).\n- **A Rust toolchain, version 1.88 or newer** (the Arrow / Parquet dependencies require it; developed with 1.100). Install it with[rustup](https://rustup.rs/) :\n - Linux / macOS: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`\n - Windows: download `rustup-init.exe` from[https://rustup.rs/](https://rustup.rs/) . It will\nask you to install the Visual Studio C++ Build Tools; accept.\n - Then **open a new terminal** and check with`rustc --version` .\n- Linux / macOS: \n- **A C linker** , which is normally already there (`build-essential` on\nDebian/Ubuntu, Xcode command-line tools on macOS, the Build Tools on\nWindows).\n\nEverything else (maturin, PyO3 and the other Rust libraries) is downloaded automatically during the build, so the first install needs internet access.\n\nTested on Debian 13 (x86_64) with Python 3.13 and Rust 1.100, on an Intel Core i7-8559U (4 cores / 8 threads) with 31 GiB of RAM. macOS and Windows have not been tested; if you try them, please open an issue with what happened.\n\n```\ngit clone https://github.com/MaxIKweeger/EuclidRust.git\ncd EuclidRust\n\npython3 -m venv .venv # a private Python environment\nsource .venv/bin/activate # Windows PowerShell: .venv\\Scripts\\Activate.ps1\n\npip install ./euclidrust # compiles the Rust code (a few minutes); GPU support is included\n```\n\nThe first compile takes a few minutes because the release profile enables full link-time optimisation; later rebuilds are faster. When it finishes, check that it worked:\n\n``` python\npython -c \"import euclidrust; print('OK, threads available:', euclidrust.detected_cpus())\"\n```\n\nIf that prints `OK, threads available: `, you are done.\n\n*Developing the Rust code?* Use `pip install maturin`, then run\n`maturin develop --release` inside `euclidrust/`. That installs the module\nin editable mode and rebuilds it after you change `src/lib.rs`.\n\nYou need a FITS file of the kind described in\n[What kind of files does it work with?](#what-kind-of-files-does-it-work-with).\nEuclid MER mosaics are of this kind; see\n[the benchmark section](#run-the-benchmark-on-your-own-fits-file) for where\nto get one.\n\n``` python\nimport numpy as np\nimport euclidrust\n\npath = \"tile.fits\"\n\n# 0. What is in the file? (reads the headers only, instant)\nfor hdu in euclidrust.list_hdus(path):\n print(hdu)\n\n# 1. Global statistics over the whole image\ns = euclidrust.fits_stats(path)\nprint(s.min, s.max, s.mean) # physical values (BZERO/BSCALE applied)\nprint(s.naxis1, s.naxis2, s.bitpix) # image shape and data type\nprint(s.threads_used, s.elapsed_ms) # how it ran\n\n# 2. A cutout around a sky position (RA/Dec in degrees)\nc = euclidrust.extract_cutout(path, ra_deg=273.0716597, dec_deg=68.0,\n width=50, height=50)\nimage = np.array(c.data).reshape(c.height, c.width) # row-major -> 2-D array\nprint(c.pixel_x_center, c.pixel_y_center) # FITS 1-indexed pixel position\n\n# 3. The same cutout written to a Parquet file\nr = euclidrust.export_cutout_parquet(path, 273.0716597, 68.0, 50, 50, \"cutout.parquet\")\nprint(r.rows, r.out_path)\n\nimport pandas as pd # pip install pandas pyarrow\nprint(pd.read_parquet(\"cutout.parquet\").head()) # columns: x, y, value\n```\n\n**Files whose images are in extensions** (Euclid calibrated and raw frames)\nneed the `hdu` argument, a name or a 0-based index, on all three functions:\n\n```\nframe = \"EUC_NIR_W-CAL-IMAGE-BKG_H-2681-0_20240930T175032.222263Z.fits\"\n\ns = euclidrust.fits_stats(frame, hdu=\"DET11.SCI\") # by EXTNAME\ns = euclidrust.fits_stats(frame, hdu=1) # by index (same HDU)\nc = euclidrust.extract_cutout(frame, 266.09, 67.32, 50, 50, hdu=\"DET11.SCI\")\n```\n\n(names are matched case-insensitively; if several HDUs share a name, the first one is used).\n\n**Compressed files** (`.fits.fz`, as written by `fpack`) work the same way: the image is in extension 1, so pass `hdu=1`.\n\n```\nstats = euclidrust.fits_stats(\"image.fits.fz\", hdu=1)\ncutout = euclidrust.extract_cutout(\"image.fits.fz\", 150.0, 2.2, 50, 50, hdu=1) # decodes only the tiles it needs\n```\n\nLossy (quantized) floating-point images decode to exactly the values astropy/cfitsio would give, which are not the original pre-compression values; that loss belongs to the format, not to `euclidrust`.\n\nReference:\n\n| Function | Returns | Fields | \n|---|---|---|\n| `fits_stats(path, hdu=None)` | `FitsStats` | `min` ,`max` ,`mean` ,`naxis1` ,`naxis2` ,`bitpix` ,`threads_used` ,`elapsed_ms` , and`min_int` ,`max_int` (exact integers for 64-bit integer images, otherwise`None` ),`n_nan` (number of NaN pixels, which`min` ,`max` and`mean` leave out) | \n| `extract_cutout(path, ra_deg, dec_deg, width, height, hdu=None)` | `Cutout` | `data` (flat list of floats, row-major),`data_int` (the same pixels as exact Python ints for 64-bit integer images, otherwise`None` ),`width` ,`height` ,`pixel_x_center` ,`pixel_y_center` ,`elapsed_ms` | \n| `export_cutout_parquet(path, ra_deg, dec_deg, width, height, out_path, hdu=None)` | `ParquetExport` | `out_path` ,`rows` ,`elapsed_ms` | \n| `fits_stats_many(paths, hdu=None, skip_errors=False)` | list of `FitsStats` (or`None` ) | one per file, in the same order; see [Many images at once](#many-images-at-once) | \n| `fits_stats_all_hdus(path, name_contains=None)` | list of `HduStats` | per image HDU: `index` ,`name` ,`naxis1` ,`naxis2` ,`bitpix` ,`min` ,`max` ,`mean` ,`min_int` ,`max_int` ,`n_nan` ,`elapsed_ms` | \n| `extract_cutouts(path, ra_deg, dec_deg, width, height, hdu=None, skip_errors=False)` | list of `Cutout` (or`None` ) | one per position, same fields as `extract_cutout` | \n| `list_hdus(path)` | list of `HduInfo` | per HDU: `index` ,`kind` (`PRIMARY` ,`IMAGE` ,`BINTABLE` ...),`name` ,`bitpix` ,`shape` (NAXIS1, NAXIS2...),`readable` ,`note` (why it is not readable) | \n| `detected_cpus()` | `int` | number of logical cores found on this machine | \n\nNotes: `hdu=None` (the default) reads the primary HDU, exactly like before.\n`Cutout.data` is a plain Python list, so convert it with `np.array(...)` as\nabove (fine for cutouts; it is not meant for multi-megapixel regions). The\nParquet file has one row per pixel with columns `x`, `y` (absolute pixel\ncoordinates in the source image) and `value` (an exact `Int64` / `UInt64` column for 64-bit integer images, `Float64` otherwise). All errors (file not found,\nwrong kind of HDU, cutout outside the image, unsupported WCS...) are raised\nas `ValueError`.\n\nWhen you have many images, three functions do the looping in Rust, in parallel on all the cores, with the GIL released:\n\n```\n# 1. The same statistics over many files (one result per file, same order)\nresults = euclidrust.fits_stats_many(paths)\nresults = euclidrust.fits_stats_many(paths, hdu=1) # hdu applies to every file\nresults = euclidrust.fits_stats_many(paths, skip_errors=True) # None for a file that cannot be read\n\n# 2. Every image HDU of one frame (all the detectors of a Euclid frame)\nfor r in euclidrust.fits_stats_all_hdus(\"frame.fits\", name_contains=\"SCI\"):\n print(r.index, r.name, r.min, r.max, r.mean)\n\n# 3. Many cutouts from one file, e.g. a catalogue of objects\n# (ras, decs: two equal-length lists or numpy arrays, in degrees)\ncutouts = euclidrust.extract_cutouts(\"tile.fits\", ras, decs, 50, 50)\nimages = [np.array(c.data).reshape(50, 50) for c in cutouts]\n```\n\nDetails:\n\n- Results come back in the order of the inputs. `fits_stats_all_hdus` returns the HDUs in file order, skipping the ones that are not readable\nimages (the empty primary HDU, tables...);`name_contains` keeps only the\nHDUs whose`EXTNAME` contains that text, case-insensitively (for example`\"SCI\"` to leave out the`RMS` and`DQ` images).\n- By default the first problem raises a `ValueError` that names the file (or\nthe position, for cutouts). With`skip_errors=True` , the failed items are`None` instead, so you can process a whole list and look at the failures\nafterwards.\n- `extract_cutouts` opens the file and reads its header once, then\nextracts the positions in parallel.\n- **The GIL is released** during all the calls (single or batch), so other\nPython threads keep running while Rust works. For example, a counter in the\nmain thread reached 731,463 while one 155 ms`fits_stats` call on the 1.47\nGB mosaic ran in another thread.\n\nWhat this buys you, measured on real Euclid data (median of 3 runs, warm cache; the 40-file case uses 40 copies of one detector, so everything is in the OS cache):\n\n| Task | `astropy` | `euclidrust` : a loop of single calls | `euclidrust` : one batch call | \n|---|---|---|---|\n| Stats of the 48 image HDUs of a NISP frame (799 MB) | 484 ms | 102 ms | 72 ms | \n| Stats of the 144 image HDUs of a VIS frame (2.4 GB) | 1309 ms | 693 ms | 299 ms | \n| Stats of 40 files (17 MB each, uncompressed) | 274 ms | 68 ms | 55 ms | \n| Stats of 40 Rice-compressed files | 3317 ms | 583 ms | 572 ms | \n| 1000 cutouts 50×50 from the 1.47 GB mosaic | 108 ms (WCS built once) / 2835 ms (WCS and file reopened each time) | 216 ms | 39 ms | \n\nEvery result is identical to astropy's and to the single calls'. How to read the table: the batch call helps most where each item is small (many cutouts, frames with many small HDUs). On big or compressed files a single call already uses all the cores, so a batch call is barely faster than a loop; the large gain over astropy there comes from reading and decoding in parallel, not from batching.\n\n`euclidrust` is **one binary for every machine**. When a computation could use an NVIDIA\nGPU, it looks for one; if there is none (or the driver is missing, or the card is too old) it\nuses the CPU and gives the same answers. There is no separate \"GPU build\" and nothing to\ninstall besides the NVIDIA driver you already have to display graphics: **the CUDA toolkit\nis not needed**, because the GPU programs are built into the library.\n\n```\neuclidrust.gpu_info()\n# \n# or: \n\neuclidrust.fits_stats(path) # device=\"auto\" is the default: the library chooses\neuclidrust.fits_stats(path, device=\"cpu\") # always the CPU\neuclidrust.fits_stats(path, device=\"gpu\") # always the GPU; a ValueError says why if it cannot\ns = euclidrust.fits_stats(path); print(s.device) # \"cpu\" or the name of the GPU that computed it\n\nimg = euclidrust.GpuImage(path) # copy an image to the card once...\nimg.stats(); img.stats() # ...then compute on it with no copy at all\n```\n\nThe environment variable `EUCLIDRUST_DEVICE` (`cpu`, `gpu` or `auto`) sets the default for a\nwhole program, for example `EUCLIDRUST_DEVICE=cpu` to keep everything on the CPU.\n\n| Image | `auto` uses | Why | \n|---|---|---|\n| Rice-compressed ( `RICE_1` ), at least 256 tiles and 64 MB | **the GPU** | Only the compressed data (five times smaller) crosses the PCIe bus, and decoding thousands of tiles in parallel is what a GPU does well: one GPU thread decodes one tile and accumulates its statistics, without ever storing the decompressed image | \n| Raw (uncompressed) | the CPU | The 1.47 GB tile has to cross the PCIe bus and the host memory twice, which costs more than the whole CPU computation | \n| Anything else (gzip, lossless floats, 64-bit integers, small files) | the CPU | Not implemented on the GPU, or too small to repay starting CUDA (about 0.1 s, once per process) | \n\nMeasured on the Euclid Q1 tile used throughout this README (milliseconds, median, lower is\nbetter; the full table, with the machines, is in\n[`benchmarks/COMPARISON.md`](https://github.com/MaxIKweeger/EuclidRust/blob/main/benchmarks/COMPARISON.md)):\n\n| | astropy CPU i7 | astropy CPU i9 | EuclidRust CPU i7 NUC | EuclidRust CPU i9 | Euclid GPU RTX 4070 Ti | \n|---|---|---|---|---|---|\n| Raw tile 1.47 GB: min / max / mean | 478 | 1,187 | 132 | 119 | 194 | \n| same, image already on the GPU | | | | | **10.1** | \n| 50×50 cutout (always on the CPU) | 2.70 | 3.33 | 0.22 | 0.23 | (CPU) | \n| Rice-compressed tile 0.31 GB: min / max / mean | 3,603 | 3,849 | 1,211 | 384 | **67** | \n| 50×50 cutout (always on the CPU) | 17.1 | 19.3 | 3.55 | 2.51 | (CPU) | \n\nThree results look odd at first sight. Each one was investigated and measured, and none is a measurement error: the same file (1,474,565,760 bytes), the same Python and astropy versions (numpy 2.5.4, astropy 8.0.2), the data already in the operating system's cache, the median of several runs, and the same minimum, maximum and mean to the last digit on every machine.\n\n1. **astropy is slower on the i9 (1,187 ms) than on the i7 (478 ms), although the i9 is the\nmore powerful processor.** astropy and numpy compute on**one core** , so the 10 cores of the\ni9 do not help them. FITS stores numbers big-endian (`>f4` ), and numpy has to swap the bytes of\nevery value before it can compute. I measured numpy's`min()` on 64 MB of float32: on ordinary\n(little-endian) data the two machines are comparable (20.3 GB/s on the PC, 23.8 GB/s on the\nNUC), but on big-endian data the PC with Windows reaches 6.9 GB/s and the NUC with Linux\n15.0 GB/s. So that column mostly measures numpy's big-endian path on Windows, not the\nprocessor. Reading the file adds about 0.5 s on the PC. I did not investigate why numpy is\nslower there (compiler or build options are the likely suspects).\n2. **EuclidRust gains little on the i9 (119 ms) over the i7 (132 ms) for the raw tile, despite\nmore than twice the cores and a third more clock speed.** The work is limited by how fast\nthe memory delivers 1.47 GB, not by the processor. The PC's memory runs at 2933 MT/s\n(measured at about 37 GB/s with many threads, after an earlier XMP misconfiguration was\nfixed — it ran at 2133 MT/s, ~28 GB/s, before), the NUC's at 2400 MT/s. Adding cores cannot\nhelp a memory-bound computation much. On the Rice-compressed tile, decoding each tile leaves\nmore work per byte fetched, so the i9 pulls further ahead of the i7 there (384 ms vs 1,211 ms).\n3. **The GPU is slower than the CPU on the raw image (194 ms against 119 ms).** The image has to\ncross the PCIe 3.0 bus (12.9 GB/s measured, so at least 114 ms for 1.47 GB) after a first\ncopy into page-locked memory, whereas the CPU reads it once. The computation on the card\ntakes 10 ms: the time is spent moving the data. The GPU wins where the transfer is small or\npaid once: the Rice-compressed file (0.31 GB, 67 ms) and an image already in GPU memory (10 ms).\n\nThe speed-ups over astropy are real for the user, who runs astropy as it is, but they combine two things: using every core, and not byte-swapping. Compare EuclidRust with the other columns of the same machine to see the first one alone.\n\nWhat to take from it:\n\n- **Compressed images: the GPU is the clear winner** , 5.7 times faster than the same code on the\n20-thread CPU, and 57 times faster than astropy, with**identical results** (minimum,\nmaximum and mean agree to the last digit, the GPU running the same arithmetic as the CPU\ncode).\n- **Raw images, one computation: the GPU is slower than the CPU** here (194 ms against 119 ms).\nThis PC's CPU only offers PCIe 3.0 (12.9 GB/s measured, so 114 ms at best just to move the\ndata) and the file has to be read into page-locked memory first. Once the image is on the\ncard, the computation takes**10 ms** , 12 times faster than the CPU: that is where`GpuImage` pays, when you compute several things on the same image.\n- Cutouts, batches and everything else run on the CPU.\n- Astropy has no GPU support at all (it is single-threaded C code); a Python user who wants a GPU today has to read the data with astropy and send it to another library.\n\n- **Hardware and driver:** an NVIDIA GPU of compute capability 7.5 or higher (RTX 20 series and\nnewer) and a driver recent enough for the built-in kernels (PTX version 9.3, from CUDA\n13.3). With an older driver, the library recompiles the kernels itself if the CUDA toolkit\nis installed (its compiler, NVRTC), and otherwise falls back to the CPU.\n- **Only tested on one GPU:** a GeForce RTX 4070 Ti under Windows 11 (driver 610.88). Linux with an\nNVIDIA card runs the same code but has not been tried. The no-GPU case was tested on Windows (a\nsimulated machine without CUDA) and on the NUC (Debian, no NVIDIA card).\n- **Not on the GPU:** gzip-compressed images, lossless floating-point tiles, 64-bit integers\n(they need exact integer arithmetic), and everything about WCS and cutouts. With`device=\"gpu\"` these raise an error that says so; with`auto` they use the CPU.\n- **One GPU, one call at a time:** device 0 is used (`CUDA_VISIBLE_DEVICES` chooses which card\nthat is), and GPU calls from several threads wait for each other.\n- **Memory:** the library keeps 128 MiB of page-locked RAM for the transfers once the GPU has\nbeen used (two staging buffers of 64 MiB: I measured 16 MiB, 64 MiB and 256 MiB, and 64 is\nthe fastest, because the limit is memory bandwidth, not quantity), plus the image itself in\nthe card's memory for`GpuImage` .\n- **First GPU call of a process:** starting CUDA costs about 0.1 to 0.2 s. For a script that\nmakes a single call on a small file, that is more than the CPU takes:`auto` accounts for it\nwith the 64 MB / 256 tiles thresholds above.\n\nTo reproduce the table: run\n`python scripts/benchmark_vs_astropy_real.py FILE --gpu --label \"i9 (PC)\" --workload \"raw tile\" --json out.json`\non each machine, then\n`python scripts/make_comparison_table.py *.json > benchmarks/COMPARISON.md`.\nAfter editing the GPU programs (`euclidrust/src/kernels.cu`), regenerate the built-in\nPTX with `python scripts/build_ptx.py` (this one needs the CUDA toolkit) and rebuild.\n\n`tests/test_input_validation.py` builds small FITS files with astropy and\nchecks that supported files work (primary HDU, extensions, `BITPIX` 16 and\n64 including exact unsigned 64-bit values and gzip-compressed 64-bit images, `BZERO`/` BSCALE`), that unsupported ones are rejected with a clear error,\nthat every supported projection, matrix form, the TPV distortion and the SIP\ndistortion (including real SIP headers from astropy's own test data) agree with\nastropy/wcslib, and that tile-compressed images (`RICE_1`, `GZIP_1`,\n`GZIP_2`, quantization and dithering) decode exactly like astropy/cfitsio, and\nthat the batch functions return what the single calls return. It needs no download. With `euclidrust`\ninstalled and `pip install -r scripts/requirements.txt` done:\n\n```\npython tests/test_input_validation.py # prints \"all checks passed\"\n```\n\n(`pytest tests/` works too.)\n\nThe script `scripts/benchmark_vs_astropy_real.py` times `euclidrust`\nagainst `astropy` on a file of your choice, and checks that both give the\nsame answer.\n\n**1. Get a FITS file.** Use any file that matches the\n[requirements above](#what-kind-of-files-does-it-work-with). To reproduce the\nnumbers in this README, use the Euclid Q1 tile they were measured on, from\nthe public IRSA mirror (anonymous S3 bucket `nasa-irsa-euclid-q1`, no account\nneeded, 1.47 GB):\n\nDownload it with your browser, `curl` or `wget`, and put it anywhere. To try\na **calibrated frame** instead (images in extensions), these are public too:\n\n- NISP, 799 MB (`TAN` WCS):[https://nasa-irsa-euclid-q1.s3.amazonaws.com/q1/NIR/2681/EUC_NIR_W-CAL-IMAGE-BKG_H-2681-0_20240930T175032.222263Z.fits](https://nasa-irsa-euclid-q1.s3.amazonaws.com/q1/NIR/2681/EUC_NIR_W-CAL-IMAGE-BKG_H-2681-0_20240930T175032.222263Z.fits)\n- VIS, 2.4 GB (`TPV` WCS):[https://nasa-irsa-euclid-q1.s3.amazonaws.com/q1/VIS/2681/EUC_VIS_SWL-BKG-002681-00-1-0000000__20241017T042839.727792Z.fits](https://nasa-irsa-euclid-q1.s3.amazonaws.com/q1/VIS/2681/EUC_VIS_SWL-BKG-002681-00-1-0000000__20241017T042839.727792Z.fits)\n\n**2. Install the benchmark's two dependencies** (inside the same virtual\nenvironment you installed `euclidrust` in):\n\n```\npip install -r scripts/requirements.txt # numpy + astropy\n```\n\n**3. Run it**, giving the path to the file:\n\n```\npython scripts/benchmark_vs_astropy_real.py /path/to/tile.fits\npython scripts/benchmark_vs_astropy_real.py /path/to/frame.fits --hdu DET11.SCI # image in an extension\n```\n\nOptional flags: `--hdu` (the HDU holding the image, an `EXTNAME` or a 0-based\nindex; default: the primary HDU), `--ra` / `--dec` (cutout center in degrees;\ndefault: the sky position of the image center, taken from the file's own\nWCS), `--size` (cutout width and height in pixels, default 50) and `--runs`\n(repetitions per measurement, default 5). If the HDU is tile-compressed, astropy has to decompress the whole image for the global stats and reads the cutout through `.section`; the script handles both. If the file has no sky WCS (raw\ndetector frames, for example), the cutout comparison is skipped with an\nexplanation and the global-stats comparison still runs. If the file uses\n`BZERO`/` BSCALE`, astropy cannot memory-map it and reads it into RAM; the\nscript says so, and the comparison is then less favourable to astropy than\nusual for reasons unrelated to `euclidrust`.\n\n**4. Read the output.** It prints, for each of the two operations, the median\ntime of both tools, the speedup, and a correctness check:\n\n```\n--- 1. Global stats (min/max/mean over the whole file) ---\nastropy (memmap, numpy) : 476.49 ms\neuclidrust (Rust/rayon) : 130.05 ms\nSpeedup : ×3.66 (72.7% less time)\nResult check: astropy min/max/mean = (-111.43283081054688, 15220.939453125, 1.0181564092636108)\n rust min/max/mean = (-111.43283081054688, 15220.939453125, 1.0181568676826722)\n\n--- 2. 50x50 cutout extraction by RA/Dec (273.071586, 68.000028) ---\nastropy (WCS + memmap) : 2.670 ms\neuclidrust (Rust/mmap) : 0.224 ms\nSpeedup : ×11.91 (91.6% less time)\n...\n all 2500 pixels identical: True\n```\n\n(That is a real run on the development machine; your times will differ with\nyour hardware, and a re-run on the same machine moves by a few percent.)\n`min`/` max` and every cutout pixel should match exactly; `mean` differs in\nthe 7th significant figure because the two tools sum floats in a different\norder. Run it twice: the first run may be slower while the operating system\nloads the file into its cache.\n\n| Symptom | Cause and fix | \n|---|---|\n| `rustc: command not found` or pip fails with \"Cargo, the Rust package manager, is not installed\" | Rust is not installed, or the terminal was opened before installing it. Install via [rustup](https://rustup.rs/) and open a**new** terminal. | \n| Build error mentioning `edition2024` or`rust-version` | Your Rust is older than 1.88. Run `rustup update` . | \n| `linker 'cc' not found` (Linux) /`link.exe not found` (Windows) | Install a C toolchain: `sudo apt install build-essential` , or the Visual Studio C++ Build Tools. | \n| `ModuleNotFoundError: No module named 'euclidrust'` | You are in a different Python than the one you installed into. Activate the virtual environment ( `source .venv/bin/activate` ) and retry; check with`pip show euclidrust` . | \n| `ValueError: the primary HDU holds no image (NAXIS = 0); it contains: ...` | The image is in an extension (Euclid calibrated/raw frames). Pass `hdu=` with the name or index shown in the message. | \n| `ValueError: no HDU named '...'` /`no HDU with index ...` | Wrong `hdu` value. The message lists the available HDUs;`euclidrust.list_hdus(path)` shows them all. | \n| `ValueError: ... tile-compressed image uses ZCMPTYPE = ...` or`... is a BINTABLE table, not an image` | That HDU uses a compression algorithm that is not implemented (only `RICE_1` ,`GZIP_1` ,`GZIP_2` are), or it is a table. | \n| `ValueError: NAXIS = 3 is not supported` or`no END card found` | A 3-D cube, or a file that is not FITS. Only 2-D images are handled. | \n| `ValueError: unsupported WCS projection ...` | The cutout functions implement the zenithal projections ( `TAN` ,`SIN` ,`ARC` ,`STG` ,`ZEA` ,`ZPN` ),`TAN-SIP` and`TPV` . Cylindrical and all-sky projections, and WCS in other coordinate systems (galactic, ecliptic), are not supported yet. | \n| `ValueError: this header declares distortion corrections stored in lookup tables ...` | The header has HST-style `CPDIS` /`D2IMDIS` keywords. Their correction tables are not applied, so cutouts are refused.`fits_stats` still works. | \n| `ValueError: this file has no WCS header` | The cutout functions need `CRVAL1/2` ,`CRPIX1/2` and a linear matrix (`CD` , or`CDELT` with an optional`PC` ) in the chosen HDU.`fits_stats` still works without them. | \n| `ValueError: cutout outside the image` | The RA/Dec you asked for falls outside the image. Check the coordinates, and for frames make sure `hdu=` is the detector you mean (each detector covers a different patch of sky). | \n| The benchmark exits asking for `numpy` /`astropy` | Run `pip install -r scripts/requirements.txt` in the same environment. | \n\nEuclid will deliver petabytes of high-resolution imaging and catalogue\ndata over its six-year survey. The standard astronomy stack for this kind\nof workload — `astropy.io.fits` on top of the C library `cfitsio` — has\ntwo structural limitations that become painful at this scale, especially\nfor independent researchers or small labs without access to the official\nconsortium's HPC clusters:\n\n- **The GIL.** Even on a machine with 8 logical cores, a single Python\nthread holds the Global Interpreter Lock while`cfitsio` does its work,\nso \"read 1000 FITS files in parallel\" rarely scales the way you'd hope\nfrom thread count alone.\n- **Unnecessary copies.** Reading a single cutout out of a multi-gigabyte\ntile often pulls far more data into memory than the cutout itself\nrequires.\n\n`euclidrust` addresses both with a small Rust core: `mmap`-based\nzero-copy access to uncompressed FITS data, and a `rayon` thread pool that\nauto-sizes itself to the host's logical core count (the GIL is released while it works) — there is no hardcoded\nthread count, so the same binary scales itself from a laptop to a\nmany-core workstation.\n\nThree single-image operations (plus `list_hdus` to see what a file contains, and batch versions of them, see [Many images at once](#many-images-at-once)), each independently benchmarked against `astropy` on\nidentical input and compared for numerical correctness, not just speed:\n\n| Function | What it computes | Notes | \n|---|---|---|\n| `fits_stats(path, hdu=None)` | Global `min` /`max` /`mean` over one image HDU (the primary one by default) | `mmap` +`rayon` , chunked reduction tuned for this workload | \n| `extract_cutout(path, ra_deg, dec_deg, width, height, hdu=None)` | Pixel-space cutout around a sky position | Zenithal WCS (TAN, SIN, ARC, STG, ZEA, ZPN), with TPV or SIP distortion; FITS 1-indexed convention | \n| `export_cutout_parquet(path, ra_deg, dec_deg, width, height, out_path, hdu=None)` | Same cutout, written as Parquet | One row per pixel: columns `x` ,`y` (absolute pixel coords in the source image),`value` (physical value,`BZERO` /`BSCALE` applied) | \n\nAll numbers are measured on a **real Euclid Quick Data Release 1 (Q1)\nproduct** from the public IRSA mirror (see\n[Run the benchmark](#run-the-benchmark-on-your-own-fits-file) for the exact\nfile):\n\n- **File** :`EUC_MER_BGSUB-MOSAIC-NIR-Y_TILE102160339-...fits` — a NISP\nY-band, background-subtracted MER mosaic tile near the Euclid Deep\nField North. 19200×19200 px,`float32` (`BITPIX = -32` ), 1.47 GB.\n- **WCS** : standard`RA---TAN` /`DEC--TAN` with a`CD` matrix and no SIP\nterms.\n\nMeasured on the development machine this project was built on: an Intel Core i7-8559U (4 cores / 8 threads, 2.7 GHz base, up to 4.5 GHz), 31 GiB of RAM, an NVMe SSD, no discrete GPU — a laptop-class machine, not a cluster. This matters: the point of these numbers is \"what you get on hardware a working astronomer might actually have on their desk,\" not a best-case HPC result. Each number is the median of 5 runs after a warm-up pass, so the OS page cache is in a comparable state for both tools.\n\n| Operation | `astropy` | `euclidrust` | Speedup | Time saved | \n|---|---|---|---|---|\n| Global stats (min/max/mean, full 1.47 GB) | 478.30 ms | 130.35 ms | ×3.67 | −72.7% | \n| Cutout extraction (50×50 px, by RA/Dec) | 2.664 ms | 0.219 ms | ×12.16 | −91.8% | \n\nRe-runs of the same script on the same machine land within a few percent of these (for example ×3.65 and ×12.36).\n\n`min` and `max` matched bit-for-bit between the two implementations.\n`mean` matched to 7 significant figures (1.0181564... vs 1.0181568...) —\nthe residual difference is floating-point summation order (NumPy's\npairwise summation vs. this project's parallel tree reduction;\nfloating-point addition is not strictly associative), not a computational\nerror. The cutout pixels are identical.\n\nTwo things are worth noting about *why* the gap differs between the two\noperations:\n\n- The **cutout** benchmark shows the larger relative gain, because it is\ndominated by per-call Python/WCS/NumPy object overhead — exactly the\noverhead this project is designed to remove. The benchmark builds the`WCS` object inside every call, like a naive script would; an astropy\nuser who builds it once and reuses it will see a smaller gap.\n- The **global stats** benchmark is ultimately bounded by memory\nbandwidth, not CPU: scanning a full tile moves more than a gigabyte\nthrough RAM regardless of which language does the summing, so the\nceiling on the achievable speedup is lower and both tools get closer to\nit.\n\nTile-compressed images (`RICE_1`, `GZIP_1`, `GZIP_2`, with quantized or\nlossless floats) are decoded tile by tile, with the tiles spread over the\nthreads; a cutout decodes only the tiles it touches. Results are checked\nagainst astropy (cfitsio): min/max and every cutout pixel are identical, and\n`mean` agrees to ~10^-9 relative, over 40 test configurations (algorithms,\npixel types, tile shapes, dither methods and seeds, NaN pixels, unsigned\n16-bit) and on the real Euclid data below.\n\nI had no Euclid-produced compressed file (none of the Q1 images I checked\nwere compressed), so the real-data runs use real Euclid images that I\ncompressed myself with astropy's `CompImageHDU` (cfitsio underneath), with\nits defaults (quantization level 16, dithering) unless stated:\n\n| Real Euclid data, compressed with astropy | Global stats: `astropy` ->`euclidrust` | 50×50 cutout: `astropy` (`.section` ) ->`euclidrust` | \n|---|---|---|\n| NISP detector, 2040², `RICE_1` | 78.6 ms -> 14.5 ms (×5.4) | 6.96 ms -> 0.54 ms (×12.9) | \n| NISP detector, `GZIP_2` | 219 ms -> 22.4 ms (×9.8) | 10.5 ms -> 0.80 ms (×13.2) | \n| NISP detector, `GZIP_2` lossless | 176 ms -> 12.8 ms (×13.7) | 8.58 ms -> 0.48 ms (×17.8) | \n| VIS detector, 2048×2066, `RICE_1` | 72.7 ms -> 11.2 ms (×6.5) | 7.37 ms -> 0.46 ms (×15.9) | \n| MER mosaic, 19200², `RICE_1` (1.47 GB -> 308 MB) | 3719 ms -> 1216 ms (×3.06) | 14.96 ms -> 3.71 ms (×4.0) | \n\nMedian of 3 runs on the development machine, warm cache. For `astropy`, the\nglobal stats include decompressing the whole image (single-threaded), which\nis what an astropy user does; the gap is wider than for uncompressed files\nbecause `euclidrust` decompresses tiles on all the cores. On the big mosaic\nthe tiles are whole image rows (19200 pixels wide), so a cutout still has to\ndecode about fifty full rows, which is why the cutout gain is smaller there.\nThese are decoding speeds only: the files were produced by cfitsio, and\nEuclid's own compressed products (if any) may use other settings.\n\nThe speedups above are the result of iterating against measurements, not a single clever trick:\n\n1. **Naive version** — one`rayon` task per pixel via`into_par_iter()` .\nCorrect, but the per-item scheduling overhead dominates.\n2. **Chunked reduction** — split the buffer into 65,536-pixel blocks,\neach reduced by a tight sequential loop (which LLVM auto-vectorizes at`opt-level = 3` without any explicit SIMD intrinsics), with`rayon` only parallelizing across blocks. This cut the stats time by more than\nhalf (×2.24 on a ~1 GB file during development).\n3. **mmap vs. eager read** — reading the whole file into a`Vec` first was measured to be**1.87× slower** than`mmap` , confirming that\nletting the OS page cache serve data on demand beats an eager bulk\ncopy for this access pattern.\n4. **Stopping point** — once (2) and (3) were in place, further\nmicro-optimization (explicit SIMD intrinsics, finer chunk tuning)\nshowed diminishing returns against the memory-bandwidth ceiling\nidentified above. That's the deliberate stopping condition for this\nround of tuning, not an oversight.\n\nBuild settings: `cargo` release profile with `lto = \"fat\"`,\n`codegen-units = 1`, `opt-level = 3`. Thread pool size is read at runtime\nfrom the host (`rayon::current_num_threads()` / `num_cpus::get()`) —\nnever hardcoded, so the same wheel adapts to whatever machine runs it.\n\n- **2-D images only.** No tables or catalogues, no 3-D cubes, no random-groups\nfiles. Each call reads one HDU of one file (the batch functions loop over\nfiles, HDUs or positions for you).\n- **Compression: `RICE_1`, `GZIP_1` and `GZIP_2` only.**`HCOMPRESS_1` and`PLIO_1` are not implemented and are refused with a clear error. The\ncompressed files tested were produced by astropy/cfitsio, not by Euclid's\npipeline.\n- **WCS coverage.** Zenithal projections (`TAN` ,`SIN` ,`ARC` ,`STG` ,`ZEA` ,`ZPN` ), with TPV or SIP distortion. Not supported: cylindrical and all-sky\nprojections (`CAR` ,`AIT` ,`MOL` ,`HPX` ...), coordinate systems other than\nRA/Dec, alternate WCS (`A` -`Z` : only the primary one is read), and\ndistortion given by lookup tables (HST`CPDIS` /`D2IMDIS` , which makes\ncutouts fail with an error). The sky frame is not read:`RA` and`Dec` are\ntaken in the file's own frame, and`RADESYS` /`EQUINOX` are neither read\nnor converted.\n- **Null pixels.** NaN pixels are left out of the statistics; the integer`BLANK` null value is ignored (see above).\n\n- **A cutout is a crop, not a resampling.** It is a rectangle of whole\npixels whose corner is`round(centre - size / 2)` (halves round away from\nzero, whereas Python's`round` rounds halves to even, so a hand-made\ncomparison can differ by one pixel on an exact tie). The requested position\ncan therefore be up to half a pixel from the middle of the crop; the exact\nfractional position is returned in`pixel_x_center` /`pixel_y_center` .\nThere is no interpolation, rotation or sub-pixel shift, and sizes are in\npixels, not in angle.\n- **Cutouts must lie fully inside the image.** One that touches the edge or\nextends beyond it is refused (`cutout outside the image` ); there is no\npadding and no partial cutout.\n- **`Cutout.data` is a plain Python list of floats** (and`data_int` a list\nof ints): fine for thumbnails, not meant for multi-megapixel regions.\n\n- **Statistics.**`mean` is the mean of the valid (non-NaN) pixels. It\nagrees with numpy to about 7 significant figures on float32 images, not to\nthe last digit, because the sums are made in a different order. Images with\nNaN pixels take about three times longer than clean ones, because a second,\nNaN-aware pass is needed (still faster than numpy's`nanmin` +`nanmax` +`nanmean` ; measured: 70 ms against 164 ms on 256 MB with 7% NaN).\n- **64-bit integers are exact only in the integer fields.** For 64-bit\ninteger images (signed, or unsigned via`BZERO = 2^63` ; raw or\ngzip-compressed) the exact values are`min_int` /`max_int` and`data_int` , and the Parquet`value` column. The float fields (`min` ,`max` ,`mean` ,`data` ) are`float64` , exact only up to 2^53 (about 9 × 10^15),\nwhich is inherent to the type. The exact path needs whole-number`BSCALE` (|BSCALE| ≤ 2^20) and`BZERO` (|BZERO| ≤ 2^64); with a fractional`BSCALE` or`BZERO` the physical values are not integers anyway, so everything is`float64` .\n- **The sky-to-pixel inverse of TPV and SIP is iterative.** It was checked\nagainst wcslib (through astropy) on over 120 synthetic headers and on real\nones: the real Euclid VIS frame (TPV, 3 × 10^-8 pixel) and the real SIP\nheaders shipped with astropy's tests, including HST/ACS (4 × 10^-9 pixel\nat worst). It can pick the wrong root when a SIP distortion is so strong\n(hundreds of pixels) that the inverse is not unique, which does not happen\nwith real instruments.\n\n- **Real-data validation is small.** The benchmark numbers above come from\na single Q1 NISP mosaic tile out of the ~35 TB Q1 release. Correctness\n(not speed) was also checked against astropy on one NISP frame, one VIS\nframe, one raw NISP detector and one segmentation map. That is enough to\nshow the approach works on real archival products, not a representative\nsample across instruments, bands and tiles.\n- **Tested on two machines.** (1) An Intel NUC running Debian 13 (Linux 6.12, x86_64): Intel Core\ni7-8559U (4 cores / 8 threads, 2.7 GHz, up to 4.5 GHz), 31 GiB of RAM, an NVMe SSD, Python\n3.13.5, Rust 1.100, no NVIDIA GPU. (2) A desktop PC running Windows 11: Intel Core i9-10850K\n(10 cores / 20 threads), 64 GiB of RAM, an NVIDIA GeForce RTX 4070 Ti (12 GB), Python 3.12.10,\nRust 1.95. macOS and ARM have not been tried, nor Linux with an NVIDIA GPU, nor machines with\nmuch less memory (the images are memory-mapped, so a laptop with 8 GB of RAM should cope with the\n1.47 GB tile, but I have not measured it).\n- **No continuous integration, no pre-built packages.** The tests are run by hand (there is no CI workflow). The package is not on PyPI: you build\nit yourself, which needs a Rust toolchain. The interface is version 0.1.0\nand may change.\n\n- **The files are memory-mapped and must not change while they are read.** If another process truncates or overwrites a file during a call, the\noperating system can kill the Python process (this is how`mmap` works; I\nhave not tried to provoke it). Network or FUSE file systems are untested.\n- **On Windows, mapping a whole big file is expensive to undo.** Closing a view of the 1.47 GB tile took\n165 ms, more than the computation. The library therefore reads big images through many small\nmapped windows, one per thread, which spreads that cost (255 ms became 123 ms). Linux does\nnot have this problem. Astropy's`memmap` is also much slower on Windows (1.3 s against 0.48 s\non the NUC for the same statistics).\n- **All the cores are used by default.** The number of threads follows the\nmachine; to limit it (a shared node, for example) set the environment\nvariable`RAYON_NUM_THREADS` before starting Python, e.g.`RAYON_NUM_THREADS=4` . The GIL is released during the calls.\n- **Memory with compressed images.** Each worker thread decodes whole tiles\ninto`float64` (8 bytes per pixel), so an image compressed as a single huge\ntile needs about eight times its pixel count in bytes. There is no tile\ncache: many cutouts falling in the same tile decode it again each time.\n- **Batch results are held in memory.**`fits_stats_many` and`extract_cutouts` return everything as Python lists at the end: there is\nno streaming and no progress callback, so a huge list of cutouts uses\nmemory proportional to its size.`hdu=` is the same for every file of a`fits_stats_many` call.\n\nThis is the first of four planned components for Euclid-scale data processing, chosen first specifically because it needed no GPU/cluster and is the foundation the others build on:\n\n1. ✅ **FITS streaming / cutout / Parquet export** (this package). Next steps for this component, if someone needs them: the remaining compression algorithms (`HCOMPRESS_1` ,`PLIO_1` ) and the cylindrical / all-sky projections.\n2. Catalogue cross-matching at the hundred-million-to-billion-object scale (HEALPix / spherical KD-tree indexing, memory-mapped for datasets larger than available RAM).\n3. Morphology / weak-lensing profile fitting (PyO3-bound, parallel) — feasible at prototype scale on commodity hardware, not yet at billion-galaxy scale without a cluster.\n4. Interactive WASM/WebGPU visualization of large mosaics and catalogues.\n\nMIT licensed (see `LICENSE`) — free to use, copy, modify, merge, publish,\ndistribute and sublicense, with no restriction beyond keeping the\ncopyright notice. Experimental, actively developed.", "url": "https://wpnews.pro/news/euclidrust-rust-fits-reader-for-euclid-10x-faster-than-astropy", "canonical_source": "https://github.com/MaxIKweeger/EuclidRust", "published_at": "2026-11-14 17:11:53+00:00", "updated_at": "2026-11-14 17:22:48.469327+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["euclidrust", "Rust", "PyO3", "Python", "astropy", "ESA", "Euclid", "Claude Code"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/euclidrust-rust-fits-reader-for-euclid-10x-faster-than-astropy", "markdown": "https://wpnews.pro/news/euclidrust-rust-fits-reader-for-euclid-10x-faster-than-astropy.md", "text": "https://wpnews.pro/news/euclidrust-rust-fits-reader-for-euclid-10x-faster-than-astropy.txt", "jsonld": "https://wpnews.pro/news/euclidrust-rust-fits-reader-for-euclid-10x-faster-than-astropy.jsonld"}}