Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,6 @@ jobs:
echo "Using LUTs only - full imagery available at:"
echo " https://doi.org/10.5281/zenodo.18704072"

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Setup Miniconda
uses: conda-incubator/setup-miniconda@v3
with:
Expand All @@ -69,7 +64,7 @@ jobs:
- name: Install build dependencies from conda-forge
shell: bash -l {0}
run: |
conda install -c conda-forge swig gcc gxx nlopt
conda install -c conda-forge pip swig gcc gxx nlopt

- name: Install Python dependencies
shell: bash -l {0}
Expand Down
12 changes: 4 additions & 8 deletions .zenodo.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
"description": "SpiPy is a Python implementation of SPIRES (SPectral Inversion of REflectance from Snow), a spectral unmixing algorithm for analyzing snow reflectance data from satellite imagery. It retrieves snow properties (grain size, dust concentration, fractional snow-covered area) using lookup tables generated from Mie-scattering theory. This implementation features a hybrid Python/C++ architecture achieving 3000x speedup over pure Python implementations.",
"creators": [
{
"name": "Bair, Ned",
"name": "Bair, Edward H.",
"affiliation": "University of California, Santa Barbara",
"orcid": "0000-0002-7654-3210"
"orcid": "https://orcid.org/0000-0002-6554-387X"
},
{
"name": "Griessbaum, Niklas",
"affiliation": "Leidos, Inc.",
"orcid": "0000-0000-0000-0000"
"orcid": "https://orcid.org/0000-0001-5037-6266"
}
],
"license": "MIT",
Expand All @@ -32,9 +32,5 @@
"scheme": "doi"
}
],
"grants": [
{
"id": "W913E523C0002"
}
]
"notes": "This work was supported by contract W913E523C0002."
}
17 changes: 9 additions & 8 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,20 @@ message: "If you use this software, please cite both the software and the algori
title: "SpiPy: Python implementation of SPIRES snow property inversion"
authors:
- family-names: Bair
given-names: Ned
given-names: Edward H.
email: edwardbair@ucsb.edu
affiliation: "University of California, Santa Barbara"
orcid: "https://orcid.org/0000-0002-7654-3210" # Update with actual ORCID if available
orcid: "https://orcid.org/0000-0002-6554-387X"
- family-names: Griessbaum
given-names: Niklas
affiliation: "Leidos, Inc."
orcid: "https://orcid.org/0000-0000-0000-0000" # Update with actual ORCID if available
version: 0.2.0
date-released: 2024-11-06
repository-code: "https://github.com/edwardbair/SpiPy"
url: "https://github.com/edwardbair/SpiPy"
license: MIT # Update if different
orcid: "https://orcid.org/0000-0001-5037-6266"
version: 0.2.8
date-released: 2026-02-23
repository-code: "https://github.com/NiklasPhabian/SpiPy"
url: "https://spipy.readthedocs.io"
license: MIT
doi: 10.5281/zenodo.18747284
keywords:
- remote sensing
- snow properties
Expand Down
54 changes: 42 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# SpiPy

[![PyPI version](https://badge.fury.io/py/spires.svg)](https://pypi.org/project/spires/)
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18747284.svg)](https://doi.org/10.5281/zenodo.18747284)
[![Documentation Status](https://readthedocs.org/projects/spipy/badge/?version=latest)](https://spipy.readthedocs.io/en/latest/?badge=latest)
[![Build and Test](https://github.com/NiklasPhabian/SpiPy/workflows/Build%20and%20Test/badge.svg)](https://github.com/NiklasPhabian/SpiPy/actions)
[![Python 3.9-3.14](https://img.shields.io/badge/python-3.9--3.14-blue.svg)](https://github.com/NiklasPhabian/SpiPy)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**[📦 View Source on GitHub](https://github.com/NiklasPhabian/SpiPy)** | **[📖 Documentation](https://spipy.readthedocs.io)** | **[🐛 Report Issues](https://github.com/NiklasPhabian/SpiPy/issues)**

SpiPy is a Python implementation of [SPIRES](https://ieeexplore.ieee.org/document/9290428) (SPectral Inversion of REflectance from Snow), originally implemented in MATLAB ([SPIRES GitHub repository](https://github.com/edwardbair/SPIRES)).
SpiPy is a Python implementation of [SPIRES](https://ieeexplore.ieee.org/document/9290428) (Snow Property Inversion From Remote Sensing), originally implemented in MATLAB ([SPIRES GitHub repository](https://github.com/edwardbair/SPIRES)).

## Overview

Expand Down Expand Up @@ -78,16 +79,30 @@ Basic usage:

```python
import spires
import numpy as np

# Load lookup table
interpolator = spires.LutInterpolator(
lut_file='tests/data/lut_sentinel2b_b2to12_3um_dust.mat'
)

# Process imagery to get fractional snow-covered area
fsca = spires.get_fsca(...)
# Invert a single mixed spectrum
spectrum_target = np.array([0.3424, 0.366, 0.3624, 0.3893,
0.4162, 0.3957, 0.0704, 0.0627, 0.3792])
spectrum_background = np.array([0.0182, 0.0265, 0.0283, 0.0561,
0.0954, 0.1204, 0.1249, 0.0789, 0.1406])

fsca, fshade, dust, grain_size = spires.speedy_invert(
spectrum_target=spectrum_target,
spectrum_background=spectrum_background,
solar_angle=55.73,
interpolator=interpolator,
)
```

See [Getting Started](https://spipy.readthedocs.io/en/latest/getting_started.html) for batch
processing, xarray, and Dask-parallel workflows.

## Development

### Building Wheels
Expand Down Expand Up @@ -154,21 +169,23 @@ Simulated Mie-scattering snow reflectance lookup tables are available on Zenodo:

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.18701286.svg)](https://doi.org/10.5281/zenodo.18701286)

- **MODIS**: `LUT_MODIS.mat` (537 MB)
- **Sentinel-2**: `lut_sentinel2b_b2to12_3um_dust.mat` (70 MB)
- **HLS**: `lut_HLSS30_b1to13_3um_dust.mat` (101 MB)
- **MODIS**: `lut_modis_b1to7_3um_dust.mat` (537 MB)
- **Landsat OLI**: `lut_oli_b1to7_3um_dust.mat` (55 MB)

Download using the helper script:
```bash
python scripts/download_test_data.py --luts
```

Or download directly:
Or download directly, e.g.:
```bash
curl -L -o LUT_MODIS.mat https://zenodo.org/records/18701286/files/LUT_MODIS.mat
curl -L -o lut_sentinel2b_b2to12_3um_dust.mat https://zenodo.org/records/18701286/files/lut_sentinel2b_b2to12_3um_dust.mat
```

**Note:** The Sentinel-2 LUT is also included in the repository via Git LFS. Landsat lookup tables are planned.
**Note:** All LUTs above are also bundled in the repository via Git LFS — see
[tests/data/README.md](tests/data/README.md) for details.

### Test Data

Expand Down Expand Up @@ -208,6 +225,18 @@ The C++ optimizations provide significant speedups over pure Python:
- SWIG interpolator and scipy's RegularGridInterpolator behave differently when coordinates aren't linspace
- COBYLA in scipy can't set `rhobeg` per dimension individually, requiring problem scaling

### Cross-platform numerical reproducibility

Inversion results can differ by a few percent between Linux (x86_64, gcc) and macOS
(arm64, clang) for the same inputs. The cause is a combination of different ISAs
rounding transcendentals (`exp`, `pow`) at the last bit, different compilers and
libm implementations, and the fact that COBYLA is a derivative-free iterative
solver — tiny ULP-level differences in early evaluations can cascade and steer
the simplex toward a different local optimum on a flat region of the objective.
The recovered parameters still reproduce the observed spectrum within tolerance,
they just aren't bit-identical across platforms. Tests therefore assert residual
quality and physical plausibility rather than pinning optimizer coordinates.

## Roadmap

- [ ] Optimize inversion for single location over multiple timesteps (keep R_0 constant)
Expand Down Expand Up @@ -239,12 +268,13 @@ If you use this software, please cite the algorithm paper, software implementati

**Software:**
```bibtex
@software{bair2024spipy,
@software{bair2026spipy,
title={SpiPy: Python implementation of SPIRES snow property inversion},
author={Bair, Ned and Griessbaum, Niklas},
year={2024},
url={https://github.com/edwardbair/SpiPy},
version={0.2.0},
author={Bair, Edward H. and Griessbaum, Niklas},
year={2026},
url={https://github.com/NiklasPhabian/SpiPy},
version={0.2.8},
doi={10.5281/zenodo.18747284},
note={See CITATION.cff for full metadata}
}
```
Expand Down
7 changes: 5 additions & 2 deletions doc/source/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,10 @@ Some notebooks have additional requirements:

## Data Notes

- Notebooks assume data is stored in `/scratch/tristate/` or `/data/sentinel2/` directories
- Most workflows are designed for UCSB region but can be adapted
- Notebooks were authored against developer-specific data paths; you'll need to
edit the input/output paths near the top of each notebook to point at your
own data
- Most workflows were developed for the UCSB region but can be adapted
- Large notebooks (>1MB) contain embedded outputs and visualizations

## Development Notebooks
Expand All @@ -88,3 +90,4 @@ The `examples/development/` directory contains notebooks used during development
- `cobyla.ipynb` - COBYLA optimizer testing
- `invert2d.ipynb` - 2D inversion experiments
- `compress_nc.ipynb` - NetCDF compression utilities
- `spicy_snow_experiment.ipynb` - Comparison experiment against the spicy-snow approach
41 changes: 40 additions & 1 deletion doc/source/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ git lfs install

```bash
# Clone the repository
git clone https://github.com/[username]/SpiPy.git
git clone https://github.com/NiklasPhabian/SpiPy.git
cd SpiPy

# Build SWIG extensions
Expand Down Expand Up @@ -129,6 +129,45 @@ results = spires.speedy_invert_xarray(
)
```

### Parallel Inversion with Dask

For datasets too large to fit in memory or that benefit from multi-core
processing (e.g. time series of full Sentinel-2 scenes), use the Dask-parallel
entry point. The C++ inversion releases the Python GIL, so a Dask client with
`threads_per_worker > 1` gives real parallel speedup while sharing one LUT
copy per worker process.

```python
import xarray as xr
from dask.distributed import Client

import spires

# Inputs as chunked DataArrays (time, y, x, band) etc.
targets = xr.open_zarr('sentinel2_data.zarr')['reflectance']
backgrounds = xr.open_dataarray('background_r0.nc')
solar_angles = xr.open_dataarray('solar_angles.nc')

interpolator = spires.LutInterpolator(
lut_file='tests/data/lut_sentinel2b_b2to12_3um_dust.mat'
)

with Client(n_workers=4, threads_per_worker=4) as client:
ds = spires.speedy_invert_dask(
spectra_targets=targets,
spectra_backgrounds=backgrounds,
obs_solar_angles=solar_angles,
interpolator=interpolator,
client=client,
)

# Encode for compact storage (NaN -> fill value, fractions scaled to int)
encoded = spires.encode_results(ds)
encoded.to_netcdf('inversion_results.nc')
```

See `examples/05_sentinel_snow_inversion.ipynb` for a complete dask workflow.

## Understanding the Algorithm

SPIRES (SPectral Inversion of REflectance from Snow) retrieves snow properties by:
Expand Down
4 changes: 2 additions & 2 deletions doc/source/reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ namespace plus those listed in the reference are public.
.. automodsumm:: spires.invert
:template: instance_methods.rst

.. automodsumm:: spires.process
.. automodsumm:: spires.parallel
:template: instance_methods.rst

.. automodsumm:: spires.utol
Expand All @@ -32,7 +32,7 @@ namespace plus those listed in the reference are public.
:undoc-members:
:show-inheritance:

.. automodule:: spires.process
.. automodule:: spires.parallel
:members:
:undoc-members:
:show-inheritance:
Expand Down
Loading
Loading