[INSTALL]: nep 2.6.0

[mathomp4]

Package name

nep

Package version/tag

2.6.0

Build options

+geotiff +g2c

Installation timeframe

Not crucial. 2.3?

Other information

@captainkirk99 has a cool package called NEP (NetCDF Extension Pack) and I'm looking at it as a nice addition to maybe GEOS and maybe it could help others work here.

In an upcoming package update, he's added a lot of new variants:

• New variants (all default OFF):
  • fits — FITS reader via CFITSIO (HST, JWST, Chandra files)
  • geotiff — GeoTIFF reader via libgeotiff (CF-1.8 CRS metadata)
  • grib2 — GRIB2 reader via NCEPLIBS-g2c (NWP model output)
  • cdf — NASA CDF reader (space physics data)
  • pds4 — PDS4 reader via libxml2 (planetary science data)
  • parallel — parallel I/O tests (requires MPI-enabled HDF5 + netcdf-c)
  • examples — example programs
  • benchmarks — performance benchmark programs

So I'm sort of using this issue as a way to maybe gauge which variants might be useful for the consortium? I'll ask around GMAO of course, but I'm wondering if folks here use any of these file types.

This is for a future spack-stack, not for 2.2 at this point.

[captainkirk99]

Probably the most immediately useful is LZ4 compression (not listed in Matt's post above).

LZ4 is faster reading and writing than zstd.

Switching from zstd to lz4 should reduce IO output of the UFS by ~30%(?) This seems significant. Although harder to measure in practice, LZ4 would also speed all downstram applications, because reading LZ4 compressed data is almost as fast as reading uncompressed data.

LZ4 was my original choice for the UFS, but Zstandard proved much easier to get through the gauntlet into a netcdf-c release. With the NetCDF Expansion Pack, I'm not limited by the conservative tendencies of the Unidites.

For more about compression: https://github.com/Intelligent-Data-Design-Inc/NEP/blob/main/docs/compression.md

[mathomp4]

Sounds like maybe starting off with NEP with no new/optional variants on is a good place for use. Maybe have +grib2?

I'll ask around GMAO to see if people know about the other types...

[captainkirk99]

For Earth science we have new formats GRIB2 and GeoTIFF.

The other new formats, CDF, FITS, and PDS4, are used by NASA/ESA for planetary missions - i.e. data from Mars and Saturn. ;-)

[mathomp4]

For Earth science we have new formats GRIB2 and GeoTIFF.

The other new formats, CDF, FITS, and PDS4, are used by NASA/ESA for planetary missions - i.e. data from Mars and Saturn. ;-)

Ah okay then. I just looked and in your PR you have:

    depends_on("libgeotiff", when="+geotiff", type=("build", "link"))
    depends_on("g2c", when="+grib2", type=("build", "link"))

and I looked at unified-dev and we have both libgeotiff and g2c in our concretization. So doing +geotiff +grib2 would be free dependency-wise.

[mathomp4]

Also, @captainkirk99 You might as well add 2.6.0 to spack/spack-packages#5557. Rather than having to make another PR. :)

[JacobCarley-NOAA]

Linking to Issue #2071 by @BinLiu-NOAA which specifically requests lz4. I believe that issue would be addressed with the addition of nep.

[mathomp4]

Linking to Issue #2071 by @BinLiu-NOAA which specifically requests lz4. I believe that issue would be addressed with the addition of nep.

@JacobCarley-NOAA Yes. I think LZ4 you just get automatically. Which is nice. :)