Support HDF filters affecting reading

[JohnLCaron]

Quantization not neded for readers?

https://github.com/Unidata/netcdf-c/blob/main/docs/quantize.md

[JohnLCaron]

https://support.hdfgroup.org/documentation/hdf5-docs/hdf5_topics/UsingCompressionInHDF5.html

HDF5 supports gzip (deflate), Szip, n-bit, scale-offset, and shuffling (with deflate) compression filters.

[lesserwhirls]

HDF also has a filter registry with many more options:

https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filter-plugins-registered-with-the-hdf-group

[JohnLCaron]

From the Readme:

"There will be no dependencies on native libraries in the core module, but other modules or
projects that use the core are free to use dependencies as needed. We will likely add runtime discovery to facilitate this,
for example, to use HDF5 filters that link to native libraries."

[lesserwhirls]

Do you have any thoughts on how you would like to support runtime filter discovery? Kotlin will work just fine with the Java Service Provider Interface, so that's pretty easy. But if we want to target anything other than Kotlin/JVM, we'd need something different (e.g. sweet-spi).

[JohnLCaron]

I havent thought about supporting anything other than kotlin/JVM. But it could be worth exploring kotlin multiplatform to see if one could use the library from native code. Is that what you're thinking? sweet-spi <https://github.com/whyoleg/sweet-spi> looks good, but we are already using kotlin = "2.1.10", so would have to wait until "KSP2 is not supported because of KSP#1823 <google/ksp#1823> " is fixed.

…

On Mon, Jun 2, 2025 at 12:44 PM Sean Arms ***@***.***> wrote: *lesserwhirls* left a comment (JohnLCaron/netchdf#117) <#117 (comment)> Do you have any thoughts on how you would like to support runtime filter discovery? Kotlin will work just fine with the Java Service Provider Interface, so that's pretty easy. But if we want to target anything other than Kotlin/JVM, we'd need something different (e.g. sweet-spi <https://github.com/whyoleg/sweet-spi>). — Reply to this email directly, view it on GitHub <#117 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEVZQGKGIDPV3TFSCHE4J33BSLRBAVCNFSM6AAAAAB5D2WXAGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSMZSGAYTQNZTGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lesserwhirls]

Yeah, just wondering if we want to leave the door open to the possibility of native (or even wasm) or not. We can always revisit runtime loading if/when there is a jump to multiplatform...probably not too hard to swap that out later.

[JohnLCaron]

I was just looking at where netchdf is using "import java.* " since that would have to be ported to native.

Targets
Occurrences of 'import java.' in Project

Found occurrences in Project 4524
Production 4464
  clibs 4317
  core 147

Test 60
  clibs 19
  core 41
    test 28
    testFixtures 13

skip clibs
mostly you have to rewrite

com.sunya.cdm.array 28
com.sunya.netchdf.hdf4 30
com.sunya.netchdf.hdf5 49

[JohnLCaron]

PR #170 adds filters from jhdf, using a simple "register filter" call, with interface

interface H5filterIF {
    fun id() : Int
    fun name() : String
    fun apply(encodedData: ByteArray, clientValues: IntArray): ByteArray
}

We have to write some kind of kotlin wrapper for any native filter. Not sure of the advantage of "runtime filter discovery" thats more than just explitly registering?

[lesserwhirls]

I think as long as the wrapper can clearly/cleanly report to a user the case where they are trying to use a native filter but the required native library is missing, then it's all good.

[JohnLCaron]

Good point. There has to be check whether the library is loaded, with an error message rather than an exception.

Also, things should work as long as the filter is registered, ie, the filter doesnt have to be in our FilterType enumeration. As long as the filter id matches the one in the data file.