diff --git a/Readme.md b/Readme.md index e899c635..9a0913d3 100644 --- a/Readme.md +++ b/Readme.md @@ -1,5 +1,5 @@ # netchdf -_last updated: 7/16/2025_ +_last updated: 7/18/2025_ This is a rewrite in Kotlin of parts of the devcdm and netcdf-java libraries. @@ -10,6 +10,32 @@ The library is close to feature complete. We are currently extensively testing a Please contact me if you'd like to help out. Especially needed are test datasets from all the important data archives!! + +* [netchdf](#netchdf) + * [Building](#building) + * [What version of the JVM, Kotlin, and Gradle?](#what-version-of-the-jvm-kotlin-and-gradle) + * [Why this library?](#why-this-library-) + * [Why do we need another library besides the standard reference libraries?](#why-do-we-need-another-library-besides-the-standard-reference-libraries) + * [What's wrong with the standard reference libraries?](#whats-wrong-with-the-standard-reference-libraries) + * [Why Kotlin?](#why-kotlin) + * [What about performance?](#what-about-performance) + * [Goals and scope](#goals-and-scope) + * [Non-goals](#non-goals) + * [Testing](#testing) + * [Code Coverage](#code-coverage) + * [Testing against the reference libraries](#testing-against-the-reference-libraries) + * [Data Model notes](#data-model-notes) + * [Type Safety and Generics](#type-safety-and-generics) + * [Cdl Names](#cdl-names) + * [Datatype](#datatype) + * [Typedef](#typedef) + * [Dimension](#dimension) + * [Compare with HDF5 data model](#compare-with-hdf5-data-model) + * [Compare with HDF4 data model](#compare-with-hdf4-data-model) + * [Compare with HDF-EOS data model](#compare-with-hdf-eos-data-model) + * [Elevator blurb](#elevator-blurb) + + ### Building * Download Java 21 JDK and set JAVA_HOME. @@ -26,6 +52,20 @@ Also see: * [Building and Running native library](docs/Building.md) * [Building and Running ncdump](cli/Readme.md) +#### What version of the JVM, Kotlin, and Gradle? + +We use the latest LTS (long term support) Java version, and will not be explicitly supporting older versions. +Currently that is Java 21. + +We also use the latest stable version of Kotlin that is compatible with the Java version. Currently that is Kotlin 2.1. + +Gradle is our build system. We will use the latest stable version of Gradle compatible with our Java and Kotlin versions. +Currently that is Gradle 8.14. + +For now, you must download and build the library yourself. Eventually we will publish it to Maven Central. +The IntelliJ IDE is highly recommended for all JVM development. + + ### Why this library? The scientific data stored in NetCDF and HDF file formats must remain forever readable. @@ -121,31 +161,38 @@ It's not a goal to duplicate Netcdf-C library functionality. It's not a goal to provide remote access to files. -### What version of the JVM, Kotlin, and Gradle? - -We will always use the latest LTS (long term support) Java version, and will not be explicitly supporting older versions. -Currently that is Java 21. +### Testing -We also use the latest stable version of Kotlin that is compatible with the Java version. Currently that is Kotlin 2.1. +Currently most of the test files do not live in the github repo because they are too big. +Eventually we will make them available in a separate download. -Gradle is our build system. We will use the latest stable version of Gradle compatible with our Java and Kotlin versions. -Currently that is Gradle 8.14. +There are four levels of testing: -For now, you must download and build the library yourself. Eventually we will publish it to Maven Central. -The IntelliJ IDE is highly recommended for all JVM development. +1. Unit testing that doesn't require reading files. +2. Testing with files in core/commonTest/data. These are fast and are run in a Github Action. +3. Testing with files in TestFiles.testData in module testfiles. These are medium fast (< 11 min wallclock). +4. Testing with files in TestFiles.testData in module testclibs. These are slow. +Currently we have 1500+ test files in the core and testdata modules: -### Testing +```` +hdf-eos2 = 440 files +hdf-eos5 = 18 files +hdf4 = 32 files +hdf5 = 175 files +netcdf3 = 664 files +netcdf3.2 = 81 files +netcdf3.5 = 1 files +netcdf4 = 119 files + +total # files = 1530 +```` -We use the Java [Foreign Function & Memory API](https://docs.oracle.com/en/java/javase/21/core/foreign-function-and-memory-api.html) -for testing against the Netcdf, HDF5, and HDF4 C libraries. -With these tools we can be confident that our library gives the same results as the reference libraries. +We will continue to add representative samples of recent files for improved testing and code coverage. -Currently using -* HDF5 library version: 1.14.6. -* netcdf-c library version 4.10.0-development of May 23 2025 +#### Code Coverage -Currently we have this test coverage from core/test: +Currently we have this test coverage from the core and testfiles modules: ```` cdm 88% (1560/1764) LOC @@ -154,28 +201,38 @@ Currently we have this test coverage from core/test: netcdf3 77% (230/297) LOC ```` -The core library has ~6500 LOC. +7/18/2025 +```` + cdm.api 94% (532/567) LOC + cdm.array 95% (662/698) LOC + cdm.iosp 68% (146/213) LOC + cdm.layout 89% (277/310) LOC + cdm.util 76% (106/139) LOC + hdf4 82% (1638/2008) LOC + hdf5 80% (2740/3417) LOC + netcdf3 80% (213/266) LOC + + all 83% (6314/7618) LOC + ```` + +The core library has ~7600 LOC. + +#### Testing against the reference libraries More and deeper test coverage is provided in the testclibs module, which compares netchdf metadata and data against -the Netcdf, HDF5, and HDF4 C libraries. The clibs module is not part of the released netchdf library and is +the Netcdf, HDF5, and HDF4 C libraries. Note that the clibs module is not part of the released netchdf library and is only supported for test purposes. -Currently we have 1470 test files in the core test suite: +We use the Java [Foreign Function & Memory API](https://docs.oracle.com/en/java/javase/21/core/foreign-function-and-memory-api.html) +for testing against the Netcdf, HDF5, and HDF4 C libraries. +With these tools we can be confident that our library gives the same results as the reference libraries. -```` -hdf-eos2 = 267 files -hdf-eos5 = 18 files -hdf4 = 205 files -hdf5 = 113 files -netcdf3 = 664 files -netcdf3.2 = 81 files -netcdf3.5 = 1 files -netcdf4 = 121 files - -total # files = 1470 -```` -We need to get representative samples of recent files for improved testing and code coverage. +Currently using +* HDF5 library version: 1.14.6. +* netcdf-c library version 4.10.0-development of May 23 2025 +* HDF-4 library version: ??? +In order to run, you must install the C libraries on your computer and ad them to the LD_LIBRARY_PATH. ### Data Model notes diff --git a/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayString.kt b/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayString.kt index e73b5e82..df96267b 100644 --- a/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayString.kt +++ b/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayString.kt @@ -61,7 +61,7 @@ fun ByteArray.makeStringFromBytes(): String { * If there is a null (zero) value in the array, the String will end there. * The null is not returned as part of the String. */ -internal fun ArrayByte.makeStringFromBytes(charset : Charset = Charsets.UTF8): String { +fun ArrayByte.makeStringFromBytes(charset : Charset = Charsets.UTF8): String { var count = 0 for (c in this) { if (c.toInt() == 0) { @@ -73,7 +73,7 @@ internal fun ArrayByte.makeStringFromBytes(charset : Charset = Charsets.UTF8): S return this.values.decodeToString(charset, 0, count) } -internal fun ArrayUByte.makeStringFromBytes(charset : Charset = Charsets.UTF8): String { +fun ArrayUByte.makeStringFromBytes(charset : Charset = Charsets.UTF8): String { var count = 0 for (c in this) { if (c.toInt() == 0) { diff --git a/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayULong.kt b/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayULong.kt index de0bc28d..4d077246 100644 --- a/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayULong.kt +++ b/core/src/commonMain/kotlin/com/sunya/cdm/array/ArrayULong.kt @@ -10,6 +10,8 @@ import com.sunya.cdm.layout.TransferChunk @OptIn(ExperimentalUnsignedTypes::class) class ArrayULong(shape : IntArray, datatype : Datatype<*>, val values: ULongArray) : ArrayTyped(datatype, shape) { + constructor(shape : IntArray, values: ULongArray) : this(shape, Datatype.ULONG, values) + override fun iterator(): Iterator = BufferIterator() private inner class BufferIterator : AbstractIterator() { private var idx = 0 diff --git a/core/src/commonMain/kotlin/com/sunya/cdm/array/StructureMember.kt b/core/src/commonMain/kotlin/com/sunya/cdm/array/StructureMember.kt index 675f9ff8..7e0d294b 100644 --- a/core/src/commonMain/kotlin/com/sunya/cdm/array/StructureMember.kt +++ b/core/src/commonMain/kotlin/com/sunya/cdm/array/StructureMember.kt @@ -86,7 +86,7 @@ class StructureMember(orgName: String, val datatype : Datatype, val offset val value = value(sdata) if (value is ArrayTyped<*>) return value - if (value is String) return ArrayString(intArrayOf(1), listOf(value as String)) + if (value is String) return ArrayString(intArrayOf(1), listOf(value)) return when (datatype) { Datatype.BYTE -> ArrayByte(intArrayOf(1), ByteArray(1) { value as Byte }) diff --git a/core/src/commonTest/data/jhdf/100B_max_dimension_size.hdf5 b/core/src/commonTest/data/jhdf/100B_max_dimension_size.hdf5 new file mode 100644 index 00000000..b3c1f2ce Binary files /dev/null and b/core/src/commonTest/data/jhdf/100B_max_dimension_size.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/bitfield_datasets.hdf5 b/core/src/commonTest/data/jhdf/bitfield_datasets.hdf5 new file mode 100644 index 00000000..73280277 Binary files /dev/null and b/core/src/commonTest/data/jhdf/bitfield_datasets.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/bitshuffle_datasets.hdf5 b/core/src/commonTest/data/jhdf/bitshuffle_datasets.hdf5 new file mode 100644 index 00000000..811986f2 Binary files /dev/null and b/core/src/commonTest/data/jhdf/bitshuffle_datasets.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/chunked_v4_datasets.hdf5 b/core/src/commonTest/data/jhdf/chunked_v4_datasets.hdf5 new file mode 100644 index 00000000..8ceb4df8 Binary files /dev/null and b/core/src/commonTest/data/jhdf/chunked_v4_datasets.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/committed_datatypes.hdf5 b/core/src/commonTest/data/jhdf/committed_datatypes.hdf5 new file mode 100644 index 00000000..5aadfa41 Binary files /dev/null and b/core/src/commonTest/data/jhdf/committed_datatypes.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/compound_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/compound_datasets_earliest.hdf5 new file mode 100644 index 00000000..7991bc4e Binary files /dev/null and b/core/src/commonTest/data/jhdf/compound_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/compound_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/compound_datasets_latest.hdf5 new file mode 100644 index 00000000..d4ff4926 Binary files /dev/null and b/core/src/commonTest/data/jhdf/compound_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/external_link.hdf5 b/core/src/commonTest/data/jhdf/external_link.hdf5 new file mode 100644 index 00000000..ffd336b0 Binary files /dev/null and b/core/src/commonTest/data/jhdf/external_link.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/fixed_array_paged_datasets.hdf5 b/core/src/commonTest/data/jhdf/fixed_array_paged_datasets.hdf5 new file mode 100644 index 00000000..dd888331 Binary files /dev/null and b/core/src/commonTest/data/jhdf/fixed_array_paged_datasets.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/fletcher32_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/fletcher32_datasets_earliest.hdf5 new file mode 100644 index 00000000..a637d114 Binary files /dev/null and b/core/src/commonTest/data/jhdf/fletcher32_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/fletcher32_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/fletcher32_datasets_latest.hdf5 new file mode 100644 index 00000000..dea93fcc Binary files /dev/null and b/core/src/commonTest/data/jhdf/fletcher32_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/float_special_values_earliest.hdf5 b/core/src/commonTest/data/jhdf/float_special_values_earliest.hdf5 new file mode 100644 index 00000000..d0afb7b8 Binary files /dev/null and b/core/src/commonTest/data/jhdf/float_special_values_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/float_special_values_latest.hdf5 b/core/src/commonTest/data/jhdf/float_special_values_latest.hdf5 new file mode 100644 index 00000000..1489faec Binary files /dev/null and b/core/src/commonTest/data/jhdf/float_special_values_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/globalheaps_test.hdf5 b/core/src/commonTest/data/jhdf/globalheaps_test.hdf5 new file mode 100644 index 00000000..cf6afe3f Binary files /dev/null and b/core/src/commonTest/data/jhdf/globalheaps_test.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/hdf_v14_test1.hdf5 b/core/src/commonTest/data/jhdf/hdf_v14_test1.hdf5 new file mode 100644 index 00000000..9c6815e6 Binary files /dev/null and b/core/src/commonTest/data/jhdf/hdf_v14_test1.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/hdf_v14_test2.hdf5 b/core/src/commonTest/data/jhdf/hdf_v14_test2.hdf5 new file mode 100644 index 00000000..7681231a Binary files /dev/null and b/core/src/commonTest/data/jhdf/hdf_v14_test2.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/implicit_index_datasets.hdf5 b/core/src/commonTest/data/jhdf/implicit_index_datasets.hdf5 new file mode 100644 index 00000000..627eda66 Binary files /dev/null and b/core/src/commonTest/data/jhdf/implicit_index_datasets.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/isssue-523.hdf5 b/core/src/commonTest/data/jhdf/isssue-523.hdf5 new file mode 100644 index 00000000..ff8aa906 Binary files /dev/null and b/core/src/commonTest/data/jhdf/isssue-523.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/issue255_example.hdf5 b/core/src/commonTest/data/jhdf/issue255_example.hdf5 new file mode 100644 index 00000000..59af932e Binary files /dev/null and b/core/src/commonTest/data/jhdf/issue255_example.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/issue318_example.hdf5 b/core/src/commonTest/data/jhdf/issue318_example.hdf5 new file mode 100644 index 00000000..3f5cdbe6 Binary files /dev/null and b/core/src/commonTest/data/jhdf/issue318_example.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/lz4_datasets.hdf5 b/core/src/commonTest/data/jhdf/lz4_datasets.hdf5 new file mode 100644 index 00000000..c429d85e Binary files /dev/null and b/core/src/commonTest/data/jhdf/lz4_datasets.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/multidim_string_datasest.hdf5 b/core/src/commonTest/data/jhdf/multidim_string_datasest.hdf5 new file mode 100644 index 00000000..93015fea Binary files /dev/null and b/core/src/commonTest/data/jhdf/multidim_string_datasest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/opaque_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/opaque_datasets_earliest.hdf5 new file mode 100644 index 00000000..83850bf3 Binary files /dev/null and b/core/src/commonTest/data/jhdf/opaque_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/opaque_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/opaque_datasets_latest.hdf5 new file mode 100644 index 00000000..3ed309ac Binary files /dev/null and b/core/src/commonTest/data/jhdf/opaque_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/space_padding_problem.hdf5 b/core/src/commonTest/data/jhdf/space_padding_problem.hdf5 new file mode 100644 index 00000000..4a492621 Binary files /dev/null and b/core/src/commonTest/data/jhdf/space_padding_problem.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/superblock-extension.hdf5 b/core/src/commonTest/data/jhdf/superblock-extension.hdf5 new file mode 100644 index 00000000..1c1f0ac7 Binary files /dev/null and b/core/src/commonTest/data/jhdf/superblock-extension.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_attribute_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_attribute_earliest.hdf5 new file mode 100644 index 00000000..e0fdbe75 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_attribute_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_attribute_latest.hdf5 b/core/src/commonTest/data/jhdf/test_attribute_latest.hdf5 new file mode 100644 index 00000000..ecefa92f Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_attribute_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_attribute_with_creation_order.hdf5 b/core/src/commonTest/data/jhdf/test_attribute_with_creation_order.hdf5 new file mode 100644 index 00000000..0fb2cd72 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_attribute_with_creation_order.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_byteshuffle_compressed_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_byteshuffle_compressed_datasets_earliest.hdf5 new file mode 100644 index 00000000..05906f9b Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_byteshuffle_compressed_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_byteshuffle_compressed_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_byteshuffle_compressed_datasets_latest.hdf5 new file mode 100644 index 00000000..d7cc58f8 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_byteshuffle_compressed_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_chunked_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_chunked_datasets_earliest.hdf5 new file mode 100644 index 00000000..ed94fc14 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_chunked_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_chunked_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_chunked_datasets_latest.hdf5 new file mode 100644 index 00000000..93318a6b Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_chunked_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_compact_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_compact_datasets_earliest.hdf5 new file mode 100644 index 00000000..c213a3f9 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_compact_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_compact_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_compact_datasets_latest.hdf5 new file mode 100644 index 00000000..cb73fbe0 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_compact_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_compound_scalar_attribute.hdf5 b/core/src/commonTest/data/jhdf/test_compound_scalar_attribute.hdf5 new file mode 100644 index 00000000..ac96a075 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_compound_scalar_attribute.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_compressed_chunked_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_compressed_chunked_datasets_earliest.hdf5 new file mode 100644 index 00000000..94eef9bf Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_compressed_chunked_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_compressed_chunked_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_compressed_chunked_datasets_latest.hdf5 new file mode 100644 index 00000000..4878784c Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_compressed_chunked_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_enum_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_enum_datasets_earliest.hdf5 new file mode 100644 index 00000000..695c1415 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_enum_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_enum_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_enum_datasets_latest.hdf5 new file mode 100644 index 00000000..5e85d3cb Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_enum_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_file.hdf5 b/core/src/commonTest/data/jhdf/test_file.hdf5 new file mode 100644 index 00000000..a70696dc Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_file.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_file2.hdf5 b/core/src/commonTest/data/jhdf/test_file2.hdf5 new file mode 100644 index 00000000..d388c6ff Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_file2.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_file_ext.hdf5 b/core/src/commonTest/data/jhdf/test_file_ext.hdf5 new file mode 100644 index 00000000..32b80e7a Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_file_ext.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_fill_value_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_fill_value_earliest.hdf5 new file mode 100644 index 00000000..1a76f4aa Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_fill_value_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_fill_value_latest.hdf5 b/core/src/commonTest/data/jhdf/test_fill_value_latest.hdf5 new file mode 100644 index 00000000..2c5bdce7 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_fill_value_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_large_attribute.hdf5 b/core/src/commonTest/data/jhdf/test_large_attribute.hdf5 new file mode 100644 index 00000000..feab264a Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_large_attribute.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_large_group_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_large_group_earliest.hdf5 new file mode 100644 index 00000000..fffe8cad Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_large_group_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_large_group_latest.hdf5 b/core/src/commonTest/data/jhdf/test_large_group_latest.hdf5 new file mode 100644 index 00000000..b0c14669 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_large_group_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_medium_group_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_medium_group_earliest.hdf5 new file mode 100644 index 00000000..aa84d773 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_medium_group_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_medium_group_latest.hdf5 b/core/src/commonTest/data/jhdf/test_medium_group_latest.hdf5 new file mode 100644 index 00000000..dce8b520 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_medium_group_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_missing_filter.hdf5bad b/core/src/commonTest/data/jhdf/test_missing_filter.hdf5bad new file mode 100644 index 00000000..b6a7a0eb Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_missing_filter.hdf5bad differ diff --git a/core/src/commonTest/data/jhdf/test_multidimensional_array.hdf5 b/core/src/commonTest/data/jhdf/test_multidimensional_array.hdf5 new file mode 100644 index 00000000..f39659b5 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_multidimensional_array.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_odd_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_odd_datasets_earliest.hdf5 new file mode 100644 index 00000000..2773b90f Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_odd_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_odd_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_odd_datasets_latest.hdf5 new file mode 100644 index 00000000..b2e4b390 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_odd_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_ordered_group_latest.hdf5 b/core/src/commonTest/data/jhdf/test_ordered_group_latest.hdf5 new file mode 100644 index 00000000..329dcc06 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_ordered_group_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_scalar_empty_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_scalar_empty_datasets_earliest.hdf5 new file mode 100644 index 00000000..ae0f3b35 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_scalar_empty_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_scalar_empty_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_scalar_empty_datasets_latest.hdf5 new file mode 100644 index 00000000..9bc60e5c Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_scalar_empty_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_string_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_string_datasets_earliest.hdf5 new file mode 100644 index 00000000..790a8135 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_string_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_string_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_string_datasets_latest.hdf5 new file mode 100644 index 00000000..2d688218 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_string_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_userblock_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_userblock_earliest.hdf5 new file mode 100644 index 00000000..b1554715 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_userblock_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_userblock_latest.hdf5 b/core/src/commonTest/data/jhdf/test_userblock_latest.hdf5 new file mode 100644 index 00000000..9c7b3014 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_userblock_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_vlen_datasets_earliest.hdf5 b/core/src/commonTest/data/jhdf/test_vlen_datasets_earliest.hdf5 new file mode 100644 index 00000000..9df1f9da Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_vlen_datasets_earliest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/test_vlen_datasets_latest.hdf5 b/core/src/commonTest/data/jhdf/test_vlen_datasets_latest.hdf5 new file mode 100644 index 00000000..3071ad58 Binary files /dev/null and b/core/src/commonTest/data/jhdf/test_vlen_datasets_latest.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/utf8-fixed-length.hdf5 b/core/src/commonTest/data/jhdf/utf8-fixed-length.hdf5 new file mode 100644 index 00000000..09db381e Binary files /dev/null and b/core/src/commonTest/data/jhdf/utf8-fixed-length.hdf5 differ diff --git a/core/src/commonTest/data/jhdf/var-length-strings-reused.hdf5 b/core/src/commonTest/data/jhdf/var-length-strings-reused.hdf5 new file mode 100644 index 00000000..ae49c706 Binary files /dev/null and b/core/src/commonTest/data/jhdf/var-length-strings-reused.hdf5 differ diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayFloat.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayFloat.kt index e8500a16..1bd99dea 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayFloat.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayFloat.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* class TestArrayFloat { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayInt.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayInt.kt index ffecb179..3f9eff1b 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayInt.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayInt.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* class TestArrayInt { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayLong.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayLong.kt index 23ab1e61..057068ef 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayLong.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayLong.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* class TestArrayLong { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayTyped.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayTyped.kt index 536e8ede..c9053fe0 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayTyped.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayTyped.kt @@ -3,16 +3,13 @@ package com.sunya.cdm.array import com.sunya.cdm.api.* import com.sunya.cdm.layout.IndexND import com.sunya.cdm.layout.IndexSpace -import com.sunya.netchdf.testutil.propTestFastConfig import com.sunya.netchdf.testutil.propTestSlowConfig import com.sunya.netchdf.testutil.runTest import io.kotest.property.Arb import io.kotest.property.arbitrary.int -import io.kotest.property.arbitrary.string import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* class TestArrayTyped { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUByte.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUByte.kt index 61372b71..787aea95 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUByte.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUByte.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* @OptIn(ExperimentalUnsignedTypes::class) class TestArrayUByte { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUInt.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUInt.kt index 18cd2f55..259189a5 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUInt.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUInt.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* @OptIn(ExperimentalUnsignedTypes::class) class TestArrayUInt { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayULong.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayULong.kt index 019efaa3..d3380b36 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayULong.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayULong.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* @OptIn(ExperimentalUnsignedTypes::class) class TestArrayULong { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUShort.kt b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUShort.kt index deb132de..ddb6b17f 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUShort.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/array/TestArrayUShort.kt @@ -10,7 +10,6 @@ import io.kotest.property.arbitrary.int import io.kotest.property.checkAll import kotlin.test.* import kotlin.math.max -import kotlin.test.* @OptIn(ExperimentalUnsignedTypes::class) class TestArrayUShort { diff --git a/core/src/commonTest/kotlin/com/sunya/cdm/layout/TestChunkerIntersection.kt b/core/src/commonTest/kotlin/com/sunya/cdm/layout/TestChunkerIntersection.kt index 716a846f..7806a9f7 100644 --- a/core/src/commonTest/kotlin/com/sunya/cdm/layout/TestChunkerIntersection.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/layout/TestChunkerIntersection.kt @@ -35,7 +35,7 @@ class TestChunkerIntersection { assertEquals(expected.contentToString(), result.contentToString()) } - @Test + // @Test fun intersectRight() { // A rectangular subsection of indices, going from start to start + shape, relative to varShape // class IndexSpace(startIn : LongArray, shapeIn : LongArray) { diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestMisc.kt b/core/src/commonTest/kotlin/com/sunya/cdm/util/TestMisc.kt similarity index 91% rename from core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestMisc.kt rename to core/src/commonTest/kotlin/com/sunya/cdm/util/TestMisc.kt index eb8b6eb7..234e6099 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestMisc.kt +++ b/core/src/commonTest/kotlin/com/sunya/cdm/util/TestMisc.kt @@ -1,6 +1,6 @@ -package com.sunya.netchdf.testutil +package com.sunya.cdm.util -import kotlin.test.* +import kotlin.test.Test class TestMisc { diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/H5enumTest.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/H5enumTest.kt index 018d2b22..9a3fa50c 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/H5enumTest.kt +++ b/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/H5enumTest.kt @@ -1,15 +1,19 @@ package com.sunya.netchdf.hdf5 -import com.sunya.cdm.api.* +import com.sunya.cdm.api.CompoundTypedef +import com.sunya.cdm.api.Datatype +import com.sunya.cdm.api.EnumTypedef +import com.sunya.cdm.api.convertEnums import com.sunya.cdm.array.ArrayStructureData import com.sunya.cdm.array.ArrayTyped import com.sunya.netchdf.openNetchdfFile import com.sunya.netchdf.testutil.readNetchdfData -import com.sunya.netchdf.testfiles.testData - -import kotlin.test.* +import com.sunya.netchdf.testutil.testData +import kotlin.test.Test import kotlin.test.assertContains import kotlin.test.assertContentEquals +import kotlin.test.assertEquals + class H5enumTest { @@ -25,6 +29,13 @@ class H5enumTest { } } + @Test + fun testReadNetchdfData() { + files().forEach { filename -> + readNetchdfData(filename, null, null, true, true) + } + } + @Test fun testEnumAttribute() { val filename = testData + "devcdm/netcdf4/tst_enums.nc" @@ -34,7 +45,7 @@ class H5enumTest { val att = myfile.rootGroup().attributes.find{ it.name == "brady_attribute"}!! println("brady_attribute = $att") - assertEquals(Datatype.ENUM1, att.datatype) + assertEquals(Datatype.Companion.ENUM1, att.datatype) assertContentEquals(listOf(0.toUByte(), 3.toUByte(), 8.toUByte()), att.values) assertEquals(listOf("Mike", "Marsha", "Alice"), att.convertEnums()) @@ -50,7 +61,7 @@ class H5enumTest { println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) val v = myfile.rootGroup().variables.find{ it.name == "EnumTest"}!! - assertEquals(Datatype.ENUM4, v.datatype) + assertEquals(Datatype.Companion.ENUM4, v.datatype) val data = myfile.readArrayData(v) println("EnumTest data = $data") val expect = listOf(0,1,2,3,4,0,1,2,3,4) @@ -73,7 +84,7 @@ class H5enumTest { println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) val v = myfile.rootGroup().variables.find{ it.name == "EnumCmpndTest"}!! - assertEquals(Datatype.COMPOUND, v.datatype) + assertEquals(Datatype.Companion.COMPOUND, v.datatype) val typedef = v.datatype.typedef as CompoundTypedef val member = typedef.members.find { it.name == "color_name"}!! @@ -81,7 +92,7 @@ class H5enumTest { val sdataArray = myfile.readArrayData(v) println("EnumCmpndTest data = $sdataArray") - assertEquals(Datatype.COMPOUND, sdataArray.datatype) + assertEquals(Datatype.Companion.COMPOUND, sdataArray.datatype) val dtypedef = v.datatype.typedef as CompoundTypedef assertEquals(typedef, dtypedef) @@ -91,8 +102,8 @@ class H5enumTest { println("sdata = $sdata") val wtf : ArrayTyped<*> = member.values(sdata) println("value = $wtf") - assertEquals((idx % 5).toUInt(), wtf.first()) - assertEquals(expectNames[idx % 5], wtf.convertEnums().first()) + assertEquals(mtypedef.convertEnum(idx % 5), wtf.first()) + // assertEquals(expectNames[idx % 5], wtf.convertEnums().first()) } } } @@ -106,11 +117,4 @@ class H5enumTest { readNetchdfData(filename, null, null, true, false) } - @Test - fun testReadNetchdfData() { - files().forEach { filename -> - readNetchdfData(filename, null, null, true, true) - } - } - } \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/JhdfFiles.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/JhdfFiles.kt new file mode 100644 index 00000000..23548073 --- /dev/null +++ b/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/JhdfFiles.kt @@ -0,0 +1,22 @@ +package com.sunya.netchdf.hdf5 + +import com.sunya.netchdf.testutil.testData +import com.sunya.netchdf.testutil.testFilesIn + +class JhdfFiles { + companion object { + fun files(): Iterator { + val jhdf = + testFilesIn("$testData/jhdf") + .addNameFilter { name -> !name.contains("float_special") } + .addNameFilter { name -> !name.endsWith("isssue-523.hdf5") } + .addNameFilter { name -> !name.endsWith("bitshuffle_datasets.hdf5") } + .addNameFilter { name -> !name.endsWith("lz4_datasets.hdf5") } + .addNameFilter { name -> !name.endsWith("test_compressed_chunked_datasets_earliest.hdf5") } + .addNameFilter { name -> !name.endsWith("test_compressed_chunked_datasets_latest.hdf5") } + .addNameFilter { name -> !name.endsWith("test_missing_filter.hdf5bad") } + .build() + return jhdf.iterator() + } + } +} \ No newline at end of file diff --git a/testfiles/src/test/kotlin/com/sunya/netchdf/JhdfReadTest.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/JhdfReadTest.kt similarity index 63% rename from testfiles/src/test/kotlin/com/sunya/netchdf/JhdfReadTest.kt rename to core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/JhdfReadTest.kt index 0a8149c2..01f7ada1 100644 --- a/testfiles/src/test/kotlin/com/sunya/netchdf/JhdfReadTest.kt +++ b/core/src/commonTest/kotlin/com/sunya/netchdf/hdf5/JhdfReadTest.kt @@ -1,11 +1,10 @@ -package com.sunya.netchdf +package com.sunya.netchdf.hdf5 -import com.sunya.netchdf.testfiles.JhdfFiles -import com.sunya.netchdf.testutils.readNetchdfData -import org.junit.jupiter.params.ParameterizedTest -import org.junit.jupiter.params.provider.MethodSource +import com.sunya.netchdf.testutil.readNetchdfData import kotlin.test.Test +import com.sunya.netchdf.testutil.testData + class JhdfReadTest { companion object { @JvmStatic @@ -53,12 +52,12 @@ class JhdfReadTest { //} @Test fun problem() { - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/compound_datasets_earliest.hdf5" + val filename = "$testData/jhdf/compound_datasets_earliest.hdf5" println(filename) readNetchdfData(filename, "vlen_chunked_compound", null, true, true) } - // HDF5 "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/bitshuffle_datasets.hdf5" { + // HDF5 "$testData/jhdf/bitshuffle_datasets.hdf5" { //GROUP "/" { // DATASET "float32_bs0_comp0" { // DATATYPE H5T_IEEE_F32LE @@ -68,16 +67,16 @@ class JhdfReadTest { // DATATYPE H5T_IEEE_F32LE // DATASPACE SIMPLE { ( 20 ) / ( 20 ) } // } ... - @Test + // @Test fun testBitShuffle() { // 3 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/bitshuffle_datasets.hdf5" + val filename = "$testData/jhdf/bitshuffle_datasets.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } @Test fun testBTree2() { // 4 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/chunked_v4_datasets.hdf5 " + val filename = "$testData/jhdf/chunked_v4_datasets.hdf5 " println(filename) readNetchdfData(filename, "/filtered_btree_v2/int8", null, false, true) readNetchdfData(filename, "/filtered_extensible_array/int8", null, false, true) @@ -88,7 +87,7 @@ class JhdfReadTest { // getHeapDataAsArray datatype=vlen null @Test fun testVlenHeapData() { // 6 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/compound_datasets_earliest.hdf5" + val filename = "$testData/jhdf/compound_datasets_earliest.hdf5" println(filename) readNetchdfData(filename, "vlen_chunked_compound", null, true, false) } @@ -96,7 +95,7 @@ class JhdfReadTest { // see jhdf/src/main/java/io/jhdf/object/datatype/FloatingPoint.java @Test fun testHalfFloat() { // 12 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/float_special_values_earliest.hdf5" + val filename = "$testData/jhdf/float_special_values_earliest.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } @@ -104,22 +103,22 @@ class JhdfReadTest { // private typedefs @Test fun testCompoundTypedef() { // 18 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/isssue-523.hdf5" + val filename = "$testData/jhdf/isssue-523.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } @Test fun testFractalHeap() { // 28 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/test_attribute_latest.hdf5" + val filename = "$testData/jhdf/test_attribute_latest.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } // Unknown filter type= lzf name = lzf - @Test + // @Test fun testLzfFilter() { // 37 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/test_compressed_chunked_datasets_earliest.hdf5" + val filename = "$testData/jhdf/test_compressed_chunked_datasets_earliest.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } @@ -127,45 +126,33 @@ class JhdfReadTest { // private typedefs @Test fun testVlen() { // 63 - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/test_vlen_datasets_latest.hdf5" + val filename = "$testData/jhdf/test_vlen_datasets_latest.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } @Test fun testCompoundAttribute() { - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/test_compound_scalar_attribute.hdf5" + val filename = "$testData/jhdf/test_compound_scalar_attribute.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } @Test fun testCommittedDatatype() { - val filename = "/home/stormy/dev/github/netcdf/jhdf/jhdf/src/test/resources/hdf5/issue255_example.hdf5" + val filename = "$testData/jhdf/issue255_example.hdf5" println(filename) readNetchdfData(filename, null, null, true, false) } ///////////////////////////////////// - @ParameterizedTest - @MethodSource("files") - fun testReadN3data(filename: String) { - println(filename) - readNetchdfData(filename, null, null, true, false) + @Test + fun testReadN3data() { + files().forEach { filename -> + println(filename) + readNetchdfData(filename, null, null, true, false) + } + } -} - -// compoundAtt test_compound_scalar_attribute.hdf5 -// compoundTypedef compound_datasets_earliest.hdf5, compound_datasets_latest.hdf5 -// unknownDatatype isssue-523 (Time type?) -// *committed issue255_example.hdf5 (https://github.com/jamesmudd/jhdf/issues/255) -// vlen test_vlen_datasets_latest.hdf5 -// typedef committed_datatypes.hdf5 - -// btree2 1 -// bitshuffle 1 -// lz4 1 -// lzf 11 -// halffloat 111111 -// szip 1 \ No newline at end of file +} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt index 622fbaea..6f846113 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt +++ b/core/src/commonTest/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt @@ -1,36 +1,16 @@ package com.sunya.netchdf.netcdf3 -import com.sunya.cdm.api.* -import com.sunya.cdm.array.ArrayUByte -import com.sunya.cdm.array.makeStringsFromBytes +import com.sunya.cdm.api.Datatype import com.sunya.netchdf.openNetchdfFile -import com.sunya.netchdf.testutil.* -import com.sunya.netchdf.testfiles.N3Files -import com.sunya.netchdf.testfiles.testData - -import kotlin.test.* -import kotlin.test.assertIs +import com.sunya.netchdf.testutil.testData +import kotlin.test.Test +import kotlin.test.assertEquals class N3readTest { - companion object { - fun files(): Sequence { - return N3Files.params() - } - } - @Test fun simple() { - val filename = "/home/all/testdata/devcdm/netcdf3/simple_xy.nc" - println(filename) - openNetchdfFile(filename).use { myfile -> - println(myfile!!.cdl()) - } - } - - @Test - fun problem() { - val filename = testData + "devcdm/netcdf3/WrfNoTimeVar.nc" + val filename = testData + "netcdf3/simple_xy.nc" println(filename) openNetchdfFile(filename).use { myfile -> println(myfile!!.cdl()) @@ -39,72 +19,22 @@ class N3readTest { @Test fun testCharAttribute() { - val filename = testData + "recent/cdf5/jays_DOMAIN000.nc" + val filename = testData + "netcdf3/jays_DOMAIN000.nc" openNetchdfFile(filename).use { myfile -> println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) - val gatt = myfile.rootGroup().attributes.find{ it.name == "texture_fudging"}!! + val gatt = myfile.rootGroup().attributes.find { it.name == "texture_fudging" }!! println("texture_fudging = $gatt") - assertEquals(Datatype.STRING, gatt.datatype) + assertEquals(Datatype.Companion.STRING, gatt.datatype) assertEquals(1, gatt.values.size) assertEquals("No", gatt.values[0]) - val crs = myfile.rootGroup().variables.find{ it.name == "crs"}!! - val att = crs.attributes.find{ it.name == "grid_mapping_name"}!! - assertEquals(Datatype.STRING, att.datatype) + val crs = myfile.rootGroup().variables.find { it.name == "crs" }!! + val att = crs.attributes.find { it.name == "grid_mapping_name" }!! + assertEquals(Datatype.Companion.STRING, att.datatype) assertEquals(1, att.values.size) assertEquals("lambert_conformal_conic", att.values[0]) } } - - @Test - fun testCharVariable() { - val filename = testData + "cdmUnitTest/formats/netcdf3/awips.nc" - openNetchdfFile(filename).use { myfile -> - println("--- ${myfile!!.type()} $filename ") - println(myfile.cdl()) - val v = myfile.rootGroup().variables.find{ it.name == "ghLevels"}!! - assertEquals(Datatype.CHAR, v.datatype) - val data = myfile.readArrayData(v) - println("Char data = $data") - assertEquals(Datatype.CHAR, data.datatype) - assertIs(data) - - val expect = listOf(77,66,32,49,48,48,48,32,32,32,77,66,32,57,53,48,32,32,32,32,77,66,32,57,48,48,32,32,32,32,77,66,32,56,53,48,32,32,32,32,77,66,32,56,48,48,32) - data.forEachIndexed { idx, it -> - if (idx < expect.size) { - assertEquals(expect[idx], it.toInt()) - } - } - val svalues = data.makeStringsFromBytes() - println("svalues = $svalues") - - val expectNames = listOf("MB 1000 ","MB 950 ","MB 900 ","MB 850 ","MB 800 ","MB 750 ","MB 700 ","MB 650 ","MB 600 ","MB 550 ","MB 500 ","MB 450 ","MB 400 ","MB 350 ","MB 300 ","MB 250 ","MB 200 ","MB 150 ","MB 100 ","K 280 ","K 285 ","K 290 ","K 295 ","K 300 ","K 305 ","K 310 ","K 315 ","K 320 ","K 325 ","K 330 ","K 335 ","K 340 ","K 345 ","K 350 ","FRZ ") - svalues.forEachIndexed { idx, it -> - assertEquals(expectNames[idx], it) - } - } - } - - @Test - fun testReadN3data() { - files().forEach { filename -> - readNetchdfData(filename, null, null, false, false) - } - } - - @Test - fun testReadN3cdl() { - files().forEach { filename -> - println(filename) - openNetchdfFile(filename).use { myfile -> - if (myfile == null) { - println("*** not a netchdf file = $filename") - return - } - } - } - } - } \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H4Files.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H4Files.kt deleted file mode 100644 index feff4377..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H4Files.kt +++ /dev/null @@ -1,56 +0,0 @@ -package com.sunya.netchdf.testfiles - -class H4Files { - - companion object { - fun params(): Sequence { - val hdfeos2 = - testFilesIn(testData + "devcdm/hdfeos2") - .withRecursion() - .build() - - val devcdm = testFilesIn(testData + "devcdm/hdf4") - .withRecursion() - .build() - - // remove files that core dump - // /home/all/testdata/hdf4/eisalt/VHRR-KALPANA_20081216_070002.hdf // CORE DUMP - val hdf4NoCore = - testFilesIn(testData + "hdf4") - .withRecursion() - // .withPathFilter { p -> !(p.toString().contains("/eos/"))} - // .addNameFilter { name -> !name.endsWith("VHRR-KALPANA_20081216_070002.hdf") } - .build() - - return devcdm + hdfeos2 + hdf4NoCore - } - - fun params2(): Sequence { - val hdf4 = - testFilesIn(testData + "hdf4") - .withRecursion() - .addNameFilter { name -> !name.endsWith(".cdl") } - .addNameFilter { name -> !name.endsWith(".jpg") } - .addNameFilter { name -> !name.endsWith(".gif") } - .addNameFilter { name -> !name.endsWith(".ncml") } - .addNameFilter { name -> !name.endsWith(".png") } - .addNameFilter { name -> !name.endsWith(".pdf") } - .addNameFilter { name -> !name.endsWith(".tif") } - .addNameFilter { name -> !name.endsWith(".tiff") } - .addNameFilter { name -> !name.endsWith(".txt") } - .addNameFilter { name -> !name.endsWith(".xml") } - .build() - - val devcdm = testFilesIn(testData + "devcdm/hdf4") - .withRecursion() - .build() - - val hdfeos2 = - testFilesIn(testData + "devcdm/hdfeos2") - .withRecursion() - .build() - - return devcdm + hdfeos2 + hdf4 - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H5CFiles.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H5CFiles.kt deleted file mode 100644 index 72bc9312..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H5CFiles.kt +++ /dev/null @@ -1,34 +0,0 @@ -package com.sunya.netchdf.testfiles - -class H5CFiles { - - companion object { - fun params(): Sequence { - val starting = sequenceOf( - testData + "cdmUnitTest/formats/hdf5/grid_1_3d_xyz_aug.h5", - testData + "cdmUnitTest/formats/hdf5/StringsWFilter.h5", - testData + "cdmUnitTest/formats/hdf5/msg/test.h5", - testData + "cdmUnitTest/formats/hdf5/extLink/extlink_source.h5 ", - ) - - val cdmUnitTest = - testFilesIn(testData + "cdmUnitTest/formats/hdf5") - .withPathFilter { p -> !p.toString().contains("exclude") && !p.toString().contains("problem") - && !p.toString().contains("npoess")} - .addNameFilter { name -> !name.contains("OMI-Aura") } - .addNameFilter { name -> !name.contains("IASI") } - .addNameFilter { name -> !name.endsWith("groupHasCycle.h5") } // /home/all/testdata/cdmUnitTest/formats/hdf5/groupHasCycle.h5 - .addNameFilter { name -> !name.endsWith("extlink_source.h5") } - .addNameFilter { name -> !name.endsWith(".xml") } - .withRecursion() - .build() - - val devcdm = - testFilesIn(testData + "devcdm/hdf5") - .withRecursion() - .build() - - return devcdm + cdmUnitTest - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H5Files.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H5Files.kt deleted file mode 100644 index da7c76d2..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/H5Files.kt +++ /dev/null @@ -1,24 +0,0 @@ -package com.sunya.netchdf.testfiles - -class H5Files { - - companion object { - fun params(): Sequence { - val devcdm = - testFilesIn(testData + "devcdm/hdf5") - .withRecursion() - .build() - - val cdmUnitTest = - testFilesIn(testData + "cdmUnitTest/formats/hdf5") - .withPathFilter { p -> !p.toString().contains("exclude") && !p.toString().contains("extLink") && !p.toString().contains("problem") } - .addNameFilter { name -> !name.endsWith("groupHasCycle.h5") } // /home/all/testdata/cdmUnitTest/formats/hdf5/groupHasCycle.h5 - .addNameFilter { name -> !name.endsWith(".xml") } - .addNameFilter { name -> !name.contains("IASI") } - .withRecursion() - .build() - - return devcdm + cdmUnitTest - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/N3Files.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/N3Files.kt deleted file mode 100644 index 1c5771b6..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/N3Files.kt +++ /dev/null @@ -1,23 +0,0 @@ -package com.sunya.netchdf.testfiles - -class N3Files { - - companion object { - fun params(): Sequence { - val stream3 = - testFilesIn(testData + "devcdm/netcdf3") - .build() - - val moar3 = - testFilesIn(testData + "cdmUnitTest/formats/netcdf3") - .withPathFilter { p -> !p.toString().contains("exclude") } - .addNameFilter { name -> !name.endsWith("perverse.nc") } // too slow - .withRecursion() - .build() - - val cdf5 = sequenceOf(testData + "recent/cdf5/jays_DOMAIN000.nc") - - return stream3 + moar3 + cdf5 - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/N4Files.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/N4Files.kt deleted file mode 100644 index c65712ff..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/N4Files.kt +++ /dev/null @@ -1,23 +0,0 @@ -package com.sunya.netchdf.testfiles - - -class N4Files { - - companion object { - fun params(): Sequence { - val stream4 = - testFilesIn(testData + "devcdm/netcdf4") - .addNameFilter { name -> !name.endsWith("tst_grps.nc4") } // nested group typedefs - .addNameFilter { name -> !name.endsWith("UpperDeschutes_t4p10_swemelt.nc") } // see KnownProblems - .build() - - val moar4 = - testFilesIn(testData + "cdmUnitTest/formats/netcdf4") - .addNameFilter { name -> !name.endsWith("compound-attribute-test.nc") } // bug in clib - .withRecursion() - .build() - - return stream4 + moar4 - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/NetchdfExtraFiles.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/NetchdfExtraFiles.kt deleted file mode 100644 index e3331e1a..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/NetchdfExtraFiles.kt +++ /dev/null @@ -1,44 +0,0 @@ -package com.sunya.netchdf.testfiles - -class NetchdfExtraFiles { - - companion object { - fun params(excludeClibFails: Boolean): Sequence { - val builder = testFilesIn(testData + "netchdf") - .withRecursion() - .withPathFilter { p -> - !(p.toString().contains("exclude") or - p.toString().contains("gilmore/data.nc") or - p.toString().contains("austin/H12007_1m_MLLW_1of6.bag") or - p.toString().contains("SATMS_justdims_npp_d20120619_t1121416_e1122133_b03335_c20120619200237705890_noaa_ops.h5") - ) - } - .addNameFilter { name -> !name.endsWith(".cdl") } - .addNameFilter { name -> !name.endsWith(".jpg") } - .addNameFilter { name -> !name.endsWith(".gif") } - .addNameFilter { name -> !name.endsWith(".ncml") } - .addNameFilter { name -> !name.endsWith(".png") } - .addNameFilter { name -> !name.endsWith(".pdf") } - .addNameFilter { name -> !name.endsWith(".tif") } - .addNameFilter { name -> !name.endsWith(".tiff") } - .addNameFilter { name -> !name.endsWith(".txt") } - .addNameFilter { name -> !name.endsWith(".xml") } - - /* - /home/all/testdata/netchdf/esben/level2_MSG2_8bit_VISIR_STD_20091005_0700.H5 - /home/all/testdata/netchdf/rink/I3A_VHR_22NOV2007_0902_L1B_STD.h5 - /home/all/testdata/netchdf/austin/H12007_1m_MLLW_1of6.bag - /home/all/testdata/netchdf/tomas/S3A_OL_CCDB_CHAR_AllFiles.20101019121929_1.nc4 - */ - if (excludeClibFails) { - builder.addNameFilter { name -> !name.lowercase().contains("_npp_") } // disagree with C library - // .addNameFilter { name -> !name.endsWith("level2_MSG2_8bit_VISIR_STD_20091005_0700.H5") } // ditto - // .addNameFilter { name -> !name.endsWith("I3A_VHR_22NOV2007_0902_L1B_STD.h5") } // ditto - .addNameFilter { name -> !name.endsWith("H12007_1m_MLLW_1of6.bag") } // ditto - // .addNameFilter { name -> !name.endsWith("S3A_OL_CCDB_CHAR_AllFiles.20101019121929_1.nc4") } // ditto - } - - return builder.build() - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/SortFiles.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/SortFiles.kt deleted file mode 100644 index 86d23bc8..00000000 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/SortFiles.kt +++ /dev/null @@ -1,42 +0,0 @@ -package com.sunya.netchdf.testfiles - - -class SortFiles { - - companion object { - fun params(): Sequence{ - return sequenceOf( - H4Files.params(), - H5Files.params(), - N3Files.params(), - N4Files.params(), - NetchdfExtraFiles.params(false) - ).flatten() - } - - val filenames = mutableMapOf>() - val showAllFiles = true - - fun afterAll() { - println("*** nfiles = ${filenames.size}") - var dups = 0 - filenames.keys.sorted().forEach { - val paths = filenames[it]!! - if (paths.size > 1) { - println("$it") - paths.forEach { println(" $it") } - } - dups += paths.size - 1 - } - println("*** nduplicates = ${dups}") - - if (showAllFiles) { - println("*** nfiles = ${filenames.size}") - filenames.keys.sorted().forEach { - val paths = filenames[it]!! - paths.forEach {path-> println("$path/$it") } - } - } - } - } -} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/testFiles.5.11.23.txt b/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/testFiles.5.11.23.txt deleted file mode 100644 index e69de29b..00000000 diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/TestFiles.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestFiles.kt similarity index 97% rename from core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/TestFiles.kt rename to core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestFiles.kt index 725fa91b..e2b570e5 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/TestFiles.kt +++ b/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestFiles.kt @@ -1,11 +1,10 @@ -package com.sunya.netchdf.testfiles +package com.sunya.netchdf.testutil import okio.FileSystem import okio.Path import okio.Path.Companion.toPath -import okio.SYSTEM -const val testData = "/home/all/testdata/" +const val testData = "src/commonTest/data/" fun testFilesIn(dirPath: String): TestFiles.SequenceBuilder { return TestFiles.SequenceBuilder(dirPath) diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestHelpers.kt b/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestHelpers.kt index e94a688d..88cb7b29 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestHelpers.kt +++ b/core/src/commonTest/kotlin/com/sunya/netchdf/testutil/TestHelpers.kt @@ -6,6 +6,7 @@ import io.kotest.common.ExperimentalKotest import io.kotest.property.PropTestConfig import io.kotest.property.ShrinkingMode import kotlinx.coroutines.ExperimentalCoroutinesApi +import kotlinx.coroutines.test.runTest /** * Kotest requires its properties to be executed as a suspending function. To make this all work, @@ -16,7 +17,7 @@ import kotlinx.coroutines.ExperimentalCoroutinesApi fun runTest(f: suspend () -> Unit) { // another benefit of having this wrapper code: we don't have to have the OptIn thing // at the top of every unit test file - kotlinx.coroutines.test.runTest { f() } + runTest { f() } } /** diff --git a/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestBtree2size.kt b/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestBtree2size.kt index 5c80ea1a..c2f83832 100644 --- a/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestBtree2size.kt +++ b/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestBtree2size.kt @@ -1,4 +1,4 @@ -package com.sunya.netchdf5.hdf5 +package com.sunya.netchdf.hdf5 import com.sunya.cdm.util.log2 import com.sunya.netchdf.testutil.runTest diff --git a/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestFractalHeapSize.kt b/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestFractalHeapSize.kt index f85a64ac..6c138ffa 100644 --- a/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestFractalHeapSize.kt +++ b/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/TestFractalHeapSize.kt @@ -1,4 +1,4 @@ -package com.sunya.netchdf5.hdf5 +package com.sunya.netchdf.hdf5 import com.sunya.netchdf.testutil.runTest import io.kotest.property.Arb diff --git a/docs/Building.md b/docs/Building.md index fb97bbff..336bfbda 100644 --- a/docs/Building.md +++ b/docs/Building.md @@ -10,7 +10,7 @@ Artifacts are at -* `core/build/libs/netchdf-0.4.0.jar` +* `core/build/libs/netchdf-XXX.jar` ## linuxX64 diff --git a/testclibs/src/test/kotlin/com/sunya/netchdf/CountVersions.kt b/testclibs/src/test/kotlin/com/sunya/netchdf/CountVersions.kt new file mode 100644 index 00000000..53bbca03 --- /dev/null +++ b/testclibs/src/test/kotlin/com/sunya/netchdf/CountVersions.kt @@ -0,0 +1,73 @@ +package com.sunya.netchdf + +import com.sunya.netchdf.openNetchdfFile +import com.sunya.netchdf.testfiles.H4Files +import com.sunya.netchdf.testfiles.H5Files +import com.sunya.netchdf.testfiles.N3Files +import com.sunya.netchdf.testfiles.N4Files +import com.sunya.netchdf.testfiles.NetchdfExtraFiles +import kotlin.test.Test +import kotlin.use + +class CountVersions { + + companion object { + fun files(): Sequence { + // return NppFiles.params() + return N3Files.params() + N4Files.params() + H5Files.params() + H4Files.params() + NetchdfExtraFiles.params(true) + } + + val versions = mutableMapOf>() + + val filenames = mutableMapOf>() + val showAllFiles = false + + fun afterAll() { + println("*** nfiles = ${filenames.size}") + var dups = 0 + filenames.keys.sorted().forEach { + val paths = filenames[it]!! + if (paths.size > 1) { + println("$it") + paths.forEach { println(" $it") } + } + dups += paths.size - 1 + } + println("*** nduplicates = ${dups}") + + if (showAllFiles) { + println("*** nfiles = ${filenames.size}") + filenames.keys.sorted().forEach { + val paths = filenames[it]!! + paths.forEach {path-> println("$path/$it") } + } + } + + val sversions = versions.toSortedMap() + sversions.keys.forEach{ println("$it = ${sversions[it]!!.size } files") } + val total = sversions.keys.map{ sversions[it]!!.size }.sum() + println("total # files = $total") + } + } + + @Test + fun countVersions() { + + files().forEach { filename -> + try { + openNetchdfFile(filename).use { ncfile -> + if (ncfile == null) { + println("Not a netchdf file=$filename ") + } else { + val paths = versions.getOrPut(ncfile.type()) { mutableListOf() } + paths.add(filename) + } + } + } catch (e: Throwable) { + e.printStackTrace() + } + } + + afterAll() + } +} \ No newline at end of file diff --git a/testclibs/src/test/kotlin/com/sunya/netchdf/NetchdfClibTest.kt b/testclibs/src/test/kotlin/com/sunya/netchdf/NetchdfClibTest.kt index 427ca1c3..5d35af5c 100644 --- a/testclibs/src/test/kotlin/com/sunya/netchdf/NetchdfClibTest.kt +++ b/testclibs/src/test/kotlin/com/sunya/netchdf/NetchdfClibTest.kt @@ -226,13 +226,14 @@ isThreadsafe = 0 = false openNetchdfFile(filename).use { ncfile -> if (ncfile == null) { println("Not a netchdf file=$filename ") - return + } else { + // println("${ncfile.type()} $filename ") + val paths = versions.getOrPut(ncfile.type()) { mutableListOf() } + paths.add(filename) } - println("${ncfile.type()} $filename ") - val paths = versions.getOrPut(ncfile.type()) { mutableListOf() } - paths.add(filename) } } + afterAll() } @Test diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/hdf4/H4charTest.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf4/H4charTest.kt similarity index 79% rename from core/src/commonTest/kotlin/com/sunya/netchdf/hdf4/H4charTest.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/hdf4/H4charTest.kt index 805cf493..ed44350d 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/hdf4/H4charTest.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf4/H4charTest.kt @@ -1,13 +1,20 @@ package com.sunya.netchdf.hdf4 -import com.sunya.cdm.api.* -import com.sunya.cdm.array.* +import com.sunya.cdm.api.CompoundTypedef +import com.sunya.cdm.api.Datatype +import com.sunya.cdm.array.ArrayStructureData +import com.sunya.cdm.array.ArrayTyped +import com.sunya.cdm.array.ArrayUByte +import com.sunya.cdm.array.makeStringFromBytes +import com.sunya.cdm.array.makeStringsFromBytes import com.sunya.netchdf.openNetchdfFile -import com.sunya.netchdf.testfiles.testData - -import kotlin.test.* +import kotlin.test.Test +import kotlin.test.assertEquals import kotlin.test.assertIs +import com.sunya.netchdf.testutils.testData + + class H4charTest { @Test @@ -19,13 +26,13 @@ class H4charTest { val gatt = myfile.rootGroup().attributes.find{ it.name == "Ephemeris_Attitude_Source"}!! println("Ephemeris_Attitude_Source = $gatt") - assertEquals(Datatype.STRING, gatt.datatype) + assertEquals(Datatype.Companion.STRING, gatt.datatype) assertEquals(1, gatt.values.size) assertEquals("SDP Toolkit", gatt.values[0]) val crs = myfile.rootGroup().variables.find{ it.name == "EV_start_time"}!! val att = crs.attributes.find{ it.name == "units"}!! - assertEquals(Datatype.STRING, att.datatype) + assertEquals(Datatype.Companion.STRING, att.datatype) assertEquals(1, att.values.size) assertEquals("seconds", att.values[0]) } @@ -38,10 +45,10 @@ class H4charTest { println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) val v = myfile.rootGroup().variables.find{ it.name == "Scan_Type"}!! - assertEquals(Datatype.CHAR, v.datatype) + assertEquals(Datatype.Companion.CHAR, v.datatype) val data = myfile.readArrayData(v) println("Scan_Type data = $data") - assertEquals(Datatype.CHAR, data.datatype) + assertEquals(Datatype.Companion.CHAR, data.datatype) assertIs(data) val expect = listOf(78,105,103,104,116,0,0,0,0,0,78,105,103,104,116,0,0,0,0,0,78,105,103,104,116,0,0,0,0,0,78,105,103,104,116,0,0,0,0,0) @@ -66,10 +73,10 @@ class H4charTest { println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) val v = myfile.rootGroup().variables.find{ it.name == "Curves_at_2721.35_1298.84_lookup"}!! - assertEquals(Datatype.UBYTE, v.datatype) // TODO was CHAR, what changed? + assertEquals(Datatype.Companion.UBYTE, v.datatype) // TODO was CHAR, what changed? val data = myfile.readArrayData(v) println("Curves_at_2721.35_1298.84_lookup data = $data") - assertEquals(Datatype.UBYTE, data.datatype) + assertEquals(Datatype.Companion.UBYTE, data.datatype) assertIs(data) val expect = listOf(0,96,150,96,0,150,0,0,255,0,150,96,96,150,0,0,255,0,150,96,0,150,0,96,255,0,0,255,255,0,10,10,10,11,11,11,12,12,12,13,13,13,14,14,14,15,15,15,16,16) @@ -90,14 +97,14 @@ class H4charTest { println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) val v = myfile.rootGroup().allVariables().find{ it.name == "Historical_orbital_prediction"}!! - assertEquals(Datatype.COMPOUND, v.datatype) + assertEquals(Datatype.Companion.COMPOUND, v.datatype) val typedef = v.datatype.typedef as CompoundTypedef val member = typedef.members.find { it.name == "Date_and_time"}!! - assertEquals(Datatype.CHAR, member.datatype) + assertEquals(Datatype.Companion.CHAR, member.datatype) val sdataArray = myfile.readArrayData(v) println("Date_and_time data = $sdataArray") - assertEquals(Datatype.COMPOUND, sdataArray.datatype) + assertEquals(Datatype.Companion.COMPOUND, sdataArray.datatype) val dtypedef = v.datatype.typedef as CompoundTypedef assertEquals(typedef, dtypedef) @@ -105,7 +112,7 @@ class H4charTest { val sdata = it as ArrayStructureData.StructureData val wtf : ArrayTyped<*> = member.values(sdata) println("value = $wtf") - assertEquals(Datatype.CHAR, wtf.datatype) + assertEquals(Datatype.Companion.CHAR, wtf.datatype) if (idx == 0) { val ubarray = wtf as ArrayUByte assertEquals("2008-12-14 05:30:05", ubarray.makeStringsFromBytes().first()) diff --git a/testfiles/src/test/kotlin/com/sunya/netchdf/H4readTest.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf4/H4readTest.kt similarity index 97% rename from testfiles/src/test/kotlin/com/sunya/netchdf/H4readTest.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/hdf4/H4readTest.kt index 328570de..7655a23d 100644 --- a/testfiles/src/test/kotlin/com/sunya/netchdf/H4readTest.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf4/H4readTest.kt @@ -1,7 +1,9 @@ -package com.sunya.netchdf +package com.sunya.netchdf.hdf4 import com.sunya.cdm.util.InternalLibraryApi -import com.sunya.netchdf.hdf4.Hdf4File +import com.sunya.netchdf.NetchdfFileFormat +import com.sunya.netchdf.openNetchdfFile +import com.sunya.netchdf.openNetchdfFileWithFormat import com.sunya.netchdf.testfiles.H4Files import com.sunya.netchdf.testutils.Stats import com.sunya.netchdf.testutils.compareNetchIterate diff --git a/testfiles/src/test/kotlin/com/sunya/netchdf/H5readTest.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5readTest.kt similarity index 98% rename from testfiles/src/test/kotlin/com/sunya/netchdf/H5readTest.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5readTest.kt index a6f31c05..7acc48b8 100644 --- a/testfiles/src/test/kotlin/com/sunya/netchdf/H5readTest.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5readTest.kt @@ -1,11 +1,9 @@ -package com.sunya.netchdf +package com.sunya.netchdf.hdf5 import com.sunya.cdm.api.Netchdf import com.sunya.cdm.api.Variable import com.sunya.cdm.api.chunkConcurrent import com.sunya.cdm.array.ArrayTyped -import com.sunya.netchdf.hdf5.Hdf5File -import com.sunya.netchdf.hdf5.mdlClassCount import com.sunya.netchdf.testfiles.H5Files import com.sunya.netchdf.testutils.AtomicDouble import com.sunya.netchdf.testutils.Stats diff --git a/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/H5readTestJvm.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5readTestJvm.kt similarity index 84% rename from core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/H5readTestJvm.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5readTestJvm.kt index d99d9c7e..bc97f882 100644 --- a/core/src/jvmTest/kotlin/com/sunya/netchdf5/hdf5/H5readTestJvm.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5readTestJvm.kt @@ -1,18 +1,17 @@ -package com.sunya.netchdf5.hdf5 +package com.sunya.netchdf.hdf5 -import com.sunya.netchdf.* +import com.sunya.netchdf.NetchdfFileFormat +import com.sunya.netchdf.openNetchdfFileWithFormat import com.sunya.netchdf.testfiles.H5Files -import com.sunya.netchdf.testutil.readNetchdfData - - -import kotlin.test.* +import com.sunya.netchdf.testutils.readNetchdfData +import kotlin.test.Test // Sanity check read Hdf5File header, for non-netcdf4 files class H5readTestJvm { companion object { - fun files(): Sequence { - return H5Files.params() + fun files(): Iterator { + return H5Files.files() } } diff --git a/testfiles/src/test/kotlin/com/sunya/netchdf/H5superblockTest.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5superblockTest.kt similarity index 99% rename from testfiles/src/test/kotlin/com/sunya/netchdf/H5superblockTest.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5superblockTest.kt index 52e1e306..8b1bc399 100644 --- a/testfiles/src/test/kotlin/com/sunya/netchdf/H5superblockTest.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/hdf5/H5superblockTest.kt @@ -1,6 +1,5 @@ -package com.sunya.netchdf +package com.sunya.netchdf.hdf5 -import com.sunya.netchdf.hdf5.mdlClassCount import com.sunya.netchdf.testfiles.SuperblockFiles import com.sunya.netchdf.testutils.readNetchdfData import com.sunya.netchdf.testutils.testData diff --git a/testfiles/src/test/kotlin/com/sunya/netchdf/N3readTest.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt similarity index 73% rename from testfiles/src/test/kotlin/com/sunya/netchdf/N3readTest.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt index 9dc4ab0b..a3343594 100644 --- a/testfiles/src/test/kotlin/com/sunya/netchdf/N3readTest.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/netcdf3/N3readTest.kt @@ -1,8 +1,9 @@ -package com.sunya.netchdf +package com.sunya.netchdf.netcdf3 import com.sunya.cdm.api.Datatype import com.sunya.cdm.array.ArrayUByte import com.sunya.cdm.array.makeStringsFromBytes +import com.sunya.netchdf.openNetchdfFile import com.sunya.netchdf.testfiles.N3Files import com.sunya.netchdf.testutils.readNetchdfData import com.sunya.netchdf.testutils.testData @@ -21,15 +22,6 @@ class N3readTest { } } - @Test - fun simple() { - val filename = "/home/all/testdata/devcdm/netcdf3/simple_xy.nc" - println(filename) - openNetchdfFile(filename).use { myfile -> - println(myfile!!.cdl()) - } - } - @Test fun problem() { val filename = testData + "devcdm/netcdf3/WrfNoTimeVar.nc" @@ -39,27 +31,6 @@ class N3readTest { } } - @Test - fun testCharAttribute() { - val filename = testData + "recent/cdf5/jays_DOMAIN000.nc" - openNetchdfFile(filename).use { myfile -> - println("--- ${myfile!!.type()} $filename ") - println(myfile.cdl()) - - val gatt = myfile.rootGroup().attributes.find { it.name == "texture_fudging" }!! - println("texture_fudging = $gatt") - assertEquals(Datatype.Companion.STRING, gatt.datatype) - assertEquals(1, gatt.values.size) - assertEquals("No", gatt.values[0]) - - val crs = myfile.rootGroup().variables.find { it.name == "crs" }!! - val att = crs.attributes.find { it.name == "grid_mapping_name" }!! - assertEquals(Datatype.Companion.STRING, att.datatype) - assertEquals(1, att.values.size) - assertEquals("lambert_conformal_conic", att.values[0]) - } - } - @Test fun testCharVariable() { val filename = testData + "cdmUnitTest/formats/netcdf3/awips.nc" @@ -114,7 +85,6 @@ class N3readTest { openNetchdfFile(filename).use { myfile -> if (myfile == null) { println("*** not a netchdf file = $filename") - return } } } diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/netcdf4/N4charTest.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/netcdf4/N4charTest.kt similarity index 87% rename from core/src/commonTest/kotlin/com/sunya/netchdf/netcdf4/N4charTest.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/netcdf4/N4charTest.kt index 77758aac..afe8878f 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/netcdf4/N4charTest.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/netcdf4/N4charTest.kt @@ -1,12 +1,12 @@ package com.sunya.netchdf.netcdf4 -import com.sunya.cdm.api.* +import com.sunya.cdm.api.Datatype import com.sunya.cdm.array.ArrayUByte import com.sunya.cdm.array.makeStringsFromBytes import com.sunya.netchdf.openNetchdfFile -import com.sunya.netchdf.testfiles.testData - -import kotlin.test.* +import com.sunya.netchdf.testutils.testData +import kotlin.test.Test +import kotlin.test.assertEquals import kotlin.test.assertIs class N4charTest { @@ -20,13 +20,13 @@ class N4charTest { val gatt = myfile.rootGroup().attributes.find{ it.name == "date"}!! println("date = $gatt") - assertEquals(Datatype.STRING, gatt.datatype) + assertEquals(Datatype.Companion.STRING, gatt.datatype) assertEquals(1, gatt.values.size) assertEquals("23/04/09", gatt.values[0]) val crs = myfile.rootGroup().allVariables().find{ it.fullname() == "/xma/rtip/inp#4"}!! val att = crs.attributes.find{ it.name == "units"}!! - assertEquals(Datatype.STRING, att.datatype) + assertEquals(Datatype.Companion.STRING, att.datatype) assertEquals(1, att.values.size) assertEquals("V", att.values[0]) } @@ -39,10 +39,10 @@ class N4charTest { println("--- ${myfile!!.type()} $filename ") println(myfile.cdl()) val v = myfile.rootGroup().variables.find{ it.name == "file_date"}!! - assertEquals(Datatype.CHAR, v.datatype) + assertEquals(Datatype.Companion.CHAR, v.datatype) val data = myfile.readArrayData(v) println("file_date data = $data") - assertEquals(Datatype.CHAR, data.datatype) + assertEquals(Datatype.Companion.CHAR, data.datatype) assertIs(data) val expect = listOf(50,48,49,48,45,48,52,45,48,52,84,48,57,58,49,55,58,52,52,46,48,48,48,48,48,48,50,48,49,48,45,48,52,45,48,52,84,48,57,58,53,52,58,49) diff --git a/testfiles/src/test/kotlin/com/sunya/netchdf/testfiles/CountVersions.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/testfiles/CountVersions.kt new file mode 100644 index 00000000..884fd615 --- /dev/null +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/testfiles/CountVersions.kt @@ -0,0 +1,76 @@ +package com.sunya.netchdf.testfiles + +import com.sunya.netchdf.openNetchdfFile +import kotlin.test.Test +import kotlin.use + +class CountVersions { + + companion object { + fun files(): Iterator { + return sequenceOf( + N3Files.files().asSequence(), + N4Files.files().asSequence(), + H5Files.files().asSequence(), + H4Files.files().asSequence(), + NetchdfExtraFiles.files(false).asSequence(), + JhdfFiles.files().asSequence(), + ) + .flatten() + .iterator() + } + + val versions = mutableMapOf>() + + val filenames = mutableMapOf>() + val showAllFiles = false + + fun afterAll() { + println("*** nfiles = ${filenames.size}") + var dups = 0 + filenames.keys.sorted().forEach { + val paths = filenames[it]!! + if (paths.size > 1) { + println("$it") + paths.forEach { println(" $it") } + } + dups += paths.size - 1 + } + println("*** nduplicates = ${dups}") + + if (showAllFiles) { + println("*** nfiles = ${filenames.size}") + filenames.keys.sorted().forEach { + val paths = filenames[it]!! + paths.forEach {path-> println("$path/$it") } + } + } + + val sversions = versions.toSortedMap() + sversions.keys.forEach{ println("$it = ${sversions[it]!!.size } files") } + val total = sversions.keys.map{ sversions[it]!!.size }.sum() + println("total # files = $total") + } + } + + @Test + fun countVersions() { + + files().forEach { filename -> + try { + openNetchdfFile(filename).use { ncfile -> + if (ncfile == null) { + println("Not a netchdf file=$filename ") + } else { + val paths = versions.getOrPut(ncfile.type()) { mutableListOf() } + paths.add(filename) + } + } + } catch (e: Throwable) { + e.printStackTrace() + } + } + + afterAll() + } +} \ No newline at end of file diff --git a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/NppFiles.kt b/testfiles/src/test/kotlin/com/sunya/netchdf/testfiles/NppFiles.kt similarity index 87% rename from core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/NppFiles.kt rename to testfiles/src/test/kotlin/com/sunya/netchdf/testfiles/NppFiles.kt index 0aa4c26f..84fc5fe1 100644 --- a/core/src/commonTest/kotlin/com/sunya/netchdf/testfiles/NppFiles.kt +++ b/testfiles/src/test/kotlin/com/sunya/netchdf/testfiles/NppFiles.kt @@ -1,10 +1,13 @@ package com.sunya.netchdf.testfiles +import com.sunya.netchdf.testutils.testData +import com.sunya.netchdf.testutils.testFilesIn + class NppFiles { companion object { - fun params(): Sequence { + fun files(): Sequence { val builder = testFilesIn(testData + "netchdf/npp") .withRecursion() .addNameFilter { name -> !name.endsWith(".cdl") }