Adds support for more types of FractalHeap (used for storing attributes)#244
Adds support for more types of FractalHeap (used for storing attributes)#244bnlawrence wants to merge 5 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #244 +/- ##
==========================================
- Coverage 78.65% 78.48% -0.18%
==========================================
Files 15 15
Lines 3345 3416 +71
Branches 534 546 +12
==========================================
+ Hits 2631 2681 +50
- Misses 579 593 +14
- Partials 135 142 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR extends pyfive’s Fractal Heap support to handle additional huge/tiny object encodings encountered in real-world netCDF/HDF5 files (notably CMIP6), including huge-object lookup through a v2 B-tree, and adds tests/fixtures to exercise these branches.
Changes:
- Implement decoding for tiny heap IDs and huge heap IDs (direct + indirect via v2 B-tree) in
FractalHeap.get_data(). - Add
BTreeV2HugeObjectsIndirectto parse/iterate huge-object records stored in HDF5 v2 B-trees. - Add tests including a real CMIP6 fixture-based test and synthetic tiny-object decoding tests; update pre-commit config to allow the large fixture file.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
pyfive/misc_low_level.py |
Adds huge/tiny Fractal Heap ID decoding and indirect huge-object lookup via a v2 B-tree. |
pyfive/btree.py |
Introduces a v2 B-tree reader for indirectly accessed huge objects. |
tests/test_fractal_heap.py |
Adds coverage for tiny IDs and a real CMIP6 huge-object case; updates huge-object expectations. |
.pre-commit-config.yaml |
Excludes the large CMIP fixture from the “added large files” pre-commit hook. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
davidhassell
left a comment
There was a problem hiding this comment.
Looks good, as far as I can tell (I've eyeballed the code and tests, and run the test, and played on the command line)
|
bit more testing would be nice though; RTD fails unrelated to here see #245 |
|
Can't test half of this because we can't create synthetic data with these properties. At some point mocking it becomes pointless ... |
no probs, cov hit is not big 😃 |
|
docs build issue sorted, this is ready for merge when @bnlawrence is happy to 🍺 |
Description
This pull request includes implementation and tests for two types of FractalHeap which we had not previously encountered (as described in #243).
In practice we have added some more code within the implementation of the
Fractal_Heapclass, which replaces some "NotImplementedError" branches, and support for the particular kind of b-tree used in the fractal heaps. One large test file which requires one of the two new branches is included - we couldn't shrink it as any attempt to rewrite it invoked a differnt kind of FractalHeap. We have not been able to generate a real file using the other branch, so some synthetic testing has been included.Closes #243
Checklist