[ZFDB] Refactoring of Chunking + Documentation Update

[tbkr]

Description

Refactor of ZFDB handling chunking besides non and individual.

Contributor Declaration

By opening this pull request, I affirm the following:

• All authors agree to the Contributor License Agreement.
• The code follows the project's coding standards.
• I have performed self-review and added comments where needed.
• I have added or updated tests to verify that my changes are effective and functional.
• I have run all existing tests and confirmed they pass.

🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/fdb/pull-requests/PR-290

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.24405% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.08%. Comparing base (e93a532) to head (41278f8).
⚠️ Report is 24 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/chunked_data_view/GribExtractor.cc	0.00%	34 Missing ⚠️
src/chunked_data_view/ChunkedDataViewBuilder.cc	81.63%	9 Missing ⚠️
src/chunked_data_view/mapping/AxisMapper.cc	87.27%	7 Missing ⚠️
src/chunked_data_view/ChunkedDataViewImpl.cc	92.18%	5 Missing ⚠️
src/chunked_data_view/ViewPart.cc	94.38%	5 Missing ⚠️
...d_data_view/exception/UnknownExtractorException.cc	0.00%	3 Missing ⚠️
...nked_data_view/exception/GribExtractorException.cc	0.00%	2 Missing ⚠️
...ata_view/exception/RequestManipulationException.cc	33.33%	2 Missing ⚠️
src/chunked_data_view/Axis.cc	50.00%	1 Missing ⚠️
src/chunked_data_view/Axis.h	94.11%	1 Missing ⚠️
... and 6 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #290      +/-   ##
===========================================
+ Coverage    71.47%   72.08%   +0.61%     
===========================================
  Files          372      378       +6     
  Lines        23510    23908     +398     
  Branches      2464     2509      +45     
===========================================
+ Hits         16803    17235     +432     
+ Misses        6707     6673      -34     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Refactors Z3FDB/chunked_data_view chunking to support fixed-size “individual chunk” shapes, reworks chunk access around bounding-box intersections, and updates the Python/Zarr interface and documentation accordingly.

Changes:

• Add IndividualChunk chunking (with configurable chunk size) across C++ core, Python wrapper, and bindings.
• Introduce bounding-box based request slicing / buffer mapping and add new C++ + Python tests.
• Propagate fillValue into Zarr metadata and expose it through the Python API; expand/refresh the docs set.

Reviewed changes

Copilot reviewed 61 out of 61 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/z3fdb/test_store_v3_parts.py	Formatting adjustments in existing Z3FDB tests.
tests/z3fdb/test_store_v3_individual_chunking.py	New Z3FDB tests covering IndividualChunk across multi-part/axis layouts.
tests/pychunked_data_view/test_chunked_data_view.py	Adds Python-side tests for IndividualChunk and fill value behavior.
tests/conftest.py	Adds function-scoped temp fixture and changes FDB setup fixture scope/storage location.
tests/chunked_data_view/test_view.cc	Refactors tests to shared mock helpers + updated builder/extractor API.
tests/chunked_data_view/test_view_individual_chunking.cc	New C++ tests for IndividualChunking behavior.
tests/chunked_data_view/test_request_manipulation_bounding_box.cc	New tests for request slicing via bounding boxes.
tests/chunked_data_view/test_mock_helpers.h	New shared mock helper utilities for chunked_data_view tests.
tests/chunked_data_view/test_index_mapper.cc	Updates include path + adds/changes computeBufferIndex tests.
tests/chunked_data_view/test_builder.cc	Removes old builder-focused tests (replaced by other suites).
tests/chunked_data_view/test_bounding_box.cc	New unit tests for BoundingBox behavior.
tests/chunked_data_view/test_axis.cc	Updates RequestManipulation API usage in axis tests.
tests/chunked_data_view/CMakeLists.txt	Registers new/renamed C++ test targets.
src/z3fdb/_internal/zarr.py	Uses ChunkedDataView’s configured fill value in Zarr metadata.
src/pychunked_data_view/chunked_data_view.py	Adds Chunking.IndividualChunk, fill value plumbing, and translation updates.
src/chunked_data_view/ViewPart.h	Introduces BoundingBox + ViewPart refactor toward bounding-box based access.
src/chunked_data_view/ViewPart.cc	Implements BoundingBox operations and ViewPart construction/slicing.
src/chunked_data_view/RequestManipulation.h	Adds selectRequest + range-based updateRequest API.
src/chunked_data_view/RequestManipulation.cc	Implements bounding-box selection logic and alignment checks.
src/chunked_data_view/mapping/IndexMapper.h	New header for buffer-index and delinearization utilities.
src/chunked_data_view/mapping/IndexMapper.cc	Implements new computeBufferIndex and index mapping logic.
src/chunked_data_view/mapping/AxisMapper.h	New mapper from AxisDefinition to (Axis, AxisChunks).
src/chunked_data_view/mapping/AxisMapper.cc	Implements mapping + chunk-size validation.
src/chunked_data_view/IndexMapper.h	Removes old IndexMapper interface header.
src/chunked_data_view/include/chunked_data_view/Types.h	Introduces BoundingBox type aliases for coordinate spaces.
src/chunked_data_view/include/chunked_data_view/ListIterator.h	Documentation/comments for iterator interface/wrapper.
src/chunked_data_view/include/chunked_data_view/LibChunkedDataView.h	Adds/clarifies library-level documentation comments.
src/chunked_data_view/include/chunked_data_view/Fdb.h	Documents FDB abstraction and factory.
src/chunked_data_view/include/chunked_data_view/Extractor.h	Refactors Extractor API to extractInto based on bounding boxes.
src/chunked_data_view/include/chunked_data_view/ChunkedDataViewBuilder.h	Builder now supports fill value + shared extractor ownership + optional config path.
src/chunked_data_view/include/chunked_data_view/ChunkedDataView.h	Adds fillValue() to the view interface and improves docs.
src/chunked_data_view/include/chunked_data_view/DataLayout.h	Adds clarifying docs/comments.
src/chunked_data_view/include/chunked_data_view/AxisDefinition.h	Adds SingleValue vs Individual chunking and improves docs.
src/chunked_data_view/GribExtractor.h	Refactors GRIB extractor to new layout/extractInto API.
src/chunked_data_view/GribExtractor.cc	Implements new extraction path + buffer-index mapping.
src/chunked_data_view/Extractor.cc	Removes old extractor factory implementation.
src/chunked_data_view/ChunkedDataViewBuilder.cc	Builds ViewParts via AxisMapper + new Extractor API; offsets/extension handling.
src/chunked_data_view/ChunkedDataViewImpl.h	Holds (ViewPart, Extractor) pairs + fill value; updated shape/chunk computations.
src/chunked_data_view/ChunkedDataViewImpl.cc	Implements chunk intersection routing and per-part extraction.
src/chunked_data_view/Axis.h	Adds AxisChunks and refactors Axis to remove “chunked” flag.
src/chunked_data_view/Axis.cc	Updates Axis constructor and error text.
src/chunked_data_view/include/chunked_data_view/Buffer.h	Removes old Buffer type.
src/chunked_data_view/Buffer.h	Removes old Buffer type.
src/chunked_data_view/include/chunked_data_view/exception/*.h	Adds new exception types for clearer failure modes.
src/chunked_data_view/exception/*.cc	Implements new exception types.
src/chunked_data_view/CMakeLists.txt	Removes Buffer/old IndexMapper; adds mapping + exception sources.
src/chunked_data_view_bindings/bindings.cc	Updates pybind API for new chunking types and builder/extractor wiring.
docs/z3fdb/technical_insights.rst	New internal mechanics explanation (bounding boxes, buffer mapping).
docs/z3fdb/index.rst	Restructures landing page and adds “getting_started” + “technical_insights”.
docs/z3fdb/getting_started.rst	New guided tutorial for first-time users.
docs/z3fdb/dimension_mapping.rst	Expands data model/coordinate system + documents IndividualChunk + fill value.
docs/z3fdb/architecture.rst	Adds layer descriptions to architecture overview.
docs/z3fdb/api.rst	Documents IndividualChunk and fill value API.

Comments suppressed due to low confidence (3)

src/chunked_data_view/mapping/IndexMapper.cc:66

• The reverse loop uses int i = axes.size() - 1, which is fragile (underflow if axes.size()==0) and can trigger signed/unsigned warnings. Use the standard size_t i = axes.size(); i-- > 0; idiom.
  src/chunked_data_view/ChunkedDataViewImpl.cc:124
• ChunkedDataViewImpl::at() never initializes the output buffer with fillValue_. Any positions not written by an extractor (e.g. uncovered by any part, or out-of-array in edge chunks) will contain uninitialized data. Also consider validating len before writing.

void ChunkedDataViewImpl::at(const std::vector<size_t>& chunkIndex, float* ptr, size_t len) {

    if (chunkIndex.size() != chunks_.size()) {
        std::ostringstream ss;
        ss << "ChunkedDataViewImpl::at: Expected chunk index of dimension " << chunks_.size() << ", got dimension "
           << chunkIndex.size();
        throw eckit::UserError(ss.str());

src/chunked_data_view/ChunkedDataViewImpl.cc:21

• The suggested fix for initializing the buffer uses std::fill, but this file currently doesn't include <algorithm>, so it won't compile once added. Add the include.

#include "chunked_data_view/exception/GribExtractorException.h"

#include <cstddef>
#include <ostream>
#include <sstream>
#include <utility>
#include <vector>

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[caraghbiner]

It's a very big PR. I only looked at the docs as I said I would. Overall they looked nice to me.

[Copilot]

Pull request overview

Copilot reviewed 73 out of 73 changed files in this pull request and generated 10 comments.

[caraghbiner]

Some fairly trivial changes requested.