[Core] Adding new data containers

[loumalouomega]

---

name: ✨ Feature
about: New applications, new features or wider changes

---

📝 Description

Phase I of a two-phase effort to port a standalone, type-erased, variable-indexed data storage prototype (DataContainer / DataChunk / DataValuePolicy / DataHistoryPolicy / DataAccessor) into the Kratos core, replacing the facilities it mimicked (error macros, VariableData/Variable<T>, the Registry) with the real ones already provided by Kratos.

This PR ports the data structure itself with minimal behavioural modification, adapts it to core style and conventions, adds full C++ and Python test coverage, and exposes it to Python via pybind11. The class is not yet wired into Node, Element, Condition, Geometry or MasterSlaveConstraint — that is Phase II, tracked separately. A few // TODO(Phase II) markers are left at the obvious future integration points, but nothing is implemented there yet.

Tags: Api Addition (new opt-in core module; no existing API is touched)

Key changes

New module kratos/containers/data_container/, organized as four logically separable pieces (mirrored in the commit history):

1. DataAccessor / DataValuePolicy / SparseDataValuePolicy — a lightweight, index-caching accessor and the type-erased value-policy hierarchy (dense, layered, sparse) that defines what is stored per entity and how it is cloned/copied/printed.
2. DataHistoryPolicy — NonHistoricalDataPolicy / HistoricalDataPolicy, defining how many steps are kept per chunk via a StepCategory-scoped ring buffer.
3. DataChunk — the contiguous, manually-managed (new[]/delete[]) raw storage for one variable's values across all entities and step slots.
4. DataContainer — ties the above together: Add/GetAccessor/Has/GetDataSpan, sparse-storage growth (UpdateSparseStorage/AddToSparseStorage), step cloning, plus the pybind11 bindings and cross-cutting tests.

Header/source split. Non-template classes (DataValuePolicyBase, DataHistoryPolicyBase + its two concrete policies, DataChunkBase, and the non-template members of DataContainer) now have a matching .cpp, mirroring how VariableData (non-template) has variable_data.cpp while Variable<T> (template) stays fully inline. Template classes (DataAccessor<T>, DataValuePolicy<T>, LayeredDataValuePolicy<T>, SparseDataValuePolicy<T>, DataChunk<T>, and DataContainer::Add/GetAccessor/GetDataSpan/Has) stay header-only, since the module must remain generic over arbitrary user value types (exercised by the tests with a custom CustomClass).

Design decisions / judgment calls for reviewers:

• DataContainer::Add canonicalizes variables through the Kratos Registry (variables.all.<name>), so only already-registered variables can be added; it does not auto-register them as a side effect. Unregistered variables raise a clear error instead of the opaque Registry::GetValue throw the prototype would have hit.
• std::mutex replaced by LockObject; DataContainer copy/move are explicit and documented as shallow (chunks are shared via std::shared_ptr).
• SparseDataValuePolicy intentionally does not override IsSameType — a sparse and a dense policy of the same value type compare as the same type (sparseness is queried through IsSparse() instead), matching the prototype's semantics.
• HistoricalDataPolicy::Clone() resets the running step index to 0 (prototype-faithful, documented) — freshly created chunks (CreateNew, Initialize) always start their history at slot 0.
• LayeredDataValuePolicy is not exposed to Python in this phase (no sane NumPy mapping for a per-entity std::vector).
• Design note added at docs/pages/Kratos/For_Developers/Data_Structures/Data_Container.md.

Prototype bugs fixed during the port (the prototype used stub macros that only printed instead of throwing, which hid these):

• HistoricalDataPolicy::CloneStepIndex was missing override.
• Zero-entity chunks (how sparse chunks legitimately start) would have thrown in FullDebug builds due to overly strict N > 0 debug checks — relaxed and regression-tested.
• DataChunk::ResizeDataContainer(0) left a dangling pointer, causing a double delete in the destructor — fixed and regression-tested.
• The resize copy/zero-fill was computed from the total size but applied per step slot, reading out of bounds for any chunk with more than one history step — fixed to a per-step min(old, new) copy and regression-tested.
• DataContainer::Initialize validated the wrong variable (a member instead of its own argument) and never stored the requested chunk size — fixed and regression-tested.
• Two throw std::runtime_error sites converted to KRATOS_ERROR.
• NumberOfEntitiePerStep() typo renamed to NumberOfEntitiesPerStep() (no existing callers outside this PR).

Validation

• Full rebuild of KratosCore, KratosCoreTest and the Kratos Python module (Release, gcc 13.3) with zero warnings/errors introduced.
• KratosCoreTest --gtest_filter='*DataContainer*:*DataChunk*:*DataAccessor*:*DataValuePolicy*:*DataHistoryPolicy*:*HistoricalDataPolicy*:*NonHistoricalDataPolicy*:*SparseDataValuePolicy*' → 31/31 passed, covering type/equality comparisons, variable compatibility, raw type-erased operations, sparse growth (UpdateSparseStorage/AddToSparseStorage reproducing the original voxel-mesh scenarios directly on a 125-entity DataContainer), step-history cycling/cloning, accessor mismatch handling, shallow-copy semantics, and variable-lifetime independence from the Registry.
• kratos/tests/test_data_container.py (registered in test_KratosCore.py's small suite) → 10/10 passed, mirroring the C++ scenarios through the pybind11 bindings (NumPy view semantics, custom zero values, array_1d shape, sparse workflow, error paths).
• Full KratosCoreTest regression run (1609 tests) → all green, no pre-existing test broken by the addition.
• Every new header and .cpp file compiled standalone (its own includes only, outside the aggregated build) to verify the trimmed include lists are actually self-sufficient and not silently relying on include order from other translation units.

🆕 Changelog

• Added kratos/containers/data_container/ — a chunked, type-erased, variable-keyed data storage module (DataContainer, DataChunk, DataValuePolicy/LayeredDataValuePolicy/SparseDataValuePolicy, DataHistoryPolicy, DataAccessor), standalone in this phase (not yet used by Node/Element/Condition/Geometry/MasterSlaveConstraint)
• Added pybind11 bindings for the new module (DataContainer, StepCategory, history and value policies) for double, int, array_1d<double,3> and std::string, with zero-copy NumPy views for GetDataSpan
• Added C++ (kratos/tests/cpp_tests/containers/data_container/) and Python (kratos/tests/test_data_container.py) test coverage for the new module
• Added a developer design note at docs/pages/Kratos/For_Developers/Data_Structures/Data_Container.md

[matekelemen]

Can you please explain the idea behind such a gigantic new system?

• Where will it be used?
• How is it different from existing data structures?
• What are the new features required by its application that are not provided by existing data structures in core?

This PR ports the data structure itself ...

Ports it from where? Is this an existing system somewhere? You should take a look at their license if so.

[loumalouomega]

Can you please explain the idea behind such a gigantic new system?

* Where will it be used?

Good question...

* How is it different from existing data structures?

Well, currently the data structure is holded by object. the idea behind thsi is to managed sxternally so data can be processed in a more performance way. Think for example in GPU parallelism. Copy our current data strcuture to a GPU is greatly ineficient because of the memory copying.

* What are the new features required by its application that are not provided by existing data structures in core?

The current data structure is designed for a CPU architectures and paradigms of 15 years ago, performance can be greatly improved.

This PR ports the data structure itself ...

Ports it from where? Is this an existing system somewhere? You should take a look at their license if so.

It ports from a @pooyan-dadvand side project, it is designed originally for structured data, but can be used for non structured as well. The usage is in another PR in the future space. Maybe I should have put this in future to begin with, but the structure looks generic enought to be as it is.

[philbucher]

This looks interesting. Is there a roadmap or sth like this somewhere? That would help externals to see where the project is going

[loumalouomega]

This looks interesting. Is there a roadmap or sth like this somewhere? That would help externals to see where the project is going

Jokes aside, yes, there is a roadmap, but not written in stone. I think if you want more details contact @pooyan-dadvand.

[matekelemen]

Think for example in GPU parallelism.

So the idea is to

1. build a model on the host (CPU)
2. convert a model to this data structure
3. copy the converted model to the device (GPU)
4. solve on the device side
5. retrieve results from the device
   ?

I imagine you'd need to mirror these data structures on device code. What toolchain are you planning for device code? I highly recommend SYCL.

[loumalouomega]

I imagine you'd need to mirror these data structures on device code. What toolchain are you planning for device code? I highly recommend SYCL.

Well, for the details I don't know much, but mainly we are using old reliable closed coded and evil CUDA.

[RiccardoRossi]

to addmy 2 cents, the idea would be to ideally have the best of both worlds. that is, to be able to still get the data using current API, but to have an alternative (FAST) view access on the line of what the tensor adaptors are doing, but ideally without copies in the middle if you are willing to use another interface

the big question here is if the access through the current interface would add (big) overhead with respect to what we have now, or if that is not the case