Deterministic, cross-platform-reproducible integration tests (WMTK_FP_STRICT) - #949
Open
danielepanozzo wants to merge 23 commits into
Open
Deterministic, cross-platform-reproducible integration tests (WMTK_FP_STRICT)#949danielepanozzo wants to merge 23 commits into
danielepanozzo wants to merge 23 commits into
Conversation
Adds an opt-in, reproducible integration-test check on top of the existing
integration test list (data/integration_tests/integration_tests.json):
* tests/integration_hashes.py drives each test through wmtk_app in an isolated
working directory, forcing single-threaded execution, and SHA-256 hashes every
output file. `generate` blesses tests (stores hashes in integration_tests.json);
`check` re-runs and verifies, exiting non-zero on any mismatch. Entries in the
test list may be a plain string (not yet blessed; run for termination only) or
an object {"file", "hashes"} (blessed; hash-checked), so tests can be added to
the golden set one at a time.
* WMTK_FP_STRICT CMake option compiles the toolkit with no FMA contraction /
no fast-math (MSVC /fp:strict) so outputs are bit-identical across compilers
and operating systems.
* The `integration_hashes` ctest runs the blessed checks (Python3 + wmtk_app).
Golden hashes live in the data2 repo's integration_tests.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e hashes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-arch testing (macOS-arm64 vs Linux-x86_64) showed that -ffp-contract=off alone leaves ~1 ULP differences because ARM (NEON) and x86 (SSE/AVX) sum SIMD reductions in different orders, which can cascade into different meshes. Adding EIGEN_DONT_VECTORIZE forces Eigen's scalar path so the op order is identical on every architecture. Because that also changes Eigen's alignment/ABI, the whole fp-strict block is now applied globally (before any target is configured) so the toolkit, components, app and third-party code (Eigen, libigl, geogram, polysolve) are all built consistently. With this, the fast surface-op integration tests are byte-identical across macOS-arm64 and Linux-x86_64. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
constrained_delaunay2D used geogram's plain CDT2d, which breaks ties between the two valid diagonals of a cocircular quad with floating-point-dependent logic, so it produced different (but equally valid) triangulations of the same input on different CPU architectures (e.g. arm64 vs x86_64). That cascaded into a completely different output mesh, making triwild non-reproducible. Two changes make it deterministic: * Use GEO::ExactCDT2d (exact predicates + symbolic perturbation) so the triangulation is the same set of triangles everywhere. This needs geogram's predicate kernel initialized, so add the GEO::initialize() call the plain CDT2d did not require (its absence aborted on Linux). * Canonicalize the output triangle order: even with an identical triangulation, geogram stores the triangles in an order that depends on the floating-point point-location walk. Rotate each triangle so its smallest vertex is first (preserving orientation) and sort the rows, so the mesh built from them is identical on every machine. With this, the triwild integration tests are byte-identical across macOS-arm64 and Linux-x86_64 (verified). Note ExactCDT2d yields a different (and, on the multi-input case, somewhat heavier-to-optimize) triangulation than the old CDT2d, so triwild output meshes change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 3D AMIPS energy/gradient/hessian need det^(2/3) (a cube root). std::cbrt and std::pow are not correctly rounded and differ by ~1 ULP between libm implementations (macOS libm vs glibc), which makes the iterative optimization take divergent discrete decisions on different OSes/architectures. Add wmtk::utils::deterministic_cbrt: range reduction with frexp/ldexp (exact) plus a fixed number of Newton iterations using only the IEEE-754 correctly-rounded ops (+ - * /), so the result is bit-identical everywhere (within ~1 ULP of std::cbrt, ~3x slower per call but a small fraction of a full run). Replace std::cbrt and pow(x, -1/3) in AMIPS.cpp/.h, and the decision/output std::cbrt in TetWildMesh.cpp and SimWildMesh.cpp. This is one necessary component of cross-platform-deterministic 3D meshing (the operation scheduler's non-determinism is addressed separately by the TBB removal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_edges() sorts the per-tet edge tuples by their (v0, v1) vertex pair and then std::unique's them down to one tuple per undirected edge. The comparator orders only by (v0, v1), so the several tuples representing the same edge (one per incident tet) compare equal. std::sort leaves equal elements in an implementation-defined order, so std::unique kept a different representative tuple under libc++ vs libstdc++. That representative feeds the operation scheduler's priority-queue tie-break, making the whole 3D optimization take divergent decisions across platforms. stable_sort keeps the equal tuples in their deterministic enumeration order (by tet id, then local edge), so the kept representative -- and every downstream decision -- is identical on every compiler/OS/CPU. This is the core fix that makes tetwild output bit-identical across architectures. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
libm's cbrt/atan2 are not correctly rounded and differ by a few ULP between
platforms (macOS libm vs glibc, x86 SSE/AVX vs ARM NEON); feeding their result
into a discrete decision (energy comparison, winding-number sign) makes the
pipeline diverge across OSes. Rather than always pay for determinism, add thin
wrappers wmtk::cbrt / wmtk::atan2 / wmtk::atan (Transcendentals.hpp) that use the
fast architecture-specific std:: functions by default and switch to hand-rolled
deterministic implementations only when the WMTK_FP_STRICT CMake option is set
(it now also defines a WMTK_FP_STRICT macro).
* deterministic_atan2.hpp: new, fdlibm-derived atan/atan2 using only IEEE-754
correctly-rounded ops (max abs err ~4.4e-16 vs std::atan2). deterministic_cbrt
already existed.
* WindingNumber.hpp: the winding number is now always the direct solid-angle
sum (copied from igl::solid_angle, MPL-2.0) whose only transcendental is
routed through wmtk::atan2 -- so it reduces to the same atan2 gate. The
non-deterministic igl AABB accelerator is dropped.
* AMIPS / SimWildMesh / TetWildMesh: call wmtk::cbrt instead of
deterministic_cbrt directly, so the fast path is used unless WMTK_FP_STRICT.
Under WMTK_FP_STRICT the output is bit-identical to the previous always-
deterministic build (tetwild_sphere out_final.msh unchanged: 284996d9...).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
clean_triangle_mesh merged near-duplicate input vertices with igl::remove_duplicate_vertices, which rounds vertices to a grid and keeps one representative per cell using igl::sortrows. sortrows sorts with a std::sort whose comparator compares only the (rounded) coordinates, so vertices that fall in the SAME cell compare equal and std::sort leaves them in an implementation-defined order -- a different vertex then survives under libc++ (macOS) vs libstdc++ (Linux), and the cleaned mesh (hence the whole tetwild pipeline) diverges across OSes. Meshes with no in-cell duplicates were unaffected (distinct keys sort deterministically), which is why sphere and double_sphere already reproduced but octocat did not. Replace the igl call with a local deterministic dedup that reproduces igl's result exactly -- same grid rounding, unique cells emitted in ascending coordinate order, representative position taken verbatim from V -- but breaks ties in the sort on the original vertex index, so the lowest-indexed vertex in a cell always survives and the output is bit-identical on every platform. tetwild octocat now produces identical out_final.msh / out_surface.obj across macOS (arm64/libc++) and Linux (x86-64/libstdc++). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
execute_offset splits edges in order of decreasing length, sorting the candidate edges with sort_edges_by_length. The std::sort comparator ordered purely on edge length, so edges that share the exact same length compared equal and std::sort left them in an implementation-defined order (libc++ vs libstdc++) -- a cross-compiler non-determinism in the split order. Break ties on simplex::Edge's total order (its vertex pair) in both the tet and tri variants so the split order is identical on every STL. (Note: this removes the cross-compiler sort ambiguity; manifold_extraction_3d has a separate, address/ASLR-dependent non-determinism in its own path and is not yet part of the reproducible golden-hash set.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… runner
The golden-hash framework rewrites each entry of integration_tests.json from a
bare filename to a {"file": ..., "hashes": {...}} object once blessed. The C++
Integration_Tests runner deserialized the list straight into
std::vector<std::string>, which throws on the object form -- so the test would
fail as soon as any entry was blessed. Parse each entry as either a string or an
object with a "file" field.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…shes Points wmtk_data at wildmeshing/data2@9d8232b (branch danielepanozzo/integration-test-hashes), which adds the per-test output hashes that tests/integration_hashes.py checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The three cross-platform-determinism sort fixes each cost performance, so gate
them on WMTK_FP_STRICT like the transcendental wrappers -- the default build
keeps the faster original path, and only a reproducible build pays for
determinism:
* TetMesh::get_edges -- std::stable_sort (strict) vs std::sort (default)
* sort_edges_by_length -- Edge-index tie-break (strict) vs plain length
comparator (default), for both the tet and tri offset meshes
* read_triangle_mesh -- deterministic vertex dedup (strict) vs
igl::remove_duplicate_vertices (default)
The strict build's output is unchanged (tetwild_sphere out_final.msh still
284996d9...), so the blessed integration hashes stay valid.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The integration_hashes golden-hash ctest is only meaningful in a reproducible build, so register it only when WMTK_FP_STRICT is ON, and build the Release integration-test CI configs (Linux, macOS, Windows) with -DWMTK_FP_STRICT=ON so the check runs and validates bit-identical output across OS/compiler. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply clang-format v21.1.8 (the pre-commit-pinned version) to the files touched by the determinism work so the pre-commit hook passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
libigl's writeOBJ (and other text writers) open output files in text mode, so on Windows every '\n' is written as '\r\n' and the raw bytes -- hence the sha256 -- differ from the Unix output. Normalize CRLF -> LF before hashing, but only for text outputs: binary outputs (.msh / .vtu) legitimately contain 0x0D 0x0A byte pairs as data and are hashed verbatim (detected by the presence of a NUL byte). On Unix this is a no-op, so the stored hashes are unchanged; on Windows the .obj outputs now match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
macos-latest already runs on Apple Silicon, so the separate macos-14 build/test (continuous.yml) and pip (pip.yml) jobs are redundant. The release wheel build (wheels.yml) keeps macos-14 for the macosx_arm64 wheel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ion_3d) TopoOffsetTetMesh::m_has_envelope had no initializer, and init_from_image only sets it (to true) inside `if (V_env.rows() > 0)` -- there is no else. Callers that pass an empty envelope (manifold_extraction) therefore left it uninitialized, and write_msh_groups branches on it (`if (m_has_envelope)`) to decide whether to emit the EnvelopeSurface physical group. The garbage value depended on stack/ASLR state, so manifold_extraction_3d produced a different entity partition (3 vs 4 node blocks) from run to run and across machines -- deterministic only with ASLR disabled. Found with valgrind --track-origins. topological_offset always passes a real envelope, which is why it was unaffected. Default m_has_envelope (and m_tags_count, same pattern) to a defined value. manifold_extraction_3d is now bit-stable across runs and machines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 32 integration tests are now bit-reproducible; point wmtk_data at wildmeshing/data2@8f0081f, which blesses the last remaining test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deterministic, cross-platform-reproducible integration tests
Adds a golden-hash integration-test framework and the determinism fixes needed to make the integration outputs bit-identical across OS, compiler, and CPU architecture (macOS arm64/libc++, Linux x86-64/libstdc++, Windows MSVC), gated behind a new
WMTK_FP_STRICTCMake option so normal builds pay nothing for it.All 32 integration tests are verified byte-identical on macOS, Linux, and Windows by CI.
The framework
tests/integration_hashes.py— runs each blessed test throughwmtk_appsingle-threaded, sha256s its outputs, and checks them against reference hashes stored insidedata/integration_tests/integration_tests.json(schema: bare string = unblessed,{"file","hashes"}= blessed). Subcommands:generate/check/list.tests/README.md— how to (re)generate hashes, the schema, and the data2 workflow.integration_hashesctest, only whenWMTK_FP_STRICT=ON(a default build's fast paths are intentionally not cross-OS reproducible). The Release integration CI jobs build with-DWMTK_FP_STRICT=ONso the check runs on all three OSes..obj(written in text mode → CRLF) matches the Unix output; binary outputs (.msh/.vtu) are hashed verbatim.WMTK_FP_STRICT(new, OFF by default)Global, applied before any dependency is configured:
-ffp-contract=off -fno-fast-math(GCC/Clang) //fp:strict(MSVC) — kills FMA-contraction divergence.EIGEN_DONT_VECTORIZE— fixes SIMD reduction order (ARM NEON vs x86 SSE/AVX differ ~1 ULP).WMTK_FP_STRICTmacro, which switches the wrappers/paths below from fast to deterministic.Determinism root causes fixed
Each was a fast-but-non-reproducible operation; the FP/sort ones are gated so the default build keeps the faster path:
cbrt,atan2) are not correctly-rounded and differ across platforms →wmtk::cbrt/atan2/atanwrappers (Transcendentals.hpp) usestd::by default, hand-rolled deterministic versions (IEEE ops only) under strict. The winding number is a direct solid-angle sum (copied from libigl, MPL-2.0) reduced to that oneatan2gate.TetMesh::get_edges— non-stablestd::sortleftstd::uniquea different representative edge per STL, flipping the whole 3D scheduler →stable_sortunder strict.igl::sortrowsin the input vertex dedup — non-stable sort picked a different survivor per STL for vertices merged within the 2e-3 grid (this is whyoctocatdiverged butspheredidn't) → deterministic dedup under strict.sort_edges_by_length(topological_offset) — equal-length edges split in an implementation-defined order →simplex::Edgetie-break under strict.GEO::ExactCDT2d+ canonical triangle ordering.manifold_extraction_3d—TopoOffsetTetMesh::m_has_envelopewas never initialized when the caller passed an empty envelope, andwrite_msh_groupsbranches on it to decide whether to emit anEnvelopeSurfacegroup; the garbage value was ASLR-dependent, so the output entity partition (3 vs 4 node blocks) varied run-to-run. Found withvalgrind --track-origins; fixed by initializing the member (an unconditional correctness fix, not gated).Performance
Crown model (76k verts / 171k faces), tetwild, strict (deterministic) vs default (fast) build: negligible overhead (macOS +1.4%; on Linux the two builds' different convergence dominates the measurement). cbrt/atan2 are a tiny fraction of runtime and
EIGEN_DONT_VECTORIZEbarely affects the small fixed-size matrices.Notes for review
wildmeshing/data2@8f0081f(branchdanielepanozzo/integration-test-hashes);cmake/wmtk_data.cmakeis pinned to it.🤖 Generated with Claude Code