Fix false positive triangle-triangle intersections

[gyuun]

Type

• Bug fix (non-breaking change which fixes an issue): Fixes TriangleMesh - is_self_intersecting return wrong result #5117
• New feature (non-breaking change which adds functionality). Resolves #
• Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

TriangleTriangle3d applies an absolute tolerance to plane-equation values computed with unnormalized triangle normals. For the triangles reported in #5117, this changes one of three same-sign values to zero, bypasses the same-side rejection, and produces a false positive.
The result also changes after a rigid rotation because the per-axis normalization changes the magnitude of these plane-equation values.

Checklist:

• I have run python util/check_style.py --apply to apply Open3D code style
  to my code.
• This PR changes Open3D behavior or adds new functionality.
  • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
    updated accordingly.
  • I have added or updated C++ and / or Python unit tests OR included test
    results (e.g. screenshots or numbers) here.
• I will follow up and update the code if CI fails.
• For fork PRs, I have selected Allow edits from maintainers.

Description

This change adds a conservative half-space rejection before the existing triangle-triangle predicate.

Signed distances are computed using a unit triangle normal. If all three vertices of either triangle are unambiguously on the same side of the other triangle's supporting plane, the triangles are reported as separated. Ambiguous, coplanar, near-parallel, degenerate, and non-finite cases continue to use the existing predicate.

This is an early rejection only: it returns false solely when plane separation is unambiguous. All uncertain cases fall through to the existing predicate unchanged.

An alternative would be to scale the epsilon used inside NoDivTriTriIsect by the corresponding normal magnitude, which is equivalent to applying the tolerance to unit-normal signed distances. That approach could eliminate the additional precheck, but it would require modifying vendored third-party code and could change near-coplanar classification throughout the existing predicate.

This PR therefore leaves the vendored implementation unchanged and adds only a conservative wrapper-level rejection for geometrically unambiguous separation.

No public API or third-party code is changed.

Tests added:

• Direct regression for the coordinates from TriangleMesh - is_self_intersecting return wrong result #5117.
• Rotation, translation, scale, vertex-order, and triangle-order coverage.
• Near-parallel intersecting-triangle regression.
• C++ TriangleMesh integration coverage.
• Python TriangleMesh API coverage.

Test results:

• 7 focused C++ tests passed.
• 1 Python regression test passed.
• tests and python-package targets built successfully.

This is my first contribution to Open3D, so I would appreciate any feedback on the approach or test coverage. Thank you for reviewing!

[update-docs]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.