Share the proxy transfer operator across a swarm's variables; measure the SWARM-15 seam

[lmoresi]

Two follow-ups from #430, both of which that PR left explicitly open: the ~5x order=1 refresh cost, and the unmeasured rank-local seam.

1. Shared transfer operator

The order=1 weights depend only on geometry, so every proxied variable whose proxy has the same degree and continuity on the same mesh needs the same operator. Each was solving for it independently.

Measured first, to decide whether to build anything:

• refresh cost scales linearly with the number of proxied variables on one swarm — 4 variables cost 3.95x one in 2D, 4.08x in 3D;
• the weight solve is 74–76% of a refresh, which bounds the saving;
• the kd-tree survives a refresh but not a migrate(), so an advecting model gets one miss per timestep and K−1 hits.

Swarm now caches the sparse particle → proxy-node operator, keyed by (mesh version, proxy degree, proxy continuity, nnn, p, order). Same-degree proxies were confirmed to have byte-identical node coordinates in distinct objects, so that key shares exactly and never conflates different node sets.

Validity is tied to the kd-tree instance, not a flag: the entry carries the tree it was built from and lookup compares identity, so migrate() invalidates it structurally. No new invalidation contract to remember.

Measured under a realistic timestep — invalidate the tree, then refresh every proxied variable:

	uncached	cached	saving
2D, 1 variable	26.99 ms	27.09 ms	−0.4%
2D, 2 variables	50.90 ms	27.00 ms	47.0%
2D, 4 variables	99.33 ms	28.12 ms	71.7%
3D, 1 variable	131.28 ms	132.06 ms	−0.6%
3D, 2 variables	239.48 ms	133.28 ms	44.3%
3D, 4 variables	448.89 ms	135.69 ms	69.8%

A swarm carrying one proxied variable gains nothing — the ~5x order=1 cost over inverse distance is unchanged there. The win is entirely the sharing, which is what a Lagrangian DDt (multiple psi_star history levels) or any multi-field swarm actually does.

monotone bypasses the cache: the limiter is data-dependent and cannot ride on a geometry-only operator. nnn/order resolution moved into SwarmVariable._resolve_stencil so the direct and cached paths cannot disagree.

2. SWARM-15 seam, measured

#430 stated the curved-field seam error was unmeasured. SWARM-15's own migration plan asks for it as step 1. Max relative error against the analytic field, same mesh and particles at each np:

field	scheme	np=1	np=2	np=4
linear	order=0	2.0e-3	2.4e-3	4.5e-3
linear	order=1	5.1e-16	5.1e-16	5.5e-16
curved	order=0	9.5e-3	1.1e-2	1.7e-2
curved	order=1	1.5e-4	1.9e-4	2.3e-4

A linear field is exactly np-independent under order=1. For a curved field np-dependence persists in both schemes, roughly doubling from np=1 to np=4 — but the absolute error at np=4 is 73x smaller under order=1. So order=1 reduces the seam by about two orders of magnitude without removing it; the halo exchange is still the real fix. Recorded in subsystems/interpolation.md, replacing the "unmeasured" wording.

Regression

level_1 623 passed, level_2 322 passed, np=2 parallel 6 passed, 0 failures. Six new tests in tests/test_0117_swarm_proxy_weight_cache.py covering value-identity with the uncached path, sharing, non-sharing across degrees, invalidation on migrate, linear exactness through the cached path, and the monotone bypass.

Artifacts: ~/+Simulations/linear-rbf-proxy/step5_*, step6_*, step7_*.

Underworld development team with AI support from Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lmoresi]

Adversarial review

The cache holds a reference to a dead kd-tree

Entries carry the tree they were built from so that identity comparison can invalidate them. That means a stale entry keeps its tree — and therefore its no-copy view of the particle coordinate buffer — alive until the key is next used. One tree per distinct key, so bounded, but for a swarm whose proxy discretisations change (an adapt loop varying degree, or callers passing different nnn) the dict only ever grows: there is no eviction. _invalidate_canonical_data does not clear it, deliberately, because identity comparison makes that unnecessary for correctness — but it would be the natural place to bound memory.

Not a leak in any run I measured; a slow one in a long adapt sequence is plausible and untested.

mesh._mesh_version is assumed to change on remesh, and I did not verify it

The key uses it to separate node layouts across an adapt. MeshVariable._get_kdtree uses the same field for the same purpose, so the assumption is consistent with existing code — but "consistent with existing code" is not "checked". If a mesh mutation bumped coordinates without bumping the version, a cached operator would survive a geometry change and silently transfer to the wrong nodes. The proxy remesh_policy="reinit" path recreates the proxy variable, which changes nothing about the cache key.

That is the failure mode worth worrying about, and it is the one thing here without a test.

The saving is real but narrower than the headline

Zero for a single proxied variable, which is a common configuration. The table shows this, but "70% saving" is what people will remember. It is also entirely conditional on variables sharing a proxy degree — a swarm mixing degree 1 and degree 2 proxies gets two cache entries and no sharing between them.

The measurement uses four identical variables. Real swarms carry variables of mixed shape (scalar, vector) and possibly mixed degree; component count does not affect the operator, but degree does.

The seam measurement compares different node sets

Node counts grow with np (728 / 758 / 790) because shared partition-boundary nodes are counted on more than one rank. So the np=4 maximum is drawn from ~8% more node instances than np=1. The doc says this. It is small against a 2.2x growth in error, but it means the reported growth is an upper bound rather than a clean measurement of the seam alone.

A cleaner design would compare values at matched coordinates against an np=1 reference, rather than error-against-analytic at each np independently. That would isolate the seam from ordinary interpolation error. I did not do it because it needs a cross-run reference file.

order=1 does not reduce relative np-sensitivity

Worth stating plainly, because the PR body leads with the 73x absolute improvement. The growth factor from np=1 to np=4 is 2.2x for order=0 and 1.5x for order=1 on the curved field — but on the quadratic field it is 1.08x for order=0 and 3.2x for order=1. So order=1 is not uniformly less np-sensitive in relative terms; it is uniformly smaller in absolute terms. The claim that survives is the absolute one.

What holds up

• The decision to build was made from a measurement, and the measurement said "no win for one variable" before any code was written.
• Invalidation is structural rather than a contract to remember, which is the specific failure mode SWARM_MODERNIZATION_DESIGN_2026-07.md criticises in the existing cache design.
• The value-identity test compares cached against freshly-rebuilt output with array_equal, not a tolerance.

[lmoresi]

Fixed the memory point from the review (c43c1f8), which turned out to be real rather than plausible.

Keying on mesh._mesh_version kept every dead entry forever — one set per mesh generation, each holding a kd-tree, which is a no-copy view of a particle coordinate buffer. Measured: two entries after a single deform.

The version leaves the key and becomes an explicit clear, so the two invalidation mechanisms each guard one thing and both stay structural:

what changed	detected by	action
mesh geometry	_mesh_version bump on deform/adapt	drop the whole cache
particle positions	migrate() replaces the kd-tree	entry not carrying the current tree is stale

Also closed the other untested assumption from the review: mesh.deform() does bump _mesh_version (0 → 1) — verified rather than inferred from MeshVariable._get_kdtree using the same field — and the operator is genuinely rebuilt afterwards, not merely re-keyed.

Two tests added: the cache stays at one entry across three successive deforms, and the operator object is not reused across one. level_1 625 passed.

The remaining review points stand as stated: no saving for a single proxied variable, sharing conditional on matching proxy degree, and the seam measurement comparing error-against-analytic at each np rather than values at matched coordinates.