dy_d_best: fit NNS.stack once per call (major speedup, identical resu…#133
Merged
Conversation
…lts)
The stack fit (CV n.best, dimension-reduction coefficients, blend weight)
depends only on (x, y), not on the evaluation points, and every regressor shares
the same X* = rowMeans(x) training design. Previously dy_d_best ran a full
5-fold NNS.stack CV once per (regressor x bandwidth): dy_d_best(wrt=1:5) issued
20 stack fits to predict a handful of points.
Now every regressor's bandwidth blocks (and mixed-derivative corners) are
gathered and predicted from a SINGLE NNS.stack fit:
* _dy_d_best_scalar -> _dy_d_best_plan: builds the test chunks and returns a
reducer instead of calling the engine;
* dy_d_best gathers all plans' chunks, runs one _dy_d_best_reg_estimates call,
and hands each regressor's slice back to its reducer.
On Example 2 (n=1001, wrt=1:5): 24.2s/20 stack calls -> 1.55s/1 stack call
(~15x), with byte-for-byte identical output [0.9396, 0.9476, 0.9331, 0.92,
0.9284]. Restores tractability of the large paper examples on the CV side (the
remaining cost is the engine's all-pairs RPM prediction).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rediction
Universal fix below dy.d_ in the shared multivariate prediction kernels, so
NNS.reg, NNS.stack, NNS.boost and dy.d_ all benefit.
_mreg_predict_path previously materialised the full query-by-RPM distance matrix
and ran a full stable sort of every RPM row for every query, then computed the
whole k = 1..kmax path. For a Stack CV fold at n = 100k that is a ~20k x 80k
(12.8 GB) matrix plus 20k full 80k-element sorts.
Now:
* _stable_topk(): exact partial selection of the k nearest rows -
O(n + k log k) per query instead of O(n log n) - byte-identical to
np.argsort(kind='stable')[:, :k], including heavy distance ties (the
duplicated 1-D synthetic coordinates fed by dy.d_ cbind(X*, X*));
* _mreg_predict_path() streams queries in row-chunks (bounded memory,
~O(chunk x n) instead of O(m x n)) and uses _stable_topk;
* _mreg_predict() (single selected n.best) also uses _stable_topk.
Exact: 344 invariant tests pass unchanged; dy_d_best output is bit-identical.
At n = 8000, kmax = 90 the CV-scale path is ~2.4x faster (grows with n) and the
materialised matrix drops from 128 MB to a 32 MB chunk. New tests pin
_stable_topk and the chunked path to a full-sort reference.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…lts)
The stack fit (CV n.best, dimension-reduction coefficients, blend weight) depends only on (x, y), not on the evaluation points, and every regressor shares the same X* = rowMeans(x) training design. Previously dy_d_best ran a full 5-fold NNS.stack CV once per (regressor x bandwidth): dy_d_best(wrt=1:5) issued 20 stack fits to predict a handful of points.
Now every regressor's bandwidth blocks (and mixed-derivative corners) are gathered and predicted from a SINGLE NNS.stack fit:
On Example 2 (n=1001, wrt=1:5): 24.2s/20 stack calls -> 1.55s/1 stack call (~15x), with byte-for-byte identical output [0.9396, 0.9476, 0.9331, 0.92, 0.9284]. Restores tractability of the large paper examples on the CV side (the remaining cost is the engine's all-pairs RPM prediction).
🤖 Generated with Claude Code