feat(data_service): per-cell loss trajectory + curve-derived shape (GetMetaData)#261
Merged
Merged
Conversation
Port the demo's loss_trajectory embedding into dev's GetMetaData path and integrate the shape classification: for each visible sample, attach its loss curve as a 'loss_trajectory' array stat AND classify that same curve into an authoritative tag:loss_shape, so the on-cell sparkline and its shape-haze are one computation and cannot disagree. Guarded no-op when no per-sample loss signal exists; signal name overridable via WL_LOSS_TRAJ_SIGNAL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AlexGrayBox
force-pushed
the
feat/per-sample-signal-endpoints
branch
from
July 21, 2026 19:42
d94d4e2 to
e590235
Compare
…ger snapshot) Two independent clusters have been red on dev's Code CI for many commits — both stale tests left behind by removed features, not real bugs: A) ui_docker_bridge: f45510b ('Docker-free UI') renamed ui_docker_bridge.py -> cli.py and moved cert scripts to ui/utils/, but a later merge re-added the orphaned ui_docker_bridge.py (imported by nothing, pointing at deleted docker/docker/utils/ paths + a build-and-deploy.sh that no longer exists). cli.py + test_cli.py (29 green) supersede it -> delete the dead file + test. B) logger snapshot: 671270b (#257) replaced the chunked-zstd logger snapshot with DuckDB (loggers.duckdb via flush_logger_to_disk), removing save_logger_snapshot but leaving its tests + one caller behind: - repoint test_logger_queue_saved_with_weights to assert loggers.duckdb - delete test_logger_snapshot_rotation (tested only the removed writer) - experiment_service: note-persist path guarded hasattr(save_logger_snapshot) -> always False -> notes silently not persisted; swap to flush_logger_to_disk Verified: repointed test passes against DuckDB; test_checkpoint_workflow 15/15; test_cli 29/29; full suite collects 1075 clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment on lines
+3574
to
+3624
| # -- Per-sample loss trajectory + its shape (one source for the UI) --- | ||
| # For each visible sample attach its loss curve as an 'array' DataStat | ||
| # (the on-cell sparkline) AND classify THAT SAME curve into tag:loss_shape, | ||
| # so the drawn trajectory and its shape-haze come from one computation and | ||
| # can never disagree. The curve-derived shape is authoritative: it replaces | ||
| # any dataframe-provided tag:loss_shape for rows that have a trajectory; | ||
| # rows with no loss history keep whatever the dataframe carries. Guarded and | ||
| # a no-op for experiments with no per-sample "loss" signal. | ||
| traj_signal = os.environ.get("WL_LOSS_TRAJ_SIGNAL", "loss") | ||
| try: | ||
| from weightslab.backend import ledgers | ||
| from weightslab.src import classify_loss_shape | ||
|
|
||
| logger_q = ledgers.get_logger() | ||
| if logger_q is not None: | ||
| traj = {} | ||
| for sid, step, val, _ in logger_q.query_per_sample( | ||
| traj_signal, sample_ids=sample_ids): | ||
| traj.setdefault(str(sid), []).append((int(step), float(val))) | ||
| if traj: | ||
| logger.info("[loss_traj] page_ids=%d with_history=%d signal=%s", | ||
| len(sample_ids), len(traj), traj_signal) | ||
| for i, sid in enumerate(sample_ids): | ||
| pts = traj.get(str(sid)) | ||
| if not pts: | ||
| continue | ||
| pts.sort() | ||
| vals = [v for _, v in pts] | ||
| try: | ||
| shape = classify_loss_shape(vals, min_points=3) | ||
| except Exception: | ||
| shape = None | ||
| # Downsample the curve for display (endpoints preserved). | ||
| if len(vals) > 32: | ||
| idx = np.linspace(0, len(vals) - 1, 32).round().astype(int) | ||
| curve = [float(vals[j]) for j in idx] | ||
| else: | ||
| curve = [float(v) for v in vals] | ||
| row_stats[i].append( | ||
| _DataStat(name="loss_trajectory", type="array", | ||
| shape=[len(curve)], value=curve)) | ||
| if shape: | ||
| # Curve-derived shape wins: drop any df-provided one. | ||
| row_stats[i] = [s for s in row_stats[i] | ||
| if s.name != "tag:loss_shape"] | ||
| row_stats[i].append( | ||
| _DataStat(name="tag:loss_shape", type="string", | ||
| shape=[1], value_string=str(shape))) | ||
| except Exception: | ||
| logger.debug("loss_trajectory/shape stat skipped", exc_info=True) | ||
|
|
Member
There was a problem hiding this comment.
We should classify Loss Shape directly in the logger, such that global signal per sample shape classification occurs only when a new signal is added
guillaume-byte
requested changes
Jul 22, 2026
guillaume-byte
left a comment
guillaume-byte
left a comment
Member
There was a problem hiding this comment.
Need to move the signal per sample shape cls to the logger
Per review (Guillaume): don't classify loss shapes inside _build_metadata_only_response — that runs per GetMetaData request. The shape is already computed on the write/train path by enable_loss_shape_signal / write_loss_shapes (cadenced + exp_hash-scoped), which maintains the tag:loss_shape dataframe column. So ship ONLY the loss_trajectory curve here and let the haze read the pre-computed tag column — no per-request classification, correct exp_hash scoping, one source for the shape. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… levels) Extract the query/group/downsample into guard-claused helpers (_loss_trajectory_curves + _downsample_curve). The GetMetaData call site is now a flat 'for sid: if curve: append' (was try>if>if>for>if). Behavior-preserving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AlexGrayBox
force-pushed
the
feat/per-sample-signal-endpoints
branch
from
July 22, 2026 13:22
4032964 to
206e499
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a per-cell loss trajectory to the
GetMetaDatapath and integrates the shape classification with it, in_build_metadata_only_response:loss_trajectoryarray DataStat (the on-cell sparkline the studio already renders), viaquery_per_sample(WL_LOSS_TRAJ_SIGNAL, sample_ids).classify_loss_shapeand emit it as the authoritativetag:loss_shape— so the drawn trajectory and its shape-haze come from one computation and can never disagree. Rows with no loss history keep whatevertag:loss_shapethe dataframe carries.Guarded, additive (+51/-0, one file). No-op for any experiment without a per-sample loss signal. Signal name overridable via
WL_LOSS_TRAJ_SIGNAL(default"loss").Why
This capability existed only as a local patch in the LIBERO showcase's pinned 1.3.3 SDK — so the demo would lose its per-cell curves on current
dev. This lifts it intodevcleanly, built ondev's existingquery_per_sampleprimitive (not the superseded__history__sentinel fromsignal-executor-wip).Verification (end-to-end on dev SDK)
Ran the LIBERO showcase on this branch (
dev+ this port) with the demo's real logger snapshot:load_snapshot→ 13 per-sample graphs restored;query_per_sample("loss")→ 21,724 rows / 5,431 samples;classify_loss_shape→ correct shapes (decreasing curves →monotonic).[loss_traj] page_ids=231 with_history=231 signal=loss;GetHistogramworks; auto-pause intact; no crashes.loss_trajectory+tag:loss_shape.Follow-up (not in this PR)
Frontend
LOSS_SHAPE_PRESETis missing a curated color forhigh_variance(the 6thLOSS_SHAPESvalue) — a ~2-line studio tweak, tracked separately.🤖 Generated with Claude Code