Measure Stage 2 training cost (#59, #84): 3.5 h/epoch not 36, and the paper run was ~12 epochs

[jonfroehlich]

Corrects a ~10× error in docs/curb_ramp_data_sourcing.md §7, and rescues the only surviving copy of RampNet 1.0's training telemetry into the repo. Prompted by a planning question — how long would retraining take if we hit the 500k target? — which §7 could not answer honestly, because it had no measurement behind it.

Headline: one Stage 2 epoch is 3.49 h on 16 GPUs (~56 GPU-h), not the ≳36 h / ≳580 GPU-h §7 estimated. That estimate read the README's "will take a very long time (> 24 hours)" as a per-epoch figure. The ">24 h" in fact describes the paper's whole ~12-epoch, preemption-riddled run — 44.7 h of active compute over 74.6 h of calendar time.

What this adds

The paper run's TensorBoard events survived at /gscratch/makelab/jsomeara/RampNet/stage_two/runs/experiment_1 — another student's scratch directory, one rm -rf from gone, and the only copy. All 18 files are committed here under docs/data/rampnet1_stage2_run/ (6.4 MB) with a SHA256SUMS manifest and a .gitattributes binary guard, so the measurement is replicable from a clean clone with no cluster access.

scripts/analysis/stage2_train_cost.py parses the TFRecord and protobuf framing directly — no TensorFlow, no TensorBoard, no GPU, no network:

Median step time (rank 0)	1.341 s (p25 1.339 / p75 1.345, n = 119,902)
Global batch	16 panoramas (16 GPUs × batch 1)
Steps per epoch	9,378 (150,063 train panos ÷ 16)
Wall-clock per epoch	3.49 h (~56 GPU-h)

The interquartile band is 6 ms wide across 119,902 samples — the signature of a hard I/O bound, not a compute bound (~7.7 TFLOPs/pano against an A100 is roughly 3% MFU; the cost is 8.4 MP JPEG decode + resize on ~3 cores per rank). Two consequences: cost scales with panoramas, not labels, which is what makes the projection below meaningful; and it is fixable — pre-resized panos or more dataloader CPU could cut it several-fold. Nothing here is a floor imposed by the model.

Two things the record did not have

The run went ~12 epochs, not 1. Max step 112,434 = 11.99 × 9,378, cancelled mid-epoch-12 across 15 preemptions on ckpt-all (×1.67 calendar overhead). Validation scalars land on exact multiples of 9,378, so the epoch length is confirmed independently of the step-rate arithmetic.

This is half of #84's epoch curve, already run:

epoch	1	2	3	4	5	6	7	8	9	10	11
auto-label val loss	.000520	.000478	.000463	.000466	.000458	.000468	.000470	.000473	.000484	.000487	.000487

Auto-label val loss bottoms at epoch 5 (−12% vs epoch 1), then rises monotonically through epoch 11. So at 1 epoch the model is not converged — but the headroom is small and it reverses.

The released model is nonetheless epoch 1. best_model.pth is byte-identical (cmp) to checkpoints/epoch_1_step_9378.pth, copied back by hand on 2025-06-21 over the val-best file the script had saved automatically. The documented "1 epoch" recipe describes the released artifact correctly; what it omits is that 11 further epochs were run and discarded.

Projection to the 500k target (#59, #96)

"500k" counts government ramp records, not labels — RampNet 1.0 is 278,544 records → 214,376 panos carrying 849,895 point labels (~3 views per ramp). 500k records is 1.80× the records and implies ~1.5M labels.

Schedule	Compute (16 GPUs)	Calendar on preemptible ckpt
1 epoch (the released recipe)	6.3 h (~100 GPU-h)	~10.5 h
5 epochs (1.0's auto-val optimum)	31.4 h	~2.2 d
12 epochs (what 1.0 actually ran)	75.2 h	~5.2 d

Stage 2 retraining is an overnight job, and the epoch count is worth ~12× what the data doubling is worth. §7's "about a week" stands as an order of magnitude — but it is essentially all Stage 1 (~170k new panos × 32 tiles ≈ 5.5M tile requests, rate-limit bound, still unmeasured).

Replication

python scripts/analysis/stage2_train_cost.py                  # the measurement
python scripts/analysis/stage2_train_cost.py --records 500000 # the projection
python scripts/analysis/stage2_train_cost.py --verify         # re-check SHA256SUMS

What this does not claim, and what is missing

• Does not close Stage 2 trains for 1 epoch at constant LR (~9.4k steps): measure the epoch curve, and with it the auto-label ceiling #84. The human half of the epoch curve is still unrun — evaluating epochs 1/3/5/8 against manual_gold. If auto-label val loss keeps improving while human F1 does not, that is the Stage 1 label ceiling. The epoch-N weights are gone (a 2025-07-11 cleanup left only epoch 1), so that half needs a re-run — ~28 h of compute on the 1.0 data at the measured rate.
• The code that produced the run is not in git. The public history begins with a squashed "Initial Commit" dated 2025-07-15, a month after the run, already carrying num_epochs = 1. The run-time value was ≥ 12 and is unrecoverable.
• The Slurm record is incomplete. sacct -u jsomeara -S 2025-06-16 -E 2025-06-22 shows only three train_curb_ramp_detector jobs of 2–3 h each, because --requeue overwrites the accounting record; the event files are the only evidence of the other 12 segments. The event-file hostnames (g3050, g3060, g3082) are what tie the logs to those 4-node jobs.
• The hardware claim is looser than the README's. README says 16× L40s; the surviving Slurm records show those incarnations holding A100s, under --constraint='l40s|l40|a40|a100'. Immaterial while I/O-bound (1.34–1.53 s/step across segments on mixed hardware), but the paper's phrasing is more specific than the evidence supports.
• The projection carries 1.0's ratios. 0.770 panos/record and the 70.0% train share will shift with city density; and at a fixed 1 epoch, 1.8× the data silently becomes a 1.8× longer schedule — a change in training regime, not only in cost.

No code paths change; docs/ and scripts/analysis/ only.

🤖 Generated with Claude Code (claude-opus-5[1m])