Scored scene-timing benchmark on the real clock

.cursor/rules/docgen-benchmark.mdc

@@ -0,0 +1,27 @@
+---
+description: Clock and scene-compile changes must pass docgen benchmark against the committed baseline.
+alwaysApply: true
+---
+
+# Required gate: `docgen benchmark`
+
+String checks on compiled `scenes.py` and `simulate_reveal_timeline` are **not**
+enough. Production uses `_TimedScene._clock` inside `construct()`.
+
+**After any change** to scene compile, clock math, `_TimedScene` / bootstrap
+helpers, dwell / hold pulses, `scene_asset_validate`, or the benchmark corpus
+itself, you **must**:
+
+1. Run **`docgen benchmark`** (or **`./scripts/benchmark-scenes.sh`**).
+2. Leave it green vs **`src/docgen/benchmark_data/baseline.json`** (exit 0).
+3. Keep the **`benchmark`** job in **`.github/workflows/ci.yml`** (it must
+   invoke `docgen benchmark`). Do not delete or skip that job.
+4. Keep **`tests/test_scene_benchmark.py`** in the default pytest run.
+
+`--update-baseline` is allowed **only** when the scorecard change is the point
+of the PR. Review the baseline JSON diff. Do not bump the baseline to hide a
+regression.
+
+When a consumer render fails in a new way, **add a case** to
+`standard_cases()` in `src/docgen/scene_benchmark.py` and update the baseline
+in the same PR.

.cursor/rules/docgen-tools-only.mdc

@@ -23,7 +23,10 @@ how a *consumer* bundle (typically `docs/demos/`) is expected to be operated.
 
 **Manim diagram segments:** use **`docgen scene-spec-generate`** / **`scene-compile`** with committed **`animations/specs/*.scene.yaml`** instead of hand-editing **`scenes.py`** inside generated regions. Subject-beat coverage (hold board on same topic; cover topic shifts; no invented labels) is enforced by **`scene-spec-generate`** and **`validate`** — not a blind label count.
 
-**Surviving CLI commands (indicative):** `init`, `wizard`, `tts`, `timestamps`, `manim`, `compose`, `validate`, `lint`, `narration-generate`, `scene-spec-generate`, `scene-compile`, `yaml-generate`, `clean-bundle`, `concat`, `pages`, `generate-all`, `rebuild-after-audio`.
+**Surviving CLI commands (indicative):** `init`, `wizard`, `tts`, `timestamps`, `manim`, `compose`, `validate`, `lint`, `narration-generate`, `scene-spec-generate`, `scene-compile`, `yaml-generate`, `clean-bundle`, `concat`, `pages`, `generate-all`, `rebuild-after-audio`, `benchmark`.
+
+**Clock / compile changes:** run **`docgen benchmark`** and keep the CI
+`benchmark` job. See **`.cursor/rules/docgen-benchmark.mdc`**.
 
 **If the right command does not exist:** implement it in **docgen** (or a thin wrapper script committed to the consumer repo), then use it — do not bypass with manual file edits.
 

.github/workflows/ci.yml

@@ -60,3 +60,16 @@ jobs:
           done
       - run: pip install --no-cache-dir ".[dev]"
       - run: pytest tests/ -v --tb=short
+
+  # Required gate for clock / scene-compile changes. Do not remove: tests
+  # assert this job runs `docgen benchmark` against the committed baseline.
+  benchmark:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+      - run: pip install --no-cache-dir "."
+      - name: Scene-timing benchmark
+        run: docgen benchmark

AGENTS.md

@@ -58,26 +58,40 @@ Commands registered on the **`docgen`** CLI include:
 - **`pages`** — emit static HTML for demo assets.
 - **`generate-all`** — orchestrated pipeline: TTS → timestamps → **scene specs** (auto `scene-spec-generate` when `animations/specs/` is empty; otherwise offline retime) → images → Manim → compose → validate → concat → pages. `--regen-scene-specs` forces LLM rewrite; `--skip-scene-retime` keeps legacy hand `scenes.py` only.
 - **`rebuild-after-audio`** — same as generate-all with TTS skipped (still retimes scenes after timestamps).
+- **`benchmark`** — score the packaged scene-timing corpus (no bundle / Manim / OpenAI). Executes compiled ``construct()`` on the real ``_TimedScene`` clock and diffs ``src/docgen/benchmark_data/baseline.json``. Exit 1 on regression. ``--update-baseline`` only after an intentional improvement.
 
 ## Implications for changes here
 
 - **Manim / `scenes.py` (marker blocks):** Fix generators under `src/docgen/**` (`manim_scene_support.py`, `scene_spec.py`, `scene_spec_generate.py`, `validate`, `yaml_generate`, tests). **Do not** patch generated classes inside a consumer's **`animations/scenes.py`** between **`BEGIN/END GENERATED SCENE`** markers; re-run **`scene-spec-generate`** / **`scene-compile --retime`** and **`manim`** instead. Preferred consumer order: narration → TTS → timestamps → scene-spec/compile → Manim → compose.
 - **Beat sync (fail-closed):** when `timing.json` has words, every story box label must match a spoken phrase (`wait_word`); unmatched labels and leftover LLM indices are rejected. Opt out with ``pace: none``. Fuzzy containment matching is not used. **`scene-compile` clamps FadeIn / page-fade `run_time` against the next word start** so `_TimedScene._clock` cannot race past waits (issue #66 — do not emit cascading first-board dumps). After a reveal, a **dwell** slot may play `Indicate` / `Circumscribe` when the gap to the next `wait_word` is long enough (also clamped). Long holds emit additional mid-hold pulses (`timed_wait` + emphasis) so the board does not freeze after the first Indicate. Optional box fields: `shape` (rounded/pill/diamond), `reveal` (fade/grow/slide), `emphasis` (none/pulse/ring). Page transitions FadeOut revealed boxes, not the parent `VGroup`. `scene-compile` refreshes stale `_box` / `_arrow` / `_TimedScene` helpers in `scenes.py`.
 - **Subject-beat coverage:** implemented in `scene_spec.layout_density_violations` / `cluster_subject_beats`; enforced by **`scene-spec-generate`** and **`validate`** (`validation.subject_beat_coverage.enabled`, default true). Not a blind label count.
+- **`docgen benchmark` is required** after clock / compile / `_TimedScene` / dwell changes. Pytest string assertions are not a substitute. Do not remove the CI `benchmark` job. See **Required gate** below.
 - Prefer **stable CLI / library contracts** and **documented exit codes** so CI can depend on them.
 - **`narration_from_source`:** hints in config + **`docgen narration-generate`** — owner-supplied context paths, not opaque bulk edits to outputs.
 - Avoid duplicating long orchestration docs here; **link** to downstream repos when describing their publish pipelines.
 
+## Required gate: `docgen benchmark`
+
+Clock, compile, scene-spec motion, `_TimedScene` helpers, and dwell/hold changes **must** stay green on the packaged corpus:
+
+1. Run **`docgen benchmark`** (or **`./scripts/benchmark-scenes.sh`**). Exit 0 vs ``src/docgen/benchmark_data/baseline.json``.
+2. Keep **`tests/test_scene_benchmark.py`** in the default ``pytest tests/`` run.
+3. Keep the **`benchmark`** job in **`.github/workflows/ci.yml`** (it must invoke ``docgen benchmark``). Do not delete or skip it.
+4. **`--update-baseline`** only when the scorecard change is the point of the PR; review the JSON diff. Do not bump the baseline to hide a regression.
+5. When production fails in a new way, add a case to ``standard_cases()`` and update the baseline in the same PR.
+
+String assertions on compiled ``scenes.py`` and ``simulate_reveal_timeline`` are not a substitute — the harness executes ``construct()`` on the real clock.
+
 ## Testing (downstream relevance)
 
-Tests should cover **CLI-visible behavior** and contracts that adopters rely on: **`yaml-generate`**, **`scene-spec-generate`**, **`scene-compile`**, **`validate`**, **`compose`**, **`generate-all`**, **`pages`**, **`init`**, **config** loading (`repo_root`, `env_file`), and package exports. Use small in-tree fixtures; this library does not ship a dogfood bundle.
+Tests should cover **CLI-visible behavior** and contracts that adopters rely on: **`yaml-generate`**, **`scene-spec-generate`**, **`scene-compile`**, **`validate`**, **`compose`**, **`generate-all`**, **`pages`**, **`init`**, **`benchmark`**, **config** loading (`repo_root`, `env_file`), and package exports. Use small in-tree fixtures; this library does not ship a dogfood bundle. Clock / compile changes must keep ``docgen benchmark`` at or above ``benchmark_data/baseline.json`` — string assertions on ``scenes.py`` are not enough.
 
 ## Cursor Cloud specific instructions
 
 - **Virtualenv:** the project is installed editable into **`/workspace/.venv`** (created by the startup update script). Shells do **not** auto-activate it — run `. /workspace/.venv/bin/activate` (or prefix the venv path) before `docgen`, `pytest`, or `ruff`. The `docgen` console script lives at `/workspace/.venv/bin/docgen`.
 - **System deps are pre-baked in the VM snapshot** (not the update script): `ffmpeg` + `tesseract-ocr` (validation/compose/OCR), plus `build-essential`, `python3-dev`, `libcairo2-dev`, `libpango1.0-dev`, `pkg-config` (needed to build the `manim` extra's `manimpango`/`pycairo` wheels). If a fresh VM ever lacks these, reinstall via apt before `pip install`.
-- **Standard commands** are in `README.md` / `pyproject.toml` / `.github/workflows/ci.yml`: lint `ruff check src/ tests/`; tests `pytest tests/ -v --tb=short`; the CI unit job also exports `PYTHONPATH=src` (not needed locally because of the editable install, but harmless).
-- **OpenAI-gated vs offline commands:** `tts`, `timestamps --engine whisper`, `image-generate`, `narration-generate`, `scene-spec-generate`, and `yaml-generate --llm` call OpenAI and need `OPENAI_API_KEY` (integration tests auto-skip without it). Fully offline: `init`, `scene-compile`, `manim`, `compose`, `validate`, `lint`, `pages`, `concat`, `yaml-generate` (no `--llm`), and `timestamps` (default `local` engine).
+- **Standard commands** are in `README.md` / `pyproject.toml` / `.github/workflows/ci.yml`: lint `ruff check src/ tests/`; tests `pytest tests/ -v --tb=short`; **required** `docgen benchmark` (CI job `benchmark`). The CI unit job also exports `PYTHONPATH=src` (not needed locally because of the editable install, but harmless).
+- **OpenAI-gated vs offline commands:** `tts`, `timestamps --engine whisper`, `image-generate`, `narration-generate`, `scene-spec-generate`, and `yaml-generate --llm` call OpenAI and need `OPENAI_API_KEY` (integration tests auto-skip without it). Fully offline: `init`, `scene-compile`, `manim`, `compose`, `validate`, `lint`, `pages`, `concat`, `yaml-generate` (no `--llm`), `timestamps` (default `local` engine), and `benchmark`.
 - **`scene-compile` gotcha:** paced specs (`wait_word`) need a `timing.json` entry for that stem (`docgen timestamps` after TTS). Prefer `scene-compile --retime` after fresh timestamps; for a fully offline smoke render, author rows without wait indices only if you accept unpaced reveals.
 - **No in-repo dogfood bundle:** exercise the pipeline against a scratch bundle (`docgen init /tmp/<name> --defaults` in a throwaway git dir). Do not hand-edit consumer generated assets (see `.cursor/rules/no-asset-edits.mdc`).
 - **Wizard:** `docgen wizard --port 8501` is an optional local Flask app (long-lived); run it from a bundle directory that contains `docgen.yaml`.

README.md

@@ -101,9 +101,11 @@ python3 -m venv .venv
 source .venv/bin/activate
 pip install -e ".[dev]"
 pytest
+docgen benchmark          # required: scene-timing corpus vs committed baseline
 ```
 
-CI installs `ffmpeg` and `tesseract` via apt — see `.github/workflows/ci.yml`.
+CI runs **`ruff`**, **`pytest`**, and a required **`benchmark`** job (`docgen benchmark`).
+CI also installs `ffmpeg` and `tesseract` via apt for unit tests — see `.github/workflows/ci.yml`.
 
 **Roadmap:** [milestones/README.md](milestones/README.md).
 
@@ -140,6 +142,7 @@ docgen validate --pre-push
 | `docgen yaml-generate [--merge-defaults] [--llm] [--dry-run] [--list-gaps]` | Merge defaults into `docgen.yaml`; optional OpenAI refresh of `tts.instructions` / `wizard.system_prompt` (rewrites the file — review in Git) |
 | `docgen scene-compile [SPEC.scene.yaml \| --all] [--retime] [--dry-run]` | Compile declarative scene YAML into `animations/scenes.py`. **`--all --retime`** re-derives `wait_word` from current `timing.json` with no OpenAI; unmatched labels fail closed (or set `pace: none`) |
 | `docgen scene-spec-generate [--segment 01 \| --all] [--compile] [--print-only] [--output PATH] [--hint …] [--model …]` | Call OpenAI to emit YAML only (same schema as `scene-compile`); rejects frame-budget overflow and **subject-beat coverage** failures (hold board on same topic; cover topic shifts; no invented labels — not a blind count); auto-paginate + word-alignment; optionally writes `animations/specs/<stem>.scene.yaml` and `--compile`s into `scenes.py` |
+| `docgen benchmark [--case ID] [--format text\|json] [--update-baseline]` | Score the **standard scene-timing corpus** (execute compiled `construct()` on the real `_TimedScene` clock, no Manim). Diffs a committed baseline so clock changes are measured, not guessed |
 
 ## Configuration
 

pyproject.toml

@@ -37,7 +37,7 @@ docgen = "docgen.cli:main"
 where = ["src"]
 
 [tool.setuptools.package-data]
-docgen = ["templates/*.html", "static/*.css", "static/*.js"]
+docgen = ["templates/*.html", "static/*.css", "static/*.js", "benchmark_data/*"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

scripts/benchmark-scenes.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# Score the standard scene-timing corpus (no bundle, no Manim, no OpenAI).
+# Usage:
+#   ./scripts/benchmark-scenes.sh
+#   ./scripts/benchmark-scenes.sh --format json
+#   ./scripts/benchmark-scenes.sh --update-baseline   # after an intentional improvement
+set -euo pipefail
+root="$(cd "$(dirname "$0")/.." && pwd)"
+if [[ -x "${root}/.venv/bin/docgen" ]]; then
+  exec "${root}/.venv/bin/docgen" benchmark "$@"
+fi
+exec docgen benchmark "$@"

src/docgen/benchmark_data/README.md

@@ -0,0 +1,33 @@
+# Scene-timing benchmark corpus
+
+Standard scripts scored by ``docgen benchmark`` / ``scripts/benchmark-scenes.sh``.
+
+Each case is a **fixed spec + Whisper-shaped word list** that encodes a
+production failure we have already shipped by accident:
+
+| id | What it proves |
+| --- | --- |
+| `issue66_tight_clamped` | Tight spoken labels + long authored `run_time` must **not** skip waits after compile clamp |
+| `issue66_tight_unclamped` | **Control** — same spec compiled *without* words still dumps (scorer is not blind) |
+| `early_title` | Title `Write` must shrink so a 0.55s first label is not skipped |
+| `wide_hold` | Long subject-beat holds get more than one pulse |
+| `emphasis_none` | Author opt-out stays still (no false “stuck” fail) |
+| `paged_slide` | Slide page transition does not skip the next `wait_word` |
+| `flow_edges` | Grow + edge-to-edge arrow still paces |
+| `audio_tail` | Last box keeps moving through a long tail |
+
+`baseline.json` is the last accepted scorecard. A change that raises
+`defect_points` or drops `quality_points` fails the command (exit 1).
+
+This command is a **required gate**: CI job `benchmark` in
+`.github/workflows/ci.yml`, `tests/test_scene_benchmark.py` in the default
+pytest run, and `.cursor/rules/docgen-benchmark.mdc` for agents. Do not
+remove those hooks.
+
+After an **intentional** improvement:
+
+```bash
+docgen benchmark --update-baseline
+```
+
+Review the baseline diff in Git. Do not bump it to hide a regression.

src/docgen/benchmark_data/baseline.json

@@ -0,0 +1,117 @@
+{
+  "version": 1,
+  "cases": {
+    "issue66_tight_clamped": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 9,
+      "box_reveals": 3,
+      "last_motion_frac": 0.9877,
+      "defect_points": 0,
+      "quality_points": 20,
+      "score": 100
+    },
+    "issue66_tight_unclamped": {
+      "role": "control",
+      "wait_skips": 2,
+      "overshoots": 2,
+      "hold_idle_violations": 0,
+      "cadence_violations": 1,
+      "sim_drift": 1,
+      "mid_hold_pulses": 0,
+      "box_reveals": 3,
+      "last_motion_frac": 0.1407,
+      "defect_points": 80,
+      "quality_points": 0,
+      "score": 20
+    },
+    "early_title": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 4,
+      "box_reveals": 2,
+      "last_motion_frac": 0.9855,
+      "defect_points": 0,
+      "quality_points": 16,
+      "score": 100
+    },
+    "wide_hold": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 8,
+      "box_reveals": 3,
+      "last_motion_frac": 0.9951,
+      "defect_points": 0,
+      "quality_points": 20,
+      "score": 100
+    },
+    "emphasis_none": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 0,
+      "box_reveals": 2,
+      "last_motion_frac": 0.5399,
+      "defect_points": 0,
+      "quality_points": 4,
+      "score": 100
+    },
+    "paged_slide": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 4,
+      "box_reveals": 2,
+      "last_motion_frac": 0.9885,
+      "defect_points": 0,
+      "quality_points": 16,
+      "score": 100
+    },
+    "flow_edges": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 4,
+      "box_reveals": 2,
+      "last_motion_frac": 0.9903,
+      "defect_points": 0,
+      "quality_points": 16,
+      "score": 100
+    },
+    "audio_tail": {
+      "role": "quality",
+      "wait_skips": 0,
+      "overshoots": 0,
+      "hold_idle_violations": 0,
+      "cadence_violations": 0,
+      "sim_drift": 0,
+      "mid_hold_pulses": 5,
+      "box_reveals": 1,
+      "last_motion_frac": 0.9935,
+      "defect_points": 0,
+      "quality_points": 18,
+      "score": 100
+    }
+  }
+}