feat(site): show the vendor documents a driver was built from

CHANGELOG.md

@@ -10,6 +10,7 @@ Driver versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html
 ### Added
 - **`upstream_docs`** — an optional manifest field (`url`, `title`, `kind`, `url_stability`) naming the vendor documents a driver was built against — a register map, a parameter changelog — at a semi-persistent URL. A driver decodes a device by following that document; when it moves upstream, the driver can fall behind and nobody notices for months. Recording the URL lets a watcher poll it and flag the driver for review. `url_stability` (`committed`/`stable`/`volatile`/`unknown`) records whether the manufacturer keeps the URL put, so a broken `stable` link reads as a real signal and a broken `volatile` one as routine. `nibe_local` and `myuplink` are the first to declare it, both pointing at NIBE's myUplink register changelog. `tools/validate_manifest.py` holds each entry to an `http(s)` URL and known `kind`/`url_stability` values; the field is descriptive metadata only and never reaches `index.yaml` or how a driver installs. Specified in `spec/manifest-v2.md` (V2.3), and `docs/WRITING-A-DRIVER.md` and `AGENTS.md` now tell a driver author to record it beside the older instruction to write down what they learned about the registers — where the knowledge came from is the half that was never asked for, and a field nobody is told to populate stays at two manifests out of 80. Both say what it cannot do: a document behind a login is not fetchable and belongs in a driver comment instead. Tested in `tests/test_upstream_docs.py`
 - **`watch-upstream-docs`** — the automation half of `upstream_docs`. `tools/check_upstream_docs.py` fetches every watched URL, hashes it, and diffs it against a committed baseline (`upstream-docs-state.json`); the weekly `.github/workflows/watch-upstream-docs.yml` opens a tracking issue when a document changes (registers to review) or goes missing (link rotted), weighted by `url_stability`. It notifies **once** per event — a change updates the baseline so it can't re-fire, an outage alerts only on crossing a three-run failure threshold, and the workflow skips a title that already has an open issue. Detection is a raw-byte hash, so a flagged change is a prompt to look, not proof the registers moved — the issue says so. Tested in `tests/test_upstream_docs_watch.py`
+- **The catalog page shows the vendor documents a driver was decoded from.** `upstream_docs` reached the manifests and the weekly watcher but never the page, so the one artefact most readers actually open still named 80 drivers without saying where any of them came from — and the field's whole purpose is to make that source findable again. Each declared document is now a link in the driver's card, under its title, with the kind (`Changelog`, `Register map`, …) and what `url_stability` claims. The stability wording is spelled out rather than printed as the bare enum: "stable" beside a link reads as a promise that the *registers* are settled, which is the opposite of why the field exists, so the page says "URL stable in practice". A short note explains that a flagged change is a prompt to review, not evidence the driver is wrong. Document titles join the search index, so the changelog a driver follows is findable by name. The generator re-checks the `http(s)` scheme it renders — `validate_manifest.py` holds the same line, but it is a different tool run at a different time, and this is the one place where the value becomes an `href` somebody clicks. `test_the_card_renders_the_documents_it_collects` guards the rendering rather than the payload, because collecting the documents and silently dropping them from the card is exactly the state that made this necessary
 - **The release's version rule now runs on every pull request, where it can still be acted on.** The signed channel refuses to republish changed artifact bytes under a version it has already published — the rule that makes a version name a set of bytes an operator can pin. Asking it needed the published channel, so it only ever ran from the release job on `main`, *after* the merge: #53 merged with five green checks and took the release down, and the repair (#55) had to be a second pull request against a `main` that was already broken. Nothing about the question needs the signing key, though — the published manifest is a release asset and its signature verifies with the public key — so `ftw_repository.py check-versions` builds the artifacts in memory, signs nothing, and answers exactly what the release would. The new `signed channel accepts this tree` job runs it on every pull request, including from forks, which need no secret to read a public asset. It reports **every** colliding driver rather than the first, each with the `make bump-driver` command that fixes it. `_version_collisions` is the single statement of the rule that both the release and this check read, and `test_check_versions_answers_what_the_release_would_answer` holds them to the same verdict on the same tree — a rule stated twice is a rule that drifts, and a check that drifts from the release is worse than no check. `test_check_versions_catches_a_manifest_only_edit` is #53 in the shape it arrived in: a manifest field corrected, no Lua touched, and the bytes moved anyway
 
 ### Fixed

tests/test_site.py

@@ -24,7 +24,7 @@
 sys.path.insert(0, str(ROOT / "tools"))
 
 import generate_site  # noqa: E402
-from manifest_parser import parse_yaml_simple  # noqa: E402
+from manifest_parser import parse_upstream_docs, parse_yaml_simple  # noqa: E402
 
 
 @pytest.fixture(scope="module")
@@ -99,6 +99,77 @@ def test_tested_devices_survive_intact(catalog: dict) -> None:
             assert device["model_family"], driver["id"]
 
 
+def test_upstream_docs_reach_the_page(catalog: dict) -> None:
+    """A recorded source document is only useful if a reader can open it.
+
+    The manifest field exists so the register map a driver follows can be
+    found again. Collecting it and not rendering it would leave the page
+    exactly as silent about provenance as it was before the field existed.
+    """
+    for driver in catalog["drivers"]:
+        declared = parse_upstream_docs(
+            (ROOT / "manifests" / f"{driver['id']}.yaml").read_text(encoding="utf-8"))
+        assert [doc["url"] for doc in driver["upstream_docs"]] == \
+            [doc["url"] for doc in declared], driver["id"]
+
+    total = sum(len(d["upstream_docs"]) for d in catalog["drivers"])
+    assert total, "no manifest's upstream_docs reached the page"
+
+
+def test_every_document_label_has_wording(catalog: dict) -> None:
+    """An unmapped kind or stability would reach the page as a bare enum."""
+    for driver in catalog["drivers"]:
+        for doc in driver["upstream_docs"]:
+            assert doc["kind"] in generate_site.DOC_KIND_LABELS, (
+                f"{driver['id']}: doc kind {doc['kind']!r} has no wording")
+            assert doc["url_stability"] in generate_site.URL_STABILITY_LABELS, (
+                f"{driver['id']}: url_stability {doc['url_stability']!r} "
+                "has no wording")
+
+
+def test_the_card_renders_the_documents_it_collects() -> None:
+    """Guard the rendering, not just the data behind it.
+
+    `test_upstream_docs_reach_the_page` passes just as well when the payload
+    carries the documents and the card silently drops them — which is the
+    state this test was written to end.
+    """
+    script = generate_site.SCRIPT
+    card = script[script.index("function driverCard"):
+                  script.index("function manufacturerView")]
+    assert "upstreamDocs(d)" in card, "the driver card no longer renders the documents"
+    assert "doc.url" in script and "doc.title" in script
+
+
+def test_a_document_url_must_be_http() -> None:
+    """The page turns this value into an href a reader clicks.
+
+    `validate_manifest.py` holds the same line, but it is a different tool run
+    at a different time, and a javascript: URL reaching the rendered card
+    would be this generator's bug to have prevented.
+    """
+    manifest = (
+        'upstream_docs:\n'
+        '  - url: "javascript:alert(1)"\n'
+        '    title: "not a document"\n'
+        '  - url: "https://example.invalid/registers.pdf"\n'
+        '    title: "a real one"\n'
+        'min_host_version: "2.0.0"\n'
+    )
+    kept = generate_site.upstream_documents(manifest)
+    assert [doc["url"] for doc in kept] == ["https://example.invalid/registers.pdf"]
+
+
+def test_a_document_without_a_title_still_reads_as_a_link(catalog: dict) -> None:
+    """`title` is optional in the manifest; an empty link label is not."""
+    kept = generate_site.upstream_documents(
+        'upstream_docs:\n  - url: "https://example.invalid/map.pdf"\n')
+    assert kept[0]["title"] == "https://example.invalid/map.pdf"
+    for driver in catalog["drivers"]:
+        for doc in driver["upstream_docs"]:
+            assert doc["title"].strip(), driver["id"]
+
+
 def test_every_driver_says_something(catalog: dict) -> None:
     """A row with no prose tells a reader nothing the id didn't already."""
     silent = [d["id"] for d in catalog["drivers"]

tools/generate_site.py

@@ -25,7 +25,8 @@
 from pathlib import Path
 
 sys.path.insert(0, str(Path(__file__).resolve().parent))
-from manifest_parser import parse_tested_devices, parse_yaml_simple
+from manifest_parser import (
+    parse_tested_devices, parse_upstream_docs, parse_yaml_simple)
 
 ROOT = Path(__file__).resolve().parents[1]
 REPO_URL = "https://github.com/srcfl/device-drivers"
@@ -53,6 +54,26 @@
     "community": "Community",
     "oem": "OEM",
 }
+# The vendor material a driver was decoded from. A reader checking whether a
+# register is still right needs the document itself, not the note that one
+# exists, so the manifest's URL is rendered as a link.
+DOC_KIND_LABELS = {
+    "changelog": "Changelog",
+    "register_map": "Register map",
+    "manual": "Manual",
+    "api_docs": "API docs",
+    "firmware_notes": "Firmware notes",
+    "other": "Document",
+}
+# url_stability describes the URL, not the registers behind it. Left as the bare
+# word, "stable" beside a link reads as a claim that the document's contents are
+# settled, which is the opposite of why the field exists.
+URL_STABILITY_LABELS = {
+    "committed": "Vendor commits to this URL",
+    "stable": "URL stable in practice",
+    "volatile": "URL known to rotate",
+    "unknown": "URL durability not assessed",
+}
 # A driver's own verification_status is the only hardware claim on the page, so
 # it is never flattened into a single "verified" mark. Most drivers are ported
 # from a reference implementation and say so; the page has to say so too.
@@ -168,6 +189,28 @@ def read_driver_source(path: Path) -> dict:
     return info
 
 
+def upstream_documents(text: str) -> list[dict]:
+    """The vendor documents a manifest declares, as the page will link them.
+
+    The scheme is checked here rather than assumed. `validate_manifest.py`
+    already holds these to `http(s)`, but that is a different tool run at a
+    different time, and this one turns the value into an `href` a reader
+    clicks — the one place where trusting the input is not free.
+    """
+    documents = []
+    for entry in parse_upstream_docs(text):
+        url = (entry.get("url") or "").strip()
+        if not re.match(r"^https?://", url, re.IGNORECASE):
+            continue
+        documents.append({
+            "url": url,
+            "title": (entry.get("title") or "").strip() or url,
+            "kind": entry.get("kind") or "other",
+            "url_stability": entry.get("url_stability") or "unknown",
+        })
+    return documents
+
+
 def collect_drivers() -> list[dict]:
     """Build one record per driver from the manifest, the source and the status."""
     support = json.loads(
@@ -231,6 +274,7 @@ def collect_drivers() -> list[dict]:
             "size_bytes": data.get("size_bytes", 0),
             "sha256": data.get("sha256", ""),
             "tested_devices": tested,
+            "upstream_docs": upstream_documents(text),
             "verification": verification,
             "hardware_verified": bool(
                 verification and verification["status"] in HARDWARE_VERIFIED),
@@ -275,6 +319,8 @@ def build_catalog() -> dict:
             "ders": DER_LABELS,
             "protocols": PROTOCOL_LABELS,
             "tiers": TIER_LABELS,
+            "doc_kinds": DOC_KIND_LABELS,
+            "url_stability": URL_STABILITY_LABELS,
         },
         "totals": {
             "drivers": len(drivers),
@@ -425,6 +471,12 @@ def embed_json(payload: dict) -> str:
 .facts div { min-width: 0; }
 .facts dt { color: var(--muted); font-family: var(--mono); font-size: 9px; letter-spacing: 0.1em; text-transform: uppercase; }
 .facts dd { margin: 3px 0 0; font-family: var(--mono); font-size: 12px; overflow-wrap: anywhere; }
+.docs { list-style: none; margin: 0; padding: 0; max-width: 88ch; }
+.docs li { padding: 9px 0; border-bottom: 1px dashed var(--line); }
+.docs li:last-child { border-bottom: 0; }
+.docs a { color: var(--signal); font-size: 13px; overflow-wrap: anywhere; }
+.docs .doc-meta { display: block; margin-top: 3px; color: var(--muted); font-family: var(--mono); font-size: 10px; letter-spacing: 0.08em; text-transform: uppercase; }
+.docs-note { max-width: 78ch; margin: 10px 0 0; color: var(--muted); font-size: 12px; }
 .links { display: flex; gap: 10px; flex-wrap: wrap; margin-top: 22px; }
 .links a { display: inline-flex; align-items: center; gap: 8px; min-height: 38px; padding: 0 14px; border: 1px solid var(--ink); font-size: 13px; text-decoration: none; }
 .links a:hover { background: var(--ink); color: var(--cream); }
@@ -494,6 +546,8 @@ def embed_json(payload: dict) -> str:
   function derLabel(d) { return labels.ders[d] || d; }
   function protocolLabel(p) { return labels.protocols[p] || p; }
   function tierLabel(t) { return labels.tiers[t] || t; }
+  function docKindLabel(k) { return labels.doc_kinds[k] || k; }
+  function stabilityLabel(s) { return labels.url_stability[s] || s; }
   function bytes(n) { return n >= 1024 ? (n / 1024).toFixed(1) + " kB" : n + " B"; }
 
   // One flat haystack per driver so a search matches a model number or a
@@ -506,6 +560,7 @@ def embed_json(payload: dict) -> str:
       parts.push(dev.manufacturer, dev.model_family, dev.notes);
       parts.push(dev.variants.join(" "), dev.regions.join(" "));
     });
+    d.upstream_docs.forEach(function (doc) { parts.push(doc.title, docKindLabel(doc.kind)); });
     d.haystack = parts.join(" ").toLowerCase();
   });
 
@@ -557,6 +612,27 @@ def embed_json(payload: dict) -> str:
       "<th>Stable package</th><th>Note</th></tr></thead><tbody>" + rows + "</tbody></table></div>";
   }
 
+  // The vendor material the driver decodes. Reading a register map is how
+  // somebody checks whether this driver is still right about a device, and
+  // until now the page named 80 drivers without once saying where any of them
+  // came from.
+  function upstreamDocs(d) {
+    if (!d.upstream_docs.length) return "";
+    var items = d.upstream_docs.map(function (doc) {
+      return "<li>" +
+        '<a href="' + esc(doc.url) + '" rel="noopener nofollow">' + esc(doc.title) +
+          ' <span aria-hidden="true">↗</span></a>' +
+        '<span class="doc-meta">' + esc(docKindLabel(doc.kind)) + " &middot; " +
+          esc(stabilityLabel(doc.url_stability)) + "</span>" +
+      "</li>";
+    }).join("");
+    return '<p class="subhead">Vendor documents this driver was built from</p>' +
+      '<ul class="docs">' + items + "</ul>" +
+      '<p class="docs-note">Watched weekly: when one of these changes or stops resolving, ' +
+      "the repository opens an issue to review the driver against it. A change is a prompt " +
+      "to look, not evidence that this driver is wrong.</p>";
+  }
+
   function driverCard(d) {
     var ders = d.ders.map(function (x) { return '<span class="badge">' + esc(derLabel(x)) + "</span>"; }).join("");
     var control = d.control
@@ -606,6 +682,7 @@ def embed_json(payload: dict) -> str:
         verified +
         modelsTable(d) +
         targetsTable(d) +
+        upstreamDocs(d) +
         '<p class="subhead">Facts</p><dl class="facts">' + facts + "</dl>" +
         '<div class="links">' +
           '<a href="' + esc(d.source_url) + '" rel="noopener">Read the Lua source <span aria-hidden="true">↗</span></a>' +