ci: add DBR LTS install check to catch ES-1960554-class regressions

[vikrantpuppala]

What

Adds a DBR LTS Install CI check that builds the connector wheel and installs it inside real DBR LTS clusters (via the PECO workspace Jobs API — no PyPI publish needed), then runs a SELECT 1 smoke test. Matrix = supported LTS {13.3, 14.3, 15.4, 16.4, 17.3} × install target {base, pyarrow, kernel}.

Also adds an incident-linked comment on the thrift pin in pyproject.toml so nobody re-widens it before the upstream fix ships.

Why

The thrift 0.23.0 bump (PR #796, shipped in 4.2.7) broke pip install on DBR LTS — SEV0 ES-1960554. thrift ships sdist-only, and 0.23.0's setup.py calls sys.exit(0) on the build-success path, killing the PEP 517 backend before pip writes output.json. On the old setuptools shipped by DBR 14.3/15.4 LTS this is a hard install failure. 4.2.7 was yanked and the bump reverted (PR #840).

Our CI never caught it because every job installs via poetry install on a modern runner — it never does a fresh pip install of the built wheel on an LTS toolchain, which is the real customer path that failed. This PR closes exactly that gap.

How it works

Per matrix leg, scripts/dbr_lts_install_check.py (driver, runs on the GH runner):

1. builds the wheel and uploads it to a UC Volume on the PECO workspace,
2. imports scripts/dbr_lts_smoke_notebook.py into the workspace,
3. submits a one-off Job on an ephemeral single-node SINGLE_USER cluster pinned to the target spark_version; the notebook pip installs the wheel (+ extras) and runs SELECT 1,
4. polls to completion (tolerating transient API errors) and exits non-zero on any non-SUCCESS,
5. cleans up the per-run wheel dir + notebook in a finally (every exit path).

Several non-obvious DBR-cluster constraints are baked in and commented (notebook_task not spark_python_task-from-Volume; SINGLE_USER access mode for UC/Volume access; dbutils.fs.cp the wheel off /Volumes; dbutils.library.restartPython() after install).

Gating

Runs on pull_request, but the cluster matrix runs only when dependency-affecting files change (pyproject.toml / poetry.lock / this workflow / the two scripts) — the only surface that can introduce this failure class. Informational check (not a required merge-queue gate). Uses only secrets already present in the azure-prod environment (DATABRICKS_HOST, DATABRICKS_TOKEN, TEST_PECO_WAREHOUSE_HTTP_PATH); the notebook-import dir is derived from the token's own identity.

Validation

Exercised end-to-end against the PECO workspace:

• ✅ Green on the current pin (thrift 0.22.0) — install + SELECT 1 pass on 15.4 for both base and [pyarrow].
• ✅ Guard proven: re-widening the pin to <0.24.0 fails on 14.3 and 15.4 with the exact incident error (Downloading thrift-0.23.0.tar.gz → OSError ... output.json) — a true guard, not a check that always passes.
• ✅ Cleanup verified: wheel dir + notebook confirmed removed after a run.

Follow-ups (external / not in this PR)

• Confirm the CI service principal behind the azure-prod DATABRICKS_TOKEN can create job clusters and write to the peco.default.ci_wheels Volume.
• Re-widen the thrift pin (restoring the CVE remediation Bump thrift to 0.23.0 #796 intended) once a fixed thrift (0.23.1 / 0.24.0, per THRIFT-6067 / THRIFT-6067: [py] Remove sys.exit() from setup.py to fix PEP 517 builds on setuptools < 69 apache/thrift#3584) ships — this check will guard that change.

This pull request and its description were written by Isaac.