docs: support Sphinx 9

[Himanshuagrawal4]

AI-assisted draft: OpenAI Codex assisted with the initial implementation and validation. The contributor manually reviewed and approved the changes before this draft was opened. This PR will remain a draft until all prose and the PR description have been rewritten and finalized by the contributor in accordance with the project policy.

Closes #9189

Summary

• remove the temporary Sphinx 8 upper bound
• replace the removed autosummary get_documenter helper with local registry-based documenter selection
• retain the existing priority behavior and DataDocumenter fallback

Validation

• Ruff check and format checks pass

• the complete documentation tree renders under Sphinx 9.1.0 without the previous import/API failure

• the build still reports existing optional-dependency and ambiguous-reference warnings unrelated to this change

• Tests added / passed

• Passes pixi run lint (focused Ruff checks pass; the full environment was not available locally)

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    39 files   -     1      39 suites   - 1   14h 11m 31s ⏱️ - 51m 42s
 4 159 tests ±    0   3 980 ✅  -     1    178 💤 ±  0  1 ❌ +1 
78 229 runs   - 2 711  74 131 ✅  - 2 572  4 097 💤  - 140  1 ❌ +1 

For more details on these failures, see this check.

Results for commit e6fc223. ± Comparison against base commit 40fcd99.