Add markdown button and make llms.txt available

[neon60]

Motivation

AI coding assistants and agents increasingly need documentation in machine-readable formats. Two patterns are emerging as standards: per-page Markdown download (so users and agents can copy page content directly into an AI context window) and llms.txt (a combined documentation file that AI agents can discover and ingest). This PR adds both capabilities to make the documentation more accessible to AI-assisted workflows.

Technical Details

Three changes to docs/conf.py, plus a new docs/llms.txt:

Enable the theme download button

Sets "use_download_button": True in html_theme_options, which activates the built-in per-page download action provided by the Sphinx book theme. This allows readers to download the current page as Markdown or RST depending on the source format.

Add a curated llms.txt index

Adds a base docs/llms.txt that serves as a curated index of all documentation pages, including those sourced from RST. Sets html_extra_path = ["llms.txt"] so Sphinx copies it to the output root, making it discoverable at /llms.txt.

Add llms-full.txt generation

Adds a Sphinx build-finished hook (generate_combined_markdown) that filters and combines Markdown and RST source files into llms-full.txt. Each file is filtered through a prose detection pass that strips code fences, directive options, MyST/RST anchors, and inline HTML. Files with fewer than 10 prose lines are skipped entirely to avoid including navigation-only or boilerplate pages. Files in build, static, template, and VCS directories are excluded.

This follows the https://llmstxt.org/ for AI agent documentation discovery, and is compatible with https://docs.readthedocs.com/platform/stable/reference/llms-txt.html.

Test Plan

• Verify the download button appears on documentation pages
• Verify llms.txt is available at {project_url}/llms.txt and contains only the curated index
• Verify llms-full.txt is available at {project_url}/llms-full.txt
• Confirm the curated index section appears at the top of llms-full.txt
• Confirm excluded directories (.git, _build, _static, _templates, .venv) are not represented in llms-full.txt
• Confirm navigation-only pages do not appear in the stitched output

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[yuva29]

Thanks for this PR! The curated llms.txt index looks well-organized and the llmstxt.org convention is a nice addition for AI-assisted workflows.

A couple of questions:

1. The commit messages mention "sync llms filter fixes from rocm-docs-core" — does rocm-docs-core already provide this functionality (or plan to)? If so, it might be cleaner to use that instead of inlining ~180 lines of filtering logic into conf.py. That way all ROCm projects get it for free and filter fixes only need to happen in one place.

2. The GPU Operator doesn't have this yet. If this is valuable, would it make more sense to add it at the rocm-docs-core level so all projects benefit?

[neon60]

Thanks for this PR! The curated llms.txt index looks well-organized and the llmstxt.org convention is a nice addition for AI-assisted workflows.

A couple of questions:

1. The commit messages mention "sync llms filter fixes from rocm-docs-core" — does rocm-docs-core already provide this functionality (or plan to)? If so, it might be cleaner to use that instead of inlining ~180 lines of filtering logic into conf.py. That way all ROCm projects get it for free and filter fixes only need to happen in one place.
2. The GPU Operator doesn't have this yet. If this is valuable, would it make more sense to add it at the rocm-docs-core level so all projects benefit?

@yuva29 Finally managed to add to rocm-docs-core. Please check the latest PR: #105