Summary
https://doc.owncloud.com/robots.txt currently 404s. The retired docs-ui repo shipped one (src/static/robots.txt):
User-agent: *
Sitemap: https://doc.owncloud.com/sitemap.xml
It was never carried over when the docs site moved to this Antora monorepo. sitemap.xml itself still builds and serves fine (200, via antora-extensions/sitemap-cleanup.js) — nothing just points crawlers at it anymore. Came up during review of #97.
Proposed fix
Add a robots.txt (the legacy content above is a reasonable starting point) wherever this Antora setup publishes root-level static assets from — e.g. via ui/supplemental/ or a static_files playbook entry.
Also worth discussing: llms.txt
https://doc.owncloud.com/llms.txt also 404s. Unlike robots.txt, this isn't a regression — no repo in the docs family (docs-main, docs-server, docs-webui, docs-ui) has ever had one. llms.txt is an emerging convention for giving LLM crawlers a curated, markdown-friendly entry point into a site. Flagging it here for visibility rather than proposing a specific fix — worth splitting into its own issue if there's appetite for it.
Verification
curl -o /dev/null -w '%{http_code}' https://doc.owncloud.com/robots.txt → 404curl -o /dev/null -w '%{http_code}' https://doc.owncloud.com/llms.txt → 404curl -o /dev/null -w '%{http_code}' https://doc.owncloud.com/sitemap.xml → 200
🤖 Generated with Claude Code
Summary
https://doc.owncloud.com/robots.txtcurrently 404s. The retireddocs-uirepo shipped one (src/static/robots.txt):It was never carried over when the docs site moved to this Antora monorepo.
sitemap.xmlitself still builds and serves fine (200, viaantora-extensions/sitemap-cleanup.js) — nothing just points crawlers at it anymore. Came up during review of #97.Proposed fix
Add a
robots.txt(the legacy content above is a reasonable starting point) wherever this Antora setup publishes root-level static assets from — e.g. viaui/supplemental/or astatic_filesplaybook entry.Also worth discussing:
llms.txthttps://doc.owncloud.com/llms.txtalso 404s. Unlikerobots.txt, this isn't a regression — no repo in the docs family (docs-main,docs-server,docs-webui,docs-ui) has ever had one.llms.txtis an emerging convention for giving LLM crawlers a curated, markdown-friendly entry point into a site. Flagging it here for visibility rather than proposing a specific fix — worth splitting into its own issue if there's appetite for it.Verification
curl -o /dev/null -w '%{http_code}' https://doc.owncloud.com/robots.txt→ 404curl -o /dev/null -w '%{http_code}' https://doc.owncloud.com/llms.txt→ 404curl -o /dev/null -w '%{http_code}' https://doc.owncloud.com/sitemap.xml→ 200🤖 Generated with Claude Code