Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions .github/skills/audit-sdk-docs/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
---
name: audit-sdk-docs
description: Use when the user wants to audit this repo's documentation and code annotations so agents get accurate, current, non-redundant info. Covers stale references (broken links / dead URLs / missing paths), code annotations that no longer match the code, duplicated or contradictory docs about the same concept, and doc simplification. Skips the auto-generated sdk/ folder and centrally-synced eng/common. Triggers on "check docs", "audit documentation", "delete/update outdated docs", "keep docs up to date", "docs don't match code".
---

# Audit SDK Docs

Audit the documentation **and code annotations** of the Azure SDK for Python repo
so that agents reading repo docs (`AGENTS.md`, `.github/copilot-instructions.md`,
`.github/skills/`, `doc/dev/`, docstrings, ...) get accurate, current, and
non-redundant information. Better docs => higher agent task-success rate (#48112).

This skill is **agent-maintained and self-improving**: the final step of every run
is to reflect on what worked, and fold any reusable pattern back into this skill
(see [Step 6](#step-6---self-improve-mandatory)).

## Global rules

- **SKIP the `sdk/` folder** - auto-generated; its docs/annotations are not
hand-maintained. The scanner excludes it by default.
- **Do NOT edit `eng/common*`** - centrally synced from `azure-sdk-tools` and
overwritten by automation. The scanner excludes it.
- **DO NOT edit too many files at one time.** Fix the highest-confidence items in
a small batch; report the rest as follow-ups rather than mass-editing.
- **Every candidate needs agent judgement.** Tools surface *candidates*, not
confirmed bugs. Legitimate, leave-alone cases include placeholders
(`sdk/mypackage/...`), generated paths (`conda/noarch`), gitignored user files
(`testsettings_local.cfg`), code-snippet regex noise (`{{`, `or`), and
**historical references** - a path introduced with wording like "Previously in",
"formerly at", "moved from", or "deprecated path" is describing history on
purpose, so leave it even though the target no longer exists.
- **Preserve meaning.** When simplifying or de-duplicating, never drop information;
consolidate it into a single source of truth and link to it.

## Audit dimensions

Run the dimensions that fit the request. Each is a separate concern; do them as
separate small batches, not one giant edit.

### Dimension 1 - Stale references (automated)

Broken relative links, dead in-repo GitHub URLs, and inline path refs that no
longer exist. Run the bundled read-only scanner (audits the repo it lives in):

```
python .github/skills/audit-sdk-docs/scripts/check_outdated_docs.py
```

Scans root `*.md`/`*.rst` + `doc/` + `eng/` (excluding `sdk/`, `eng/common*`,
`node_modules`). Options: `REPO_ROOT` positional, `--scan-dir DIR` (repeatable),
`--include-sdk`, `--org`/`--repo`. Output groups: `BROKEN_RELATIVE_LINK`,
`DEAD_REPO_URL`, `MISSING_PATH_REF`. For each real hit, find where the target
*moved to* (`grep` the symbol/filename, `git log -- <old-path>`) before editing.

### Dimension 2 - Annotation vs code drift

Check that code annotations describe the real code (focus on hand-maintained
tooling under `eng/`, `scripts/`, `tools/`; skip auto-generated `sdk/`).
Method (agent-driven; verify each candidate against the actual code):

- **Docstring params vs signature.** For a documented function, compare the params
named in the docstring (`:param x:`, `Args:`) against the real signature; flag
removed/renamed/added params.
- **Referenced symbols exist.** `grep` for class/function/module names mentioned in
docstrings and doc prose; flag ones that no longer exist.
- **Runnable examples.** Import/run code snippets and `>>> doctest` examples where
cheap; flag imports/attribute paths that fail.
- **CLI help drift.** When a doc quotes a command's flags/output, run the tool's
`--help` and diff; flag divergence (e.g. `azpysdk`, `sdk_build_conda`).
- **Type-hint claims.** When prose states a return/param type, confirm against the
annotation.

Prefer `grep`/`git log`/running the tool over guessing. Fix by correcting the
annotation to match the code (do **not** change working code to match a stale
comment unless the code is the bug).

### Dimension 3 - Duplicated or contradictory docs

Find the same concept documented in multiple places that have drifted apart.
Method:

- **Build a concept index.** For a topic (a command, config key, env var, pipeline
id, path), `grep -rn` it across all in-scope docs to list every doc that covers
it.
- **Duplication.** Multiple docs explaining the same procedure => pick a single
source of truth, keep the fullest/most-correct copy, and replace the others with
a link to it.
- **Contradiction.** Same command/flag/config with **different values** across docs
(e.g. two different pipeline `definitionId`s, differing env-var names, conflicting
steps) => determine the correct one from the code/pipeline, fix all copies (or
consolidate), and note which was authoritative.

### Dimension 4 - Simplification

Where a doc is redundant, stale-by-accretion, or needlessly long:

- Remove dead sections describing removed features (verify removal first).
- Collapse duplicated prose into the single source of truth from Dimension 3.
- Tighten wording without dropping any real instruction or caveat.

Keep edits surgical and review with `git diff` before committing.

## Workflow

1. **Scope.** Decide which dimensions apply to the request.
2. **Detect.** Run the scanner (D1) and/or the grep/index methods (D2-D4) to gather
candidates.
3. **Judge.** Read each candidate in context; discard legitimate cases per the
global rules.
4. **Fix a small batch.** Update/delete/consolidate. Verify moved targets first.
5. **Verify.** Re-run the scanner, re-run any affected examples/`--help`, and show
`git diff` of edited docs. Confirm no new candidates were introduced.
6. **Self-improve (see below).**
7. **Report.** Summarize fixes by dimension and list follow-ups (and why they were
left).

### Step 6 - Self-improve (MANDATORY)

Before finishing, **review the process you just ran** and ask: did I discover a
reusable pattern or solution that isn't yet captured here? For example: a new
staleness class worth adding to the scanner, a reliable heuristic for detecting a
contradiction, a grep recipe for a concept index, a known false-positive to
whitelist, or a moved-file mapping worth recording.

If yes:
- Update this `SKILL.md` (and/or extend `scripts/check_outdated_docs.py`) with the
new pattern/solution.
- **Commit the skill change in the SAME PR** as the doc fixes, so the skill gets
better every time it is used.

If nothing new was learned, state that explicitly in the report and skip the edit.

## Example findings (run relating to #48112)

- `doc/dev/conda-builds.md`: `eng/conda_env.yml` -> `conda/conda-recipes/conda_env.yml`
(moved in #31804; verified via `eng/tools/azure-sdk-tools/ci_tools/conda/conda_functions.py`
`get_version_from_config`). *(Dimension 1)*
- `doc/dev/mgmt/tests.md`: sample `conftest.py` link pointed at the removed
`sdk/advisor/azure-mgmt-advisor/tests/` folder; repointed to
`sdk/apimanagement/azure-mgmt-apimanagement/tests/conftest.py`. *(Dimension 1)*
- `doc/dev/conda-builds.md`: its "CI Build Process" bullets duplicated and partly
contradicted (manual version bump) the now-authoritative, largely-automated
process in `conda-release.md`; consolidated to a pointer, keeping this page
focused on local builds. *(Dimensions 3 + 4)*
- `sdk_build_conda` flags (`-c/--config`, `--channel`) in `conda-builds.md`
verified against `ci_tools/conda/conda_functions.py` argparse - accurate, no
change. *(Dimension 2, no fix)*
168 changes: 168 additions & 0 deletions .github/skills/audit-sdk-docs/scripts/check_outdated_docs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
#!/usr/bin/env python
"""Audit an Azure SDK repo for outdated documentation references.

Scans Markdown/reStructuredText docs (excluding the auto-generated ``sdk/`` tree
by default) and reports three classes of high-signal staleness for an agent to
judge and fix:

1. BROKEN_RELATIVE_LINK - ``[text](relative/path.md)`` whose target is missing.
2. DEAD_REPO_URL - ``https://github.com/<org>/<repo>/blob|tree/main/<path>``
pointing at a file/dir that no longer exists locally.
3. MISSING_PATH_REF - an inline-code token like ``eng/foo.yml`` that looks
like a repo-relative path but does not exist.

The script only *reports*; it never edits. Many hits are legitimate placeholders
(e.g. ``sdk/path-to-your-package/_version.py``) or code snippets, so the agent
must read each candidate in context before deciding to update or delete.

Usage:
python check_outdated_docs.py [REPO_ROOT] [--scan-dir DIR ...] [--include-sdk]

Defaults:
REPO_ROOT the repo this script lives in (../../../.. from the script)
scan dirs repo-root-level *.md/*.rst files + doc/ + eng/ (recursive)
"""
import argparse
import os
import re
import sys

# repo root = <root>/.github/skills/audit-sdk-docs/scripts/check_outdated_docs.py
DEFAULT_ROOT = os.path.abspath(
os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "..", "..")
)

MD_LINK = re.compile(r"\[[^\]]*\]\(([^)]+)\)")
INLINE = re.compile(r"`([^`\n]+)`")
PATH_PREFIXES = ("sdk/", "eng/", "scripts/", "doc/", "common/", "tools/", "conda/")
# repo url matcher is built at runtime from --org/--repo

DOC_EXT = (".md", ".rst")


def collect_docs(root, scan_dirs, include_sdk):
targets = []
# top-level doc files
for f in os.listdir(root):
full = os.path.join(root, f)
if os.path.isfile(full) and f.lower().endswith(DOC_EXT):
targets.append(full)
for d in scan_dirs:
base = os.path.join(root, d)
if not os.path.isdir(base):
continue
for dp, dn, fns in os.walk(base):
low = dp.lower()
if (os.sep + "node_modules") in low:
continue
# eng/common* is centrally synced from azure-sdk-tools; do not hand-edit
if (os.sep + "eng" + os.sep + "common") in low:
continue
if not include_sdk and (os.sep + "sdk" + os.sep) in (low + os.sep):
continue
for fn in fns:
if fn.lower().endswith(DOC_EXT):
targets.append(os.path.join(dp, fn))
# de-dup preserving order
seen, out = set(), []
for t in targets:
if t not in seen:
seen.add(t)
out.append(t)
return out


def check_broken_links(doc, text, root):
base = os.path.dirname(doc)
hits = []
for m in MD_LINK.finditer(text):
link = m.group(1).strip().split(" ")[0].strip("<>")
if not link or link.startswith(("http://", "https://", "#", "mailto:")):
continue
path_part = link.split("#")[0]
if not path_part:
continue
# skip obvious code-snippet noise (parens, quotes, commas, equals)
if any(c in path_part for c in "\"'=,"):
continue
tgt = os.path.normpath(os.path.join(base, path_part))
if not os.path.exists(tgt):
hits.append(("BROKEN_RELATIVE_LINK", link, os.path.relpath(tgt, root)))
return hits


def check_repo_urls(doc, text, root, url_re):
hits = []
for m in url_re.finditer(text):
rel = m.group(1).rstrip("/")
local = os.path.normpath(os.path.join(root, rel))
if not os.path.exists(local):
hits.append(("DEAD_REPO_URL", rel, rel))
return hits


def check_path_refs(doc, text, root):
hits = []
for m in INLINE.finditer(text):
tok = m.group(1).strip()
if " " in tok or not any(tok.lower().startswith(p) for p in PATH_PREFIXES):
continue
if any(c in tok for c in "*?<>|{}\"'"):
continue
cand = tok.split("#")[0].split(":")[0]
if not os.path.exists(os.path.normpath(os.path.join(root, cand))):
hits.append(("MISSING_PATH_REF", tok, cand))
return hits


def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("root", nargs="?", default=DEFAULT_ROOT)
ap.add_argument("--scan-dir", action="append", default=None,
help="Directory (relative to root) to scan recursively. Repeatable.")
ap.add_argument("--include-sdk", action="store_true",
help="Also scan the auto-generated sdk/ tree (off by default).")
ap.add_argument("--org", default="Azure")
ap.add_argument("--repo", default="azure-sdk-for-python")
args = ap.parse_args()

root = os.path.abspath(args.root)
if not os.path.isdir(root):
print(f"ERROR: repo root not found: {root}", file=sys.stderr)
return 2
scan_dirs = args.scan_dir if args.scan_dir else ["doc", "eng"]
url_re = re.compile(
r"https://github\.com/" + re.escape(args.org) + "/" + re.escape(args.repo)
+ r"/(?:blob|tree)/main/([^)\s\"'>#]+)")

docs = collect_docs(root, scan_dirs, args.include_sdk)
all_hits = []
for doc in docs:
with open(doc, encoding="utf-8", errors="ignore") as fh:
text = fh.read()
rel_doc = os.path.relpath(doc, root)
for kind, shown, tgt in (
check_broken_links(doc, text, root)
+ check_repo_urls(doc, text, root, url_re)
+ check_path_refs(doc, text, root)
):
all_hits.append((kind, rel_doc, shown, tgt))

if not all_hits:
print("No outdated doc references found.")
else:
for kind in ("BROKEN_RELATIVE_LINK", "DEAD_REPO_URL", "MISSING_PATH_REF"):
group = [h for h in all_hits if h[0] == kind]
if not group:
continue
print(f"\n=== {kind} ({len(group)}) ===")
for _, doc, shown, tgt in group:
print(f"{doc}\n ref: {shown}\n missing: {tgt}")
print(f"\nScanned {len(docs)} doc files. Total candidates: {len(all_hits)}")
print("NOTE: candidates are NOT auto-fixed. Read each in context; many are "
"intentional placeholders or code snippets.")
return 0


if __name__ == "__main__":
sys.exit(main())
11 changes: 6 additions & 5 deletions doc/dev/conda-builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@ Follow the instructions [here](https://docs.conda.io/projects/conda-build/en/lat

## CI Build Process

- Update `CondaArtifacts` parameters as necessary within `eng/pipelines/templates/stages/conda-sdk-client.yml` .
- If necessary, add or update the conda recipes present under `/conda/conda-recipes`.
- Update `eng/conda_env.yml` variable `AZURESDK_CONDA_VERSION` to the target version you wish to release.
- Invoke [python - conda](https://dev.azure.com/azure-sdk/internal/_build?definitionId=6321) manually, checking off which packages you wish to release.
- Once built, approve the packages for release individually, there will be pending approval stages.
The quarterly CI build/release process is documented in full in
[conda-release.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/conda-release.md) —
the recipe/version updates are produced automatically by the Conda Update
Pipeline, and the packages are built and published by the Conda Build/Release
Pipeline. Refer to that document as the single source of truth for releasing;
this page focuses on building a conda package locally.

## How to Build an Azure SDK Conda Package Locally

Expand Down
2 changes: 1 addition & 1 deletion doc/dev/mgmt/tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ Management plane SDKs are those that are formatted `azure-mgmt-xxxx`, otherwise

### Tips:
After the migration of the test proxy, `conftests.py` needs to be configured under the tests folder.<br/>
* For a sample about `conftest.py`, see [conftest.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/advisor/azure-mgmt-advisor/tests/conftest.py). <br/>
* For a sample about `conftest.py`, see [conftest.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/apimanagement/azure-mgmt-apimanagement/tests/conftest.py). <br/>
* For more information about test proxy, see [TestProxy][testproxy].

### Example 1: Basic Azure service interaction and recording
Expand Down
Loading