Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions .github/skills/code-coverage-onboarding/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
name: code-coverage-onboarding
description: >-
Onboard a team/repo to automated code-coverage tracking and reporting. Use when a user wants
to set up code coverage measurement, add coverage to CI/CD, track coverage over time
(week-over-week), ingest coverage into Kusto/Azure Data Explorer, generate a coverage report
or email, wire coverage into a weekly/scheduled pipeline, port an existing coverage setup to
another team, or raise/increase coverage. Handles JaCoCo (Gradle/Maven/Android),
Cobertura (.NET/Python/JS/Go), and LCOV (JS/TS/C++) reports, Azure DevOps and GitHub Actions
pipelines, Kusto ingestion with cost control, report/email integration (including Azure
Communication Services), and ranking uncovered classes/files to boost coverage. Triggers
include "set up code coverage", "add coverage tracking", "coverage
report", "coverage in the pipeline", "track coverage in Kusto", "onboard to code coverage",
"weekly coverage email", "increase/raise code coverage", "where should I add tests",
"code coverage skill".
---

# Code Coverage Onboarding

Set up end-to-end code-coverage tracking for a repo: generate coverage → parse to a normalized
schema → publish a summary → (scheduled) ingest into Kusto → surface a week-over-week trend in
a report/email. Portable across CI systems and languages. The core is `scripts/coverage_report.py`
plus a Kusto table; the pipeline wiring is adaptable to Azure DevOps or GitHub Actions.

## Bundled resources
- `scripts/coverage_report.py` — the portable engine. Four subcommands: `parse` (coverage
report → normalized NDJSON), `report` (NDJSON → Markdown summary), `gaps` (report → ranked
worklist of the least-covered classes/files, for raising coverage), `wow` (Kusto → HTML trend
fragment with grouped tables + Overall rows). Parses JaCoCo XML, Cobertura XML, and LCOV.
Stdlib only; runs on any image with Python 3.
- `references/coverage-generation.md` — make the build emit JaCoCo/Cobertura/LCOV (Gradle,
Android AGP, Maven, .NET, Python, Node, C/C++, Go).
- `references/kusto-setup.md` — table schema, WIF/MI auth, ingestor grants, inline ingest, and
the Sunday-only cost-control gate.
- `references/pipeline-integration.md` — Azure DevOps and GitHub Actions patterns, plus how to
create a scheduled build if none exists.
- `references/report-integration.md` — inject the trend into an existing report, a Kusto
dashboard, or a new ACS email; grouped-table configuration.
- `references/increasing-coverage.md` — the find-gaps → write-tests → gate loop for *raising*
coverage. Read when the user wants to boost numbers, not just track them.

## Workflow

This skill has five capabilities. Confirm scope with the user, then implement in order. Steps
1–4 are the tracking pipeline; step 5 (raising coverage) is optional and builds on them.

### Step 0 — Scope the setup
Ask (only what's not already clear):
- **Build tool & language** → determines report format (JaCoCo / Cobertura / LCOV).
- **CI system** (Azure DevOps / GitHub Actions / other).
- **Is there a recurring/scheduled build?** If not, one is needed (see step 2).
- **Track history in Kusto?** Default **yes**; a team can opt for summary-artifact-only.
- **Existing reporting** to inject into, or start fresh?

### Step 1 — Generate coverage
Get the build to emit JaCoCo, Cobertura, or LCOV output. See `references/coverage-generation.md`.
**Reuse an existing test run** rather than adding a second one — re-running tests just for
coverage doubles CI time. Gate coverage behind a flag so normal builds aren't slowed.

### Step 2 — Parse & summarize in CI
Add steps that run `coverage_report.py parse` (per module → shared NDJSON) then `report`
(NDJSON → Markdown summary), and publish both as an artifact. These run every build and are
**non-fatal** (warn, don't fail). See `references/pipeline-integration.md`. If the team has no
scheduled build, create a minimal weekly one whose job is: test + parse + ingest.

### Step 3 — Ingest into Kusto (default on, cost-gated)
Create the `CodeCoverageData` table, grant the CI identity Ingestor, and add an ingest step
that converts NDJSON → CSV and POSTs an inline `.ingest`. **Only ingest on the scheduled
reporting run** (e.g. Sunday) with an operator override parameter — see the gate in
`references/kusto-setup.md`. Ingestion **should fail loudly** if it can't write (unlike the
reporting steps). Skip this whole step only if the team declined history.

### Step 4 — Surface the trend
Render the week-over-week trend with `coverage_report.py wow` and get it in front of the team.
Priority: (1) inject the HTML fragment into an existing recurring report, (2) a Kusto
dashboard, (3) a new ACS email. See `references/report-integration.md`. Use `--group-file` to
split modules into labeled tables, each with an Overall row.

### Step 5 — Increasing coverage (integrated, optional)
When the user wants to *raise* coverage (not just track it), run the find-gaps loop: use
`coverage_report.py gaps` on the latest report to rank the least-covered classes/files, write
tests for the highest-value targets, re-run to confirm, then lock gains in with a no-regression
gate. See `references/increasing-coverage.md`. This is a separate effort from steps 1–4 — do it
once a baseline is visible so progress can be verified.

## Key conventions (carry these when porting)
- **Normalized schema** is fixed: `Date, Repo, Module, Metric, Covered, Missed, Percentage,
CommitId, BuildId, Branch`. The KQL and ingest CSV depend on these names.
- **Reporting is non-fatal; ingestion is fatal.** The report/email must still ship on a
coverage hiccup, but a silent ingest failure that greens the run is a bug.
- **WoW uses calendar semantics**, not a rolling 7-day window: the baseline is last week's
start-of-week run (`startofweek()`), so same-week manual re-runs don't skew the delta.
- **Overall rows sum Covered/Total** and derive the delta from summed prior Covered/Missed —
never average per-module percentages.
- **Ingest on a schedule only** to control Kusto cost; keep cheap parse/publish per build.

## Validating the script
`coverage_report.py` is stdlib-only. Sanity-check after any edit:
```bash
python3 -m py_compile scripts/coverage_report.py
python3 scripts/coverage_report.py parse --input sample.xml --repo r --module m --out rows.ndjson
python3 scripts/coverage_report.py report --input rows.ndjson --metric LINE --goal 75
python3 scripts/coverage_report.py gaps --input sample.xml --top 10
```
The `wow` subcommand needs a live Kusto table + token; test rendering logic with mock rows if
no cluster is available.
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Coverage Report Generation

How to make a build emit a machine-readable coverage report that
`scripts/coverage_report.py` can parse. The parser auto-detects **JaCoCo XML**,
**Cobertura XML**, and **LCOV** tracefiles, so the only job here is to get the build
tool to produce one of those.

## Table of contents
- [Which format do I have?](#which-format-do-i-have)
- [Gradle + JaCoCo (Java/Kotlin/Android)](#gradle--jacoco-javakotlinandroid)
- [Android Gradle Plugin specifics](#android-gradle-plugin-specifics)
- [Maven + JaCoCo](#maven--jacoco)
- [.NET (Coverlet / VSTest → Cobertura)](#net-coverlet--vstest--cobertura)
- [Python (coverage.py → Cobertura)](#python-coveragepy--cobertura)
- [Node/JS (nyc/jest → Cobertura or LCOV)](#nodejs-nycjest--cobertura-or-lcov)
- [C/C++ (lcov / gcovr)](#cc-lcov--gcovr)
- [Go (gocover-cobertura)](#go-gocover-cobertura)
- [Finding the report in CI](#finding-the-report-in-ci)

## Which format do I have?
- Root element `<report>` → **JaCoCo** (`--format jacoco`, or let auto-detect handle it).
- Root element `<coverage>` → **Cobertura** (`--format cobertura`).
- Text lines like `SF:`, `DA:`, `LF:`, `end_of_record` (usually `lcov.info` / `*.info`) →
**LCOV** (`--format lcov`).

The parser reads the top-level aggregate `<counter>` totals (JaCoCo), the
`lines-covered`/`lines-valid`/`branches-*` attributes (Cobertura), or the summed
`LF`/`LH` + `BRF`/`BRH` records (LCOV; falls back to counting `DA:` records when the
summaries are absent). It does **not** need per-class/per-file detail, so any
correctly-formed report works.

## Gradle + JaCoCo (Java/Kotlin/Android)
Apply the plugin and ensure a report task emits XML:

```groovy
plugins { id 'jacoco' }
jacoco { toolVersion = "0.8.10" }

tasks.named('jacocoTestReport') {
dependsOn test // or the flavor-specific unit test task
reports {
xml.required = true // REQUIRED - this is what the parser reads
html.required = true // optional, for humans
}
}
```

Run: `./gradlew jacocoTestReport` → XML at
`build/reports/jacoco/jacocoTestReport/jacocoTestReport.xml`.

**Gate coverage behind a flag** so normal builds aren't slowed down:
```groovy
def enableCodeCoverage = project.hasProperty("codeCoverageEnabled")
? codeCoverageEnabled.toBoolean() : false
tasks.withType(Test) { jacoco { enabled = enableCodeCoverage } }
```
Then in CI: `./gradlew jacocoTestReport -PcodeCoverageEnabled=true`.

**Reuse existing test runs.** If the build already runs a coverage task (many Android
setups have `<flavor>UnitTestCoverageReport`), do NOT add a second test run — point the
parser at the XML that task already produces. Re-running tests just to get coverage
doubles CI time.

## Android Gradle Plugin specifics
- AGP unit-test coverage is per **build variant**: `testDebugUnitTest` → JaCoCo exec →
`create<Variant>UnitTestCoverageReport`. Use the variant your CI actually builds
(e.g. `dist`/`release`), not `localDebug`, or the `.exec` won't exist and the report
will be empty.
- `includeNoLocationClasses = true` is required for **Robolectric** tests.
- Modules with no `src/test` sources produce no report — skip them (don't fail the build).
- Robolectric/instrumented mixes: only unit-test coverage flows through JaCoCo XML here;
instrumented (Espresso) coverage needs a connected device and is out of scope for the
weekly trend.

## Maven + JaCoCo
```xml
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.10</version>
<executions>
<execution><goals><goal>prepare-agent</goal></goals></execution>
<execution><id>report</id><phase>test</phase><goals><goal>report</goal></goals></execution>
</executions>
</plugin>
```
Run `mvn test` → XML at `target/site/jacoco/jacoco.xml`.

## .NET (Coverlet / VSTest → Cobertura)
```bash
dotnet test --collect:"XPlat Code Coverage" -- \
DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura
```
Emits `**/TestResults/**/coverage.cobertura.xml`. Parse with `--format cobertura`.

## Python (coverage.py → Cobertura)
```bash
coverage run -m pytest
coverage xml # -> coverage.xml (Cobertura)
```

## Node/JS (nyc/jest → Cobertura or LCOV)
Either format works — the parser reads both natively.
```bash
# jest — Cobertura
jest --coverage --coverageReporters=cobertura # -> coverage/cobertura-coverage.xml
# jest — LCOV (often already the default)
jest --coverage --coverageReporters=lcov # -> coverage/lcov.info
# nyc
nyc --reporter=cobertura npm test # or --reporter=lcovonly -> coverage/lcov.info
```

## C/C++ (lcov / gcovr)
```bash
# lcov -> lcov.info (parse with --format lcov / auto-detect)
lcov --capture --directory . --output-file coverage.info
# or gcovr -> Cobertura
gcovr --cobertura -o coverage.xml
```

## Go (gocover-cobertura)
```bash
go test -coverprofile=cover.out ./...
go run github.com/boumenot/gocover-cobertura < cover.out > coverage.xml
```

## Finding the report in CI
When report paths vary per module, locate JaCoCo reports by content rather than a fixed path:
```bash
grep -rlZ --include='*.xml' 'JACOCO//DTD' "$SOURCES_DIR" | tr '\0' '\n'
```
For Cobertura, match on the `<coverage` root or the well-known filename
(`coverage.cobertura.xml` / `cobertura-coverage.xml` / `coverage.xml`). For LCOV, match the
well-known filename (`lcov.info` / `*.info`) or grep for `end_of_record`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Increasing Code Coverage

Read this file when the user wants to **raise/improve/boost** coverage numbers (not just set
up tracking). It is an integrated capability of this skill, but a *distinct effort* from
onboarding: establish measurement + a visible baseline first (steps 1–4 of the skill) so the
effort can be verified. If tracking is already in place, jump straight in.

## Table of contents
- [The loop](#the-loop)
- [Finding gaps with `coverage_report.py gaps`](#finding-gaps)
- [Choosing targets](#choosing-targets)
- [Writing tests that count](#writing-tests-that-count)
- [Locking in gains with a gate](#gate)
- [Anti-patterns](#anti-patterns)

## The loop
Raising coverage is an iterative loop, not a one-shot:
1. **Baseline** — record current per-module numbers (the weekly report already does this).
2. **Find gaps** — run `coverage_report.py gaps` on the latest report to get a ranked worklist
of the least-covered classes/files.
3. **Pick targets** — take the highest-value items off the list (see below).
4. **Write tests** for real behavior in those units; re-run coverage.
5. **Re-run `gaps`** to confirm the target dropped off / shrank, and watch the module % rise in
the weekly trend.
6. **Lock it in** — once a module reaches a healthy level, add a no-regression gate so it can't
slide back.

<a id="finding-gaps"></a>
## Finding gaps with `coverage_report.py gaps`
The `gaps` subcommand turns a raw coverage report into a prioritized, per-class/per-file
worklist. It reads the **same** JaCoCo / Cobertura / LCOV reports the build already produces —
no extra tooling.

```bash
# Biggest absolute wins first (default): classes with the most uncovered lines.
python3 coverage_report.py gaps --input '**/build/reports/**/*.xml' --metric LINE --top 25

# Lowest-coverage classes first, ignoring anything already >60% or tiny (<5 missed lines).
python3 coverage_report.py gaps --input coverage.cobertura.xml \
--sort pct --max-pct 60 --min-missed 5

# Branch gaps for a JS module from LCOV, saved as a checklist artifact.
python3 coverage_report.py gaps --input coverage/lcov.info --metric BRANCH \
--out-md gaps.md --out-json gaps.json
```

Key flags:
- `--metric LINE|BRANCH` — line gaps are the usual target; branch gaps expose untested
conditionals/error paths.
- `--sort missed` (default) ranks by **most uncovered units** = biggest number bump per test
written. `--sort pct` ranks by **lowest coverage** = worst-tested code first.
- `--max-pct N` hides units already well covered; `--min-missed N` hides trivial ones. Together
they focus attention on "big and poorly covered".
- `--top N` caps the list (0 = all). `--out-md` / `--out-json` persist it as a work artifact.

Format is auto-detected; pass `--format jacoco|cobertura|lcov` to force it. The output columns
are `Class/File | Coverage | Missed | Covered/Total`.

<a id="choosing-targets"></a>
## Choosing targets
The ranked list tells you *where the uncovered code is*; combine it with judgment on *what is
worth covering*:
- **Prefer big + low-coverage + high-churn** modules — best return on effort. Cross-reference
the `gaps` list against `git log`/churn if available.
- **Cover behavior that matters**: core logic, error/exception paths, boundary conditions,
regression tests for recently-fixed bugs, and any newly-added code.
- **Skip / exclude the denominator noise**: generated code, DTOs/`data class`es, `toString`,
builders, and test code. Exclude these in the build tool so the number reflects meaningful
coverage instead of chasing 100% on trivial code.
- Use `--sort pct` for the "worst offenders" view when you want to eliminate near-zero classes,
and `--sort missed` when you want the fastest overall percentage gain.

<a id="writing-tests-that-count"></a>
## Writing tests that count
- Assert on observable outcomes (return values, state changes, thrown exceptions, emitted
events) — never write tests that execute code without asserting just to move the number.
- One behavior per test; name tests for the behavior, not the method.
- For each `gaps` target, open the class and cover its untested branches first (constructors and
simple getters are low value even if uncovered).
- Re-run tests + `gaps` after each batch so progress is visible and you don't over-invest in one
class.

<a id="gate"></a>
## Locking in gains with a gate
Prevent backsliding once a module improves. **Prefer a no-regression gate over a fixed absolute
threshold** — it's fair to modules that start low and doesn't block unrelated PRs.
- Compare a PR's coverage to the baseline/dev branch; **fail only if it *lowers* coverage**.
- Make the gate flip via a variable (e.g. `ENFORCE_COVERAGE_GATE`) between report-only and
enforcing, with an emergency off-switch.
- Introduce it gradually: report-only first, then enforce once numbers are stable.
- Exclude generated/test/DTO code from the denominator where the build tool supports it.

<a id="anti-patterns"></a>
## Anti-patterns
- Tests that assert nothing (executing code just to raise the number).
- Chasing a global percentage instead of covering risky code paths.
- Hard-gating an absolute threshold on day one — blocks unrelated work and breeds resentment.
- Padding coverage with generated/DTO code left in the denominator.
Loading
Loading