feat: add test-placement skill for the plugin testing strategy - #58
Merged
kadel merged 4 commits intoJul 21, 2026
Merged
Conversation
This was referenced Jul 7, 2026
gustavolira
commented
Jul 7, 2026
gustavolira
added a commit
to gustavolira/rhdh-skill
that referenced
this pull request
Jul 7, 2026
- Quote literal trigger phrases in the frontmatter description, matching the nfs-migration/backstage-upgrade convention - Add test-placement to the orchestrator intake menu (item 10) and skills_index; renumber release/general items and routes - Cross-reference skills/rhdh/references/rhdh-repos.md with the standalone-install caveat (backstage-upgrade pattern) - Drop the layer-matrix URL that 404s until rhdh#5044 merges; keep the repo path + PR reference - Remove the hardcoded date from the References header - Add parametrized frontmatter validation for every skills/*/SKILL.md (name matches directory, description within spec limits) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 10, 2026
durandom
reviewed
Jul 10, 2026
durandom
left a comment
durandom
left a comment
Member
There was a problem hiding this comment.
Review
Clean, well-scoped skill that fills a real gap — developers repeatedly ask "where should I test this?" and get it wrong (adding cluster e2e tests for pure UI logic, or missing the overlays native smoke entirely).
What's good
- Decision table (Step 2) — the 10-row table mapping "what you want to verify" → repo, harness, cluster?, Docker? is the core value of this skill. Concrete and actionable.
- Layer ladder (Step 3) — L1→L4b progression with "cheapest that catches the bug wins" as the guiding rule is clear.
- "Not possible today" section — documenting researched dead-ends (standalone Scalprum host, `@backstage/frontend-dynamic-feature-loader`, catalog modules in minimal `startTestBackend`) saves developers from wasting time. This is the kind of tribal knowledge that's hardest to discover independently.
- PR status awareness — "PR statuses are as of the last edit — verify before citing as merged" is responsible. The References section correctly notes which PRs have merged vs. are in review.
- Output format — specifying what the recommendation should include (Repo, Layer/harness, Location, Scaffolding, Why not elsewhere, Cost) gives consistent, useful answers.
Test additions (bonus)
The `TestAllSkillsFrontmatter` class is a great addition:
- `test_name_matches_directory` — catches skill name/directory mismatches
- `test_description_within_spec_limits` — enforces 20-1024 char description length
- Parameterized via `ALL_SKILL_MD_FILES` glob — automatically covers new skills
Issues
-
Version conflict with #54. Both PRs bump 0.6.1 → 0.7.0. Only one can merge with that version. Coordinate — whoever merges second needs to bump to 0.7.1 or 0.8.0.
-
No version bump needed for tests? Per AGENTS.md: "Do NOT bump for docs-only or CI-only changes." The test file changes (`test_skill_structure.py`) are CI-only. But the new skill does warrant the minor bump, so this is correct overall.
-
Orchestrator menu renumbering. Items 10→11, 11→12, 12→13 — correct. The routing table matches.
Recommendation
Ship it (after version coordination with #54). This is content-only, well-scoped, and the test additions are a bonus that benefits all future skills.
durandom
pushed a commit
to gustavolira/rhdh-skill
that referenced
this pull request
Jul 10, 2026
- Quote literal trigger phrases in the frontmatter description, matching the nfs-migration/backstage-upgrade convention - Add test-placement to the orchestrator intake menu (item 10) and skills_index; renumber release/general items and routes - Cross-reference skills/rhdh/references/rhdh-repos.md with the standalone-install caveat (backstage-upgrade pattern) - Drop the layer-matrix URL that 404s until rhdh#5044 merges; keep the repo path + PR reference - Remove the hardcoded date from the References header - Add parametrized frontmatter validation for every skills/*/SKILL.md (name matches directory, description within spec limits) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
durandom
force-pushed
the
feat/test-placement-skill
branch
from
July 10, 2026 14:03
edf53d0 to
6718d01
Compare
Centralizes the test-placement skill here instead of duplicating it across rhdh, rhdh-plugins, and rhdh-plugin-export-overlays (supersedes overlays#2726, rhdh#5045, plugins#3670). Given a change, bug, or feature in the RHDH dynamic-plugin ecosystem, the skill proposes the right repo, the right test layer (L1 unit to L4b cluster e2e), the location, and how to scaffold the test. Encodes the RHIDP-13501 per-repo responsibility split and researched dead-ends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Quote literal trigger phrases in the frontmatter description, matching the nfs-migration/backstage-upgrade convention - Add test-placement to the orchestrator intake menu (item 10) and skills_index; renumber release/general items and routes - Cross-reference skills/rhdh/references/rhdh-repos.md with the standalone-install caveat (backstage-upgrade pattern) - Drop the layer-matrix URL that 404s until rhdh#5044 merges; keep the repo path + PR reference - Remove the hardcoded date from the References header - Add parametrized frontmatter validation for every skills/*/SKILL.md (name matches directory, description within spec limits) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… merged The cluster-free harness expansion (10 specs / 14 test cases) and the native-smoke workspace mode both merged on 2026-07-07; the References section now cites them so the decision-table capabilities stay backed by merged PRs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gustavolira
force-pushed
the
feat/test-placement-skill
branch
from
July 20, 2026 14:57
6718d01 to
be2da3f
Compare
gustavolira
changed the title
lholmquist
approved these changes
Jul 20, 2026
subhashkhileri
approved these changes
Jul 21, 2026
kadel
reviewed
Jul 21, 2026
- rhdh#4864 is merged, not closed — corrected in both the L3 section and References, and the L3 entry now names the concrete templates that landed (packages/app/src/**/*.test.tsx) instead of telling the reader to go hunt for merged examples. - rhdh#5044 merged 2026-07-20 — layer matrix marked merged and the blob URL restored, since the path now resolves on main. - Dropped the first "When to Use" bullet: it restated the trigger phrases already quoted in the frontmatter description. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kadel
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
test-placementskill (skills/test-placement/): when a developer asks "where should I test this?", the assistant gathers the context and proposes the right repo (rhdh-plugins/rhdh-plugin-export-overlays/rhdh), the right test layer (L1 unit → L4b cluster e2e), the location, and how to scaffold the test.It encodes the per-repository testing responsibility split consolidated on epic RHIDP-13501: artifact bugs → overlays native smoke (merged in overlays#2714 — install from OCI +
startTestBackendboot, workspace mode, bundle presence) · source bugs → the plugin's source repo · integration/UI-render bugs → the rhdh cluster-free harness (merged in rhdh#5005) · platform bugs → cluster e2e. Guiding rule: the cheapest environment that catches the bug wins. Includes the researched dead-ends (catalog-extending modules in minimalstartTestBackend, local-path artifacts, standalone Scalprum host) so devs don't burn time on them.Why here
Per review feedback on overlays#2726: the skill covers the entire RHDH ecosystem (three repos, five test layers), not any single repo — exactly what this centralized repo exists to hold. Supersedes and replaces the three per-repo sibling PRs:
docs(ai): add test-placement skill for the plugin testing strategy (RHIDP-13501) rhdh-plugin-export-overlays#2726docs(ai): add test-placement skill for the plugin testing strategy (RHIDP-13501) rhdh#5045docs(ai): add test-placement skill for the plugin testing strategy (RHIDP-13501) rhdh-plugins#3670Compared to those PRs, the References section was refreshed: overlays#2714 and rhdh#5005 have since merged.
Changes
skills/test-placement/SKILL.md(content-only, no scripts)skills/rhdh/SKILL.md): new "Test Placement Routes" entry0.6.1→0.7.0(new skill = minor), in sync acrosspyproject.toml,plugin.json,marketplace.jsonRelated
Epic RHIDP-13501 · overlays#2714 · rhdh#5005 · rhdh#5044 (layer matrix) · rhdh#4967 (catalog-index sanity)
🤖 Generated with Claude Code