feat: platform/module test selection with capability x module matrix#530
Draft
abegnoche wants to merge 30 commits into
Draft
feat: platform/module test selection with capability x module matrix#530abegnoche wants to merge 30 commits into
abegnoche wants to merge 30 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
abegnoche
force-pushed
the
feat/capability-module-selection
branch
from
July 16, 2026 20:11
0a4575d to
9ac5daf
Compare
…dule) Reframe test selection around the capability x module matrix without a directory move, manifest, or catalog-kind redesign. - Add tests.kind (capability|module) to the 10 suites; provider configs inherit it via import. Govern labels in validate_suite_wiring.py: derive the capability/module axes from suites' kind+platform, allowlist modifier labels, and require <=1 capability label per check (scanning provider configs too). - Add --capability (run a whole column: capability config + all module configs, with capability-scoped excludes and continue-on-failure) and --module (run one module suite). Resolve by effective kind, never by filename. Plumb CLI exclude_labels through Orchestrator.run() so they apply even when include/pytest selection bypasses config excludes. PR 485 --label discovery is unchanged. Add my-isv k8s/slurm capability wrappers. - Re-home host syslog + BMC SEL/telemetry observability checks from the observability suite to bare_metal (they need a live host/BMC); update aws and my-isv provider configs and sync docs/test-plan.yaml. - Label the 4 K8s CSI checks with "storage" and allowlist it as a module label. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Replace the redundant tests.platform + tests.kind pairing in suites with a single axis key whose value is also the runtime platform: - ValidationConfig gains capability/module (mutually exclusive) and derives platform from whichever is set; the standalone kind field is removed. The 10 suites now declare `capability: <name>` or `module: <name>` only. - Classification (capability_resolution, validate_suite_wiring) reads the axis key; the validator requires exactly one per suite. - get_platform_from_config (raw-read upload path that skips imports) falls back to the capability/module value, so nico's standalone configs drop the redundant explicit platform and declare only `module: <name>`. Observability provider configs declare `module: observability` explicitly (inherited axis keys are invisible to the raw path). - Update schema JSON, docs, and tests accordingly. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Avoid a competing vocabulary: platform is already the runtime, commands key, upload/report term, and YAML axis key throughout the codebase. Service-line suites declare tests.platform; modules declare tests.module (whose value is the derived runtime platform). - Drop the capability field from ValidationConfig; classify suites by platform-or-module. Revert the four capability suites to platform:. - Rename capability_resolution to platform_resolution; --capability becomes --platform. Platform/module label axes, wiring validation, schema, docs, provider comments, and tests follow the same terminology. - GPU compute capability, DPU capability inventory, and test-plan taxonomy nodes are unchanged. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Move the tests.platform / tests.module axis key to the first line of the tests: block in every suite so classification is visible at a glance and the block shape is consistent across suites. Pure YAML key reordering; no behavior change (validate-suites and demo-test still pass). Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Wrap the catalog document in a versioned envelope: a schemaVersion (independent of the isvtest package version), plus the platform (capability) and module axis label lists derived from the suite axis keys. These let a consumer build the capability x module matrix straight from the catalog instead of a hardcoded taxonomy. isvreporter forwards the new fields on upload; the per-entry label universe is intentionally not summarized at the top level since it is derivable from the entries. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Remove the hand-maintained label allowlists (MODIFIER_LABELS, EXTRA_MODULE_LABELS, SUITE_REQUIRED_LABELS) from the wiring validator and catalog. These duplicated data already declared in the suite YAML: the required suite label is now derived from each suite's tests.platform / tests.module axis key, and the module axis is derived purely from module: keys. Labels are otherwise free-form (they originate in the wiring YAML, so there is no external allowlist to guard). The at-most-one-platform-label rule and the per-check axis-label requirement are preserved. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Consolidate the platform/module selection code paths: - Make isvtest.catalog.build_axis_taxonomy the single canonical scanner of suite axis keys (tests.platform / tests.module); platform_resolution and validate_suite_wiring now delegate to it instead of carrying mirrored copies. - cli/test.py: drop the duplicated JUnit-path helper and thread the shared run() options through one forward_kwargs dict used by both the label-discovery loop and the planned-run loop. - resolve_module_configs classifies the provider once per invocation instead of once per --module flag. - platform_resolution: fold the copy-pasted no-match/multi-match error blocks into _single_config, replace the half-used kind constants with a Literal type, derive the CLI platform aliases from the reporter's canonical alias table, and reject configs declaring both axis keys (matching the schema's mutual-exclusion rule). - label_discovery.provider_config_dir now owns the providers/<name>/config path convention. - validate_suite_wiring: read and parse each file once, share the line-attribution bookkeeping between the suite and provider loops, and simplify the single-error governance helper. - Move the duplicated axis suite/provider fixture builders into isvctl/tests/conftest.py. - Describe commands keys by the axis-key rule instead of enumerating platform names in schema descriptions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
The per-entry platforms field was built from hardcoded PLATFORM_CONFIGS / LABEL_TO_PLATFORM tables that predated the platform/module axis split and treated all ten suites as platforms - so module-suite checks (e.g. the control-plane IAM checks) were mis-tagged platforms: [CONTROL_PLANE]. Derive per-entry axis placement from each suite's declared tests.platform / tests.module key instead (reusing the canonical axis scanner), and emit a separate modules field alongside platforms. Drop the barely-used per-entry module (Python import path) field that also clashed with the module axis name. The document schemaVersion stays 1 (backend consumers move in step on their dev branches). Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
A module suite's derived tests.platform is its module name, so module runs uploaded IAM/NETWORK/... as the test target type, polluting the capability namespace. Runs now report both axes: platform runs upload (capability, -), module runs inside a --platform column inherit the column as their capability via PlannedRun.column_platform, and standalone module runs upload (-, module) with no capability. isvctl deploy resolves the axis pair through import: (new effective_axes helper) instead of reading raw keys, and isvreporter create gains --module plus pair auto-detection that passes unknown axis values through instead of coercing them to kubernetes. Dry-run plans now print the upload axes per run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Generic check classes (StepSuccessCheck, GpuCheck, ...) were wired under one bare name across many suites, so the class-keyed catalog unioned unrelated labels/test_ids onto a single entry: the IAM matrix row counted checks that only run under bare_metal, and per-column counts were artifacts of reuse, not coverage. Every duplicated wiring (121 across 12 files) now uses a distinct variant name per wiring (StepSuccessCheck-iam_teardown, GpuCheck-bm_gpu), so catalog entries, JUnit case names, and matrix cells are 1:1 with wirings. The wiring validator enforces global name uniqueness. The catalog no longer propagates variant attributes to base names, variant entries carry only their own wiring's labels/test_ids, classes wired only under variant names get no bare entry, and the released-only filter accepts variants of released classes (mirroring the runtime gating). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
The unique-name refactor (2452add) relies on generic check classes (FieldValueCheck, StepSuccessCheck, ...) never being wired under their bare class name, but nothing enforced it: two bare FieldValueCheck wirings in the NICo network provider config passed validation while silently smearing unrelated labels/test_ids onto one catalog entry. Add two guardrails: - variant_required on the reusable classes in generic.py, enforced at resolve time (bare name -> INVALID_CONFIG) and on the raw --config pytest discovery path. Pre-resolved runs are exempt since resolution already gated them and leaves unique names bare for pytest storage. - validate_suite_wiring.py now scans provider configs for within-file duplicate names and rejects bare variant_required wirings, in addition to the existing cross-suite uniqueness check. Rename the offending bare wirings to unique variants and update the orchestrator/resolution tests that wired bare generic names. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
main added suites/storage.yaml (DATASVC + HSS checks) after this branch last merged; its axis key was re-keyed to 'module: storage' during the rebase so storage is a matrix row, not a bogus capability column. Give its reusable generic wirings (StepSuccessCheck/FieldExistsCheck/ CrudOperationsCheck) the globally-unique variant names the branch now enforces, and expect 'storage' on the module axis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
The observability run now provisions a single g4dn.metal instance (launch_host) that serves every host-scoped check: host syslogs and the BMC SEL/telemetry checks move back from the bare_metal suite/configs to observability, wired to the run's own host alongside the telemetry and storage checks that already used it. Dropping their bare_metal platform label means the module runs in full under every capability column instead of splitting its host checks into the bare_metal lifecycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Rebasing onto v0.9.0 surfaced a semantic conflict: upstream NVIDIA#469 models AUTH03-01 as one requirement spanning vm and bare_metal (two check classes sharing the test_id, union-labeled per docs/test-plan.yaml), which collides with this branch's "one platform label per wiring" invariant. Trimming labels to satisfy validate_suite_wiring breaks test_plan_coverage's label_sync, and vice versa. Stopgap: exempt the shared test_id in validate_suite_wiring so both validators pass without touching the plan. See the TODO(AUTH03-01) note in the file for the proper follow-ups (split into per-platform test_ids, or reconcile the rules). Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
abegnoche
force-pushed
the
feat/capability-module-selection
branch
from
July 16, 2026 20:56
9ac5daf to
def663d
Compare
…ction Syncs the branch with upstream/main (4d61b18), which added NVIDIA#547 "register STORAGE as a capability platform" on top of v0.9.0. NVIDIA#547 and this branch are two divergent designs of the same test-taxonomy area. Per decision, this branch's design wins: - Storage stays a MODULE (module: storage), not a capability platform. - Taxonomy is derived from suite YAML (build_axis_taxonomy) rather than the hardcoded PLATFORM_CONFIGS registry NVIDIA#547 introduced. Kept the branch versions of scripts/validate_suite_wiring.py, isvtest/catalog.py, and their tests; dropped the orphaned isvtest/catalog_platforms.py registry and its platform_registration/registry_consistency validators. Kept upstream's compatible, non-taxonomy additions: - isvreporter.platform STORAGE constant + its config test (coexists with the branch's get_axes_from_config). - .pre-commit-config.yaml validate-suites hook tweak. Verified: ruff, validate_suite_wiring, test_plan_coverage label_sync, make test (all packages), catalog build, and make demo-test all pass. Catalog axes: platforms=[bare_metal, kubernetes, slurm, vm], modules=[..., storage]. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Plain-text dry-run summaries replace JSON as the default output, with --json retained for machine consumption. Summaries show validation counts, skipped checks, and wrapped check lists for config, platform-column, and provider-discovery paths. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
…forms: declaration A module-suite check now declares the platform columns it supports with platforms: [vm, bare_metal] on its wiring instead of carrying a platform label that opt-out exclusion interprets. Omitted means compatible with every column; subsets no longer require duplicate wirings. - Column runs plumb column_platform into resolution and skip checks whose declared platforms miss the column; the universe-minus-platform exclude_labels derivation is removed (user --exclude-label unchanged). - validate_suite_wiring: platform names banned from module-suite labels, platforms: values must belong to the platform axis, platforms: rejected on platform-suite checks. Replaces the single-platform-label rule and its temporary AUTH03-01 exemption. - Catalog entry platform badges come from suite placement or the declared platforms: field; wire format unchanged. - Migrated the two bare_metal-restricted CAP04 security checks; standalone --module runs still apply no platform filtering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Both flags now build and save the catalog locally without uploading, using --dry-run as the primary name for consistency with other commands. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Reuse the same steps, validations, and skipped-check rendering from -f dry-run so --provider --platform/--module plans show what each config would execute, not just one-line summaries. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
With --platform, --no-modules skips the module fan-out and runs just the platform base config. Rejected without --platform. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Modules that fit under no runtime capability (iam, control_plane - pure provider-API tests) move to a new foundational capability instead of running identically under every column. - suites/foundational.yaml is a validation-less platform suite that exists only to put foundational on the suite-derived capability axis; a validation-less suite's column has no platform run, so providers ship no config for it and --platform foundational plans modules only. - Eligibility for a modules-only column requires a positive platforms: declaration; omitted platforms: means every real environment, not synthetic columns. iam and control-plane checks declare platforms: ["foundational"] and run once, there. - Column plans omit module configs with no column-eligible check (previously they paid setup/teardown to run zero checks); omissions are surfaced in the human and JSON dry-run output. - --platform X --module m1 --module m2 is now legal (intersect): only the requested modules run, each under column X with platform-scoped filtering, and upload reports the (X, module) pair; the platform config does not run. --no-modules remains the platform-only complement and cannot be combined with --module. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Flip the capability-compatibility declaration from optional to required (strict mode): the wiring validator now errors on a module-suite check with a missing or empty platforms: declaration - there is no implicit default in this repo. Runtime resolution stays lenient so older or external configs with omitted declarations keep meaning every real environment. The five previously-undeclared suites (network, security, observability, image-registry, storage) are bulk-filled with the behavior-preserving platforms: ["bare_metal", "kubernetes", "slurm", "vm"]; dry-run column plans are byte-identical before and after. Existing declarations (CAP04 pair bare_metal, iam/control-plane foundational) are untouched. Narrowing the filled declarations to honest per-check subsets is now deliberate module-team work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Place image registry checks in the foundational column so they run once per lab instead of once per runtime platform. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Run the self-contained network suite once per lab instead of repeating it for every runtime platform. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Run the self-contained observability suite once per lab instead of repeating it for every runtime platform. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Run provider-wide and self-contained security checks once per lab instead of repeating them for every runtime platform. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Run the self-contained storage suite once per lab while keeping platform-bound storage checks inline with their environments. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
… suite The capacity checks moved to platforms: ["foundational"] with the rest of the security suite; the test still pinned the old bare_metal declaration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
…ket (PROTOTYPE)
Reopens decision 6 (see test-structure-doc-v2.md "Platform sessions"):
after strict-mode narrowing sent every module check to foundational,
the capability x module matrix had no intersections. A platform run can
now bracket module runs, so a concern team writes per-environment
checks in its own suite file and they execute against the live
platform environment - kubernetes x storage is the working example for
team evaluation before wider adoption.
Execution model:
- `--platform <col>` runs the platform config's pre-teardown phases,
then each compatible module's session run INSIDE the bracket, then
the platform teardown, always last (Orchestrator.run gains
pre_teardown_hook; hook errors never block teardown; a failed
platform setup reports session runs as SKIP instead of executing
them against a broken environment).
- A session run executes a provider commands[<module>@<column>]
lifecycle (commands_key) with the platform run's step outputs
mounted read-only at {{ session.* }} for step args and validation
params; steps.* stays scoped to the run's own lifecycle.
- Opt-in is the existing positive platforms: declaration - a module
check declaring a real column is a session check. The planner admits
a module to a column only with a column-eligible check AND the
session commands block; otherwise it is omitted with the reason
surfaced in --dry-run.
- Intersect under a real column (--platform X --module m) is the
column plan filtered to the requested modules: platform bracket plus
only those session runs.
- In a run with no column (bare --module / -f), session-only checks
(declaring only real columns) are skipped with a pointer to
--platform; foundational-declared and undeclared checks keep today's
behavior.
kubernetes x storage example:
- The 15 CSI / shared-filesystem checks move from k8s.yaml to
storage.yaml as session checks (platforms: ["kubernetes"], params
reading {{ session.setup.csi.* }}), restoring storage-team ownership
without losing the cluster context.
- providers/aws/config/storage.yaml gains commands[storage@kubernetes]
(verify cluster access on setup, sweep leaked isvtest-* namespaces
on teardown).
Also: _run_single_config extracted from the run() CLI command (verbatim
move) so the fan-out can pass session parameters; dry-run text/JSON
show session runs, commands keys, and the bracket order; plan-coverage
guardrails accept a platforms: declaration for platform-family test
ids and compare non-platform labels only (decision 7's label split).
Not adopted policy: this is the prototype for the reopened decision;
revert this commit to withdraw it wholesale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reframes test selection around a platform x module matrix and cleans up the surrounding classification/catalog/reporting machinery. No directory moves or manifest redesign — suites are classified by a single axis key declared in their YAML.
Key changes:
tests:—platform: <name>for a service-line platform (vm,bare_metal,kubernetes,slurm) ormodule: <name>for an operational concern (iam,network,security,observability,control_plane,image_registry, ...). Provider configs inherit the key viaimport:.test rungains--platform(run a platform column: the platform config plus every module config, with platform-scoped excludes) and--module(run one module suite); both require--providerand are mutually exclusive with-f.platformvocabulary (runtime,commandskey, upload term, YAML axis key) instead of introducing a competingcapability/kindpairing.MODIFIER_LABELS,EXTRA_MODULE_LABELS,SUITE_REQUIRED_LABELS); the required suite label and module axis are derived from each suite's axis key. Labels are otherwise free-form.schemaVersionplusplatformsandmodulesaxis lists derived from the suites, so a consumer can build the matrix straight from the catalog. Per-entry axes are split intoplatformsandmodules.StepSuccessCheck-iam_teardown,GpuCheck-bm_gpu) so catalog entries, JUnit case names, and matrix cells are 1:1 with wirings.variant_requiredon the reusable classes plusvalidate_suite_wiring.pyenforce global name uniqueness and reject bare wirings.(capability, module)pair — platform runs report(platform, -), module runs inside a--platformcolumn inherit the column as capability, and standalone module runs report(-, module).isvctl deployresolves the pair throughimport:via a neweffective_axeshelper;isvreportergains--moduleand passes unknown axis values through instead of coercing tokubernetes.bare_metal(they need a live host/BMC).Test plan
make lint&&make testmake demo-testpython scripts/validate_suite_wiring.pymake planrenders cleanlyOpened as a draft for review.