Validate array/span parameters in the WinMD generator and add failure tests

.github/skills/testing/SKILL.md

@@ -13,7 +13,7 @@ Before adding tests, always check whether tests for the same functionality alrea
 
 ## Test project overview
 
-CsWinRT 3.0 has 5 primary test project areas, each serving a different purpose. Additional specialized test projects also exist under `src/Tests/`:
+CsWinRT 3.0 has 6 primary test project areas, each serving a different purpose. Additional specialized test projects also exist under `src/Tests/`:
 
 ### 1. Unit tests (`src/Tests/UnitTest/`)
 
@@ -213,6 +213,45 @@ public async Task InvalidType_Warns()
 - **Release x64:** NativeAOT self-contained mode
 - Build-time validation (compilation succeeds = test passes)
 
+### 6. WinMD generator tests (`src/Tests/WinMDGeneratorTest/`)
+
+**What it tests:** End-to-end behavior of the `cswinrtwinmdgen` build tool (the WinMD generator). It focuses on **failure cases** — unsupported authored component shapes and invalid tool invocations — that can be asserted without breaking a real build.
+
+**When to add tests here:** For testing a WinMD generator failure mode — a new `CSWINRTWINMDGEN` validation error, an unsupported authoring pattern, or a malformed/invalid invocation. Authored shapes that produce a valid `.winmd` are covered by the authoring tests (`AuthoringTest/`) instead.
+
+**Project settings:**
+- **Test framework:** MSTest (`[TestClass]`, `[TestMethod]`, `Assert.*`)
+- **TFM:** `net10.0`, multi-platform (x64/x86)
+- **Output type:** Exe, `CsWinRTEnabled=false`
+- **Key dependencies:** MSTest packages, `Basic.Reference.Assemblies.Net100` and `Microsoft.CodeAnalysis.CSharp` (used to compile the C# inputs)
+- **References:** `WinRT.WinMD.Generator` via `ProjectReference` with `ReferenceOutputAssembly="false"` and `GlobalPropertiesToRemove="RuntimeIdentifier"` — the tool is **built but not referenced**, and tests invoke it as a separate process. Its path is passed to the tests via an `AssemblyMetadata` item (`WinMDGeneratorAssemblyPath`).
+
+**Test classes:**
+| Test class | What it tests |
+|------------|---------------|
+| `Test_ParameterConventions` | Unsupported array/span parameter shapes (`ref`/`in` arrays, `out` spans) → `CSWINRTWINMDGEN0011`/`0012`, plus a valid smoke test |
+| `Test_InvalidInputs` | Invalid invocations: malformed/missing response files, bad arguments, missing/corrupt input assemblies, an unwritable output path, a missing debug-repro directory |
+
+**Test helper (in `Helpers/`):**
+- `WinMDGeneratorRunner` — compiles a C# input, runs the actual tool as a subprocess, and asserts the outcome. Public entry points: `AssertSuccess`, `AssertFailure` (from component source or full response-file content), `AssertFailureForMissingResponseFile`, and `CompileComponent`.
+
+**Pattern:**
+```csharp
+[TestMethod]
+public void RefArrayParameter_IsRejected()
+{
+    WinMDGeneratorRunner.AssertFailure("""
+        public interface IComponent
+        {
+            void Method(ref int[] values);
+        }
+        """, error: "CSWINRTWINMDGEN0011");
+}
+```
+
+- Each test is a single `AssertSuccess`/`AssertFailure` call; the runner makes the exit-code and error-output assertions
+- Failure cases assert the tool exits non-zero and its output contains the expected `CSWINRTWINMDGEN` error id
+
 ## Deciding where to add tests
 
 | You want to test... | Add test to... |
@@ -227,6 +266,7 @@ public async Task InvalidType_Warns()
 | GC/reference tracking behavior | `ObjectLifetimeTests/` |
 | XAML visual tree element lifetime | `ObjectLifetimeTests/` |
 | WinRT component authoring patterns | `AuthoringTest/` (when enabled) |
+| A WinMD generator failure mode (a `CSWINRTWINMDGEN` error) | `WinMDGeneratorTest/` (add to `Test_ParameterConventions` or `Test_InvalidInputs`) |
 | Generated projection code patterns or cross-ABI control flow | Update `TestComponentCSharp/` and add tests in `UnitTest/` or `FunctionalTests/` |
 
 ## Test component: TestComponentCSharp (`src/Tests/TestComponentCSharp/`)

.github/skills/update-testing-instructions/SKILL.md

@@ -76,7 +76,19 @@ Launch an explore agent to verify:
 - **"Referenced from" list** is current
 - **"When to update" guidance** is still correct
 
-### Step 8: verify the "deciding where to add tests" table
+### Step 8: verify WinMD generator tests (`src/Tests/WinMDGeneratorTest/`)
+
+Launch an explore agent to analyze the WinMD generator test project. Verify:
+
+- **Project settings** are current: test framework, TFM, platforms, output type, `CsWinRTEnabled` setting
+- **Tool wiring** is accurate: the `WinRT.WinMD.Generator` `ProjectReference` is built but not referenced (`ReferenceOutputAssembly="false"` + `GlobalPropertiesToRemove="RuntimeIdentifier"`), and the `AssemblyMetadata` item (`WinMDGeneratorAssemblyPath`) passes the built tool's path to the tests
+- **Key dependencies** are current (MSTest packages, `Basic.Reference.Assemblies.Net100`, `Microsoft.CodeAnalysis.CSharp`)
+- **Test class table** is complete and accurate — check all `Test_*.cs` files, verify no classes have been added, removed, or renamed, and verify each class's description matches its actual content
+- **Test helper** (`WinMDGeneratorRunner` in `Helpers/`) is accurately described, including its public entry points (`AssertSuccess`, `AssertFailure`, `AssertFailureForMissingResponseFile`, `CompileComponent`)
+- **Pattern example** matches the actual single-call `AssertSuccess`/`AssertFailure` style
+- **Error ids** referenced (`CSWINRTWINMDGEN*`) are still produced by the generator
+
+### Step 9: verify the "deciding where to add tests" table
 
 Check every row in the routing table against the actual test projects. Verify:
 
@@ -85,16 +97,17 @@ Check every row in the routing table against the actual test projects. Verify:
 - No new test project categories exist that should be added as rows
 - The `TestComponentCSharp` guidance is still accurate
 
-### Step 9: verify code examples
+### Step 10: verify code examples
 
 For each code example in the testing skill, verify it matches the conventions actually used in the test files:
 
 - **Unit test pattern**: verify the example uses the correct assert methods, attributes, and structure
 - **Functional test pattern**: verify the example uses the correct exit codes and top-level statement style
 - **Generator test pattern**: verify the `VerifySources` call matches the actual method signature and style. Check an actual test method in `Test_CustomPropertyProviderGenerator` to confirm
 - **Analyzer test pattern**: verify the `VerifyAnalyzerAsync` call matches the actual method signature and style. Check an actual test method in the analyzer test files to confirm
+- **WinMD generator test pattern**: verify the single-call `AssertSuccess`/`AssertFailure` example matches the actual style. Check an actual test method in `Test_ParameterConventions` or `Test_InvalidInputs` to confirm
 
-### Step 10: update the testing instructions
+### Step 11: update the testing instructions
 
 Apply surgical edits to `.github/skills/testing/SKILL.md` to fix any discrepancies found. Typical updates include:
 
@@ -115,17 +128,17 @@ Apply surgical edits to `.github/skills/testing/SKILL.md` to fix any discrepanci
 - Do not add unnecessary commentary or explanation beyond what's needed for test placement guidance
 </style_rules>
 
-### Step 11: update this skill if needed
+### Step 12: update this skill if needed
 
 If significant changes to the test suite were discovered (e.g. test projects added or removed, new categories of tests, changed validation criteria), also update this skill file (`.github/skills/update-testing-instructions/SKILL.md`) to reflect those changes. In particular:
 
-- The **per-project verification steps** (steps 2–7) must stay in sync with the actual test projects. If a test project is added or removed, add or remove its verification step accordingly.
+- The **per-project verification steps** (steps 2–8) must stay in sync with the actual test projects. If a test project is added or removed, add or remove its verification step accordingly.
 - The **verification criteria** for each step should reflect what is actually worth checking. If a project gains new aspects worth validating (e.g. new test helper classes, new build settings), add those to the checklist.
-- The **code example verification step** (step 9) should list all code examples present in the testing skill.
+- The **code example verification step** (step 10) should list all code examples present in the testing skill.
 
 This ensures the skill remains useful and accurate for future runs.
 
-### Step 12: summarize changes
+### Step 13: summarize changes
 
 After editing, provide a clear summary of what was updated and why, so the user can review the changes before committing.
 

src/Tests/WinMDGeneratorTest/Helpers/WinMDGeneratorRunner.cs

@@ -0,0 +1,226 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+using System;
+using System.Diagnostics;
+using System.IO;
+using System.Linq;
+using System.Reflection;
+using Basic.Reference.Assemblies;
+using Microsoft.CodeAnalysis;
+using Microsoft.CodeAnalysis.CSharp;
+using Microsoft.CodeAnalysis.Emit;
+
+namespace WinMDGeneratorTest.Helpers;
+
+/// <summary>
+/// Runs the WinMD generator (<c>cswinrtwinmdgen</c>) end-to-end and asserts its outcome.
+/// </summary>
+/// <remarks>
+/// Each entry point compiles a small C# input (or builds a raw response file), runs the actual tool as
+/// a separate process, and asserts the result. Tests should call <see cref="AssertSuccess"/> or one of
+/// the <c>AssertFailure</c> overloads so each scenario stays a single call.
+/// </remarks>
+internal static class WinMDGeneratorRunner
+{
+    /// <summary>
+    /// Asserts that the generator succeeds (exit code <c>0</c>) for the given component source.
+    /// </summary>
+    /// <param name="source">The C# source defining the authored component types.</param>
+    /// <param name="useWindowsUIXamlProjections">Whether to use the UWP (<c>Windows.UI.Xaml</c>) projections.</param>
+    public static void AssertSuccess(string source, bool useWindowsUIXamlProjections = false)
+    {
+        (int exitCode, string output) = Run(source, useWindowsUIXamlProjections);
+
+        Assert.AreEqual(0, exitCode, output);
+    }
+
+    /// <summary>
+    /// Asserts that the generator fails for the given component source, reporting the expected error.
+    /// </summary>
+    /// <param name="source">The C# source defining the authored component types.</param>
+    /// <param name="error">The expected error id (e.g. <c>"CSWINRTWINMDGEN0011"</c>) the output must contain.</param>
+    /// <param name="useWindowsUIXamlProjections">Whether to use the UWP (<c>Windows.UI.Xaml</c>) projections.</param>
+    public static void AssertFailure(string source, string error, bool useWindowsUIXamlProjections = false)
+    {
+        AssertFailureResult(Run(source, useWindowsUIXamlProjections), error);
+    }
+
+    /// <summary>
+    /// Asserts that the generator fails for a caller-provided response file, reporting the expected error.
+    /// </summary>
+    /// <remarks>
+    /// The <paramref name="responseFileFactory"/> receives a fresh temporary working directory and
+    /// returns the full response file content to run with (one <c>--argument value</c> pair per line). It
+    /// can use <see cref="CompileComponent"/> to produce an input assembly, or reference any other path
+    /// under the directory (e.g. to test invalid inputs).
+    /// </remarks>
+    /// <param name="responseFileFactory">Builds the full response file content for a given temporary directory.</param>
+    /// <param name="error">The expected error id (e.g. <c>"CSWINRTWINMDGEN0002"</c>) the output must contain.</param>
+    public static void AssertFailure(Func<string, string> responseFileFactory, string error)
+    {
+        AssertFailureResult(Run(responseFileFactory), error);
+    }
+
+    /// <summary>
+    /// Asserts that the generator fails when pointed at a response file path that does not exist,
+    /// reporting the expected error.
+    /// </summary>
+    /// <param name="error">The expected error id (e.g. <c>"CSWINRTWINMDGEN0001"</c>) the output must contain.</param>
+    public static void AssertFailureForMissingResponseFile(string error)
+    {
+        string missingResponseFilePath = Path.Combine(Path.GetTempPath(), $"WinMDGeneratorTest_{Guid.NewGuid():N}.rsp");
+
+        AssertFailureResult(RunTool(GetGeneratorPath(), missingResponseFilePath), error);
+    }
+
+    /// <summary>
+    /// Compiles the given C# <paramref name="source"/> into <c>TestInput.dll</c> in <paramref name="directory"/>.
+    /// </summary>
+    /// <remarks>
+    /// This is exposed for the <see cref="AssertFailure(Func{string, string}, string)"/>
+    /// scenarios that need a valid input assembly before triggering a later-stage failure (e.g. an
+    /// unwritable output path).
+    /// </remarks>
+    /// <param name="source">The C# source defining the authored component types.</param>
+    /// <param name="directory">The directory to emit the assembly into.</param>
+    /// <returns>The full path to the compiled assembly.</returns>
+    public static string CompileComponent(string source, string directory)
+    {
+        string outputPath = Path.Combine(directory, "TestInput.dll");
+
+        CSharpParseOptions parseOptions = new(LanguageVersion.Preview);
+
+        // The target framework attribute lets the generator probe the .NET runtime version. It is added
+        // in a separate syntax tree so it does not interfere with any 'using' directives in the test
+        // source (an assembly attribute must precede type declarations but follow 'using' directives).
+        SyntaxTree sourceTree = CSharpSyntaxTree.ParseText(source, parseOptions);
+        SyntaxTree assemblyInfoTree = CSharpSyntaxTree.ParseText(
+            """[assembly: System.Runtime.Versioning.TargetFramework(".NETCoreApp,Version=v10.0")]""",
+            parseOptions);
+
+        CSharpCompilation compilation = CSharpCompilation.Create(
+            assemblyName: "TestInput",
+            syntaxTrees: [sourceTree, assemblyInfoTree],
+            references: Net100.References.All,
+            options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary, allowUnsafe: true));
+
+        EmitResult result = compilation.Emit(outputPath);
+
+        Assert.IsTrue(result.Success, $"Input compilation failed:\n{string.Join("\n", result.Diagnostics)}");
+
+        return outputPath;
+    }
+
+    /// <summary>
+    /// Compiles the given C# <paramref name="source"/> into an assembly and runs the WinMD generator
+    /// against it with a standard response file.
+    /// </summary>
+    private static (int ExitCode, string Output) Run(string source, bool useWindowsUIXamlProjections)
+    {
+        return Run(temporaryDirectory =>
+        {
+            string inputAssemblyPath = CompileComponent(source, temporaryDirectory);
+
+            return $"""
+                --input-assembly-path {inputAssemblyPath}
+                --reference-assembly-paths {inputAssemblyPath}
+                --output-winmd-path {Path.Combine(temporaryDirectory, "TestInput.winmd")}
+                --assembly-version 1.0.0.0
+                --use-windows-ui-xaml-projections {useWindowsUIXamlProjections}
+                """;
+        });
+    }
+
+    /// <summary>
+    /// Runs the WinMD generator against a caller-provided response file.
+    /// </summary>
+    private static (int ExitCode, string Output) Run(Func<string, string> responseFileFactory)
+    {
+        string toolPath = GetGeneratorPath();
+        string temporaryDirectory = Directory.CreateTempSubdirectory("WinMDGeneratorTest_").FullName;
+
+        try
+        {
+            string responseFileContent = responseFileFactory(temporaryDirectory);
+            string responseFilePath = Path.Combine(temporaryDirectory, "args.rsp");
+
+            File.WriteAllText(responseFilePath, responseFileContent);
+
+            return RunTool(toolPath, responseFilePath);
+        }
+        finally
+        {
+            TryDeleteDirectory(temporaryDirectory);
+        }
+    }
+
+    /// <summary>
+    /// Asserts that a generator run failed (non-zero exit code) and its output contains the expected error.
+    /// </summary>
+    private static void AssertFailureResult((int ExitCode, string Output) result, string error)
+    {
+        Assert.AreNotEqual(0, result.ExitCode, result.Output);
+        StringAssert.Contains(result.Output, error);
+    }
+
+    /// <summary>
+    /// Runs <c>dotnet exec &lt;tool&gt; &lt;argument&gt;</c> and captures the exit code and output.
+    /// </summary>
+    private static (int ExitCode, string Output) RunTool(string toolPath, string argument)
+    {
+        ProcessStartInfo startInfo = new("dotnet")
+        {
+            RedirectStandardOutput = true,
+            RedirectStandardError = true,
+            UseShellExecute = false,
+            CreateNoWindow = true,
+        };
+
+        startInfo.ArgumentList.Add("exec");
+        startInfo.ArgumentList.Add(toolPath);
+        startInfo.ArgumentList.Add(argument);
+
+        using Process process = Process.Start(startInfo) ?? throw new InvalidOperationException("Failed to start the WinMD generator process.");
+
+        string standardOutput = process.StandardOutput.ReadToEnd();
+        string standardError = process.StandardError.ReadToEnd();
+
+        process.WaitForExit();
+
+        return (process.ExitCode, standardOutput + standardError);
+    }
+
+    /// <summary>
+    /// Resolves the path to the built <c>cswinrtwinmdgen</c> tool from assembly metadata.
+    /// </summary>
+    private static string GetGeneratorPath()
+    {
+        string? path = typeof(WinMDGeneratorRunner).Assembly
+            .GetCustomAttributes<AssemblyMetadataAttribute>()
+            .FirstOrDefault(static attribute => attribute.Key == "WinMDGeneratorAssemblyPath")?.Value;
+
+        Assert.IsFalse(string.IsNullOrEmpty(path), "The 'WinMDGeneratorAssemblyPath' assembly metadata was not found.");
+
+        string fullPath = Path.GetFullPath(path!);
+
+        Assert.IsTrue(File.Exists(fullPath), $"The WinMD generator was not found at '{fullPath}'.");
+
+        return fullPath;
+    }
+
+    /// <summary>
+    /// Deletes a directory recursively, ignoring failures (best effort cleanup).
+    /// </summary>
+    private static void TryDeleteDirectory(string directory)
+    {
+        try
+        {
+            Directory.Delete(directory, recursive: true);
+        }
+        catch (IOException)
+        {
+            // Best effort cleanup; a leftover temp directory must not fail the test
+        }
+    }
+}