Add MEAI: Otel GenAI Manager agentic workflow

[jeffhandley]

Summary

This PR adds MEAI: Otel GenAI Manager, a scheduled GitHub Agentic Workflow (gh-aw) that continuously integrates OpenTelemetry GenAI semantic-conventions updates into Microsoft.Extensions.AI. A deterministic manager resolves the upstream scan target and fans out to an agentic producer that wraps the existing update-otel-genai-conventions skill and maintains a single draft pull request tracking the upstream open-telemetry/semantic-conventions-genai repository, incrementally folding in new convention changes as they land upstream.

It also lands the gh-aw scaffolding this repository needs to host agentic workflows (agent definitions, the Copilot PAT pool, and MCP configuration), plus small refinements to the update-otel-genai-conventions skill so it delegates tracking-PR lifecycle to the maintaining workflow.

How it works

1. Otel GenAI Manager (.github/workflows/meai-otel-genai-manager.yml + meai-otel-genai-manager-discover.sh) -- deterministic discovery, no agent. It scans open-telemetry/semantic-conventions-genai at the default-branch HEAD (or an explicit upstream_ref when dispatched), resolving it to a concrete commit SHA with automatic retry, and emits a single fully-specified target. It invokes the producer as a reusable workflow (workflow_call) once.

2. Otel GenAI Producer (.github/workflows/meai-otel-genai-producer.md -> compiled .lock.yml, + meai-otel-genai-producer-setup.sh) -- a host-side setup step locates the standing draft tracking PR, classifies it, and computes the recommended lifecycle action from the recorded scan ref vs. the freshly resolved upstream SHA. The agent then delegates all convention analysis and implementation to the update-otel-genai-conventions skill, builds and tests the change, folds in reviewer feedback, and a safe-output layer refreshes the draft PR for a human to review and merge.

Behavior

• Lifecycle decision (deterministic, from the recorded scan ref vs. the resolved upstream SHA):
  • Caught up on the SHA -- the run no-ops when there is no new review activity and no new upstream release, leaving the PR exactly as it is. When new review activity has arrived since the last run, a feedback-only update folds it in (see Feedback below) as a body/comment-only update.
  • Behind, draft PR open -- incremental update: implement the differential convention changes, push a commit, refresh the body and tracking block, and post a single summary comment.
  • Behind, PR is non-draft -- advisory: leave a comment describing what is pending, keeping the in-review PR focused on human review.
  • No matching PR (absent or previously closed) -- open a fresh draft PR from scratch. If the canonical update-otel-genai-to-latest branch still lingers from a previously closed PR, the new PR is opened on a run-scoped branch to preserve the existing one, and a > [!NOTE] block documenting the deviation is added to the PR body.
• Build-and-test gate: every change is built and tested (./build.sh -vs AI then ./build.sh -build -test) before a PR is opened or updated. When the internal NuGet feeds are unreachable and the change cannot be validated, the run emits a report_incomplete signal and fails, keeping every published PR build-and-test validated.
• Feedback: whenever there is new PR review activity since the previous run -- submitted review summaries and their inline (file/line-level) review comments -- the run folds that feedback into the same batch of commits it pushes on the incremental path, or into a dedicated feedback-only update when it is already caught up on the SHA, and rejects out-of-scope requests with a brief rationale in the single summary comment. Trust is enforced by gh-aw integrity filtering (min-integrity: approved): only reviews and comments from write-access reviewers (OWNER/MEMBER/COLLABORATOR) -- plus any external comment a maintainer has endorsed with a 👍/❤️ reaction -- reach the agent. Conflicting comments resolve to the most recent guidance. The tracking block's feedback-processed-through watermark advances to the run's start time each run, so feedback is processed exactly once across runs -- this cross-run dedup is the "already handled" signal.
• Human-owned guard: if a non-automation PR already occupies the tracking branch, the automation stands down (blocked). Removing the automation label is an intentional pause -- the required-labels safe-output gate holds all writes until the label is restored.
• Labels + identity: the tracking PR is labeled automation + area-ai and carries a machine-readable state block delimited by # meai-otel-genai-producer:state:begin / :state:end sentinel comments in its body. Before publishing, a post-run guardrail validates that every full-body PR write still carries that state block -- the sentinel markers, a non-empty upstream-scan-ref, and a feedback-processed-through watermark -- matching the sentinels as whole comment lines and each field the same quote/comment-tolerant way the next run's setup parses it, so every published body is guaranteed resumable by the next run.
• Scope: the workflow only runs on a schedule for the canonical (non-fork) repository; manual dispatch is allowed anywhere for testing. The upstream_ref input exists primarily for time-travel testing and targeted re-scans; scheduled runs always scan the upstream default-branch HEAD.

Supporting changes

New agentic workflow (manager + producer)

• .github/workflows/meai-otel-genai-manager.yml, .github/scripts/meai-otel-genai-manager-discover.sh -- the deterministic discovery manager.
• .github/workflows/meai-otel-genai-producer.md -> compiled .github/workflows/meai-otel-genai-producer.lock.yml, .github/scripts/meai-otel-genai-producer-setup.sh -- the agentic producer and its host-side setup/classification.

gh-aw + PAT pool infrastructure

• .github/agents/agentic-workflows.md, .github/skills/agentic-workflow-designer/SKILL.md, .github/skills/agentic-workflows/SKILL.md -- agent/skill definitions for authoring workflows.
• .github/aw/actions-lock.json, .github/mcp.json, .github/workflows/copilot-setup-steps.yml -- gh-aw scaffolding.
• .github/workflows/shared/pat_pool.md, .github/workflows/shared/pat_pool.README.md -- the Copilot PAT pool integration that selects a random PAT from a numbered pool to avoid single-token rate limiting.

Skill refinements (update-otel-genai-conventions)

• Standardized "Gen-AI" to "GenAI" throughout.
• Moved tracking-PR refresh ownership out of the skill's PR-body template and into the maintaining workflow (removed the "Refreshing the tracking PR" section).
• Switched the machine-readable tracking state from HTML-comment delimiters to a fenced yaml block whose first and last lines are # meai-otel-genai-producer:state:begin / :state:end sentinel comments, because safe-output processing strips HTML comments from PR/issue bodies.
• Clarified the defer rule: only brand-new attributes/metrics with no emission site are deferred; changes to already-emitted convention items (type/unit/requiredness/rename/sampling-relevance) are actionable and must be applied in the same pass.
• Tracking table now lists every open upstream PR.

Validation

The full lifecycle was exercised end-to-end against a dedicated test repository by forcing historical upstream commit SHAs into the upstream_ref dispatch input to simulate progressing through time. Every reachable scenario passed cleanly:

• Fresh PR creation, no-op when caught up, incremental update, advisory (non-draft) update, fresh-after-close (opening a new PR when the prior deterministic branch still exists), and reviewer-feedback folding with out-of-scope rejection.
• Idempotency confirmed at the timeline tip.
• The release path (mark-ready on an upstream release) has no live trigger yet because the upstream conventions repository has no tagged releases, so its deterministic logic -- tag resolution, readiness gating, and the draft mark-ready transition -- is validated in full through the chaos harness below.

Test plan executed -- chaos-monkey end-to-end validation of the deterministic host-side logic

A comprehensive, adversarial ("chaos monkey") test pass drove the manager and producer's deterministic scripts and lock-file shell steps -- upstream discovery, producer setup classification/action, the review-activity wake gate, tracking-block parsing, and the manifest guardrail -- against a mock gh harness with fixture-driven responses and failure injection. Trust and feedback selection are delegated to gh-aw integrity filtering (min-integrity: approved plus maintainer endorsement reactions), enforced by the framework itself. The pass validated the setup's resilience behavior, notably retrying the PR-body fetch to keep the tracking marker intact and our own PR correctly classified across transient API blips. It also expanded into the release-gate lifecycle, command-injection safety, forward-threaded idempotence sequences, targeted failure injection, and a full cross-script handoff round-trip, and was capped by randomized property-based fuzzing and an exhaustive combinatorial sweep of the decision state space. Along the way it surfaced and fixed one defect on the release path (an unresolvable release tag could abort a run; readiness now degrades gracefully instead). Scenarios executed:

• Classification matrix -- ours (marker present), adopt (automation labels, no marker), blocked (human-owned PR on the branch), none (no PR), and the body-fetch-blip fallback (unreadable body but automation labels present is treated as ours, preserving prior ownership). A fork PR reusing the branch name is filtered out by head-repository owner, and both discovery stages are scoped to PRs whose base is main (matching the create-pull-request base-branch), so a PR reusing the tracking branch but targeting a different base -- e.g. a backport onto a release branch -- is never mistaken for the maintained PR.
• Action-decision matrix -- caught-up no-op; behind -> produce; new review activity -> feedback-only produce; a published release on an open draft always produces, ensuring the Step 6 mark-ready transition always completes (even once the tag is already recorded in the body, e.g. after a torn prior run), while a caught-up non-draft PR with the release recorded no-ops; adopt always produces, even when the SHA looks current, so the tracking block is written on adoption; empty recorded-SHA and empty upstream-SHA guards; most-recently-updated PR selection when multiple PRs occupy the branch.
• Release-gate lifecycle -- upstream release-tag resolution across annotated tags (a two-hop dereference through the tag object to its commit), lightweight tags, and the commits-endpoint fallback; release_matches_scan, release_matches_pr, and release_ready are computed independently. When a published release exists but its tag cannot be resolved to a commit (for example a brief replication lag between the releases and git-refs APIs, or a retagged release), readiness is withheld at low confidence and the run continues normally rather than aborting. The draft mark-ready path (release_pending) always produces so the Ready-for-review transition completes, settling to a no-op only once the PR is non-draft with the release recorded in its body.
• Review-activity wake gate -- comments/reviews newer than the watermark set has_new_feedback so a caught-up PR is woken for a feedback-only update; activity at or before the watermark leaves the gate closed; an empty watermark opens the gate. Each of the three activity queries (issue comments, review comments, submitted reviews) is retried, and if any still fails the gate opens rather than under-counting, so a transient blip never drops real feedback. Bot activity (including the automation's own summary comments) is filtered out to avoid a self-wake loop. The gate is author-agnostic and body-free -- it only decides whether to wake the agent, while integrity filtering governs which feedback the agent may then see.
• Idempotence and torn-run sequences -- multi-run lifecycles threaded forward so each run's fixture reflects the previous run's actions: behind -> incremental -> caught-up -> stable no-op; the wake gate opens on new feedback and then settles once the watermark advances past that feedback; adopt -> ours is one-way, so a PR is never re-adopted once its tracking marker is written; and a torn run that refreshed the body but never advanced the recorded SHA self-heals on the next run. The no-op state is stable across identical reruns, and the watermark advances monotonically so a given piece of feedback is processed exactly once.
• Discovery -- default-branch HEAD and explicit-ref resolution; transient-blip retry; hard-fail (exit 1, no empty target set) on an unresolvable ref; slash-style refs preserved verbatim; ref argument is injection-safe (passed as a single quoted argument, no shell side effects); well-formed single-element output array. The producer's Stage-2 rediscovery of a prior tracking PR accepts the canonical branch and run-suffixed branches (update-otel-genai-to-latest_<run_id>, digits only) while rejecting near-miss suffixes (_12a, a bare trailing _, _1_2, an embedded 1a2, and a trailing underscore), and validates that each candidate's recorded scan ref is a well-formed commit SHA before selecting the most recently updated one.
• Tracking-block parsing -- missing end marker; duplicated blocks (last wins -- the live block at the body's end, so a stale block pasted above it cannot shadow the real state); quoted, trailing-whitespace, and trailing inline # comment values; CRLF line endings (GitHub returns \r\n); indented field names; an optional leading markdown list/quote marker (-, *, +, >) on field values; bare marker with no fields. The setup parser and the publish-time guardrail match the sentinels as whole comment lines and read each field the same quote/comment-tolerant way, and both scope extraction to the last # meai-otel-genai-producer:state:begin / :state:end range, so any body that passes the guard is guaranteed parseable next run -- format drift or human prose cannot wedge the state machine into a permanent re-produce / re-wake loop.
• Injection and adversarial input safety -- shell and command-substitution metacharacters ($(...), backticks, ;, |, &) embedded in PR-body tracking values, branch names, and refs are carried strictly as data; a command-execution canary confirmed no substitution ever fires along any parsing path. Branch-name injection is rejected by the numeric-suffix guard, a field-name prefix such as x-upstream-scan-ref does not false-match the real field, and oversized (20 KB), unicode, and control-character bodies parse without error. Malicious values never produce a false no-op -- an unrecognized scan ref simply drives a produce.
• Manager -> producer handoff -- empty, {}, and fully-specified TARGET_JSON all resolve to safe defaults; a missing repository/token safely falls back to a fresh-produce decision, keeping the run productive. The producer setup independently re-resolves the scan ref and hard-fails (exit 1) rather than proceeding with an empty upstream_sha, so a standalone dispatch with an unresolvable upstream_ref fails fast instead of publishing an unrecoverable body. An end-to-end round-trip feeds the manager's real discovery output straight into the producer setup, confirming the resolved SHA is passed through verbatim (never silently re-resolved to a different commit) and that the custom repo and branch fields drive the state machine, with ours classification gated on the hard-coded # meai-otel-genai-producer:state: sentinel written into the tracking block.
• Resilience asymmetry -- transient API failures are retried throughout. Where a miss would be unsafe the run fails open: discovery (both stages), the wake gate (any of the three activity queries, partial or total), and the PR-body fetch all default to producing rather than a false no-op. Where proceeding would be unrecoverable it fails fast: an unresolvable upstream scan ref exits 1 rather than publishing an empty target. Release-detection failures are non-fatal and leave the release simply unrecorded.
• Manifest guardrail -- missing agent output and comment-only / partial (append) / empty-body items are passed through untouched; a full-body PR write missing any of the sentinel markers, a well-formed upstream-scan-ref, or the feedback-processed-through watermark fails the run, including the mixed valid-plus-invalid batch case.
• Concurrency and pause semantics -- the manager's cancel-in-progress: false concurrency group serializes runs, keeping pushes to the single evergreen PR strictly sequential; removing the automation label mid-run is a coherent atomic pause (the required-labels safe-output gate holds both the code push and the tracking-block update together, keeping the PR in a consistent state, and the next run re-does the work idempotently).
• Invariant-based verification -- beyond the enumerated scenarios above, the deterministic decision was checked against a set of safety invariants: a no-op is emitted only when the PR is genuinely caught up with no pending feedback or release transition; every behind, feedback-pending, adopt, or fresh state produces; blocked stands down; and an unresolvable release withholds readiness without aborting. These invariants were exercised both by randomized property-based fuzzing over the full state space and by an exhaustive combinatorial sweep of every discrete state (PR classification x recorded-SHA x draft x release resolvability x recorded release x feedback timing), with zero violations across every case, confirming the decision logic cannot silently skip real work or crash on any modeled state.

Deployment prerequisite

The safe-output publish path creates and updates the tracking PR with GITHUB_TOKEN, so the repo/org must enable "Allow GitHub Actions to create and approve pull requests" (actions/permissions/workflow can_approve_pull_request_reviews=true). Copilot requests in the agent are funded by the copilot-pat-pool environment.

---

Draft for iteration.

[Copilot]

Pull request overview

Adds a scheduled “MEAI: Otel GenAI Manager/Producer” GitHub Agentic Workflow that continuously tracks and integrates upstream OpenTelemetry GenAI semantic-conventions changes into Microsoft.Extensions.AI, alongside repository scaffolding for hosting gh-aw workflows (PAT pool, MCP config, and supporting skills/scripts).

Changes:

• Introduces a deterministic manager workflow + discovery script that resolves upstream scan targets and invokes a reusable agentic producer workflow.
• Adds a shared Copilot PAT pool component (docs + import workflow) and a scheduled PAT pool validation workflow.
• Refines the existing update-otel-genai-conventions skill and PR-body tracking conventions (including release-tag confidence guidance).

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
.github/workflows/validate-pat-pool.yml	New scheduled workflow to validate PAT pool entries via Copilot CLI.
.github/workflows/shared/pat_pool.README.md	Documentation for PAT pool onboarding/usage and security rationale.
.github/workflows/shared/pat_pool.md	Shared gh-aw import that selects a random PAT slot and outputs its index.
.github/workflows/meai-otel-genai-producer.md	New agentic producer workflow definition (prompt + safe outputs + lifecycle rules).
.github/workflows/meai-otel-genai-producer.lock.yml	Compiled gh-aw lock workflow for the producer.
.github/workflows/meai-otel-genai-manager.yml	New deterministic manager workflow that discovers upstream target and calls producer.
.github/workflows/copilot-setup-steps.yml	Adds Copilot setup workflow for gh-aw tooling in the Copilot environment.
.github/skills/update-otel-genai-conventions/SKILL.md	Updates skill wording and guidance (release tagging analysis, tracking PR lifecycle ownership).
.github/skills/update-otel-genai-conventions/references/pr-description.md	Updates tracking PR body template (release confidence section, YAML-fenced tracking block).
.github/skills/agentic-workflows/SKILL.md	Adds router skill to direct gh-aw workflow tasks to the right prompts.
.github/skills/agentic-workflow-designer/SKILL.md	Adds interview-style workflow designer skill for gh-aw workflow creation.
.github/scripts/meai-otel-genai-producer-setup.sh	Host-side deterministic setup script to discover PR state, compute action, and write target.json.
.github/scripts/meai-otel-genai-manager-discover.sh	Deterministic upstream SHA resolution and target emission for the manager.
.github/mcp.json	Adds local MCP server wiring for gh aw mcp-server.
.github/aw/actions-lock.json	Adds pinned action SHAs used by gh-aw scaffolding.
.github/agents/agentic-workflows.md	Adds an agent dispatcher doc for gh-aw workflow work in this repo.
.gitattributes	Marks .github/workflows/*.lock.yml as generated and resolves merges with ours.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 11 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 13 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 2 comments.