Feature/task redesign

.agents/skills/anvil-task-builder/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: anvil-task-builder
-description: Builds and maintains Anvil task modules, workflows, schemas, runner behavior, SARIF-compatible detect_ tasks, and plugin templates. Use when user asks to "create an Anvil task", "edit this task", "add dry-run behavior", "record actions", "return task results", "create a SARIF task", "create a detect task", "update Anvil YAML", "modify schemas", "change account execution", or "update plugin templates".
+description: Builds and maintains Anvil task modules, workflows, schemas, runner behavior, SARIF-compatible detect_ tasks, and plugin templates. Use when user asks to "create an Anvil task", "edit this task", "add dry-run behavior", "record actions", "return task results", "create a SARIF task", "create a detect task", "update Anvil YAML", "modify schemas", "change account execution", "update plugin templates", "add concurrency for the payer account", or "build a management-account-only task".
 ---
 
 # Anvil Task Builder
@@ -47,10 +47,11 @@ Load only the reference files needed for the current task:
 - For stock task, plugin task, entry-point, and `run()` signature rules, read `references/runtime-contract.md`.
 - For dry-run behavior, action recording, logging, and result shape, read `references/dry-run-and-actions.md`.
 - For task granularity, inventory task boundaries, performance, and region concurrency, read `references/task-granularity.md`.
+- For concurrency guidance specific to workflows that target only the payer/management account, read `references/payer-management-account-tasks.md`.
 - For AWS read-only and mutating task implementation patterns, read `references/aws-task-patterns.md`.
 - For GitHub REST task helpers, repository target rules, and metadata helpers, read `references/github-task-patterns.md`.
 - For SARIF-compatible `detect_` tasks and `sarif_findings` output, read `references/sarif-detection-tasks.md`.
-- For YAML examples, dependencies, optional tasks, and validation commands, read `references/yaml-and-validation.md`.
+- For YAML examples, invocation IDs, dependencies, dependency data, and validation commands, read `references/yaml-and-validation.md`.
 
 ## Review Behavior
 

.agents/skills/anvil-task-builder/references/aws-task-patterns.md

@@ -42,6 +42,7 @@ def run(
     session,
     dry_run: bool,
     metadata: dict[str, object],
+    dependency_data: dict[str, object],
     actions: ActionRecorder,
 ) -> None:
     user_name = metadata.get("user_name")

.agents/skills/anvil-task-builder/references/payer-management-account-tasks.md

@@ -0,0 +1,111 @@
+# Payer / Management Account-Only Tasks
+
+Use this reference when a workflow's target selection resolves to only the
+payer (management) account — not when the payer/management account is one of
+several targets in a broader multi-account run. If the same workflow also
+targets other accounts, treat the payer account like any other target and
+follow the general guidance in `task-granularity.md` instead.
+
+## Why This Case Is Different
+
+`task-granularity.md` frames concurrency pressure as:
+
+```text
+max_parallel_targets * max_workers * max_parallel_regions
+```
+
+That formula assumes multiple accounts are being worked concurrently, which
+is why the general advice is to raise `max_parallel_regions` cautiously and
+benchmark first. When a run's target set is only the payer/management
+account, `max_parallel_targets` is fixed at 1 for that run. There is no other
+account's work competing for the same connection pool, worker threads, or API
+quota, so a task (or the YAML orchestration around it) has meaningfully more
+headroom to use threading, async, or higher region/worker concurrency than a
+typical multi-account task would.
+
+This headroom is about contention with *other targets*, not about the target
+service's own limits. AWS service throttling still applies per account and
+region regardless of how many other accounts Anvil happens to be running
+against in the same invocation.
+
+## Where This Actually Helps
+
+The benefit is largest for services that live only in the payer/management
+account and require one API call per item with no batch equivalent:
+
+- IAM Identity Center (`sso-admin`, `identitystore`) — enumerating or acting
+  on permission sets, account assignments, or identity store users/groups.
+- AWS Organizations — enumerating accounts, OUs, or policies attached across
+  many targets from the single management account.
+- Control Tower or an Organizations-wide Config aggregator, when querying
+  many regions from the single aggregator account.
+
+It does not help generic single-account inventory tasks that already run
+quickly, and it does not justify adding concurrency to a task that doesn't
+need it. See `python-best-practices/concurrency-and-caching.md` for the
+baseline rule that concurrency should only be added when it provides a
+meaningful, measurable benefit.
+
+## Pattern: Bounded Fan-Out For No-Batch-API Calls
+
+Use this shape whenever a payer-only task needs to make one API call per item
+and the service has no batch or single-paginated-call equivalent. It works
+for both mutating fan-out (deleting or updating N items) and read-heavy
+fan-out (describing N items after a list call):
+
+```python
+from concurrent.futures import ThreadPoolExecutor, as_completed
+
+from anvil.task_errors import TaskExecutionError
+
+# No batch API for this call, so document why this number was chosen,
+# e.g. an observed throttling point, or a conservative starting value.
+MAX_WORKERS = 5
+
+
+def _process_item(client, item: dict[str, object]) -> dict[str, object]:
+    # One provider API call per item goes here.
+    ...
+    return item
+
+
+def fan_out(
+    client, items: list[dict[str, object]]
+) -> tuple[list[dict[str, object]], list[dict[str, object]]]:
+    succeeded: list[dict[str, object]] = []
+    failed: list[dict[str, object]] = []
+
+    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
+        future_to_item = {
+            executor.submit(_process_item, client, item): item for item in items
+        }
+
+        for future in as_completed(future_to_item):
+            item = future_to_item[future]
+            try:
+                succeeded.append(future.result())
+            except Exception as error:  # narrow to the real provider exceptions
+                failed.append({**item, "error": str(error)})
+
+    return succeeded, failed
+```
+
+Raise `TaskExecutionError` with a `partial_result` carrying both `succeeded`
+and `failed` when `failed` is non-empty, following the standard
+partial-failure pattern instead of aborting the whole batch on the first
+error.
+
+## Guardrails
+
+- Keep the worker count a named, documented module constant, never an
+  unbounded pool.
+- Still respect the target service's own rate limits. "No other account is
+  competing" does not mean "no limits apply."
+- Do not reach for this pattern just because a task is payer-only. Only use
+  it when the call shape (one call per item, no batch API, a meaningful item
+  count) actually benefits from concurrency.
+- Preserve per-item error isolation: one failure should not abort the whole
+  batch unless the task's contract requires all-or-nothing behavior.
+- If the workflow's target set later expands to include other accounts
+  alongside the payer account, re-evaluate. The headroom this doc describes
+  no longer applies once `max_parallel_targets` is greater than 1.

.agents/skills/anvil-task-builder/references/runtime-contract.md

@@ -56,6 +56,7 @@ def run(
     session,
     dry_run: bool,
     metadata: dict[str, object],
+    dependency_data: dict[str, object],
     actions: ActionRecorder,
 ) -> dict:
 ```
@@ -74,9 +75,11 @@ Runtime facts:
 
 - The provided `session` is already scoped to the provider target and region.
 - Tasks run once per concrete region by default. A task module may declare
-  `TASK_SCOPE = "target"` to run once per execution target instead.
-- Providers declare which task scopes they support. AWS supports only the
-  default `region` scope; Azure, GCP, and GitHub support `region` and `target`.
+  `TASK_SCOPE = "target"` to run once per execution target or
+  `TASK_SCOPE = "configured_target"` to run once for the configured YAML target.
+- Providers declare which task scopes they support. AWS supports
+  `configured_target` and `region`; Azure, GCP, and GitHub support `region` and
+  `target`.
 - A target-scoped task receives the first resolved concrete provider location
   as `region`, and its session uses that location. No synthetic target-scope or
   global sentinel is introduced. GitHub's `global` value is a real provider
@@ -87,10 +90,12 @@ Runtime facts:
   itself, or ignores it because the API is target-wide.
 - For region-scoped tasks, `region` is the current task execution region. AWS
   sessions also expose `session.region_name`.
-- Operator-provided task inputs come from `metadata`.
-- Tasks should treat metadata as read-only configuration.
-  Anvil isolates changes to top-level metadata keys,
-  but not changes inside nested lists or dictionaries.
+- Operator-provided static task inputs come from `metadata`. Target metadata is
+  recursively merged with task metadata, with task values taking precedence.
+- Runtime dependency inputs come from `dependency_data`. They are selected from
+  direct dependency `TaskResult` objects and are never merged into `metadata`.
+- Tasks should treat both mappings as read-only. Anvil deep-copies nested
+  mappings and lists for every invocation.
 - `actions` is an `ActionRecorder` for audit-level actions.
 - Returned values are included in Anvil result JSON.
 - The engine already includes execution context such as target identity, `region`, and `dry_run` in normal results.
@@ -119,6 +124,7 @@ def run(
     session,
     dry_run: bool,
     metadata: dict[str, object],
+    dependency_data: dict[str, object],
     actions: ActionRecorder,
 ) -> dict:
     """Check the current provider target and return a simple status payload.
@@ -132,6 +138,7 @@ def run(
         session: Provider runtime session scoped to the target and region.
         dry_run: Whether Anvil is running in dry-run mode.
         metadata: Task metadata from YAML. This task does not require metadata.
+        dependency_data: Runtime dependency inputs. This task requires none.
         actions: Action recorder provided by the Anvil engine.
 
     Returns:
@@ -148,3 +155,30 @@ def run(
 ```
 
 First-party provider tasks should use the provider-neutral signature above.
+
+## Invocation identity and dependencies
+
+YAML `name` selects the discovered component. YAML `id` identifies one
+configured invocation and defaults to `name` when omitted. `depends_on` and
+`dependency_data.task_id` always reference effective invocation IDs. Component
+names are not a dependency fallback.
+
+The same component may be configured more than once only when every occurrence
+has an explicit, unique ID. Results preserve both `task_id` and `task_name`.
+
+Normal tasks run only when every dependency succeeds. An `always_run` task waits
+for every dependency to settle and then runs even after errors, interruption, or
+skips, provided its dependency chain began. Successful finalization does not
+erase an upstream failure.
+
+Use `TaskExecutionError` when a task must report failure while retaining
+JSON-serializable recovery data:
+
+```python
+from anvil.task_errors import TaskExecutionError
+
+
+raise TaskExecutionError(
+    "Mutation partially failed", partial_result={"attachments": detached_attachments}
+)
+```

.agents/skills/anvil-task-builder/references/task-granularity.md

@@ -6,7 +6,7 @@ Prefer separate tasks when:
 
 - The tasks have different safety profiles, especially read-only vs mutating.
 - They are commonly run independently.
-- They need different optional or fail-fast behavior.
+- They need different failure, cleanup, or dependency behavior.
 - A dependency relationship is meaningful to the workflow.
 - Combining them would make the result shape confusing or too broad.
 
@@ -36,4 +36,10 @@ For lightweight describe/list inventory across many accounts, especially multipl
 max_parallel_targets * max_workers * max_parallel_regions
 ```
 
-Recommend benchmarking the actual task mix before raising region concurrency.
+Recommend benchmarking the actual task mix before raising region concurrency.
+
+This caution assumes multiple accounts are in scope for the run. If the
+workflow's target set is only the payer/management account,
+`max_parallel_targets` is fixed at 1 and there is no cross-account
+contention to protect against, so there is more headroom for concurrency.
+See `references/payer-management-account-tasks.md`.

.agents/skills/anvil-task-builder/references/yaml-and-validation.md

@@ -18,15 +18,37 @@ targets:
     dry_run: true
     tasks:
       - name: inventory_users
-      - name: remove_iam_user
+      - id: remove_example_user
+        name: remove_iam_user
         depends_on:
           - inventory_users
-        optional: false
+        dependency_data:
+          users:
+            task_id: inventory_users
+            path: result.users
     metadata:
       user_name: example-user
 ```
 
-Use `depends_on` when task order matters. Use `optional: true` only when failure should not fail the account or block dependent work.
+`name` selects the discovered component. `id` identifies one configured use and
+defaults to `name` when omitted. If a component is repeated, give every use an
+explicit unique ID.
+
+Use effective IDs in `depends_on`; Anvil does not fall back to component names.
+Normal dependents require every dependency to succeed. Use `always_run: true`
+with at least one dependency for cleanup that should run after unsuccessful
+work.
+
+`dependency_data` selects runtime values from tasks listed directly in
+`depends_on`. Omit `path` for the complete producer `TaskResult`, use
+`path: result` for its returned value, or select nested mapping values such as
+`path: result.users`. Existing null values are valid; missing paths produce a
+consumer task error before its `run()` function is called.
+
+Task scope is module-declared with `TASK_SCOPE`; do not put scope in YAML.
+
+For a complete configured-target fan-out/fan-in and recovery example, see
+`examples/17-aws-config-cleanup-workflow.yaml`.
 
 ## Region Selection
 

README.md

@@ -129,6 +129,10 @@ targets:
 
 Task compatibility is determined by package location.
 
+For schema-v2 invocation IDs, dependency-data selection, `always_run` recovery,
+module-declared scopes, and configured-target fan-in/fan-out, see
+[Task workflows](docs/task-workflows.md).
+
 - `anvil.providers.tasks.<task>` is universal and can run for any provider.
 - `anvil.providers.aws.tasks.<task>` is AWS-only.
 - `anvil.providers.azure.tasks.<task>` is Azure-only.
@@ -207,7 +211,8 @@ For delegated-administrator patterns, keep the base session on the
 delegated-admin profile. Anvil uses that base session directly for the
 delegated-admin account if it appears in Organizations discovery, and assumes
 `role_name` in every other selected account, including the management/payer
-account.
+account. AWS organization targets accept `management` and `payer` as
+case-insensitive aliases for that account in `include` and `exclude` filters.
 
 ```yaml
 schema_version: 2
@@ -222,6 +227,8 @@ targets:
         role_name: SecurityAuditRole
     regions:
       - us-east-1
+    include:
+      - management
     tasks:
       - name: noop
 ```

examples/03-aws-include-exclude.yaml

@@ -9,7 +9,7 @@ targets:
   regions:
   - us-east-1
   include:
-  - '111111111111'
+  - management
   - '222222222222'
   dry_run: true
   tasks:
@@ -24,7 +24,7 @@ targets:
   regions:
   - us-east-1
   exclude:
-  - '999999999999'
+  - payer
   dry_run: true
   tasks:
   - name: count_vpc

examples/04-aws-advanced.yaml

@@ -23,11 +23,12 @@ targets:
     - python3.8
     - nodejs16.x
   tasks:
-  - name: count_vpc
-  - name: list_lambdas_by_runtime
-  - name: detect_deprecated_lambda_runtimes
-    depends_on:
-    - list_lambdas_by_runtime
+  # Advanced examples use descriptive invocation IDs so result records remain
+  # clear even when a component is configured more than once elsewhere.
+  - id: vpc_count
+    name: count_vpc
+  - id: deprecated_runtime_findings
+    name: detect_deprecated_lambda_runtimes
   post_run:
   - processor: sarif_report
     output: aws-lambda-runtime-audit.sarif

examples/08-azure-advanced.yaml

@@ -19,7 +19,8 @@ targets:
   fail_fast: true
   dry_run: true
   tasks:
-  - name: count_resource_groups
+  - id: resource_group_count
+    name: count_resource_groups
   post_run:
   - processor: html_report
     output: azure-resource-group-inventory.html

examples/12-gcp-advanced.yaml

@@ -19,7 +19,8 @@ targets:
   fail_fast: true
   dry_run: true
   tasks:
-  - name: get_project_info
+  - id: project_inventory
+    name: get_project_info
   post_run:
   - processor: html_report
     output: gcp-project-inventory.html