diff --git a/.agents/skills/anvil-task-builder/SKILL.md b/.agents/skills/anvil-task-builder/SKILL.md index b9ffe07..563dfdf 100644 --- a/.agents/skills/anvil-task-builder/SKILL.md +++ b/.agents/skills/anvil-task-builder/SKILL.md @@ -1,6 +1,6 @@ --- name: anvil-task-builder -description: Builds and maintains Anvil task modules, workflows, schemas, runner behavior, SARIF-compatible detect_ tasks, and plugin templates. Use when user asks to "create an Anvil task", "edit this task", "add dry-run behavior", "record actions", "return task results", "create a SARIF task", "create a detect task", "update Anvil YAML", "modify schemas", "change account execution", or "update plugin templates". +description: Builds and maintains Anvil task modules, workflows, schemas, runner behavior, SARIF-compatible detect_ tasks, and plugin templates. Use when user asks to "create an Anvil task", "edit this task", "add dry-run behavior", "record actions", "return task results", "create a SARIF task", "create a detect task", "update Anvil YAML", "modify schemas", "change account execution", "update plugin templates", "add concurrency for the payer account", or "build a management-account-only task". --- # Anvil Task Builder @@ -47,10 +47,11 @@ Load only the reference files needed for the current task: - For stock task, plugin task, entry-point, and `run()` signature rules, read `references/runtime-contract.md`. - For dry-run behavior, action recording, logging, and result shape, read `references/dry-run-and-actions.md`. - For task granularity, inventory task boundaries, performance, and region concurrency, read `references/task-granularity.md`. +- For concurrency guidance specific to workflows that target only the payer/management account, read `references/payer-management-account-tasks.md`. - For AWS read-only and mutating task implementation patterns, read `references/aws-task-patterns.md`. - For GitHub REST task helpers, repository target rules, and metadata helpers, read `references/github-task-patterns.md`. - For SARIF-compatible `detect_` tasks and `sarif_findings` output, read `references/sarif-detection-tasks.md`. -- For YAML examples, dependencies, optional tasks, and validation commands, read `references/yaml-and-validation.md`. +- For YAML examples, invocation IDs, dependencies, dependency data, and validation commands, read `references/yaml-and-validation.md`. ## Review Behavior diff --git a/.agents/skills/anvil-task-builder/references/aws-task-patterns.md b/.agents/skills/anvil-task-builder/references/aws-task-patterns.md index bdf7053..7b0f1d6 100644 --- a/.agents/skills/anvil-task-builder/references/aws-task-patterns.md +++ b/.agents/skills/anvil-task-builder/references/aws-task-patterns.md @@ -42,6 +42,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: user_name = metadata.get("user_name") diff --git a/.agents/skills/anvil-task-builder/references/payer-management-account-tasks.md b/.agents/skills/anvil-task-builder/references/payer-management-account-tasks.md new file mode 100644 index 0000000..d8907fb --- /dev/null +++ b/.agents/skills/anvil-task-builder/references/payer-management-account-tasks.md @@ -0,0 +1,111 @@ +# Payer / Management Account-Only Tasks + +Use this reference when a workflow's target selection resolves to only the +payer (management) account — not when the payer/management account is one of +several targets in a broader multi-account run. If the same workflow also +targets other accounts, treat the payer account like any other target and +follow the general guidance in `task-granularity.md` instead. + +## Why This Case Is Different + +`task-granularity.md` frames concurrency pressure as: + +```text +max_parallel_targets * max_workers * max_parallel_regions +``` + +That formula assumes multiple accounts are being worked concurrently, which +is why the general advice is to raise `max_parallel_regions` cautiously and +benchmark first. When a run's target set is only the payer/management +account, `max_parallel_targets` is fixed at 1 for that run. There is no other +account's work competing for the same connection pool, worker threads, or API +quota, so a task (or the YAML orchestration around it) has meaningfully more +headroom to use threading, async, or higher region/worker concurrency than a +typical multi-account task would. + +This headroom is about contention with *other targets*, not about the target +service's own limits. AWS service throttling still applies per account and +region regardless of how many other accounts Anvil happens to be running +against in the same invocation. + +## Where This Actually Helps + +The benefit is largest for services that live only in the payer/management +account and require one API call per item with no batch equivalent: + +- IAM Identity Center (`sso-admin`, `identitystore`) — enumerating or acting + on permission sets, account assignments, or identity store users/groups. +- AWS Organizations — enumerating accounts, OUs, or policies attached across + many targets from the single management account. +- Control Tower or an Organizations-wide Config aggregator, when querying + many regions from the single aggregator account. + +It does not help generic single-account inventory tasks that already run +quickly, and it does not justify adding concurrency to a task that doesn't +need it. See `python-best-practices/concurrency-and-caching.md` for the +baseline rule that concurrency should only be added when it provides a +meaningful, measurable benefit. + +## Pattern: Bounded Fan-Out For No-Batch-API Calls + +Use this shape whenever a payer-only task needs to make one API call per item +and the service has no batch or single-paginated-call equivalent. It works +for both mutating fan-out (deleting or updating N items) and read-heavy +fan-out (describing N items after a list call): + +```python +from concurrent.futures import ThreadPoolExecutor, as_completed + +from anvil.task_errors import TaskExecutionError + +# No batch API for this call, so document why this number was chosen, +# e.g. an observed throttling point, or a conservative starting value. +MAX_WORKERS = 5 + + +def _process_item(client, item: dict[str, object]) -> dict[str, object]: + # One provider API call per item goes here. + ... + return item + + +def fan_out( + client, items: list[dict[str, object]] +) -> tuple[list[dict[str, object]], list[dict[str, object]]]: + succeeded: list[dict[str, object]] = [] + failed: list[dict[str, object]] = [] + + with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor: + future_to_item = { + executor.submit(_process_item, client, item): item for item in items + } + + for future in as_completed(future_to_item): + item = future_to_item[future] + try: + succeeded.append(future.result()) + except Exception as error: # narrow to the real provider exceptions + failed.append({**item, "error": str(error)}) + + return succeeded, failed +``` + +Raise `TaskExecutionError` with a `partial_result` carrying both `succeeded` +and `failed` when `failed` is non-empty, following the standard +partial-failure pattern instead of aborting the whole batch on the first +error. + +## Guardrails + +- Keep the worker count a named, documented module constant, never an + unbounded pool. +- Still respect the target service's own rate limits. "No other account is + competing" does not mean "no limits apply." +- Do not reach for this pattern just because a task is payer-only. Only use + it when the call shape (one call per item, no batch API, a meaningful item + count) actually benefits from concurrency. +- Preserve per-item error isolation: one failure should not abort the whole + batch unless the task's contract requires all-or-nothing behavior. +- If the workflow's target set later expands to include other accounts + alongside the payer account, re-evaluate. The headroom this doc describes + no longer applies once `max_parallel_targets` is greater than 1. diff --git a/.agents/skills/anvil-task-builder/references/runtime-contract.md b/.agents/skills/anvil-task-builder/references/runtime-contract.md index b672cb1..82427fd 100644 --- a/.agents/skills/anvil-task-builder/references/runtime-contract.md +++ b/.agents/skills/anvil-task-builder/references/runtime-contract.md @@ -56,6 +56,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict: ``` @@ -74,9 +75,11 @@ Runtime facts: - The provided `session` is already scoped to the provider target and region. - Tasks run once per concrete region by default. A task module may declare - `TASK_SCOPE = "target"` to run once per execution target instead. -- Providers declare which task scopes they support. AWS supports only the - default `region` scope; Azure, GCP, and GitHub support `region` and `target`. + `TASK_SCOPE = "target"` to run once per execution target or + `TASK_SCOPE = "configured_target"` to run once for the configured YAML target. +- Providers declare which task scopes they support. AWS supports + `configured_target` and `region`; Azure, GCP, and GitHub support `region` and + `target`. - A target-scoped task receives the first resolved concrete provider location as `region`, and its session uses that location. No synthetic target-scope or global sentinel is introduced. GitHub's `global` value is a real provider @@ -87,10 +90,12 @@ Runtime facts: itself, or ignores it because the API is target-wide. - For region-scoped tasks, `region` is the current task execution region. AWS sessions also expose `session.region_name`. -- Operator-provided task inputs come from `metadata`. -- Tasks should treat metadata as read-only configuration. - Anvil isolates changes to top-level metadata keys, - but not changes inside nested lists or dictionaries. +- Operator-provided static task inputs come from `metadata`. Target metadata is + recursively merged with task metadata, with task values taking precedence. +- Runtime dependency inputs come from `dependency_data`. They are selected from + direct dependency `TaskResult` objects and are never merged into `metadata`. +- Tasks should treat both mappings as read-only. Anvil deep-copies nested + mappings and lists for every invocation. - `actions` is an `ActionRecorder` for audit-level actions. - Returned values are included in Anvil result JSON. - The engine already includes execution context such as target identity, `region`, and `dry_run` in normal results. @@ -119,6 +124,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict: """Check the current provider target and return a simple status payload. @@ -132,6 +138,7 @@ def run( session: Provider runtime session scoped to the target and region. dry_run: Whether Anvil is running in dry-run mode. metadata: Task metadata from YAML. This task does not require metadata. + dependency_data: Runtime dependency inputs. This task requires none. actions: Action recorder provided by the Anvil engine. Returns: @@ -148,3 +155,30 @@ def run( ``` First-party provider tasks should use the provider-neutral signature above. + +## Invocation identity and dependencies + +YAML `name` selects the discovered component. YAML `id` identifies one +configured invocation and defaults to `name` when omitted. `depends_on` and +`dependency_data.task_id` always reference effective invocation IDs. Component +names are not a dependency fallback. + +The same component may be configured more than once only when every occurrence +has an explicit, unique ID. Results preserve both `task_id` and `task_name`. + +Normal tasks run only when every dependency succeeds. An `always_run` task waits +for every dependency to settle and then runs even after errors, interruption, or +skips, provided its dependency chain began. Successful finalization does not +erase an upstream failure. + +Use `TaskExecutionError` when a task must report failure while retaining +JSON-serializable recovery data: + +```python +from anvil.task_errors import TaskExecutionError + + +raise TaskExecutionError( + "Mutation partially failed", partial_result={"attachments": detached_attachments} +) +``` diff --git a/.agents/skills/anvil-task-builder/references/task-granularity.md b/.agents/skills/anvil-task-builder/references/task-granularity.md index 0f6f0e6..a41be9d 100644 --- a/.agents/skills/anvil-task-builder/references/task-granularity.md +++ b/.agents/skills/anvil-task-builder/references/task-granularity.md @@ -6,7 +6,7 @@ Prefer separate tasks when: - The tasks have different safety profiles, especially read-only vs mutating. - They are commonly run independently. -- They need different optional or fail-fast behavior. +- They need different failure, cleanup, or dependency behavior. - A dependency relationship is meaningful to the workflow. - Combining them would make the result shape confusing or too broad. @@ -36,4 +36,10 @@ For lightweight describe/list inventory across many accounts, especially multipl max_parallel_targets * max_workers * max_parallel_regions ``` -Recommend benchmarking the actual task mix before raising region concurrency. \ No newline at end of file +Recommend benchmarking the actual task mix before raising region concurrency. + +This caution assumes multiple accounts are in scope for the run. If the +workflow's target set is only the payer/management account, +`max_parallel_targets` is fixed at 1 and there is no cross-account +contention to protect against, so there is more headroom for concurrency. +See `references/payer-management-account-tasks.md`. diff --git a/.agents/skills/anvil-task-builder/references/yaml-and-validation.md b/.agents/skills/anvil-task-builder/references/yaml-and-validation.md index 93c856e..926c0ab 100644 --- a/.agents/skills/anvil-task-builder/references/yaml-and-validation.md +++ b/.agents/skills/anvil-task-builder/references/yaml-and-validation.md @@ -18,15 +18,37 @@ targets: dry_run: true tasks: - name: inventory_users - - name: remove_iam_user + - id: remove_example_user + name: remove_iam_user depends_on: - inventory_users - optional: false + dependency_data: + users: + task_id: inventory_users + path: result.users metadata: user_name: example-user ``` -Use `depends_on` when task order matters. Use `optional: true` only when failure should not fail the account or block dependent work. +`name` selects the discovered component. `id` identifies one configured use and +defaults to `name` when omitted. If a component is repeated, give every use an +explicit unique ID. + +Use effective IDs in `depends_on`; Anvil does not fall back to component names. +Normal dependents require every dependency to succeed. Use `always_run: true` +with at least one dependency for cleanup that should run after unsuccessful +work. + +`dependency_data` selects runtime values from tasks listed directly in +`depends_on`. Omit `path` for the complete producer `TaskResult`, use +`path: result` for its returned value, or select nested mapping values such as +`path: result.users`. Existing null values are valid; missing paths produce a +consumer task error before its `run()` function is called. + +Task scope is module-declared with `TASK_SCOPE`; do not put scope in YAML. + +For a complete configured-target fan-out/fan-in and recovery example, see +`examples/17-aws-config-cleanup-workflow.yaml`. ## Region Selection diff --git a/README.md b/README.md index a08897d..c003cc9 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,10 @@ targets: Task compatibility is determined by package location. +For schema-v2 invocation IDs, dependency-data selection, `always_run` recovery, +module-declared scopes, and configured-target fan-in/fan-out, see +[Task workflows](docs/task-workflows.md). + - `anvil.providers.tasks.` is universal and can run for any provider. - `anvil.providers.aws.tasks.` is AWS-only. - `anvil.providers.azure.tasks.` is Azure-only. @@ -207,7 +211,8 @@ For delegated-administrator patterns, keep the base session on the delegated-admin profile. Anvil uses that base session directly for the delegated-admin account if it appears in Organizations discovery, and assumes `role_name` in every other selected account, including the management/payer -account. +account. AWS organization targets accept `management` and `payer` as +case-insensitive aliases for that account in `include` and `exclude` filters. ```yaml schema_version: 2 @@ -222,6 +227,8 @@ targets: role_name: SecurityAuditRole regions: - us-east-1 + include: + - management tasks: - name: noop ``` diff --git a/examples/03-aws-include-exclude.yaml b/examples/03-aws-include-exclude.yaml index 0b85de6..f5ac85c 100644 --- a/examples/03-aws-include-exclude.yaml +++ b/examples/03-aws-include-exclude.yaml @@ -9,7 +9,7 @@ targets: regions: - us-east-1 include: - - '111111111111' + - management - '222222222222' dry_run: true tasks: @@ -24,7 +24,7 @@ targets: regions: - us-east-1 exclude: - - '999999999999' + - payer dry_run: true tasks: - name: count_vpc diff --git a/examples/04-aws-advanced.yaml b/examples/04-aws-advanced.yaml index f13a218..7dee989 100644 --- a/examples/04-aws-advanced.yaml +++ b/examples/04-aws-advanced.yaml @@ -23,11 +23,12 @@ targets: - python3.8 - nodejs16.x tasks: - - name: count_vpc - - name: list_lambdas_by_runtime - - name: detect_deprecated_lambda_runtimes - depends_on: - - list_lambdas_by_runtime + # Advanced examples use descriptive invocation IDs so result records remain + # clear even when a component is configured more than once elsewhere. + - id: vpc_count + name: count_vpc + - id: deprecated_runtime_findings + name: detect_deprecated_lambda_runtimes post_run: - processor: sarif_report output: aws-lambda-runtime-audit.sarif diff --git a/examples/08-azure-advanced.yaml b/examples/08-azure-advanced.yaml index 1e2b74c..a4b690b 100644 --- a/examples/08-azure-advanced.yaml +++ b/examples/08-azure-advanced.yaml @@ -19,7 +19,8 @@ targets: fail_fast: true dry_run: true tasks: - - name: count_resource_groups + - id: resource_group_count + name: count_resource_groups post_run: - processor: html_report output: azure-resource-group-inventory.html diff --git a/examples/12-gcp-advanced.yaml b/examples/12-gcp-advanced.yaml index a52ee62..d313d96 100644 --- a/examples/12-gcp-advanced.yaml +++ b/examples/12-gcp-advanced.yaml @@ -19,7 +19,8 @@ targets: fail_fast: true dry_run: true tasks: - - name: get_project_info + - id: project_inventory + name: get_project_info post_run: - processor: html_report output: gcp-project-inventory.html diff --git a/examples/16-github-advanced.yaml b/examples/16-github-advanced.yaml index 88c974d..d74a597 100644 --- a/examples/16-github-advanced.yaml +++ b/examples/16-github-advanced.yaml @@ -21,12 +21,18 @@ targets: max_results: 100 includes_parents: true tasks: - - name: list_code_scanning_alerts - - name: list_secret_scanning_alerts - - name: list_dependabot_alerts - - name: audit_branch_protection - - name: audit_rulesets - - name: audit_repo_security_settings + - id: code_scanning_alerts + name: list_code_scanning_alerts + - id: secret_scanning_alerts + name: list_secret_scanning_alerts + - id: dependabot_alerts + name: list_dependabot_alerts + - id: branch_protection_audit + name: audit_branch_protection + - id: ruleset_audit + name: audit_rulesets + - id: repository_security_audit + name: audit_repo_security_settings post_run: - processor: html_report output: github-security-audit.html diff --git a/examples/17-aws-config-cleanup-workflow.yaml b/examples/17-aws-config-cleanup-workflow.yaml new file mode 100644 index 0000000..536313f --- /dev/null +++ b/examples/17-aws-config-cleanup-workflow.yaml @@ -0,0 +1,77 @@ +schema_version: 2 +max_parallel_targets: 2 + +# These component names illustrate tasks supplied by an AWS task plugin. +# Their modules declare scope; scope is never configured in YAML: +# - snapshot_org_config and summarize_config_cleanup: configured_target +# - reconcile_config_guardrails and verify_config_cleanup: region +targets: +- name: organization-config-cleanup + provider: + name: aws + mode: organization + options: + profile: delegated-admin-security + role_name: OrganizationAccountAccessRole + regions: + - us-east-1 + - us-west-2 + max_workers: 4 + max_parallel_regions: 2 + dry_run: true + tasks: + # Omitted id defaults to the component name. + - name: snapshot_org_config + + # The same component is configured twice, so both uses have explicit IDs. + # The configured-target snapshot fans out to every selected account-region. + - id: detach_guardrails + name: reconcile_config_guardrails + depends_on: + - snapshot_org_config + metadata: + attachment_state: absent + dependency_data: + organization_snapshot: + task_id: snapshot_org_config + + - id: verify_cleanup + name: verify_config_cleanup + depends_on: + - detach_guardrails + dependency_data: + detach_result: + task_id: detach_guardrails + path: result + + # Cleanup runs after a success, error, interruption, or skip. A producer that + # raises TaskExecutionError can preserve result.attachments for restoration. + - id: restore_guardrails + name: reconcile_config_guardrails + depends_on: + - detach_guardrails + always_run: true + metadata: + attachment_state: present + dependency_data: + attachments: + task_id: detach_guardrails + path: result.attachments + + # Region results fan in to this configured-target summary in configured + # account and region order. + - name: summarize_config_cleanup + depends_on: + - verify_cleanup + - restore_guardrails + dependency_data: + verification_results: + task_id: verify_cleanup + path: result + restoration_task_results: + task_id: restore_guardrails + + post_run: + - processor: html_report + output: organization-config-cleanup.html + run_on_failure: true diff --git a/examples/Results/README.md b/examples/Results/README.md index 7a7ee80..d779ee2 100644 --- a/examples/Results/README.md +++ b/examples/Results/README.md @@ -34,8 +34,9 @@ Returned data is the native baseline. It is stored under each task result's `result` field and is useful for inventory, measurements, findings, IDs, counts, timing, and other structured task-specific data. -`ActionRecorder` is optional. It is useful when a task should record what it did, -or what it would do during dry-run mode, in a concise audit-friendly form. +The `actions` parameter is required by the runtime contract. Recording actions +is optional behavior: use it when a task should record what it did, or what it +would do during dry-run mode, in a concise audit-friendly form. ## Returned Results @@ -46,17 +47,23 @@ Return data directly from `run()` for small tasks: ```python import logging +from anvil.actions import ActionRecorder + __LOGGER__ = logging.getLogger(__name__) def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], - actions=None, + dependency_data: dict[str, object], + actions: ActionRecorder, ) -> dict[str, object]: user_name = str(metadata["user_name"]) iam = session.client("iam") @@ -67,7 +74,7 @@ def run( __LOGGER__.info( f"Inspected IAM groups for user {user_name} in account " - f"{account_alias} ({account_id}), dry_run={dry_run}" + f"{execution_target_name} ({execution_target_id}), dry_run={dry_run}" ) return { @@ -84,6 +91,8 @@ For larger tasks, helper functions can build result data and return it to ```python import logging +from anvil.actions import ActionRecorder + __LOGGER__ = logging.getLogger(__name__) @@ -137,12 +146,16 @@ def cleanup_user_resources( def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], - actions=None, + dependency_data: dict[str, object], + actions: ActionRecorder, ) -> dict[str, object]: user_name = metadata.get("user_name") if not isinstance(user_name, str): @@ -156,7 +169,8 @@ The returned value appears in the task result: ```json { - "task": "function_returned_results", + "task_id": "function_returned_results", + "task_name": "function_returned_results", "region": "us-east-1", "status": "success", "started_at": "2026-05-01T18:30:12+00:00", @@ -191,7 +205,7 @@ The returned value appears in the task result: Returned data is also available in the flattened JSONL query artifact: ```console -anvil results --type task --task function_returned_results --fields target,entity_id,region,status,result --json +anvil results --type task --task function_returned_results --fields target,entity_id,region,task_id,task_name,status,result --json ``` ## Recorded Actions @@ -208,11 +222,15 @@ from anvil.actions import ActionRecorder def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: if dry_run: @@ -247,11 +265,15 @@ def cleanup_user(iam, user_name: str, dry_run: bool, actions: ActionRecorder) -> def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: iam = session.client("iam") diff --git a/examples/Results/basic_action_recorder.py b/examples/Results/basic_action_recorder.py index d1989ef..f52d576 100644 --- a/examples/Results/basic_action_recorder.py +++ b/examples/Results/basic_action_recorder.py @@ -7,13 +7,32 @@ def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: + """Check one IAM user and record the outcome as an audit action. + + Args: + provider: Provider name for the execution target. + execution_target_id: Provider-owned target identifier. + execution_target_name: Target display name. + execution_target_type: Provider-owned target type. + region: Concrete execution region. + session: AWS session scoped to the target and region. + dry_run: Whether mutations must be simulated. + metadata: Static task configuration with an optional `user_name`. + dependency_data: Runtime dependency inputs; unused by this task. + actions: Engine-provided action recorder. + """ + iam = session.client("iam") user_name = metadata.get("user_name", "example") @@ -27,6 +46,6 @@ def run( iam.get_user(UserName=user_name) __LOGGER__.info(f"IAM user exists: {user_name}") actions.record(f"IAM user exists: {user_name}") - except Exception: + except iam.exceptions.NoSuchEntityException: __LOGGER__.info(f"IAM user not found: {user_name}") actions.record(f"IAM user not found: {user_name}") diff --git a/examples/Results/basic_returned_results.py b/examples/Results/basic_returned_results.py index 0711236..8263afb 100644 --- a/examples/Results/basic_returned_results.py +++ b/examples/Results/basic_returned_results.py @@ -6,20 +6,43 @@ import logging +from anvil.actions import ActionRecorder + __LOGGER__ = logging.getLogger(__name__) def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], - actions=None, + dependency_data: dict[str, object], + actions: ActionRecorder, ) -> dict[str, object]: - """ - Return JSON-serializable task data for normal Anvil result output. + """Return JSON-serializable task data for normal Anvil result output. + + Args: + provider: Provider name for the execution target. + execution_target_id: Provider-owned target identifier. + execution_target_name: Target display name. + execution_target_type: Provider-owned target type. + region: Concrete execution region. + session: AWS session scoped to the target and region. + dry_run: Whether mutations must be simulated. + metadata: Static task configuration requiring `user_name`. + dependency_data: Runtime dependency inputs; unused by this task. + actions: Engine-provided action recorder. + + Returns: + IAM group and access-key inventory for the configured user. + + Raises: + RuntimeError: If `metadata.user_name` is not a string. """ user_name = metadata.get("user_name") if not isinstance(user_name, str): @@ -37,7 +60,7 @@ def run( __LOGGER__.info( f"Inspected IAM resources for user {user_name} in account " - f"{account_alias} ({account_id}), dry_run={dry_run}" + f"{execution_target_name} ({execution_target_id}), dry_run={dry_run}" ) return { diff --git a/examples/Results/function_recording.py b/examples/Results/function_recording.py index dd4fc96..6ba3e7b 100644 --- a/examples/Results/function_recording.py +++ b/examples/Results/function_recording.py @@ -37,13 +37,35 @@ def cleanup_user_resources( def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: + """Clean up one IAM user's resources and record each action. + + Args: + provider: Provider name for the execution target. + execution_target_id: Provider-owned target identifier. + execution_target_name: Target display name. + execution_target_type: Provider-owned target type. + region: Concrete execution region. + session: AWS session scoped to the target and region. + dry_run: Whether mutations must be simulated. + metadata: Static task configuration requiring `user_name`. + dependency_data: Runtime dependency inputs; unused by this task. + actions: Engine-provided action recorder. + + Raises: + RuntimeError: If `metadata.user_name` is not a string. + """ + user_name = metadata.get("user_name") if not isinstance(user_name, str): raise RuntimeError("example_cleanup requires metadata.user_name to be a string") diff --git a/examples/Results/function_returned_results.py b/examples/Results/function_returned_results.py index 550ab30..4fd8653 100644 --- a/examples/Results/function_returned_results.py +++ b/examples/Results/function_returned_results.py @@ -6,6 +6,8 @@ import logging +from anvil.actions import ActionRecorder + __LOGGER__ = logging.getLogger(__name__) @@ -64,15 +66,36 @@ def cleanup_user_resources( def run( *, - account_id: str, - account_alias: str, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, session, dry_run: bool, metadata: dict[str, object], - actions=None, + dependency_data: dict[str, object], + actions: ActionRecorder, ) -> dict[str, object]: - """ - Return JSON-serializable task data for normal Anvil result output. + """Return JSON-serializable task data for normal Anvil result output. + + Args: + provider: Provider name for the execution target. + execution_target_id: Provider-owned target identifier. + execution_target_name: Target display name. + execution_target_type: Provider-owned target type. + region: Concrete execution region. + session: AWS session scoped to the target and region. + dry_run: Whether mutations must be simulated. + metadata: Static task configuration requiring `user_name`. + dependency_data: Runtime dependency inputs; unused by this task. + actions: Engine-provided action recorder. + + Returns: + Planned or completed IAM cleanup details. + + Raises: + RuntimeError: If `metadata.user_name` is not a string. """ user_name = metadata.get("user_name") if not isinstance(user_name, str): diff --git a/examples/invalid/aws-configured-target-multiple-accounts.yaml b/examples/invalid/aws-configured-target-multiple-accounts.yaml new file mode 100644 index 0000000..d5f4106 --- /dev/null +++ b/examples/invalid/aws-configured-target-multiple-accounts.yaml @@ -0,0 +1,20 @@ +schema_version: 2 + +# Intentionally invalid at offline task-configuration validation time. +# An AWS accounts target with configured-target tasks must select exactly one +# explicit account; Anvil rejects this before authentication or preflight. +targets: +- name: invalid-multi-account-config-cleanup + provider: + name: aws + mode: accounts + options: + profile: security-audit + role_name: OrganizationAccountAccessRole + include: + - '111111111111' + - '222222222222' + regions: + - us-east-1 + tasks: + - name: snapshot_org_config diff --git a/pyproject.toml b/pyproject.toml index 5221592..ab35fb1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -73,10 +73,10 @@ anvil = "anvil.cli:main" dev = [ "pytest-cov>=7.1.0,<8.0.0", "pytest>=9.1.1,<10.0.0", - "prek>=0.4.5,<0.5.0", + "prek>=0.4.12,<0.5.0", "check-jsonschema>=0.37.3,<0.40.0", - "ruff>=0.15.19,<0.20.0", - "ty>=0.0.53,<0.20.0", + "ruff>=0.16.0,<0.20.0", + "ty>=0.0.60,<0.20.0", ] [tool.pytest.ini_options] diff --git a/src/anvil/execution_context.py b/src/anvil/execution_context.py index 00b7c45..c51884a 100644 --- a/src/anvil/execution_context.py +++ b/src/anvil/execution_context.py @@ -22,3 +22,4 @@ class ExecutionContext: benchmark_enabled: bool = False log_level: str | int | None = None cancel_event: threading.Event = field(default_factory=threading.Event) + fail_fast_event: threading.Event = field(default_factory=threading.Event) diff --git a/src/anvil/processors/html_report.py b/src/anvil/processors/html_report.py index 3292209..ab5ff07 100644 --- a/src/anvil/processors/html_report.py +++ b/src/anvil/processors/html_report.py @@ -106,6 +106,20 @@ def _records_from_target_dict( return [] records: list[dict[str, object]] = [] + configured_entity_record = { + "target_type": "target", + "target": target_name, + "generated_at": target_result.get("generated_at"), + "dry_run": target_result.get("dry_run"), + "entity_id": None, + "entity_name": None, + "entity_type": "configured_target", + } + records.extend( + _task_records( + tasks=target_result.get("tasks", []), entity_record=configured_entity_record + ) + ) for entity_result in entities: if not isinstance(entity_result, dict): continue @@ -129,26 +143,39 @@ def _records_from_target_dict( } ) - tasks = entity_result.get("tasks", []) - if not isinstance(tasks, list): - continue - - for task_result in tasks: - if not isinstance(task_result, dict): - continue - task_result = cast(dict[str, object], task_result) - records.append( - { - **entity_record, - "record_type": "task", - "task": task_result.get("task"), - "region": task_result.get("region"), - **_timed_status_record(task_result), - "result": task_result.get("result"), - "error": task_result.get("error"), - } + records.extend( + _task_records( + tasks=entity_result.get("tasks", []), entity_record=entity_record ) + ) + + return records + +def _task_records( + *, tasks: object, entity_record: dict[str, object] +) -> list[dict[str, object]]: + if not isinstance(tasks, list): + return [] + + records: list[dict[str, object]] = [] + for task_result in tasks: + if not isinstance(task_result, dict): + continue + task_result = cast(dict[str, object], task_result) + records.append( + { + **entity_record, + "record_type": "task", + "task_id": task_result.get("task_id"), + "task_name": task_result.get("task_name"), + "region": task_result.get("region"), + **_timed_status_record(task_result), + "result": task_result.get("result"), + "error": task_result.get("error"), + "skip_reason": task_result.get("skip_reason"), + } + ) return records @@ -403,7 +430,8 @@ def _build_html( - + +
@@ -436,10 +464,11 @@ def _build_html( target: document.getElementById("targetFilter"), entity: document.getElementById("entityFilter"), region: document.getElementById("regionFilter"), - task: document.getElementById("taskFilter"), + task_id: document.getElementById("taskIdFilter"), + task_name: document.getElementById("taskNameFilter"), search: document.getElementById("searchFilter") }}; - const fields = ["status", "record_type", "target", "entity", "region", "task"]; + const fields = ["status", "record_type", "target", "entity", "region", "task_id", "task_name"]; function value(record, field) {{ if (field === "entity") {{ @@ -500,7 +529,8 @@ def _build_html( record.entity_name, record.entity_type, record.region, - record.task, + record.task_id, + record.task_name, record.error ].some((item) => String(item || "").toLowerCase().includes(query)); }} @@ -553,7 +583,7 @@ def _build_html( cells[2].textContent = record.target || ""; cells[3].textContent = [record.entity_name, record.entity_id].filter(Boolean).join(" "); cells[4].textContent = record.region || ""; - cells[5].textContent = record.task || ""; + cells[5].textContent = [record.task_id, record.task_name].filter(Boolean).join(" / "); cells[6].textContent = record.duration_seconds === undefined || record.duration_seconds === null ? "" : `${{record.duration_seconds}}s`; @@ -570,7 +600,8 @@ def _build_html( populateSelect(controls.target, sortedValues("target"), "targets"); populateSelect(controls.entity, sortedValues("entity"), "entities"); populateSelect(controls.region, sortedValues("region"), "regions"); - populateSelect(controls.task, sortedValues("task"), "tasks"); + populateSelect(controls.task_id, sortedValues("task_id"), "task IDs"); + populateSelect(controls.task_name, sortedValues("task_name"), "task names"); renderCards(data.cards); Object.values(controls).forEach((control) => control.addEventListener("input", renderTable)); renderTable(); @@ -585,6 +616,7 @@ def _summary_cards(records: list[dict[str, object]]) -> list[dict[str, object]]: error_count = _count_status(records=records, status="error") interrupted_count = _count_status(records=records, status="interrupted") unsuccessful_count = sum(1 for record in records if _is_unsuccessful(record)) + skipped_count = _count_status(records=records, status="skipped") failed_entities = sum( 1 for record in records @@ -612,6 +644,7 @@ def _summary_cards(records: list[dict[str, object]]) -> list[dict[str, object]]: "tone": "interrupted", "mark": "INT", }, + {"label": "Skipped", "value": skipped_count, "mark": "SKIP"}, { "label": "Failed entities", "value": failed_entities, @@ -633,7 +666,7 @@ def _count_status(*, records: list[dict[str, object]], status: str) -> int: def _is_unsuccessful(record: dict[str, object]) -> bool: status = record.get("status") - return isinstance(status, str) and status.lower() != "success" + return isinstance(status, str) and status.lower() in {"error", "interrupted"} def _json_for_script(value: object) -> str: diff --git a/src/anvil/processors/sarif_report.py b/src/anvil/processors/sarif_report.py index e58c04e..66f1840 100644 --- a/src/anvil/processors/sarif_report.py +++ b/src/anvil/processors/sarif_report.py @@ -67,40 +67,24 @@ def _collect_sarif_results( rules: dict[str, dict[str, object]] = {} for target_result in _target_result_dicts(context=context): + configured_entity: dict[str, object] = {"type": "configured_target"} + for task_result in _target_task_results(target_result=target_result): + _collect_task_findings( + sarif_results=sarif_results, + rules=rules, + target_result=target_result, + entity_result=configured_entity, + task_result=task_result, + ) for entity_result in _entity_results(target_result=target_result): for task_result in _task_results(entity_result=entity_result): - result = task_result.get("result") - if not isinstance(result, dict) or "sarif_findings" not in result: - continue - - raw_findings = result.get("sarif_findings") - if not isinstance(raw_findings, list): - raise RuntimeError( - "sarif_report requires result.sarif_findings to be a list" - ) - - for raw_finding in raw_findings: - if not isinstance(raw_finding, dict): - raise RuntimeError( - "sarif_report requires every sarif_findings entry " - "to be a mapping" - ) - raw_finding = cast(dict[str, object], raw_finding) - sarif_result, rule = _convert_finding( - finding=raw_finding, - target_result=target_result, - entity_result=entity_result, - task_result=task_result, - ) - rule_id = _required_string(rule, "id", "finding.rule") - existing_rule = rules.get(rule_id) - if existing_rule is not None and existing_rule != rule: - raise RuntimeError( - f"sarif_report found conflicting metadata for rule " - f"{rule_id!r}" - ) - rules[rule_id] = rule - sarif_results.append(sarif_result) + _collect_task_findings( + sarif_results=sarif_results, + rules=rules, + target_result=target_result, + entity_result=entity_result, + task_result=task_result, + ) return sarif_results, rules @@ -143,6 +127,50 @@ def _task_results(*, entity_result: dict[str, object]) -> list[dict[str, object] ] +def _target_task_results( + *, target_result: dict[str, object] +) -> list[dict[str, object]]: + return _task_results(entity_result=target_result) + + +def _collect_task_findings( + *, + sarif_results: list[dict[str, object]], + rules: dict[str, dict[str, object]], + target_result: dict[str, object], + entity_result: dict[str, object], + task_result: dict[str, object], +) -> None: + result = task_result.get("result") + if not isinstance(result, dict) or "sarif_findings" not in result: + return + + raw_findings = result.get("sarif_findings") + if not isinstance(raw_findings, list): + raise RuntimeError("sarif_report requires result.sarif_findings to be a list") + + for raw_finding in raw_findings: + if not isinstance(raw_finding, dict): + raise RuntimeError( + "sarif_report requires every sarif_findings entry to be a mapping" + ) + raw_finding = cast(dict[str, object], raw_finding) + sarif_result, rule = _convert_finding( + finding=raw_finding, + target_result=target_result, + entity_result=entity_result, + task_result=task_result, + ) + rule_id = _required_string(rule, "id", "finding.rule") + existing_rule = rules.get(rule_id) + if existing_rule is not None and existing_rule != rule: + raise RuntimeError( + f"sarif_report found conflicting metadata for rule {rule_id!r}" + ) + rules[rule_id] = rule + sarif_results.append(sarif_result) + + def _convert_finding( *, finding: dict[str, object], @@ -278,18 +306,24 @@ def _result_properties( ) -> dict[str, object]: target_key = "target" - properties: dict[str, object] = { - "target_type": target_key, - "target": target_result.get(target_key) or target_result.get("target"), - "entity_id": entity_result.get("id"), - "entity_name": entity_result.get("name"), - "entity_type": entity_result.get("type"), - "region": task_result.get("region"), - "task": task_result.get("task"), - } raw_properties = finding.get("properties") - if isinstance(raw_properties, dict): - properties.update(cast(dict[str, object], raw_properties)) + properties: dict[str, object] = ( + dict(cast(dict[str, object], raw_properties)) + if isinstance(raw_properties, dict) + else {} + ) + properties.update( + { + "target_type": target_key, + "target": target_result.get(target_key) or target_result.get("target"), + "entity_id": entity_result.get("id"), + "entity_name": entity_result.get("name"), + "entity_type": entity_result.get("type"), + "region": task_result.get("region"), + "task_id": task_result.get("task_id"), + "task_name": task_result.get("task_name"), + } + ) return {key: value for key, value in properties.items() if value is not None} diff --git a/src/anvil/provider_lifecycle.py b/src/anvil/provider_lifecycle.py new file mode 100644 index 0000000..53144ed --- /dev/null +++ b/src/anvil/provider_lifecycle.py @@ -0,0 +1,53 @@ +"""Constant-size provider lifecycle state for task graphs.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from anvil.results import TaskResult + + +@dataclass(slots=True) +class CoordinateLifecycleState: + """Constant-size settlement and timing state for one runtime coordinate.""" + + remaining_instances: int = 0 + failed: bool = False + interrupted: bool = False + started_perf: float | None = None + region_started_perf: float | None = None + region_ended_perf: float | None = None + + def record_settlement( + self, *, result: TaskResult, region_scoped: bool, ended_perf: float + ) -> bool: + """Record one terminal result without retaining the result object. + + Args: + result: Terminal result for one task instance. + region_scoped: Whether the instance contributes to regional timing. + ended_perf: Monotonic settlement timestamp. + + Returns: + Whether every task instance for this coordinate has settled. + + Raises: + RuntimeError: If more results settle than were planned. + """ + + if self.remaining_instances <= 0: + raise RuntimeError( + "Provider coordinate settled more task instances than planned" + ) + self.failed = self.failed or result.status.is_error + self.interrupted = self.interrupted or ( + result.status.is_interrupted + or ( + result.status.is_skipped + and result.skip_reason == "cancelled_before_start" + ) + ) + self.remaining_instances -= 1 + if region_scoped: + self.region_ended_perf = ended_perf + return self.remaining_instances == 0 diff --git a/src/anvil/providers/aws/organization.py b/src/anvil/providers/aws/organization.py index a840774..b5f44b7 100644 --- a/src/anvil/providers/aws/organization.py +++ b/src/anvil/providers/aws/organization.py @@ -14,6 +14,14 @@ __LOGGER__ = logging.getLogger(__name__) +MANAGEMENT_ACCOUNT_KEYWORDS = frozenset({"management", "payer"}) + + +def is_management_account_keyword(value: str) -> bool: + """Return whether an AWS account filter names the management account.""" + + return value.casefold() in MANAGEMENT_ACCOUNT_KEYWORDS + class OrganizationResolver: """ @@ -112,7 +120,9 @@ def _build_accounts( Build executable account objects for all selected target accounts. """ all_accounts = discovered_accounts or self.discover_accounts(base_session) - target_accounts = self._filter_accounts(all_accounts) + target_accounts = self._filter_accounts( + all_accounts, management_account_id=management_account_id + ) accounts: list[Account] = [] @@ -167,7 +177,7 @@ def discover_accounts(session: boto3.Session) -> dict[str, dict[str, str]]: return accounts def _filter_accounts( - self, all_accounts: dict[str, dict[str, str]] + self, all_accounts: dict[str, dict[str, str]], *, management_account_id: str ) -> dict[str, dict[str, str]]: """ Apply include/exclude account filters to discovered organization accounts. @@ -175,7 +185,10 @@ def _filter_accounts( discovered_ids = set(all_accounts.keys()) if self.descriptor.include: - include_set = set(self.descriptor.include) + include_set = self._resolve_account_filter_keywords( + values=self.descriptor.include, + management_account_id=management_account_id, + ) unknown_include_ids = sorted(include_set - discovered_ids) if unknown_include_ids: __LOGGER__.warning( @@ -186,7 +199,10 @@ def _filter_accounts( selected_ids = sorted(include_set & discovered_ids) return {account_id: all_accounts[account_id] for account_id in selected_ids} - exclude_set = set(self.descriptor.exclude or []) + exclude_set = self._resolve_account_filter_keywords( + values=self.descriptor.exclude or [], + management_account_id=management_account_id, + ) unknown_exclude_ids = sorted(exclude_set - discovered_ids) if unknown_exclude_ids: __LOGGER__.warning( @@ -197,6 +213,17 @@ def _filter_accounts( remaining_ids = sorted(discovered_ids - exclude_set) return {account_id: all_accounts[account_id] for account_id in remaining_ids} + @staticmethod + def _resolve_account_filter_keywords( + *, values: list[str], management_account_id: str + ) -> set[str]: + """Expand AWS-owned account filter keywords to concrete account IDs.""" + + return { + management_account_id if is_management_account_keyword(value) else value + for value in values + } + def _get_effective_regions( self, session: boto3.Session, *, region_statuses: dict[str, str] | None = None ) -> list[str]: diff --git a/src/anvil/providers/aws/provider.py b/src/anvil/providers/aws/provider.py index f6ab95b..1f9c2ee 100644 --- a/src/anvil/providers/aws/provider.py +++ b/src/anvil/providers/aws/provider.py @@ -14,8 +14,11 @@ ) from anvil.providers.aws.account_resolver import AccountResolver from anvil.providers.aws.auth import auth_check, infer_auth_source -from anvil.providers.aws.config import aws_option -from anvil.providers.aws.organization import OrganizationResolver +from anvil.providers.aws.config import DEFAULT_ORGANIZATION_ROLE_NAME, aws_option +from anvil.providers.aws.organization import ( + OrganizationResolver, + is_management_account_keyword, +) from anvil.providers.base import ( ExecutionTarget, ProviderAuthResult, @@ -152,7 +155,7 @@ class AwsProvider: display_name="AWS", description="Amazon Web Services provider", default_regions=DEFAULT_REGIONS, - supported_task_scopes=frozenset({"region"}), + supported_task_scopes=frozenset({"configured_target", "region"}), ) def __init__(self, *, region_service: AwsRegionService | None = None) -> None: @@ -171,6 +174,13 @@ def validate_target(self, target: TargetDescriptor) -> None: if target.include is not None and target.exclude is not None: raise ValueError("AWS include and exclude filters are mutually exclusive") for account_id in [*(target.include or []), *(target.exclude or [])]: + if is_management_account_keyword(account_id): + if target.mode != MODE_ORGANIZATION: + raise ValueError( + f"AWS account filter keyword '{account_id}' requires " + "organization mode" + ) + continue if len(account_id) != 12 or not account_id.isdigit(): raise ValueError(f"Invalid AWS account ID: {account_id}") if target.mode == MODE_ACCOUNTS: @@ -212,6 +222,28 @@ def resolve_target_filters( self.validate_target(effective_target) return include, exclude + def validate_task_configuration( + self, *, target: TargetDescriptor, task_scopes: dict[str, str] + ) -> None: + """Validate AWS configured-target ownership before authentication.""" + + configured_task_ids = [ + task_id + for task_id, scope in task_scopes.items() + if scope == "configured_target" + ] + if not configured_task_ids or target.mode == MODE_ORGANIZATION: + return + + account_ids = target.include or [] + if len(account_ids) != 1: + task_display = ", ".join(configured_task_ids) + raise ValueError( + f"AWS target '{target.name}' has ambiguous configured-target " + f"identity for task(s) {task_display}: accounts mode requires " + "exactly one explicit account" + ) + def bootstrap_region(self, *, configured_regions: list[str]) -> str: """Return the concrete AWS region used for discovery calls.""" @@ -341,7 +373,22 @@ def resolve_execution_targets( for account in accounts ] - return ProviderExecutionPlan(execution_targets=execution_targets) + configured_target: ExecutionTarget | None = None + if effective_target.mode == MODE_ORGANIZATION and preflight_data is not None: + configured_target = self._organization_configured_target( + target=effective_target, + context=context, + preflight_data=preflight_data, + effective_regions=( + list(execution_targets[0].regions) if execution_targets else None + ), + ) + elif effective_target.mode == MODE_ACCOUNTS and len(execution_targets) == 1: + configured_target = replace(execution_targets[0], type="configured_target") + + return ProviderExecutionPlan( + execution_targets=execution_targets, configured_target=configured_target + ) def prepare_target( self, @@ -438,6 +485,78 @@ def prepare_execution_runtime( ) return AwsExecutionRuntime(account=account) + def prepare_configured_target_runtime( + self, + *, + target: TargetDescriptor, + execution_target: ExecutionTarget, + context: ExecutionContext, + ) -> ProviderExecutionRuntime: + """Prepare an AWS runtime for the provider-owned configured identity.""" + + if execution_target.type != "configured_target": + raise ValueError( + "AWS configured-target runtime requires execution target type " + "'configured_target'" + ) + return self.prepare_execution_runtime( + target=target, execution_target=execution_target, context=context + ) + + def _organization_configured_target( + self, + *, + target: TargetDescriptor, + context: ExecutionContext, + preflight_data: AwsPreflightData, + effective_regions: list[str] | None, + ) -> ExecutionTarget: + """Build the management-account identity independently of entity filters.""" + + management_info = preflight_data.discovered_accounts.get( + preflight_data.management_account_id + ) + if management_info is None: + raise ValueError( + f"AWS organization target '{target.name}' management account " + f"'{preflight_data.management_account_id}' was not present in " + "discovered organization accounts" + ) + + resolved_regions = effective_regions or self.resolve_regions( + target_name=target.name, + configured_regions=context.regions, + region_statuses=preflight_data.region_statuses, + ) + if not resolved_regions: + raise ValueError("No effective configured regions remain after validation.") + + access_strategy = ( + AccountAccessStrategy.BASE_SESSION + if preflight_data.base_session_account_id + == preflight_data.management_account_id + else AccountAccessStrategy.ASSUME_ROLE + ) + management_account = Account( + account_id=preflight_data.management_account_id, + account_alias=management_info["account_alias"], + is_management=True, + access_strategy=access_strategy, + role_name=( + aws_option(target, "role_name") or DEFAULT_ORGANIZATION_ROLE_NAME + ), + base_session=preflight_data.base_session, + context=context, + regions=resolved_regions, + session_factory=preflight_data.session_factory, + ) + return replace( + _execution_target_from_account( + account=management_account, provider_name=self.metadata.name + ), + type="configured_target", + ) + def _account_from_execution_target( self, *, execution_target: ExecutionTarget, context: ExecutionContext ) -> Account: diff --git a/src/anvil/providers/aws/tasks/compare_asg_to_cluster_instances.py b/src/anvil/providers/aws/tasks/compare_asg_to_cluster_instances.py index bf2cae1..7a8f767 100644 --- a/src/anvil/providers/aws/tasks/compare_asg_to_cluster_instances.py +++ b/src/anvil/providers/aws/tasks/compare_asg_to_cluster_instances.py @@ -37,6 +37,7 @@ def run( session: boto3.Session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: """Compare ECS container instances to corresponding Auto Scaling Groups. @@ -60,6 +61,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Task metadata containing cluster configuration. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Raises: @@ -106,25 +108,27 @@ def run( for instance in auto_scaling_group["Instances"] } - list_container_instances = ecs_client.list_container_instances(cluster=cluster) + container_instances: list[dict[str, object]] = [] + paginator = ecs_client.get_paginator("list_container_instances") + for page in paginator.paginate(cluster=cluster): + container_instance_arns = page.get("containerInstanceArns", []) + if not container_instance_arns: + continue - if list_container_instances["containerInstanceArns"]: __LOGGER__.debug( f"Gathering container instance information for cluster '{cluster}'" ) - - describe_container_instances = ecs_client.describe_container_instances( - cluster=cluster, - containerInstances=list_container_instances["containerInstanceArns"], + response = ecs_client.describe_container_instances( + cluster=cluster, containerInstances=container_instance_arns ) + container_instances.extend(response.get("containerInstances", [])) + if container_instances: __LOGGER__.debug(f"Gathering ec2InstanceIds for cluster '{cluster}'") ecs_instance_ids = { container_instance["ec2InstanceId"] - for container_instance in describe_container_instances[ - "containerInstances" - ] + for container_instance in container_instances } __LOGGER__.debug( @@ -134,9 +138,7 @@ def run( instances_with_zero_tasks: list[str] = [] - for container_instance in describe_container_instances[ - "containerInstances" - ]: + for container_instance in container_instances: if container_instance["runningTasksCount"] == 0: instances_with_zero_tasks.append( container_instance["ec2InstanceId"] diff --git a/src/anvil/providers/aws/tasks/count_subnets_with_timings.py b/src/anvil/providers/aws/tasks/count_subnets_with_timings.py index f519e50..e533d0f 100644 --- a/src/anvil/providers/aws/tasks/count_subnets_with_timings.py +++ b/src/anvil/providers/aws/tasks/count_subnets_with_timings.py @@ -70,6 +70,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Count subnets in the session's current AWS region with timing data. @@ -86,6 +87,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/aws/tasks/count_vpc.py b/src/anvil/providers/aws/tasks/count_vpc.py index 85a7897..a1853b4 100644 --- a/src/anvil/providers/aws/tasks/count_vpc.py +++ b/src/anvil/providers/aws/tasks/count_vpc.py @@ -19,6 +19,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict: """Count VPCs in the session's current AWS region. @@ -35,6 +36,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/aws/tasks/detect_deprecated_lambda_runtimes.py b/src/anvil/providers/aws/tasks/detect_deprecated_lambda_runtimes.py index bc8222c..7561223 100644 --- a/src/anvil/providers/aws/tasks/detect_deprecated_lambda_runtimes.py +++ b/src/anvil/providers/aws/tasks/detect_deprecated_lambda_runtimes.py @@ -142,6 +142,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Detect Lambda functions using configured deprecated runtimes. @@ -163,6 +164,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Task metadata containing deprecated runtime filters. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/aws/tasks/get_aws_inline_policies.py b/src/anvil/providers/aws/tasks/get_aws_inline_policies.py index 8c950d6..7f81496 100644 --- a/src/anvil/providers/aws/tasks/get_aws_inline_policies.py +++ b/src/anvil/providers/aws/tasks/get_aws_inline_policies.py @@ -199,6 +199,7 @@ def run( session: boto3.Session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Gather AWS inline policies for IAM identities and Identity Center. @@ -221,6 +222,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Task metadata containing optional policy type filters. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/aws/tasks/get_organization_structure.py b/src/anvil/providers/aws/tasks/get_organization_structure.py index f1fb857..e41cc8b 100644 --- a/src/anvil/providers/aws/tasks/get_organization_structure.py +++ b/src/anvil/providers/aws/tasks/get_organization_structure.py @@ -98,6 +98,7 @@ def run( session: boto3.Session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Gather AWS Organizations structure from the management account. @@ -116,6 +117,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/aws/tasks/list_lambdas_by_runtime.py b/src/anvil/providers/aws/tasks/list_lambdas_by_runtime.py index d241d9a..6ed953b 100644 --- a/src/anvil/providers/aws/tasks/list_lambdas_by_runtime.py +++ b/src/anvil/providers/aws/tasks/list_lambdas_by_runtime.py @@ -68,6 +68,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """List Lambda functions using any configured runtime. @@ -88,6 +89,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Task metadata containing runtime filters. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/aws/tasks/remove_disabled_idc_users.py b/src/anvil/providers/aws/tasks/remove_disabled_idc_users.py new file mode 100644 index 0000000..b3541a0 --- /dev/null +++ b/src/anvil/providers/aws/tasks/remove_disabled_idc_users.py @@ -0,0 +1,282 @@ +from __future__ import annotations + +import logging +from concurrent.futures import ThreadPoolExecutor, as_completed + +import boto3 +from botocore.config import Config +from botocore.exceptions import BotoCoreError, ClientError + +from anvil.actions import ActionRecorder +from anvil.task_errors import TaskExecutionError + +__LOGGER__ = logging.getLogger(__name__) + +BOTO_CONFIG = Config(max_pool_connections=40) + +# DeleteUser has no batch API, so removals happen one call per user. +MAX_DELETE_WORKERS = 3 + + +def _get_active_sso_instance(sso_admin_client) -> tuple[str, str, str]: + response = sso_admin_client.list_instances() + instances = response.get("Instances", []) + + active_instance = next( + (instance for instance in instances if instance.get("Status") == "ACTIVE"), None + ) + + if not active_instance: + raise RuntimeError("No active IAM Identity Center (SSO) instance found") + + return ( + active_instance["InstanceArn"], + active_instance["IdentityStoreId"], + active_instance["OwnerAccountId"], + ) + + +def _list_disabled_users( + identitystore_client, identity_store_id: str +) -> list[dict[str, object]]: + disabled_users: list[dict[str, object]] = [] + + paginator = identitystore_client.get_paginator("list_users") + + for page in paginator.paginate(IdentityStoreId=identity_store_id): + for user in page.get("Users", []): + if user.get("UserStatus") != "DISABLED": + continue + + user_id = user["UserId"] + + emails = [ + email.get("Value") + for email in user.get("Emails", []) + if email.get("Value") + ] + + disabled_users.append( + { + "UserId": user_id, + "UserName": user.get("UserName"), + "Emails": emails, + "UserType": user.get("UserType"), + "UserStatus": user.get("UserStatus"), + } + ) + + __LOGGER__.debug(f"Disabled user found: {user.get('UserName')} ({user_id})") + + return disabled_users + + +def _resolve_user_id_filters( + metadata: dict[str, object], +) -> tuple[set[str] | None, set[str] | None]: + include_raw = metadata.get("include_user_ids") + exclude_raw = metadata.get("exclude_user_ids") + + if include_raw is not None and exclude_raw is not None: + raise RuntimeError( + "remove_disabled_idc_users requires only one of " + "metadata.include_user_ids or metadata.exclude_user_ids to be set" + ) + + def _as_id_set(raw: object, key: str) -> set[str] | None: + if raw is None: + return None + + if not isinstance(raw, list) or not all(isinstance(item, str) for item in raw): + raise RuntimeError( + f"remove_disabled_idc_users requires metadata.{key} to be a " + "list of UserId strings" + ) + + return set(raw) + + return ( + _as_id_set(include_raw, "include_user_ids"), + _as_id_set(exclude_raw, "exclude_user_ids"), + ) + + +def run( + *, + provider: str, + execution_target_id: str, + execution_target_name: str, + execution_target_type: str, + region: str, + session: boto3.Session, + dry_run: bool, + metadata: dict[str, object], + dependency_data: dict[str, object], + actions: ActionRecorder, +) -> dict[str, object]: + """Remove disabled users from IAM Identity Center. + + This AWS task runs in the IAM Identity Center owner account. It enumerates + every user in the Identity Store via `list_users`, filters to users whose + `UserStatus` is `DISABLED`, optionally filter further with + `include_user_ids` or `exclude_user_ids`, and deletes every user that + remains. Deletions run concurrently. A failure deleting one user does not stop the others; + failures are collected and raised together at the end via `TaskExecutionError`. + dry-run supported. + + Metadata: + identity_center_region: Optional AWS region for the IAM Identity + Center and Identity Store clients. Defaults to the current + session region. + include_user_ids: Optional list of Identity Store `UserId` strings. + When set, only disabled users in this list are actioned; all + other disabled users are left alone. Mutually exclusive with + `exclude_user_ids`. + exclude_user_ids: Optional list of Identity Store `UserId` strings. + When set, disabled users in this list are left alone and every + other disabled user is actioned. Mutually exclusive with + `include_user_ids`. + + Args: + provider: Provider name for the current execution target. + execution_target_id: Target AWS account ID. + execution_target_name: Friendly name for the target account. + execution_target_type: Provider target type. + region: Current AWS region. + session: Boto3 session scoped to the current region. + dry_run: Whether execution is running in dry-run mode. + metadata: Task metadata containing optional Identity Center region + and optional include/exclude UserId lists. + dependency_data: Runtime data selected from declared task dependencies. + This task requires none. + actions: Action recorder provided by the engine. + + Returns: + A payload containing the Identity Center region, disabled user + count, targeted user count (after include/exclude filtering), + removed count, failed count, failed user details (with an `error` + message per entry), and disabled user details (UserId, UserName, + Emails, UserType, UserStatus), or `{"skipped": True}` for non-owner + accounts. + + Raises: + ValueError: If metadata.identity_center_region is not a string. + RuntimeError: If no active IAM Identity Center instance exists, if + both metadata.include_user_ids and metadata.exclude_user_ids are + set, or if either is set to something other than a list of + strings. + TaskExecutionError: If one or more targeted users failed to delete. + `partial_result` carries the same payload described above, + including which users succeeded and which failed. + """ + + raw_region = metadata.get("identity_center_region") + account_id = execution_target_id + + if raw_region is None: + identity_center_region = region + elif isinstance(raw_region, str): + identity_center_region = raw_region + else: + raise ValueError("metadata.identity_center_region must be a string") + + include_ids, exclude_ids = _resolve_user_id_filters(metadata) + + __LOGGER__.info(f"Using Identity Center region '{identity_center_region}'") + + sso_admin_client = session.client( + "sso-admin", region_name=identity_center_region, config=BOTO_CONFIG + ) + identitystore_client = session.client( + "identitystore", region_name=identity_center_region, config=BOTO_CONFIG + ) + + _, identity_store_id, owner_account_id = _get_active_sso_instance(sso_admin_client) + + if account_id != owner_account_id: + __LOGGER__.info( + f"Skipping account '{account_id}' because it is not " + f"the Identity Center owner account" + ) + return {"skipped": True} + + disabled_users = _list_disabled_users(identitystore_client, identity_store_id) + + if include_ids is not None: + targeted_users = [ + user for user in disabled_users if user["UserId"] in include_ids + ] + elif exclude_ids is not None: + targeted_users = [ + user for user in disabled_users if user["UserId"] not in exclude_ids + ] + else: + targeted_users = disabled_users + + removed: list[dict[str, object]] = [] + failed: list[dict[str, object]] = [] + + if dry_run: + for user in targeted_users: + __LOGGER__.info( + f"(dry-run) Would remove disabled user '{user['UserName']}' " + f"({user['UserId']})" + ) + else: + with ThreadPoolExecutor(max_workers=MAX_DELETE_WORKERS) as executor: + future_to_user = { + executor.submit( + identitystore_client.delete_user, + IdentityStoreId=identity_store_id, + UserId=user["UserId"], + ): user + for user in targeted_users + } + + for future in as_completed(future_to_user): + user = future_to_user[future] + user_id = user["UserId"] + user_name = user["UserName"] + + try: + future.result() + except (ClientError, BotoCoreError) as error: + __LOGGER__.warning( + f"Failed to remove disabled user '{user_name}' " + f"({user_id}): {error}" + ) + failed.append({**user, "error": str(error)}) + continue + + removed.append(user) + __LOGGER__.info(f"Removed disabled user '{user_name}' ({user_id})") + + if dry_run: + actions.record( + f"(dry-run) Would remove {len(targeted_users)} disabled " + "Identity Center user(s)" + ) + else: + actions.record( + f"Removed {len(removed)} disabled Identity Center user(s), " + f"{len(failed)} failed" + ) + + result = { + "identity_center_region": identity_center_region, + "disabled_count": len(disabled_users), + "targeted_count": len(targeted_users), + "removed_count": len(removed), + "failed_count": len(failed), + "failed_users": failed, + "disabled_users": disabled_users, + } + + if failed: + raise TaskExecutionError( + f"remove_disabled_idc_users failed to remove {len(failed)} of " + f"{len(targeted_users)} targeted user(s)", + partial_result=result, + ) + + return result diff --git a/src/anvil/providers/aws/tasks/remove_iam_user.py b/src/anvil/providers/aws/tasks/remove_iam_user.py index af290bc..6be27aa 100644 --- a/src/anvil/providers/aws/tasks/remove_iam_user.py +++ b/src/anvil/providers/aws/tasks/remove_iam_user.py @@ -1,6 +1,7 @@ from __future__ import annotations import logging +from typing import Any from botocore.exceptions import ClientError @@ -9,19 +10,34 @@ __LOGGER__ = logging.getLogger(__name__) -def cleanup_user_resources( - iam_client, user_name: str, dry_run: bool, actions: ActionRecorder -) -> None: - # Groups +def _list_paginated_user_resources( + iam_client, *, operation_name: str, result_key: str, user_name: str +) -> list[Any]: + """Return every page of one IAM user resource collection.""" + try: - groups_response = iam_client.list_groups_for_user(UserName=user_name) + paginator = iam_client.get_paginator(operation_name) + resources: list[Any] = [] + for page in paginator.paginate(UserName=user_name): + resources.extend(page.get(result_key, [])) + return resources except ClientError as error: if error.response["Error"]["Code"] == "NoSuchEntity": - groups_response = {"Groups": []} - else: - raise + return [] + raise + - for group in groups_response.get("Groups", []): +def cleanup_user_resources( + iam_client, user_name: str, dry_run: bool, actions: ActionRecorder +) -> None: + # Groups + groups = _list_paginated_user_resources( + iam_client, + operation_name="list_groups_for_user", + result_key="Groups", + user_name=user_name, + ) + for group in groups: name = group["GroupName"] if dry_run: __LOGGER__.debug(f"(dry-run) Would remove user from group: {name}") @@ -30,15 +46,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Removed user from group: {name}") # Access Keys - try: - access_keys = iam_client.list_access_keys(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - access_keys = {"AccessKeyMetadata": []} - else: - raise - - for key in access_keys.get("AccessKeyMetadata", []): + access_keys = _list_paginated_user_resources( + iam_client, + operation_name="list_access_keys", + result_key="AccessKeyMetadata", + user_name=user_name, + ) + for key in access_keys: key_id = key["AccessKeyId"] if dry_run: __LOGGER__.debug(f"(dry-run) Would delete access key: {key_id}") @@ -47,15 +61,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Deleted access key: {key_id}") # MFA Devices - try: - mfa_list = iam_client.list_mfa_devices(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - mfa_list = {"MFADevices": []} - else: - raise - - for device in mfa_list.get("MFADevices", []): + mfa_devices = _list_paginated_user_resources( + iam_client, + operation_name="list_mfa_devices", + result_key="MFADevices", + user_name=user_name, + ) + for device in mfa_devices: serial = device["SerialNumber"] if dry_run: __LOGGER__.debug(f"(dry-run) Would deactivate MFA device: {serial}") @@ -66,15 +78,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Deleted MFA device: {serial}") # SSH Keys - try: - ssh_list = iam_client.list_ssh_public_keys(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - ssh_list = {"SSHPublicKeys": []} - else: - raise - - for ssh in ssh_list.get("SSHPublicKeys", []): + ssh_keys = _list_paginated_user_resources( + iam_client, + operation_name="list_ssh_public_keys", + result_key="SSHPublicKeys", + user_name=user_name, + ) + for ssh in ssh_keys: ssh_id = ssh["SSHPublicKeyId"] if dry_run: __LOGGER__.debug(f"(dry-run) Would delete SSH key: {ssh_id}") @@ -102,15 +112,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Deleted service credential: {cred_id}") # Certificates - try: - certs = iam_client.list_signing_certificates(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - certs = {"Certificates": []} - else: - raise - - for cert in certs.get("Certificates", []): + certificates = _list_paginated_user_resources( + iam_client, + operation_name="list_signing_certificates", + result_key="Certificates", + user_name=user_name, + ) + for cert in certificates: cert_id = cert["CertificateId"] if dry_run: __LOGGER__.debug(f"(dry-run) Would delete certificate: {cert_id}") @@ -121,15 +129,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Deleted certificate: {cert_id}") # Attached Policies - try: - attached = iam_client.list_attached_user_policies(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - attached = {"AttachedPolicies": []} - else: - raise - - for policy in attached.get("AttachedPolicies", []): + attached_policies = _list_paginated_user_resources( + iam_client, + operation_name="list_attached_user_policies", + result_key="AttachedPolicies", + user_name=user_name, + ) + for policy in attached_policies: arn = policy["PolicyArn"] if dry_run: __LOGGER__.debug(f"(dry-run) Would detach policy: {arn}") @@ -138,15 +144,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Detached policy: {arn}") # Inline Policies - try: - inline = iam_client.list_user_policies(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - inline = {"PolicyNames": []} - else: - raise - - for name in inline.get("PolicyNames", []): + inline_policy_names = _list_paginated_user_resources( + iam_client, + operation_name="list_user_policies", + result_key="PolicyNames", + user_name=user_name, + ) + for name in inline_policy_names: if dry_run: __LOGGER__.debug(f"(dry-run) Would delete inline policy: {name}") else: @@ -154,15 +158,13 @@ def cleanup_user_resources( __LOGGER__.debug(f"Deleted inline policy: {name}") # Tags - try: - tags = iam_client.list_user_tags(UserName=user_name) - except ClientError as error: - if error.response["Error"]["Code"] == "NoSuchEntity": - tags = {"Tags": []} - else: - raise - - tag_keys = [t["Key"] for t in tags.get("Tags", [])] + tags = _list_paginated_user_resources( + iam_client, + operation_name="list_user_tags", + result_key="Tags", + user_name=user_name, + ) + tag_keys = [tag["Key"] for tag in tags] if tag_keys: if dry_run: __LOGGER__.debug(f"(dry-run) Would remove tags: {tag_keys}") @@ -194,6 +196,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> None: """Remove IAM resources attached to a configured IAM user. @@ -217,6 +220,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Task metadata containing the IAM user name. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Raises: @@ -240,4 +244,7 @@ def run( iam_client=iam_client, user_name=user_name, dry_run=dry_run, actions=actions ) - actions.record("Removed IAM user resources") + if dry_run: + actions.record(f"(dry-run) Would remove IAM user resources for {user_name}") + else: + actions.record(f"Removed IAM user resources for {user_name}") diff --git a/src/anvil/providers/aws/tasks/remove_missing_group_assignments.py b/src/anvil/providers/aws/tasks/remove_missing_group_assignments.py index 40a03c6..e2d4676 100644 --- a/src/anvil/providers/aws/tasks/remove_missing_group_assignments.py +++ b/src/anvil/providers/aws/tasks/remove_missing_group_assignments.py @@ -131,10 +131,6 @@ def _validate_groups( except identitystore_client.exceptions.ResourceNotFoundException: group_existence[group_id] = False - except ClientError as error: - __LOGGER__.error(f"Error validating group '{group_id}': {error}") - group_existence[group_id] = False - return group_existence @@ -148,6 +144,7 @@ def run( session: boto3.Session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Remove IAM Identity Center group assignments for missing groups. @@ -171,6 +168,7 @@ def run( session: Boto3 session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Task metadata containing optional Identity Center region. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: @@ -270,38 +268,36 @@ def run( for entry in missing_assignments: if dry_run: __LOGGER__.info( - f"(Dry run) Would remove GROUP '{entry['GroupId']}' " + f"(dry-run) Would remove GROUP '{entry['GroupId']}' " f"from permission set '{entry['PermissionSetName']}' " f"in account '{entry['AccountName']}'" ) continue - try: - sso_admin_client.delete_account_assignment( - InstanceArn=instance_arn, - TargetId=entry["AccountId"], - TargetType="AWS_ACCOUNT", - PermissionSetArn=entry["PermissionSetArn"], - PrincipalType="GROUP", - PrincipalId=entry["GroupId"], - ) - - removed.append(entry) + sso_admin_client.delete_account_assignment( + InstanceArn=instance_arn, + TargetId=entry["AccountId"], + TargetType="AWS_ACCOUNT", + PermissionSetArn=entry["PermissionSetArn"], + PrincipalType="GROUP", + PrincipalId=entry["GroupId"], + ) - __LOGGER__.info( - f"Removed GROUP '{entry['GroupId']}' " - f"from permission set '{entry['PermissionSetName']}' " - f"in account '{entry['AccountName']}'" - ) + removed.append(entry) - except ClientError as error: - __LOGGER__.error( - f"Failed to remove GROUP '{entry['GroupId']}' " - f"from permission set '{entry['PermissionSetName']}' " - f"in account '{entry['AccountName']}': {error}" - ) + __LOGGER__.info( + f"Removed GROUP '{entry['GroupId']}' " + f"from permission set '{entry['PermissionSetName']}' " + f"in account '{entry['AccountName']}'" + ) - actions.record(f"Missing group assignments detected: {len(missing_assignments)}") + if dry_run: + actions.record( + f"(dry-run) Would remove {len(missing_assignments)} missing group " + "assignment(s)" + ) + else: + actions.record(f"Removed {len(removed)} missing group assignment(s)") return { "identity_center_region": identity_center_region, diff --git a/src/anvil/providers/azure/tasks/count_resource_groups.py b/src/anvil/providers/azure/tasks/count_resource_groups.py index b5184f6..4846b39 100644 --- a/src/anvil/providers/azure/tasks/count_resource_groups.py +++ b/src/anvil/providers/azure/tasks/count_resource_groups.py @@ -47,6 +47,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Count resource groups in the current Azure subscription. @@ -68,6 +69,7 @@ def run( session: Azure session scoped to the subscription and region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/base.py b/src/anvil/providers/base.py index 2ce18c5..175b49e 100644 --- a/src/anvil/providers/base.py +++ b/src/anvil/providers/base.py @@ -135,6 +135,7 @@ class ProviderExecutionPlan: """Resolved provider execution targets.""" execution_targets: list[ExecutionTarget] + configured_target: ExecutionTarget | None = None @dataclass(frozen=True, slots=True) @@ -163,7 +164,13 @@ def build_session(self, *, region: str) -> object: def record_region_outcome( self, *, region: str, duration_seconds: float, failed: bool, interrupted: bool ) -> None: - """Record one region outcome for provider lifecycle decisions.""" + """Record one region outcome for provider lifecycle decisions. + + Implementations must complete promptly because graph settlement records + the outcome synchronously before admitting dependent work. Ordinary + parallel-region execution may call different region outcomes + concurrently. + """ def close(self) -> None: """Release any provider-owned runtime resources.""" @@ -228,6 +235,24 @@ def prepare_execution_runtime( """Prepare lifecycle state for one execution target.""" +class ConfiguredTargetProvider(Protocol): + """Additional contract for providers declaring configured-target support.""" + + def validate_task_configuration( + self, *, target: TargetDescriptor, task_scopes: dict[str, str] + ) -> None: + """Validate configured-target task compatibility before authentication.""" + + def prepare_configured_target_runtime( + self, + *, + target: TargetDescriptor, + execution_target: ExecutionTarget, + context: ExecutionContext, + ) -> ProviderExecutionRuntime: + """Prepare lifecycle state for the provider-owned configured target.""" + + def validate_provider_contract(provider: Provider) -> None: """Validate that a provider exposes the public provider contract.""" @@ -263,9 +288,27 @@ def validate_provider_contract(provider: Provider) -> None: }, "prepare_execution_runtime": {"target", "execution_target", "context"}, } + if "configured_target" in metadata.supported_task_scopes: + required_methods.update( + { + "validate_task_configuration": {"target", "task_scopes"}, + "prepare_configured_target_runtime": { + "target", + "execution_target", + "context", + }, + } + ) for method_name, required_parameters in required_methods.items(): if not callable(getattr(provider, method_name, None)): - raise TypeError(f"provider missing callable {method_name}()") + capability = ( + " configured_target capability" + if "configured_target" in metadata.supported_task_scopes + and method_name + in {"validate_task_configuration", "prepare_configured_target_runtime"} + else "" + ) + raise TypeError(f"provider{capability} missing callable {method_name}()") signature = inspect.signature(getattr(provider, method_name)) parameter_names = set(signature.parameters) diff --git a/src/anvil/providers/gcp/tasks/get_project_info.py b/src/anvil/providers/gcp/tasks/get_project_info.py index 5b95082..d4401c4 100644 --- a/src/anvil/providers/gcp/tasks/get_project_info.py +++ b/src/anvil/providers/gcp/tasks/get_project_info.py @@ -73,6 +73,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Return metadata for the current GCP project. @@ -89,6 +90,7 @@ def run( session: GCP session scoped to the project and region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/github/tasks/audit_branch_protection.py b/src/anvil/providers/github/tasks/audit_branch_protection.py index cbae7bb..525a0e4 100644 --- a/src/anvil/providers/github/tasks/audit_branch_protection.py +++ b/src/anvil/providers/github/tasks/audit_branch_protection.py @@ -30,6 +30,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Read branch protection settings for a GitHub repository branch.""" diff --git a/src/anvil/providers/github/tasks/audit_repo_security_settings.py b/src/anvil/providers/github/tasks/audit_repo_security_settings.py index f22c487..ed7b7ce 100644 --- a/src/anvil/providers/github/tasks/audit_repo_security_settings.py +++ b/src/anvil/providers/github/tasks/audit_repo_security_settings.py @@ -47,6 +47,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Read security-relevant repository settings from GitHub.""" diff --git a/src/anvil/providers/github/tasks/audit_rulesets.py b/src/anvil/providers/github/tasks/audit_rulesets.py index 3152977..54e9fdb 100644 --- a/src/anvil/providers/github/tasks/audit_rulesets.py +++ b/src/anvil/providers/github/tasks/audit_rulesets.py @@ -31,6 +31,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """List repository rulesets for a GitHub repository target.""" diff --git a/src/anvil/providers/github/tasks/list_code_scanning_alerts.py b/src/anvil/providers/github/tasks/list_code_scanning_alerts.py index 55679f6..4fd9cc1 100644 --- a/src/anvil/providers/github/tasks/list_code_scanning_alerts.py +++ b/src/anvil/providers/github/tasks/list_code_scanning_alerts.py @@ -40,6 +40,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """List code scanning alerts for a GitHub target.""" diff --git a/src/anvil/providers/github/tasks/list_dependabot_alerts.py b/src/anvil/providers/github/tasks/list_dependabot_alerts.py index 0943f64..91e97af 100644 --- a/src/anvil/providers/github/tasks/list_dependabot_alerts.py +++ b/src/anvil/providers/github/tasks/list_dependabot_alerts.py @@ -43,6 +43,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """List Dependabot alerts for a GitHub target.""" diff --git a/src/anvil/providers/github/tasks/list_secret_scanning_alerts.py b/src/anvil/providers/github/tasks/list_secret_scanning_alerts.py index 585f978..b8574ef 100644 --- a/src/anvil/providers/github/tasks/list_secret_scanning_alerts.py +++ b/src/anvil/providers/github/tasks/list_secret_scanning_alerts.py @@ -43,6 +43,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """List secret scanning alerts for a GitHub target.""" diff --git a/src/anvil/providers/github/tasks/search_code.py b/src/anvil/providers/github/tasks/search_code.py index 6a3d220..795ac21 100644 --- a/src/anvil/providers/github/tasks/search_code.py +++ b/src/anvil/providers/github/tasks/search_code.py @@ -208,6 +208,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict[str, object]: """Search code efficiently in the current GitHub organization or repository. @@ -227,6 +228,7 @@ def run( metadata: Search options. ``query`` is required. ``language``, ``path``, ``extension``, ``filename``, ``max_results``, and ``highlight`` are optional. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the Anvil engine. Returns: diff --git a/src/anvil/providers/tasks/noop.py b/src/anvil/providers/tasks/noop.py index 6f701d5..511569b 100644 --- a/src/anvil/providers/tasks/noop.py +++ b/src/anvil/providers/tasks/noop.py @@ -25,6 +25,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict: """Run a no-op task for validation, smoke tests, and framework checks. @@ -40,6 +41,7 @@ def run( session: Provider session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Returns: diff --git a/src/anvil/providers/tasks/noop_fail.py b/src/anvil/providers/tasks/noop_fail.py index 063e828..5d1cb17 100644 --- a/src/anvil/providers/tasks/noop_fail.py +++ b/src/anvil/providers/tasks/noop_fail.py @@ -25,6 +25,7 @@ def run( session, dry_run: bool, metadata: dict[str, object], + dependency_data: dict[str, object], actions: ActionRecorder, ) -> dict: """Raise an intentional failure for error-path validation. @@ -40,6 +41,7 @@ def run( session: Provider session scoped to the current region. dry_run: Whether execution is running in dry-run mode. metadata: Arbitrary config metadata for the task. + dependency_data: Runtime data selected from declared task dependencies. actions: Action recorder provided by the engine. Raises: diff --git a/src/anvil/result_query.py b/src/anvil/result_query.py index 64c80a0..3295aa2 100644 --- a/src/anvil/result_query.py +++ b/src/anvil/result_query.py @@ -21,7 +21,8 @@ "entity_metadata", "entity_type", "region", - "task", + "task_id", + "task_name", "error", ] FIELD_HEADERS = {"record_type": "type"} @@ -42,11 +43,13 @@ "provider", "region", "result", + "skip_reason", "started_at", "status", "target", "target_type", - "task", + "task_id", + "task_name", ] @@ -91,15 +94,36 @@ def build_jsonl_records_for_target( { **entity_record, "record_type": "task", - "task": task_result.task_name, + "task_id": task_result.task_id, + "task_name": task_result.task_name, "region": task_result.region, **_timed_status_record(task_result), "result": task_result.result, "error": task_result.error, + "skip_reason": task_result.skip_reason, "actions": list(task_result.actions), } ) + configured_record = _base_configured_target_record( + target_result=target_result, config_file=config_file + ) + for task_result in target_result.tasks: + records.append( + { + **configured_record, + "record_type": "task", + "task_id": task_result.task_id, + "task_name": task_result.task_name, + "region": task_result.region, + **_timed_status_record(task_result), + "result": task_result.result, + "error": task_result.error, + "skip_reason": task_result.skip_reason, + "actions": list(task_result.actions), + } + ) + return records @@ -176,7 +200,7 @@ def iter_filtered_records( and _matches(record, "target", filters.target) and _matches_entity(record, filters.entity) and _matches(record, "region", filters.region) - and _matches(record, "task", filters.task) + and _matches_task(record, filters.task) ): yield record @@ -360,7 +384,7 @@ def _normalize_status_filter(status: str | None) -> str | set[str] | None: def _record_is_unsuccessful(record: dict[str, object]) -> bool: status = record.get("status") - return isinstance(status, str) and status.lower() != "success" + return isinstance(status, str) and status.lower() in {"error", "interrupted"} def _matches_status(record: dict[str, object], expected: str | set[str] | None) -> bool: @@ -378,18 +402,18 @@ def _matches_status(record: dict[str, object], expected: str | set[str] | None) return normalized_actual in expected -def _task_specs_by_name(tasks: list[dict[str, object]]) -> dict[str, dict[str, object]]: +def _task_specs_by_id(tasks: list[dict[str, object]]) -> dict[str, dict[str, object]]: return { - str(task["name"]): task + str(task.get("id", task["name"])): task for task in tasks if isinstance(task.get("name"), str) and task.get("name") } -def _expand_task_names_with_dependencies( - *, selected_names: set[str], tasks: list[dict[str, object]] +def _expand_task_ids_with_dependencies( + *, selected_ids: set[str], tasks: list[dict[str, object]] ) -> list[dict[str, object]]: - task_specs = _task_specs_by_name(tasks) + task_specs = _task_specs_by_id(tasks) expanded_names: set[str] = set() def add_with_dependencies(task_name: str) -> None: @@ -406,13 +430,13 @@ def add_with_dependencies(task_name: str) -> None: add_with_dependencies(dependency) expanded_names.add(task_name) - for selected_name in selected_names: - add_with_dependencies(selected_name) + for selected_id in selected_ids: + add_with_dependencies(selected_id) return [ task for task in tasks - if isinstance(task.get("name"), str) and task["name"] in expanded_names + if str(task.get("id", task.get("name", ""))) in expanded_names ] @@ -441,9 +465,9 @@ def _narrow_target_for_failed_entity( for region in (record.get("region") for record in task_records) if isinstance(region, str) and region } - failed_task_names = { + failed_task_ids = { task - for task in (record.get("task") for record in task_records) + for task in (record.get("task_id") for record in task_records) if isinstance(task, str) and task } @@ -456,9 +480,9 @@ def _narrow_target_for_failed_entity( regions = sorted(failed_regions) tasks = target.tasks - if failed_task_names and not entity_level_failure_exists: - tasks = _expand_task_names_with_dependencies( - selected_names=failed_task_names, tasks=target.tasks + if failed_task_ids and not entity_level_failure_exists: + tasks = _expand_task_ids_with_dependencies( + selected_ids=failed_task_ids, tasks=target.tasks ) if not tasks: tasks = target.tasks @@ -472,6 +496,25 @@ def _narrow_target_for_failed_entity( def _narrow_target_for_failure_records( *, target: TargetDescriptor, records: list[dict[str, object]] ) -> list[TargetDescriptor]: + configured_records = [ + record for record in records if record.get("entity_type") == "configured_target" + ] + if configured_records: + failed_task_ids = { + task_id + for task_id in ( + record.get("task_id") + for record in records + if record.get("record_type") == "task" + and _record_is_unsuccessful(record) + ) + if isinstance(task_id, str) and task_id + } + tasks = _expand_task_ids_with_dependencies( + selected_ids=failed_task_ids, tasks=target.tasks + ) + return [replace(target, tasks=tasks or target.tasks)] + records_by_entity: dict[str, list[dict[str, object]]] = defaultdict(list) for record in records: entity_id = record.get("entity_id") @@ -492,6 +535,38 @@ def _matches(record: dict[str, object], key: str, expected: str | None) -> bool: return isinstance(actual, str) and actual.lower() == expected.lower() +def _matches_task(record: dict[str, object], expected: str | None) -> bool: + if expected is None: + return True + + expected_lower = expected.lower() + return any( + isinstance(actual, str) and actual.lower() == expected_lower + for actual in (record.get("task_id"), record.get("task_name")) + ) + + +def _base_configured_target_record( + *, target_result: TargetResult, config_file: Path | None +) -> dict[str, object]: + record: dict[str, object] = { + "target_type": "target", + "target": target_result.target_name, + "generated_at": target_result.generated_at, + "dry_run": target_result.dry_run, + "entity_id": None, + "entity_name": None, + "entity_type": "configured_target", + "provider": target_result.provider, + "entity_metadata": {}, + } + if config_file is not None: + record["config_file"] = config_file.as_posix() + record["config_file_resolved"] = config_file.resolve().as_posix() + + return record + + def _matches_entity(record: dict[str, object], expected: str | None) -> bool: if expected is None: return True diff --git a/src/anvil/results.py b/src/anvil/results.py index 98c24ae..229b3c6 100644 --- a/src/anvil/results.py +++ b/src/anvil/results.py @@ -9,6 +9,7 @@ class ExecutionStatus(str, Enum): SUCCESS = "success" ERROR = "error" INTERRUPTED = "interrupted" + SKIPPED = "skipped" @property def is_success(self) -> bool: @@ -22,9 +23,31 @@ def is_error(self) -> bool: def is_interrupted(self) -> bool: return self is ExecutionStatus.INTERRUPTED + @property + def is_skipped(self) -> bool: + return self is ExecutionStatus.SKIPPED + @property def is_unsuccessful(self) -> bool: - return self is not ExecutionStatus.SUCCESS + return self in {ExecutionStatus.ERROR, ExecutionStatus.INTERRUPTED} + + +def aggregate_execution_statuses(statuses: list[ExecutionStatus]) -> ExecutionStatus: + """Aggregate execution statuses without treating skipped work as failure. + + Args: + statuses: Task or child-entity statuses to aggregate. + + Returns: + Error when any child errored, interrupted when none errored and at least + one was interrupted, otherwise success. + """ + + if any(status.is_error for status in statuses): + return ExecutionStatus.ERROR + if any(status.is_interrupted for status in statuses): + return ExecutionStatus.INTERRUPTED + return ExecutionStatus.SUCCESS class EngineState(str, Enum): @@ -43,16 +66,19 @@ class TimedResult: @dataclass(frozen=True, slots=True) class TaskResult(TimedResult): + task_id: str task_name: str region: str status: ExecutionStatus result: object | None = None error: str | None = None + skip_reason: str | None = None actions: list[str] = field(default_factory=list) def to_dict(self) -> dict[str, object]: return { - "task": self.task_name, + "task_id": self.task_id, + "task_name": self.task_name, "region": self.region, "status": self.status.value, "started_at": self.started_at, @@ -60,6 +86,7 @@ def to_dict(self) -> dict[str, object]: "duration_seconds": self.duration_seconds, "result": self.result, "error": self.error, + "skip_reason": self.skip_reason, "actions": list(self.actions), } @@ -132,6 +159,7 @@ class TargetResult: generated_at: str dry_run: bool entities: list[EntityResult] + tasks: list[TaskResult] = field(default_factory=list) error: str | None = None benchmark: dict[str, object] | None = None @@ -153,8 +181,10 @@ def unsuccessful_entities(self) -> list[EntityResult]: @property def has_failures(self) -> bool: - return self.error is not None or any( - result.status.is_unsuccessful for result in self.entities + return ( + self.error is not None + or any(result.status.is_unsuccessful for result in self.entities) + or any(result.status.is_unsuccessful for result in self.tasks) ) def to_dict(self) -> dict[str, object]: @@ -164,6 +194,7 @@ def to_dict(self) -> dict[str, object]: "generated_at": self.generated_at, "dry_run": self.dry_run, "total_entities": self.total_entities, + "tasks": [task.to_dict() for task in self.tasks], "entities": [result.to_dict() for result in self.entities], "error": self.error, } @@ -180,6 +211,7 @@ def create( provider: str, dry_run: bool, entities: list[EntityResult], + tasks: list[TaskResult] | None = None, error: str | None = None, benchmark: dict[str, object] | None = None, ) -> TargetResult: @@ -189,6 +221,7 @@ def create( generated_at=datetime.datetime.now(datetime.UTC).isoformat(), dry_run=dry_run, entities=entities, + tasks=list(tasks or []), error=error, benchmark=benchmark, ) @@ -267,6 +300,8 @@ def build_summary(self) -> dict[str, object]: total_failed_entities = 0 total_interrupted_entities = 0 total_failed_tasks = 0 + total_interrupted_tasks = 0 + total_skipped_tasks = 0 for target_result in self.target_results: entities = target_result.entities @@ -282,16 +317,19 @@ def build_summary(self) -> dict[str, object]: if entity_result.status.is_interrupted ] - failed_tasks = sum( - 1 - for entity_result in entities - for task in entity_result.tasks - if task.status.is_error - ) + tasks = [ + *target_result.tasks, + *(task for entity_result in entities for task in entity_result.tasks), + ] + failed_tasks = sum(1 for task in tasks if task.status.is_error) + interrupted_tasks = sum(1 for task in tasks if task.status.is_interrupted) + skipped_tasks = sum(1 for task in tasks if task.status.is_skipped) total_failed_entities += len(failed_entities) total_interrupted_entities += len(interrupted_entities) total_failed_tasks += failed_tasks + total_interrupted_tasks += interrupted_tasks + total_skipped_tasks += skipped_tasks target_summaries.append( { @@ -299,7 +337,10 @@ def build_summary(self) -> dict[str, object]: "total_entities": target_result.total_entities, "failed_entities": len(failed_entities), "interrupted_entities": len(interrupted_entities), + "total_tasks": len(tasks), "failed_tasks": failed_tasks, + "interrupted_tasks": interrupted_tasks, + "skipped_tasks": skipped_tasks, "has_failures": target_result.has_failures, "error": target_result.error, **( @@ -318,6 +359,8 @@ def build_summary(self) -> dict[str, object]: "total_failed_entities": total_failed_entities, "total_interrupted_entities": total_interrupted_entities, "total_failed_tasks": total_failed_tasks, + "total_interrupted_tasks": total_interrupted_tasks, + "total_skipped_tasks": total_skipped_tasks, } if self.benchmark is not None: payload["benchmark"] = self.benchmark diff --git a/src/anvil/runner.py b/src/anvil/runner.py index 7e987c6..1f0125f 100644 --- a/src/anvil/runner.py +++ b/src/anvil/runner.py @@ -5,7 +5,7 @@ import threading import time from collections import deque -from collections.abc import Callable +from collections.abc import Callable, Collection from concurrent.futures import ( FIRST_COMPLETED, CancelledError, @@ -14,13 +14,16 @@ wait, ) from dataclasses import dataclass, field, replace +from typing import cast from anvil.benchmark import BenchmarkRecorder from anvil.actions import ActionRecorder from anvil.descriptors import TargetDescriptor from anvil.execution_context import ExecutionContext +from anvil.provider_lifecycle import CoordinateLifecycleState from anvil.provider_loader import load_provider from anvil.providers.base import ( + ConfiguredTargetProvider, ExecutionTarget, Provider, ProviderAuthResult, @@ -36,21 +39,42 @@ ExecutionStatus, TargetResult, TaskResult, + aggregate_execution_statuses, +) +from anvil.task_context import ( + TaskCallContext, + merge_task_metadata, + resolve_dependency_data, +) +from anvil.task_errors import TaskExecutionError +from anvil.task_loader import ( + ResolvedExecution, + ResolvedTask, + TaskConfigError, + TaskScope, + resolve_tasks, +) +from anvil.task_planner import TaskInstance, plan_task_instances +from anvil.task_scheduler import ( + DependencyResults, + ScheduledTaskResult, + TaskInstanceEligibility, + execute_task_instance_plan, + task_dependency_eligibility, ) -from anvil.task_context import TaskCallContext -from anvil.task_loader import ResolvedExecution, ResolvedTask, TaskScope, resolve_tasks __LOGGER__ = logging.getLogger(__name__) STATE_PRECEDENCE: dict[EngineState, int] = { EngineState.AUTH_FAILED: 4, - EngineState.CANCELLED: 3, - EngineState.COMPLETED_WITH_FAILURES: 2, + EngineState.COMPLETED_WITH_FAILURES: 3, + EngineState.CANCELLED: 2, EngineState.COMPLETED_SUCCESS: 1, } DEFAULT_AUTH_CHECK_MAX_WORKERS = 4 +_SESSION_NOT_CREATED = object() def _load_provider(provider_name: str) -> Provider: @@ -246,12 +270,13 @@ def _auth_result_from_provider_result( def _run_provider_auth_check_for_target( - *, provider: Provider, target: TargetDescriptor + *, provider: Provider, target: TargetDescriptor, validate_target: bool = True ) -> AuthResult: started_perf = time.perf_counter() started_at = datetime.datetime.now(datetime.UTC).isoformat() try: - provider.validate_target(target) + if validate_target: + provider.validate_target(target) provider_result = provider.auth_check(target) except Exception as error: ended_at = datetime.datetime.now(datetime.UTC).isoformat() @@ -274,15 +299,23 @@ def _run_provider_auth_check_for_target( def _run_cached_provider_auth_check_for_target( - *, provider: Provider, target: TargetDescriptor, auth_cache: AuthCheckCache + *, + provider: Provider, + target: TargetDescriptor, + auth_cache: AuthCheckCache, + validate_target: bool = True, ) -> AuthResult: cache_key = provider.auth_cache_key(target) if cache_key is None: - return _run_provider_auth_check_for_target(provider=provider, target=target) + return _run_provider_auth_check_for_target( + provider=provider, target=target, validate_target=validate_target + ) def check() -> AuthCheckOutcome: return _auth_outcome_from_result( - _run_provider_auth_check_for_target(provider=provider, target=target) + _run_provider_auth_check_for_target( + provider=provider, target=target, validate_target=validate_target + ) ) lookup = auth_cache.get_or_check(key=cache_key, check=check) @@ -313,7 +346,7 @@ def _build_effective_target( def _auth_result_from_config_error( - *, target: TargetDescriptor, error: ValueError + *, target: TargetDescriptor, error: Exception ) -> AuthResult: started_at = datetime.datetime.now(datetime.UTC).isoformat() return AuthResult( @@ -366,7 +399,30 @@ def prepare_target( cli_include=cli_include, cli_exclude=cli_exclude, ) - except ValueError as error: + provider.validate_target(effective_target) + + regions = configured_or_default_regions( + configured=effective_target.regions, + default=provider.metadata.default_regions, + ) + validate_resolved_regions(regions=regions) + effective_target = replace(effective_target, regions=regions) + + with recorder.phase("resolve_tasks_seconds"): + execution: ResolvedExecution = resolve_tasks( + task_specs=effective_target.tasks, + provider_name=effective_target.provider, + supported_task_scopes=provider.metadata.supported_task_scopes, + ) + tasks: list[ResolvedTask] = execution.ordered + + configured_provider = cast(ConfiguredTargetProvider, provider) + if any(task.scope is TaskScope.CONFIGURED_TARGET for task in tasks): + configured_provider.validate_task_configuration( + target=effective_target, + task_scopes={task.id: task.scope.value for task in tasks}, + ) + except (TaskConfigError, TypeError, ValueError) as error: return PreparedTarget( index=index, provider=provider, @@ -379,7 +435,10 @@ def prepare_target( effective_include = effective_target.include effective_exclude = effective_target.exclude auth_result: AuthResult = _run_cached_provider_auth_check_for_target( - provider=provider, target=effective_target, auth_cache=auth_cache + provider=provider, + target=effective_target, + auth_cache=auth_cache, + validate_target=False, ) if auth_result.is_error: @@ -394,21 +453,6 @@ def prepare_target( benchmark=recorder.data, ) - regions = configured_or_default_regions( - configured=effective_target.regions, - default=provider.metadata.default_regions, - ) - validate_resolved_regions(regions=regions) - effective_target = replace(effective_target, regions=regions) - - with recorder.phase("resolve_tasks_seconds"): - execution: ResolvedExecution = resolve_tasks( - task_specs=effective_target.tasks, - provider_name=effective_target.provider, - supported_task_scopes=provider.metadata.supported_task_scopes, - ) - tasks: list[ResolvedTask] = execution.ordered - context: ExecutionContext = _build_execution_context( target=effective_target, tasks=tasks, benchmark_enabled=benchmark_enabled ) @@ -453,10 +497,186 @@ class _ProviderRegionOutcome: failed: bool interrupted: bool duration_seconds: float + activated_task_ids: frozenset[str] = frozenset() + + +@dataclass(frozen=True, slots=True) +class _OrdinaryTaskExecution: + """Task results and runtime metrics for one ordinary execution target.""" + + task_results: list[TaskResult] + benchmark: dict[str, object] | None + + +def _aggregate_task_result_status(task_results: list[TaskResult]) -> ExecutionStatus: + """Aggregate task results while preserving cancellation-only outcomes.""" + + status = aggregate_execution_statuses( + [task_result.status for task_result in task_results] + ) + if status is ExecutionStatus.SUCCESS and any( + task_result.status.is_skipped + and task_result.skip_reason == "cancelled_before_start" + for task_result in task_results + ): + return ExecutionStatus.INTERRUPTED + return status + + +def _provider_region_outcome( + *, + region: str, + task_results: list[TaskResult], + duration_seconds: float, + status_results: Collection[TaskResult] | None = None, + interrupted: bool = False, + activated_task_ids: frozenset[str] = frozenset(), +) -> _ProviderRegionOutcome: + """Build one provider lifecycle outcome with shared status semantics.""" + + aggregate_results = task_results if status_results is None else status_results + return _ProviderRegionOutcome( + region=region, + task_results=task_results, + failed=any(result.status.is_error for result in aggregate_results), + interrupted=interrupted + or any( + result.status.is_interrupted + or ( + result.status.is_skipped + and result.skip_reason == "cancelled_before_start" + ) + for result in aggregate_results + ), + duration_seconds=duration_seconds, + activated_task_ids=activated_task_ids, + ) + + +def _task_eligibility( + *, + task: ResolvedTask, + task_results: dict[str, TaskResult], + activated_task_ids: set[str], + stop_reason: str | None, +) -> TaskInstanceEligibility: + """Return the shared execution and finalizer-activation decision.""" + + missing_dependencies = [ + dependency for dependency in task.depends_on if dependency not in task_results + ] + if missing_dependencies: + dependencies = ", ".join(missing_dependencies) + raise RuntimeError( + f"Task '{task.id}' dependencies have not settled: {dependencies}" + ) + + chain_activated = not task.depends_on or any( + dependency in activated_task_ids for dependency in task.depends_on + ) + return task_dependency_eligibility( + task=task, + dependency_results=[task_results[dependency] for dependency in task.depends_on], + chain_activated=chain_activated, + stop_reason=stop_reason, + ) -def _task_order(context: ExecutionContext) -> dict[str, int]: - return {task.name: index for index, task in enumerate(context.tasks)} +def _skipped_task_result( + *, task: ResolvedTask, region: str, skip_reason: str +) -> TaskResult: + """Create a zero-duration result for settled unstarted work.""" + + now_at = datetime.datetime.now(datetime.UTC).isoformat() + return TaskResult( + task_id=task.id, + task_name=task.name, + region=region, + status=ExecutionStatus.SKIPPED, + started_at=now_at, + ended_at=now_at, + duration_seconds=0.0, + skip_reason=skip_reason, + ) + + +def _run_task_instance( + *, + instance: TaskInstance, + dependency_results: DependencyResults, + session: object, + context: ExecutionContext, +) -> TaskResult: + """Invoke one planned task instance with isolated provider-neutral inputs.""" + + return _invoke_task( + task=instance.task, + execution_target=instance.execution_target, + region=instance.region, + session=session, + context=context, + dependency_results=dependency_results, + ) + + +def _invoke_task( + *, + task: ResolvedTask, + execution_target: ExecutionTarget, + region: str, + session: object, + context: ExecutionContext, + dependency_results: DependencyResults, +) -> TaskResult: + """Invoke one task and return its complete terminal result.""" + + task_started_perf = time.perf_counter() + task_started_at = datetime.datetime.now(datetime.UTC).isoformat() + actions = ActionRecorder(actions=[]) + try: + result = task.run( + **TaskCallContext( + provider=execution_target.provider, + execution_target_id=execution_target.id, + execution_target_name=execution_target.name, + execution_target_type=execution_target.type, + region=region, + session=session, + dry_run=context.dry_run, + metadata=merge_task_metadata( + target_metadata=context.metadata, task_metadata=task.metadata + ), + dependency_data=resolve_dependency_data( + references=task.dependency_data, + dependency_results=dependency_results, + ), + actions=actions, + ).to_kwargs() + ) + status = ExecutionStatus.SUCCESS + error_message = None + task_data = result + except Exception as task_error: + status = ExecutionStatus.ERROR + error_message = str(task_error) + task_data = ( + task_error.partial_result + if isinstance(task_error, TaskExecutionError) + else None + ) + + return TaskResult( + task_id=task.id, + task_name=task.name, + region=region, + status=status, + started_at=task_started_at, + ended_at=datetime.datetime.now(datetime.UTC).isoformat(), + duration_seconds=time.perf_counter() - task_started_perf, + result=task_data, + error=error_message, + actions=list(actions.actions), + ) def _execution_target_regions( @@ -466,6 +686,62 @@ def _execution_target_regions( return list(execution_target.regions) +def _requires_task_instance_scheduler(tasks: list[ResolvedTask]) -> bool: + """Return whether ordinary tasks contain a narrow-to-broad barrier.""" + + depends_on_region: dict[str, bool] = {} + for task in tasks: + has_region_ancestor = any( + depends_on_region.get(dependency_id, False) + for dependency_id in task.depends_on + ) + if task.scope is TaskScope.TARGET and has_region_ancestor: + return True + depends_on_region[task.id] = ( + task.scope is TaskScope.REGION or has_region_ancestor + ) + return False + + +def _task_result_barrier_stages(tasks: list[ResolvedTask]) -> dict[str, int]: + """Return stable result-order stages for topologically ordered tasks. + + A dependency from a narrower scope to a broader scope creates a fan-in + barrier. Advancing the broader consumer to the next stage keeps every + producer ahead of it while preserving the established scope-first, + region-major ordering within each stage. + + Args: + tasks: Resolved tasks in dependency order. + + Returns: + Result-order stage keyed by effective task invocation ID. + """ + + scope_breadth = { + TaskScope.REGION: 0, + TaskScope.TARGET: 1, + TaskScope.CONFIGURED_TARGET: 2, + } + task_scopes: dict[str, TaskScope] = {} + stages: dict[str, int] = {} + for task in tasks: + stages[task.id] = max( + ( + stages[dependency_id] + + int( + scope_breadth[task.scope] + > scope_breadth[task_scopes[dependency_id]] + ) + for dependency_id in task.depends_on + ), + default=0, + ) + task_scopes[task.id] = task.scope + + return stages + + def _execute_provider_region( *, execution_target: ExecutionTarget, @@ -475,108 +751,192 @@ def _execute_provider_region( target_cancel_event: threading.Event, tasks: list[ResolvedTask] | None = None, dependency_results: dict[str, TaskResult] | None = None, + dependency_activated_task_ids: frozenset[str] | None = None, + initial_stop_reason: str | None = None, + lazy_session: bool = False, ) -> _ProviderRegionOutcome: region_started = time.perf_counter() - session = runtime.build_session(region=region) + session = ( + _SESSION_NOT_CREATED if lazy_session else runtime.build_session(region=region) + ) task_results: list[TaskResult] = [] region_task_results: dict[str, TaskResult] = dict(dependency_results or {}) - optional_map = {task.name: task.optional for task in context.tasks} - interrupted = False + activated_task_ids = ( + { + task_id + for task_id, result in region_task_results.items() + if not result.status.is_skipped + or result.skip_reason == "dependency_unsuccessful" + } + if dependency_activated_task_ids is None + else set(dependency_activated_task_ids) + ) + interrupted = initial_stop_reason == "cancelled_before_start" + stop_reason = initial_stop_reason for task in tasks if tasks is not None else context.tasks: - if context.cancel_event.is_set() or target_cancel_event.is_set(): - interrupted = True - break - - dependency_failed = any( - region_task_results[dependency].status.is_error - for dependency in task.depends_on - if dependency in region_task_results + if stop_reason is None: + if context.cancel_event.is_set() or target_cancel_event.is_set(): + interrupted = True + stop_reason = "cancelled_before_start" + elif context.fail_fast_event.is_set(): + stop_reason = "fail_fast" + + eligibility = _task_eligibility( + task=task, + task_results=region_task_results, + activated_task_ids=activated_task_ids, + stop_reason=stop_reason, ) - if dependency_failed: - now_at = datetime.datetime.now(datetime.UTC).isoformat() - blocked_result = TaskResult( - task_name=task.name, + if not eligibility.should_run: + skipped_result = _skipped_task_result( + task=task, region=region, - status=ExecutionStatus.ERROR, - started_at=now_at, - ended_at=now_at, - duration_seconds=0.0, - error="Blocked: dependency failed", + skip_reason=eligibility.skip_reason or "dependency_unsuccessful", ) - region_task_results[task.name] = blocked_result - task_results.append(blocked_result) - if not task.optional: - break + region_task_results[task.id] = skipped_result + task_results.append(skipped_result) + if eligibility.chain_activated: + activated_task_ids.add(task.id) continue - task_started_perf = time.perf_counter() - task_started_at = datetime.datetime.now(datetime.UTC).isoformat() - actions = ActionRecorder(actions=[]) - try: - task_context = TaskCallContext( - provider=execution_target.provider, - execution_target_id=execution_target.id, - execution_target_name=execution_target.name, - execution_target_type=execution_target.type, - region=region, - session=session, - dry_run=context.dry_run, - metadata=context.metadata, - actions=actions, - ) - result = task.run(**task_context.to_kwargs()) - except Exception as error: - task_ended_perf = time.perf_counter() - task_ended_at = datetime.datetime.now(datetime.UTC).isoformat() - task_result = TaskResult( - task_name=task.name, - region=region, - status=ExecutionStatus.ERROR, - started_at=task_started_at, - ended_at=task_ended_at, - duration_seconds=task_ended_perf - task_started_perf, - error=str(error), - actions=list(actions.actions), - ) - region_task_results[task.name] = task_result - task_results.append(task_result) - if not task.optional: - break - continue - - task_ended_perf = time.perf_counter() - task_ended_at = datetime.datetime.now(datetime.UTC).isoformat() - task_result = TaskResult( - task_name=task.name, + activated_task_ids.add(task.id) + if session is _SESSION_NOT_CREATED: + session = runtime.build_session(region=region) + task_result = _invoke_task( + task=task, + execution_target=execution_target, region=region, - status=ExecutionStatus.SUCCESS, - started_at=task_started_at, - ended_at=task_ended_at, - duration_seconds=task_ended_perf - task_started_perf, - result=result, - actions=list(actions.actions), + session=session, + context=context, + dependency_results=region_task_results, ) - region_task_results[task.name] = task_result + region_task_results[task.id] = task_result task_results.append(task_result) + if ( + context.fail_fast + and task_result.status.is_unsuccessful + and stop_reason is None + ): + stop_reason = "fail_fast" - failed = any( - result.status.is_error and not optional_map.get(result.task_name, False) - for result in region_task_results.values() - ) duration_seconds = time.perf_counter() - region_started - runtime.record_region_outcome( - region=region, - duration_seconds=duration_seconds, - failed=failed, - interrupted=interrupted, - ) - return _ProviderRegionOutcome( + if session is not _SESSION_NOT_CREATED: + runtime.record_region_outcome( + region=region, + duration_seconds=duration_seconds, + failed=any( + result.status.is_error for result in region_task_results.values() + ), + interrupted=interrupted, + ) + return _provider_region_outcome( region=region, task_results=task_results, - failed=failed, + status_results=region_task_results.values(), interrupted=interrupted, duration_seconds=duration_seconds, + activated_task_ids=frozenset(activated_task_ids), + ) + + +def _execute_ordinary_tasks_fast( + *, + execution_target: ExecutionTarget, + runtime: ProviderExecutionRuntime, + context: ExecutionContext, + regions: list[str], + tasks: list[ResolvedTask], +) -> _OrdinaryTaskExecution: + """Preserve region-oriented execution when no fan-in barrier is required.""" + + target_tasks = [task for task in tasks if task.scope is TaskScope.TARGET] + region_tasks = [task for task in tasks if task.scope is TaskScope.REGION] + task_results: list[TaskResult] = [] + target_outcome: _ProviderRegionOutcome | None = None + target_execution_seconds = 0.0 + + if target_tasks: + target_started = time.perf_counter() + target_outcome = _execute_provider_region( + execution_target=execution_target, + runtime=runtime, + context=context, + region=regions[0], + target_cancel_event=threading.Event(), + tasks=target_tasks, + ) + target_execution_seconds = time.perf_counter() - target_started + task_results.extend(target_outcome.task_results) + + region_started = time.perf_counter() + has_regional_finalizers = any(task.always_run for task in region_tasks) + target_stop_reason = ( + "cancelled_before_start" + if (has_regional_finalizers and context.cancel_event.is_set()) + or (target_outcome is not None and target_outcome.interrupted) + else ( + "fail_fast" + if (has_regional_finalizers and context.fail_fast_event.is_set()) + or ( + target_outcome is not None + and context.fail_fast + and target_outcome.failed + ) + else None + ) + ) + if region_tasks: + region_outcomes = _execute_provider_regions( + execution_target=execution_target, + runtime=runtime, + context=context, + regions=regions, + tasks=region_tasks, + dependency_results=dict( + zip( + (task.id for task in target_tasks), + target_outcome.task_results, + strict=True, + ) + ) + if target_outcome is not None + else None, + dependency_activated_task_ids=( + target_outcome.activated_task_ids + if target_outcome is not None + else frozenset() + ), + initial_stop_reason=target_stop_reason, + ) + else: + region_outcomes = [] + region_execution_seconds = time.perf_counter() - region_started + for outcome in region_outcomes: + task_results.extend(outcome.task_results) + + region_order = {region: index for index, region in enumerate(regions)} + task_order = {task.id: index for index, task in enumerate(tasks)} + task_scope_order = { + task.id: 0 if task.scope is TaskScope.TARGET else 1 for task in tasks + } + task_results.sort( + key=lambda result: ( + task_scope_order.get(result.task_id, 1), + region_order.get(result.region, len(region_order)), + task_order.get(result.task_id, len(task_order)), + ) + ) + + return _OrdinaryTaskExecution( + task_results=task_results, + benchmark=_provider_runtime_benchmark( + runtime=runtime, + region_outcomes=region_outcomes, + region_execution_seconds=region_execution_seconds, + target_outcome=target_outcome, + target_execution_seconds=target_execution_seconds, + ), ) @@ -589,6 +949,36 @@ def _execute_provider_execution_target( ) -> EntityResult: started_perf = time.perf_counter() started_at = datetime.datetime.now(datetime.UTC).isoformat() + ordinary_tasks = [ + task for task in context.tasks if task.scope is not TaskScope.CONFIGURED_TARGET + ] + if _requires_task_instance_scheduler(ordinary_tasks): + try: + graph_result = _execute_provider_task_graph( + provider=provider, + target=target, + context=context, + execution_targets=[execution_target], + configured_execution_target=None, + benchmark_data=None, + ) + return graph_result.entities[0] + except Exception as runtime_error: + ended_at = datetime.datetime.now(datetime.UTC).isoformat() + return EntityResult( + id=execution_target.id, + name=execution_target.name, + type=execution_target.type, + provider=execution_target.provider, + metadata=dict(execution_target.metadata), + status=ExecutionStatus.ERROR, + started_at=started_at, + ended_at=ended_at, + duration_seconds=time.perf_counter() - started_perf, + tasks=[], + error=str(runtime_error), + ) + task_results: list[TaskResult] = [] runtime: ProviderExecutionRuntime | None = None benchmark: dict[str, object] | None = None @@ -599,84 +989,32 @@ def _execute_provider_execution_target( regions = _execution_target_regions( execution_target=execution_target, context=context ) - target_tasks = [ - task for task in context.tasks if task.scope is TaskScope.TARGET - ] - region_tasks = [ - task for task in context.tasks if task.scope is TaskScope.REGION - ] - target_outcome: _ProviderRegionOutcome | None = None - target_execution_seconds = 0.0 - if target_tasks: - target_started = time.perf_counter() - target_outcome = _execute_provider_region( + if not ordinary_tasks: + region_started = time.perf_counter() + region_outcomes = _execute_provider_regions( execution_target=execution_target, runtime=runtime, context=context, - region=regions[0], - target_cancel_event=threading.Event(), - tasks=target_tasks, + regions=regions, + tasks=[], ) - target_execution_seconds = time.perf_counter() - target_started - task_results.extend(target_outcome.task_results) - - region_started = time.perf_counter() - if (region_tasks or not context.tasks) and not ( - target_outcome is not None - and (target_outcome.failed or target_outcome.interrupted) - ): - region_outcomes = _execute_provider_regions( + region_execution_seconds = time.perf_counter() - region_started + benchmark = _provider_runtime_benchmark( + runtime=runtime, + region_outcomes=region_outcomes, + region_execution_seconds=region_execution_seconds, + ) + else: + ordinary_execution = _execute_ordinary_tasks_fast( execution_target=execution_target, runtime=runtime, context=context, regions=regions, - tasks=region_tasks, - dependency_results={ - result.task_name: result for result in target_outcome.task_results - } - if target_outcome is not None - else None, - ) - else: - region_outcomes = [] - region_execution_seconds = time.perf_counter() - region_started - benchmark = _provider_runtime_benchmark( - runtime=runtime, - region_outcomes=region_outcomes, - region_execution_seconds=region_execution_seconds, - target_outcome=target_outcome, - target_execution_seconds=target_execution_seconds, - ) - for outcome in region_outcomes: - task_results.extend(outcome.task_results) - - region_order = {region: index for index, region in enumerate(regions)} - task_order = _task_order(context) - task_scope_order = { - task.name: 0 if task.scope is TaskScope.TARGET else 1 - for task in context.tasks - } - task_results.sort( - key=lambda result: ( - task_scope_order.get(result.task_name, 1), - region_order.get(result.region, len(region_order)), - task_order.get(result.task_name, len(task_order)), + tasks=ordinary_tasks, ) - ) - - interrupted = ( - target_outcome.interrupted if target_outcome is not None else False - ) or any(outcome.interrupted for outcome in region_outcomes) - failed = ( - target_outcome.failed if target_outcome is not None else False - ) or any(outcome.failed for outcome in region_outcomes) - status = ( - ExecutionStatus.INTERRUPTED - if interrupted - else ExecutionStatus.ERROR - if failed - else ExecutionStatus.SUCCESS - ) + task_results = ordinary_execution.task_results + benchmark = ordinary_execution.benchmark + status = _aggregate_task_result_status(task_results) error = None except Exception as runtime_error: status = ExecutionStatus.ERROR @@ -716,6 +1054,25 @@ def _provider_runtime_benchmark( if not isinstance(benchmark, dict): return None + return _augment_provider_runtime_benchmark( + benchmark=benchmark, + region_outcomes=region_outcomes, + region_execution_seconds=region_execution_seconds, + target_outcome=target_outcome, + target_execution_seconds=target_execution_seconds, + ) + + +def _augment_provider_runtime_benchmark( + *, + benchmark: dict[str, object], + region_outcomes: list[_ProviderRegionOutcome], + region_execution_seconds: float, + target_outcome: _ProviderRegionOutcome | None = None, + target_execution_seconds: float = 0.0, +) -> dict[str, object]: + """Attach engine-owned execution timings to provider benchmark data.""" + benchmark["region_execution_seconds"] = region_execution_seconds if target_outcome is not None: benchmark["target_execution_seconds"] = target_execution_seconds @@ -737,6 +1094,25 @@ def _provider_runtime_benchmark( return benchmark +def _settled_provider_region_outcomes( + *, regions: list[str], tasks: list[ResolvedTask], skip_reason: str +) -> list[_ProviderRegionOutcome]: + """Settle unstarted region tasks without constructing runtime sessions.""" + + return [ + _provider_region_outcome( + region=region, + task_results=[ + _skipped_task_result(task=task, region=region, skip_reason=skip_reason) + for task in tasks + ], + interrupted=skip_reason == "cancelled_before_start", + duration_seconds=0.0, + ) + for region in regions + ] + + def _execute_provider_regions( *, execution_target: ExecutionTarget, @@ -745,10 +1121,24 @@ def _execute_provider_regions( regions: list[str], tasks: list[ResolvedTask] | None = None, dependency_results: dict[str, TaskResult] | None = None, + dependency_activated_task_ids: frozenset[str] | None = None, + initial_stop_reason: str | None = None, ) -> list[_ProviderRegionOutcome]: target_cancel_event = threading.Event() if context.max_parallel_regions == 1: - return _execute_provider_regions_sequential( + outcomes = _execute_provider_regions_sequential( + execution_target=execution_target, + runtime=runtime, + context=context, + regions=regions, + target_cancel_event=target_cancel_event, + tasks=tasks, + dependency_results=dependency_results, + dependency_activated_task_ids=dependency_activated_task_ids, + initial_stop_reason=initial_stop_reason, + ) + else: + outcomes = _execute_provider_regions_parallel( execution_target=execution_target, runtime=runtime, context=context, @@ -756,17 +1146,31 @@ def _execute_provider_regions( target_cancel_event=target_cancel_event, tasks=tasks, dependency_results=dependency_results, + dependency_activated_task_ids=dependency_activated_task_ids, + initial_stop_reason=initial_stop_reason, ) - return _execute_provider_regions_parallel( - execution_target=execution_target, - runtime=runtime, - context=context, - regions=regions, - target_cancel_event=target_cancel_event, - tasks=tasks, - dependency_results=dependency_results, + completed_regions = {outcome.region for outcome in outcomes} + missing_regions = [region for region in regions if region not in completed_regions] + if not missing_regions: + return outcomes + + skip_reason = ( + "fail_fast" + if context.fail_fast + and ( + context.fail_fast_event.is_set() + or any(outcome.failed for outcome in outcomes) + ) + else "cancelled_before_start" + ) + settled_tasks = tasks if tasks is not None else context.tasks + outcomes.extend( + _settled_provider_region_outcomes( + regions=missing_regions, tasks=settled_tasks, skip_reason=skip_reason + ) ) + return outcomes def _execute_provider_regions_sequential( @@ -778,6 +1182,8 @@ def _execute_provider_regions_sequential( target_cancel_event: threading.Event, tasks: list[ResolvedTask] | None = None, dependency_results: dict[str, TaskResult] | None = None, + dependency_activated_task_ids: frozenset[str] | None = None, + initial_stop_reason: str | None = None, ) -> list[_ProviderRegionOutcome]: region_outcomes: list[_ProviderRegionOutcome] = [] for region in regions: @@ -789,10 +1195,15 @@ def _execute_provider_regions_sequential( target_cancel_event=target_cancel_event, tasks=tasks, dependency_results=dependency_results, + dependency_activated_task_ids=dependency_activated_task_ids, + initial_stop_reason=initial_stop_reason, + lazy_session=initial_stop_reason is not None, ) region_outcomes.append(outcome) - if outcome.interrupted or outcome.failed: + if initial_stop_reason is None and ( + outcome.interrupted or (context.fail_fast and outcome.failed) + ): target_cancel_event.set() break @@ -808,6 +1219,8 @@ def _execute_provider_regions_parallel( target_cancel_event: threading.Event, tasks: list[ResolvedTask] | None = None, dependency_results: dict[str, TaskResult] | None = None, + dependency_activated_task_ids: frozenset[str] | None = None, + initial_stop_reason: str | None = None, ) -> list[_ProviderRegionOutcome]: pending_regions: deque[str] = deque(regions) active_futures: set[Future[_ProviderRegionOutcome]] = set() @@ -822,7 +1235,13 @@ def _execute_provider_regions_parallel( while ( pending_regions and not target_cancel_event.is_set() - and not context.cancel_event.is_set() + and ( + initial_stop_reason is not None + or ( + not context.cancel_event.is_set() + and not context.fail_fast_event.is_set() + ) + ) and len(active_futures) < region_worker_limit ): region = pending_regions.popleft() @@ -835,6 +1254,9 @@ def _execute_provider_regions_parallel( target_cancel_event=target_cancel_event, tasks=tasks, dependency_results=dependency_results, + dependency_activated_task_ids=dependency_activated_task_ids, + initial_stop_reason=initial_stop_reason, + lazy_session=initial_stop_reason is not None, ) active_futures.add(future) @@ -852,17 +1274,404 @@ def _execute_provider_regions_parallel( region_outcomes.append(outcome) - if outcome.interrupted or outcome.failed: + if initial_stop_reason is None and ( + outcome.interrupted or (context.fail_fast and outcome.failed) + ): target_cancel_event.set() pending_regions.clear() - if target_cancel_event.is_set() or context.cancel_event.is_set(): + if initial_stop_reason is None and ( + target_cancel_event.is_set() + or context.cancel_event.is_set() + or context.fail_fast_event.is_set() + ): for future in active_futures: future.cancel() return region_outcomes +def _execute_provider_task_graph( + *, + provider: Provider, + target: TargetDescriptor, + context: ExecutionContext, + execution_targets: list[ExecutionTarget], + configured_execution_target: ExecutionTarget | None, + benchmark_data: dict[str, object] | None, +) -> TargetResult: + """Execute a task graph using provider-owned runtime identities.""" + + plan = plan_task_instances( + tasks=context.tasks, + execution_targets=execution_targets, + configured_target=configured_execution_target, + ) + task_order = {task.id: index for index, task in enumerate(context.tasks)} + task_result_stages = _task_result_barrier_stages(context.tasks) + target_order = { + execution_target.id: index + for index, execution_target in enumerate(execution_targets) + } + admission_region_order = { + (execution_target.id, region): index + for execution_target in execution_targets + for index, region in enumerate(execution_target.regions) + } + + def admission_order(instance: TaskInstance) -> tuple[int, int, int]: + return ( + target_order[instance.execution_target.id], + admission_region_order[(instance.execution_target.id, instance.region)], + task_order[instance.task.id], + ) + + # Prefer completing ready work for one ordinary target-region coordinate + # before opening its next coordinate. Dependency checks still release + # cross-region fan-in when required. Configured-target instances retain + # their declaration-relative positions. + ordered_ordinary_instances = iter( + sorted( + ( + instance + for instance in plan.instances + if instance.task.scope is not TaskScope.CONFIGURED_TARGET + ), + key=admission_order, + ) + ) + plan = replace( + plan, + instances=tuple( + instance + if instance.task.scope is TaskScope.CONFIGURED_TARGET + else next(ordered_ordinary_instances) + for instance in plan.instances + ), + ) + runtime_cache = _SingleFlightCache() + session_cache = _SingleFlightCache() + created_runtimes: dict[tuple[bool, str], ProviderExecutionRuntime] = {} + created_sessions: set[tuple[bool, str, str]] = set() + runtime_benchmarks: dict[tuple[bool, str], dict[str, object]] = {} + runtime_started_at: dict[tuple[bool, str], str] = {} + runtime_started_perf: dict[tuple[bool, str], float] = {} + runtime_ended_at: dict[tuple[bool, str], str] = {} + runtime_duration_seconds: dict[tuple[bool, str], float] = {} + lifecycle_lock = threading.Lock() + lifecycle_states: dict[tuple[bool, str, str], CoordinateLifecycleState] = {} + + def lifecycle_key(instance: TaskInstance) -> tuple[bool, str, str]: + return ( + instance.task.scope is TaskScope.CONFIGURED_TARGET, + instance.execution_target.id, + instance.region, + ) + + for planned_instance in plan.instances: + key = lifecycle_key(planned_instance) + lifecycle_states.setdefault( + key, CoordinateLifecycleState() + ).remaining_instances += 1 + + def runtime_for_instance(instance: TaskInstance) -> ProviderExecutionRuntime: + is_configured = instance.task.scope is TaskScope.CONFIGURED_TARGET + runtime_key = (is_configured, instance.execution_target.id) + + def create_runtime() -> object: + started_at = datetime.datetime.now(datetime.UTC).isoformat() + started_perf = time.perf_counter() + if is_configured: + runtime = cast( + ConfiguredTargetProvider, provider + ).prepare_configured_target_runtime( + target=target, + execution_target=instance.execution_target, + context=context, + ) + else: + runtime = provider.prepare_execution_runtime( + target=target, + execution_target=instance.execution_target, + context=context, + ) + with lifecycle_lock: + created_runtimes[runtime_key] = runtime + runtime_started_at[runtime_key] = started_at + runtime_started_perf[runtime_key] = started_perf + return runtime + + runtime, _cache_hit, _shared_wait = runtime_cache.get_or_create( + key=runtime_key, create=create_runtime + ) + return cast(ProviderExecutionRuntime, runtime) + + def session_for_instance(instance: TaskInstance) -> object: + is_configured = instance.task.scope is TaskScope.CONFIGURED_TARGET + session_key = (is_configured, instance.execution_target.id, instance.region) + session_request_started_perf = time.perf_counter() + + def create_session() -> object: + runtime = runtime_for_instance(instance) + started_perf = time.perf_counter() + session = runtime.build_session(region=instance.region) + with lifecycle_lock: + created_sessions.add(session_key) + lifecycle_states[session_key].started_perf = started_perf + return session + + session, cache_hit, _shared_wait = session_cache.get_or_create( + key=session_key, create=create_session + ) + if instance.task.scope is TaskScope.REGION: + with lifecycle_lock: + lifecycle_state = lifecycle_states[session_key] + if lifecycle_state.region_started_perf is None: + lifecycle_state.region_started_perf = ( + time.perf_counter() + if cache_hit + else session_request_started_perf + ) + return session + + def execute_instance( + instance: TaskInstance, dependency_results: DependencyResults + ) -> TaskResult: + return _run_task_instance( + instance=instance, + dependency_results=dependency_results, + session=session_for_instance(instance), + context=context, + ) + + def record_settled_instance(instance: TaskInstance, result: TaskResult) -> None: + key = lifecycle_key(instance) + with lifecycle_lock: + lifecycle_state = lifecycle_states[key] + settled = lifecycle_state.record_settlement( + result=result, + region_scoped=instance.task.scope is TaskScope.REGION, + ended_perf=time.perf_counter(), + ) + if not settled or key not in created_sessions: + return + is_configured, execution_target_id, region = key + runtime = created_runtimes[(is_configured, execution_target_id)] + started_perf = lifecycle_state.started_perf + failed = lifecycle_state.failed + interrupted = lifecycle_state.interrupted + if started_perf is None: + raise RuntimeError("Created session has no lifecycle start time") + ended_perf = time.perf_counter() + runtime.record_region_outcome( + region=region, + duration_seconds=ended_perf - started_perf, + failed=failed, + interrupted=interrupted, + ) + + recorder = BenchmarkRecorder(data=benchmark_data) + try: + with recorder.phase("entity_execution_seconds"): + schedule = execute_task_instance_plan( + plan=plan, + execute=execute_instance, + max_workers=max(1, target.max_workers * context.max_parallel_regions), + max_active_execution_targets=target.max_workers, + max_active_coordinates_per_execution_target=( + context.max_parallel_regions + ), + cancel_event=context.cancel_event, + fail_fast=context.fail_fast, + external_fail_fast_event=context.fail_fast_event, + on_instance_settled=record_settled_instance, + ) + for runtime_key, runtime in created_runtimes.items(): + benchmark = getattr(runtime, "benchmark", None) + if callable(benchmark): + benchmark = benchmark() + if isinstance(benchmark, dict): + runtime_benchmarks[runtime_key] = benchmark + finally: + for runtime_key, runtime in list(created_runtimes.items()): + try: + runtime.close() + finally: + runtime_ended_at[runtime_key] = datetime.datetime.now( + datetime.UTC + ).isoformat() + runtime_duration_seconds[runtime_key] = ( + time.perf_counter() - runtime_started_perf[runtime_key] + ) + + task_results_by_execution_target: dict[str, list[ScheduledTaskResult]] = {} + configured_results: list[TaskResult] = [] + for scheduled_result in schedule.results: + if scheduled_result.key.scope is TaskScope.CONFIGURED_TARGET: + configured_results.append(scheduled_result.result) + continue + task_results_by_execution_target.setdefault( + scheduled_result.key.execution_target_id, [] + ).append(scheduled_result) + + entity_results: list[EntityResult] = [] + for execution_target in execution_targets: + target_task_results = task_results_by_execution_target.get( + execution_target.id, [] + ) + if not target_task_results: + continue + + region_order = { + region: index for index, region in enumerate(execution_target.regions) + } + target_task_results.sort( + key=lambda item: ( + task_result_stages[item.key.task_id], + 0 if item.key.scope is TaskScope.TARGET else 1, + region_order.get(item.key.region, len(region_order)), + task_order[item.key.task_id], + ) + ) + results = [item.result for item in target_task_results] + status = _aggregate_task_result_status(results) + target_scope_results = [ + item.result + for item in target_task_results + if item.key.scope is TaskScope.TARGET + ] + target_outcome = ( + _provider_region_outcome( + region=target_scope_results[0].region, + task_results=target_scope_results, + duration_seconds=sum( + result.duration_seconds for result in target_scope_results + ), + ) + if target_scope_results + else None + ) + region_results_by_region: dict[str, list[TaskResult]] = {} + for item in target_task_results: + if item.key.scope is TaskScope.REGION: + region_results_by_region.setdefault(item.key.region, []).append( + item.result + ) + + region_outcomes: list[_ProviderRegionOutcome] = [] + region_lifecycle_states: list[CoordinateLifecycleState] = [] + for region in execution_target.regions: + lifecycle_state = lifecycle_states.get((False, execution_target.id, region)) + if ( + lifecycle_state is None + or lifecycle_state.region_started_perf is None + or lifecycle_state.region_ended_perf is None + ): + continue + region_results = region_results_by_region.get(region, []) + if not region_results: + continue + region_outcomes.append( + _provider_region_outcome( + region=region, + task_results=region_results, + duration_seconds=( + lifecycle_state.region_ended_perf + - lifecycle_state.region_started_perf + ), + ) + ) + region_lifecycle_states.append(lifecycle_state) + + provider_benchmark = runtime_benchmarks.get((False, execution_target.id)) + region_execution_seconds = ( + max( + lifecycle_state.region_ended_perf + for lifecycle_state in region_lifecycle_states + if lifecycle_state.region_ended_perf is not None + ) + - min( + lifecycle_state.region_started_perf + for lifecycle_state in region_lifecycle_states + if lifecycle_state.region_started_perf is not None + ) + if region_lifecycle_states + else 0.0 + ) + runtime_benchmark = ( + _augment_provider_runtime_benchmark( + benchmark=provider_benchmark, + region_outcomes=region_outcomes, + region_execution_seconds=region_execution_seconds, + target_outcome=target_outcome, + target_execution_seconds=( + target_outcome.duration_seconds + if target_outcome is not None + else 0.0 + ), + ) + if provider_benchmark is not None + else None + ) + + runtime_key = (False, execution_target.id) + started_at = ( + runtime_started_at[runtime_key] + if runtime_key in runtime_started_at + else min(result.started_at for result in results) + ) + ended_at = ( + runtime_ended_at[runtime_key] + if runtime_key in runtime_ended_at + else max(result.ended_at for result in results) + ) + entity_results.append( + EntityResult( + id=execution_target.id, + name=execution_target.name, + type=execution_target.type, + provider=execution_target.provider, + metadata=dict(execution_target.metadata), + status=status, + started_at=started_at, + ended_at=ended_at, + duration_seconds=( + runtime_duration_seconds[runtime_key] + if runtime_key in runtime_duration_seconds + else ( + datetime.datetime.fromisoformat(ended_at) + - datetime.datetime.fromisoformat(started_at) + ).total_seconds() + ), + tasks=results, + benchmark=runtime_benchmark, + ) + ) + + entity_results.sort(key=lambda result: (result.name.lower(), result.id)) + _record_entity_execution_metrics( + recorder=recorder, + entity_results=entity_results, + submitted_entity_count=( + len(execution_targets) + if any( + task.scope is not TaskScope.CONFIGURED_TARGET for task in context.tasks + ) + else 0 + ), + target=target, + context=context, + ) + return TargetResult.create( + target_name=target.name, + provider=target.provider, + dry_run=context.dry_run, + entities=entity_results, + tasks=configured_results, + benchmark=recorder.data, + ) + + def _execute_provider_targets( *, provider: Provider, @@ -870,7 +1679,28 @@ def _execute_provider_targets( context: ExecutionContext, execution_targets: list[ExecutionTarget], benchmark_data: dict[str, object] | None, + configured_execution_target: ExecutionTarget | None = None, ) -> TargetResult: + has_configured_tasks = any( + task.scope is TaskScope.CONFIGURED_TARGET for task in context.tasks + ) + ordinary_tasks = [ + task for task in context.tasks if task.scope is not TaskScope.CONFIGURED_TARGET + ] + if has_configured_tasks or _requires_task_instance_scheduler(ordinary_tasks): + if has_configured_tasks and configured_execution_target is None: + raise ValueError( + "configured-target tasks require a provider-owned execution identity" + ) + return _execute_provider_task_graph( + provider=provider, + target=target, + context=context, + execution_targets=execution_targets, + configured_execution_target=configured_execution_target, + benchmark_data=benchmark_data, + ) + entity_results: list[EntityResult] = [] recorder = BenchmarkRecorder(data=benchmark_data) @@ -884,6 +1714,7 @@ def _execute_provider_targets( while ( pending_targets and not context.cancel_event.is_set() + and not context.fail_fast_event.is_set() and len(active_futures) < target.max_workers ): execution_target = pending_targets.popleft() @@ -912,7 +1743,7 @@ def _execute_provider_targets( entity_results.append(entity_result) if context.fail_fast and entity_result.status.is_unsuccessful: - context.cancel_event.set() + context.fail_fast_event.set() pending_targets.clear() for active_future in active_futures: active_future.cancel() @@ -921,34 +1752,13 @@ def _execute_provider_targets( raise entity_results.sort(key=lambda result: (result.name.lower(), result.id)) - - if recorder.enabled: - entity_execution_window_seconds = _entity_execution_window_seconds( - entity_results - ) - sum_entity_duration_seconds = sum( - result.duration_seconds for result in entity_results - ) - recorder.update( - { - "submitted_entity_count": len(execution_targets), - "completed_entity_count": len(entity_results), - "max_workers": target.max_workers, - "entity_execution_window_seconds": entity_execution_window_seconds, - "sum_entity_duration_seconds": sum_entity_duration_seconds, - "max_entity_duration_seconds": max( - (result.duration_seconds for result in entity_results), default=0.0 - ), - "worker_utilization": _entity_worker_utilization( - sum_entity_duration_seconds=sum_entity_duration_seconds, - max_workers=target.max_workers, - entity_execution_window_seconds=entity_execution_window_seconds, - ), - "max_parallel_regions": context.max_parallel_regions, - "entity_region_limit": target.max_workers - * context.max_parallel_regions, - } - ) + _record_entity_execution_metrics( + recorder=recorder, + entity_results=entity_results, + submitted_entity_count=len(execution_targets), + target=target, + context=context, + ) return TargetResult.create( target_name=target.name, @@ -984,6 +1794,43 @@ def _entity_worker_utilization( return sum_entity_duration_seconds / (max_workers * entity_execution_window_seconds) +def _record_entity_execution_metrics( + *, + recorder: BenchmarkRecorder, + entity_results: list[EntityResult], + submitted_entity_count: int, + target: TargetDescriptor, + context: ExecutionContext, +) -> None: + """Record shared target-worker metrics for either execution strategy.""" + + if not recorder.enabled: + return + entity_execution_window_seconds = _entity_execution_window_seconds(entity_results) + sum_entity_duration_seconds = sum( + result.duration_seconds for result in entity_results + ) + recorder.update( + { + "submitted_entity_count": submitted_entity_count, + "completed_entity_count": len(entity_results), + "max_workers": target.max_workers, + "entity_execution_window_seconds": entity_execution_window_seconds, + "sum_entity_duration_seconds": sum_entity_duration_seconds, + "max_entity_duration_seconds": max( + (result.duration_seconds for result in entity_results), default=0.0 + ), + "worker_utilization": _entity_worker_utilization( + sum_entity_duration_seconds=sum_entity_duration_seconds, + max_workers=target.max_workers, + entity_execution_window_seconds=entity_execution_window_seconds, + ), + "max_parallel_regions": context.max_parallel_regions, + "entity_region_limit": target.max_workers * context.max_parallel_regions, + } + ) + + def run_prepared_target(*, prepared_target: PreparedTarget) -> TargetExecutionOutcome: if prepared_target.context is None: raise ValueError("Prepared target is not runnable.") @@ -1033,6 +1880,7 @@ def run_prepared_target(*, prepared_target: PreparedTarget) -> TargetExecutionOu provider=provider, target=target, execution_targets=execution_plan.execution_targets, + configured_execution_target=execution_plan.configured_target, benchmark_data=benchmark_data, context=context, ) @@ -1200,13 +2048,13 @@ def _run_target_pipeline( outcome = future.result() target_results_by_index[outcome.index] = outcome.target_result - if outcome.cancelled: + if outcome.target_result.has_failures: execution_state = _elevate_state( - execution_state, EngineState.CANCELLED + execution_state, EngineState.COMPLETED_WITH_FAILURES ) - elif outcome.target_result.has_failures: + elif outcome.cancelled: execution_state = _elevate_state( - execution_state, EngineState.COMPLETED_WITH_FAILURES + execution_state, EngineState.CANCELLED ) auth_results = [ diff --git a/src/anvil/schemas/common.schema.v2.json b/src/anvil/schemas/common.schema.v2.json index e6bf071..44b9503 100644 --- a/src/anvil/schemas/common.schema.v2.json +++ b/src/anvil/schemas/common.schema.v2.json @@ -6,29 +6,132 @@ "$defs": { "taskEntry": { "type": "object", + "description": "One configured invocation of a discovered Python task component.", + "examples": [ + { + "id": "detach_guardrails", + "name": "reconcile_config_guardrails", + "metadata": { + "attachment_state": "absent" + } + } + ], "required": [ "name" ], "additionalProperties": false, "properties": { + "id": { + "type": "string", + "minLength": 1, + "description": "Unique invocation ID for this configured task. When omitted, the discovered component name is used.", + "examples": [ + "detach_guardrails" + ] + }, "name": { "type": "string", "minLength": 1, - "description": "Task name (must match the discovered task component name)." + "description": "Discovered Python task component name.", + "examples": [ + "reconcile_config_guardrails" + ] + }, + "metadata": { + "type": "object", + "default": {}, + "additionalProperties": true, + "description": "Static operator-configured context merged over target metadata for this invocation.", + "examples": [ + { + "attachment_state": "absent" + } + ] }, "depends_on": { "type": "array", + "default": [], "items": { "type": "string", - "minLength": 1 + "minLength": 1, + "description": "Effective task invocation ID that must reach a terminal state first.", + "examples": [ + "detach_guardrails" + ] }, "uniqueItems": true, - "description": "List of task names that must complete before this task runs." + "description": "Effective task invocation IDs that control ordering, eligibility, and dependency-data availability.", + "examples": [ + [ + "detach_guardrails", + "cleanup_config" + ] + ] }, - "optional": { + "always_run": { "type": "boolean", "default": false, - "description": "If true, task failure does not fail the account or block dependent tasks." + "description": "If true, run after every dependency reaches a terminal state even when a dependency was unsuccessful.", + "examples": [ + true + ] + }, + "dependency_data": { + "type": "object", + "default": {}, + "propertyNames": { + "type": "string", + "minLength": 1, + "description": "Non-empty local input name chosen by the consuming task.", + "examples": [ + "attachments" + ] + }, + "additionalProperties": { + "$ref": "#/$defs/dependencyDataReference" + }, + "description": "Runtime inputs selected from directly declared dependency TaskResults.", + "examples": [ + { + "attachments": { + "task_id": "detach_guardrails", + "path": "result.attachments" + } + } + ] + } + } + }, + "dependencyDataReference": { + "type": "object", + "required": [ + "task_id" + ], + "additionalProperties": false, + "description": "Selection of a complete dependency TaskResult or one readable dotted path within it.", + "examples": [ + { + "task_id": "detach_guardrails", + "path": "result.attachments" + } + ], + "properties": { + "task_id": { + "type": "string", + "minLength": 1, + "description": "Effective ID of a task also listed directly in the consumer's depends_on.", + "examples": [ + "detach_guardrails" + ] + }, + "path": { + "type": "string", + "pattern": "^(?:result(?:\\.[A-Za-z_][A-Za-z0-9_]*)*|status|error|actions)$", + "description": "Optional dotted selection: result, nested result fields, status, error, or actions. List indexing is not supported.", + "examples": [ + "result.attachments", + "status" + ] } } }, @@ -137,7 +240,7 @@ "minLength": 1 }, "uniqueItems": true, - "description": "Provider-owned execution target ID list." + "description": "Provider-owned execution target IDs or filter keywords." }, "baseTargetProperties": { "name": { diff --git a/src/anvil/task_context.py b/src/anvil/task_context.py index a291e75..57ebe02 100644 --- a/src/anvil/task_context.py +++ b/src/anvil/task_context.py @@ -1,8 +1,102 @@ from __future__ import annotations +from collections.abc import Mapping, Sequence +from copy import deepcopy from dataclasses import dataclass, fields - +from typing import cast from anvil.actions import ActionRecorder +from anvil.results import TaskResult + + +class TaskInputResolutionError(ValueError): + """Raised when configured dependency data cannot be resolved.""" + + +def merge_task_metadata( + *, target_metadata: Mapping[str, object], task_metadata: Mapping[str, object] +) -> dict[str, object]: + """Recursively merge task metadata over target metadata. + + Args: + target_metadata: Metadata inherited from the configured target. + task_metadata: Static metadata declared for one task invocation. + + Returns: + A recursively merged metadata mapping. ``TaskCallContext.to_kwargs()`` + performs the invocation's required deep copy. + """ + + merged = dict(target_metadata) + for key, task_value in task_metadata.items(): + target_value = merged.get(key) + if isinstance(target_value, Mapping) and isinstance(task_value, Mapping): + merged[key] = merge_task_metadata( + target_metadata=cast(Mapping[str, object], target_value), + task_metadata=cast(Mapping[str, object], task_value), + ) + else: + merged[key] = task_value + return merged + + +def _select_dependency_path( + *, local_name: str, path: str, task_result: TaskResult +) -> object: + """Select one configured dotted path from a task result.""" + + root, *nested_keys = path.split(".") + selected: object = getattr(task_result, root) + for key in nested_keys: + if not isinstance(selected, Mapping) or key not in selected: + raise TaskInputResolutionError( + f"dependency_data.{local_name} path '{path}' does not exist" + ) + selected = cast(Mapping[str, object], selected)[key] + return selected + + +def resolve_dependency_data( + *, + references: Mapping[str, Mapping[str, str]], + dependency_results: Mapping[str, TaskResult | Sequence[TaskResult]], +) -> dict[str, object]: + """Resolve configured dependency-data references. + + Args: + references: Local input names mapped to producer IDs and optional paths. + dependency_results: Available results keyed by effective producer ID. + + Returns: + Dependency values keyed by the configured local input names. + + Raises: + TaskInputResolutionError: If a producer result or selected path is missing. + """ + + resolved: dict[str, object] = {} + for local_name, reference in references.items(): + task_id = reference["task_id"] + if task_id not in dependency_results: + raise TaskInputResolutionError( + f"dependency_data.{local_name} has no result for task '{task_id}'" + ) + + available = dependency_results[task_id] + is_multiple = not isinstance(available, TaskResult) + results = list(available) if is_multiple else [available] + path = reference.get("path") + values: list[object] = [ + ( + result + if path is None + else _select_dependency_path( + local_name=local_name, path=path, task_result=result + ) + ) + for result in results + ] + resolved[local_name] = values if is_multiple else values[0] + return resolved @dataclass(frozen=True, slots=True) @@ -17,6 +111,7 @@ class TaskCallContext: session: object dry_run: bool metadata: dict[str, object] + dependency_data: dict[str, object] actions: ActionRecorder @classmethod @@ -31,5 +126,6 @@ def to_kwargs(self) -> dict[str, object]: invocation_kwargs = { field.name: getattr(self, field.name) for field in fields(self) } - invocation_kwargs["metadata"] = dict(self.metadata) + invocation_kwargs["metadata"] = deepcopy(self.metadata) + invocation_kwargs["dependency_data"] = deepcopy(self.dependency_data) return invocation_kwargs diff --git a/src/anvil/task_errors.py b/src/anvil/task_errors.py new file mode 100644 index 0000000..c5f996f --- /dev/null +++ b/src/anvil/task_errors.py @@ -0,0 +1,28 @@ +from __future__ import annotations + +import json + + +class TaskExecutionError(RuntimeError): + """Report task failure while retaining JSON-serializable partial output.""" + + def __init__(self, message: str, *, partial_result: object) -> None: + """Initialize a task execution failure. + + Args: + message: Actionable failure detail recorded on the task result. + partial_result: JSON-serializable output produced before failure. + + Raises: + TypeError: If the partial result is not JSON-serializable. + """ + + try: + json.dumps(partial_result) + except (TypeError, ValueError) as error: + raise TypeError( + "TaskExecutionError partial_result must be JSON-serializable" + ) from error + + super().__init__(message) + self.partial_result = partial_result diff --git a/src/anvil/task_loader.py b/src/anvil/task_loader.py index 03a4e03..46e4fae 100644 --- a/src/anvil/task_loader.py +++ b/src/anvil/task_loader.py @@ -2,10 +2,12 @@ import importlib import importlib.util +import json +import re import sys from collections import defaultdict, deque from collections.abc import Callable, Mapping, Sequence -from dataclasses import dataclass +from dataclasses import dataclass, field from enum import StrEnum from functools import lru_cache from importlib.metadata import entry_points @@ -38,6 +40,7 @@ class TaskScope(StrEnum): REGION = "region" TARGET = "target" + CONFIGURED_TARGET = "configured_target" @dataclass(frozen=True, slots=True) @@ -45,21 +48,45 @@ class ResolvedTask: name: str run: Callable depends_on: list[str] - optional: bool scope: TaskScope = TaskScope.REGION + id: str = "" + always_run: bool = False + metadata: dict[str, object] = field(default_factory=dict) + dependency_data: dict[str, dict[str, str]] = field(default_factory=dict) + + def __post_init__(self) -> None: + """Default an omitted invocation ID to the component name.""" + + if not self.id: + object.__setattr__(self, "id", self.name) @dataclass(frozen=True, slots=True) class TaskSpec: """Immutable normalized task declaration.""" + id: str name: str depends_on: tuple[str, ...] - optional: bool + always_run: bool + metadata_json: str + dependency_data: tuple[tuple[str, str, str | None], ...] TaskSpecKey = tuple[TaskSpec, ...] -CachedOrderedTask = tuple[tuple[str, Callable, tuple[str, ...], bool, TaskScope], ...] +CachedOrderedTask = tuple[ + tuple[ + str, + str, + Callable, + tuple[str, ...], + bool, + str, + tuple[tuple[str, str, str | None], ...], + TaskScope, + ], + ..., +] CachedAdjacency = tuple[tuple[str, tuple[str, ...]], ...] @@ -283,45 +310,151 @@ def _clear_task_caches() -> None: TaskSpecInput = Mapping[str, object] +DEPENDENCY_PATH_PATTERN = re.compile( + r"^(?:result(?:\.[A-Za-z_][A-Za-z0-9_]*)*|status|error|actions)$" +) + + +def _metadata_json(*, task_id: str, value: object) -> str: + """Return a deterministic cache representation of task metadata.""" + + if not isinstance(value, dict): + raise TaskConfigError(f"Task '{task_id}' metadata must be a mapping") + try: + return json.dumps(value, sort_keys=True, separators=(",", ":")) + except (TypeError, ValueError) as error: + raise TaskConfigError( + f"Task '{task_id}' metadata must contain JSON-serializable values" + ) from error + + +def _normalize_dependency_data( + *, task_id: str, value: object +) -> tuple[tuple[str, str, str | None], ...]: + """Validate dependency-data references and return a stable cache value.""" + + if not isinstance(value, dict): + raise TaskConfigError(f"Task '{task_id}' dependency_data must be a mapping") + + normalized: list[tuple[str, str, str | None]] = [] + for local_name, raw_reference in value.items(): + if not isinstance(local_name, str) or not local_name: + raise TaskConfigError( + f"Task '{task_id}' dependency_data names must be non-empty strings" + ) + if not isinstance(raw_reference, dict): + raise TaskConfigError( + f"Task '{task_id}' dependency_data.{local_name} must be a mapping" + ) + unknown_properties = set(raw_reference) - {"task_id", "path"} + if unknown_properties: + unknown_display = ", ".join( + sorted(str(name) for name in unknown_properties) + ) + raise TaskConfigError( + f"Task '{task_id}' dependency_data.{local_name} has unknown " + f"properties: {unknown_display}" + ) + + producer_id = raw_reference.get("task_id") + if not isinstance(producer_id, str) or not producer_id: + raise TaskConfigError( + f"Task '{task_id}' dependency_data.{local_name}.task_id " + "must be a non-empty string" + ) + path = raw_reference.get("path") + if path is not None and ( + not isinstance(path, str) or DEPENDENCY_PATH_PATTERN.fullmatch(path) is None + ): + raise TaskConfigError( + f"Task '{task_id}' dependency_data.{local_name}.path must be " + "result, a dotted result field, status, error, or actions" + ) + normalized.append((local_name, producer_id, path)) + + return tuple(sorted(normalized)) def _normalize_task_specs(task_specs: Sequence[TaskSpecInput]) -> TaskSpecKey: """Validate task declarations once while preserving declaration order.""" normalized_specs: list[TaskSpec] = [] - seen_names: set[str] = set() + seen_ids: set[str] = set() + duplicate_ids: set[str] = set() + component_occurrences: dict[str, list[bool]] = defaultdict(list) for spec in task_specs: name = spec.get("name") if not isinstance(name, str) or not name: raise TaskConfigError("task name must be a non-empty string") - if name in seen_names: - raise TaskConfigError(f"Duplicate task name detected: '{name}'") - seen_names.add(name) + + explicit_id = "id" in spec + raw_id = spec.get("id", name) + if not isinstance(raw_id, str) or not raw_id: + raise TaskConfigError("task id must be a non-empty string") + task_id = raw_id + if task_id in seen_ids: + duplicate_ids.add(task_id) + seen_ids.add(task_id) + component_occurrences[name].append(explicit_id) raw_depends_on = spec.get("depends_on", []) if not isinstance(raw_depends_on, list): - raise TaskConfigError(f"Task '{name}' depends_on must be a list of strings") + raise TaskConfigError( + f"Task '{task_id}' depends_on must be a list of strings" + ) depends_on: list[str] = [] for dependency in raw_depends_on: - if not isinstance(dependency, str): + if not isinstance(dependency, str) or not dependency: raise TaskConfigError( - f"Task '{name}' depends_on must be a list of strings" + f"Task '{task_id}' depends_on must be a list of non-empty strings" ) depends_on.append(dependency) if len(set(depends_on)) != len(depends_on): raise TaskConfigError( - f"Task '{name}' depends_on must not contain duplicates" + f"Task '{task_id}' depends_on must not contain duplicates" ) - optional = spec.get("optional", False) - if not isinstance(optional, bool): - raise TaskConfigError(f"Task '{name}' optional must be a boolean") + always_run = spec.get("always_run", False) + if not isinstance(always_run, bool): + raise TaskConfigError(f"Task '{task_id}' always_run must be a boolean") + if always_run and not depends_on: + raise TaskConfigError( + f"Task '{task_id}' sets always_run but has no dependencies" + ) + + metadata_json = _metadata_json(task_id=task_id, value=spec.get("metadata", {})) + dependency_data = _normalize_dependency_data( + task_id=task_id, value=spec.get("dependency_data", {}) + ) normalized_specs.append( - TaskSpec(name=name, depends_on=tuple(depends_on), optional=optional) + TaskSpec( + id=task_id, + name=name, + depends_on=tuple(depends_on), + always_run=always_run, + metadata_json=metadata_json, + dependency_data=dependency_data, + ) ) + repeated_without_explicit_ids = [ + name + for name, explicit_ids in component_occurrences.items() + if len(explicit_ids) > 1 and not all(explicit_ids) + ] + if repeated_without_explicit_ids: + names = ", ".join(sorted(repeated_without_explicit_ids)) + raise TaskConfigError( + f"Duplicate task name detected: '{names}'. When a component name is " + "configured more than once, every occurrence must have an explicit " + "unique ID" + ) + if duplicate_ids: + duplicates = ", ".join(sorted(duplicate_ids)) + raise TaskConfigError(f"Duplicate task ID detected: '{duplicates}'") + return tuple(normalized_specs) @@ -337,7 +470,7 @@ def _task_scope(*, task_name: str, run: Callable) -> TaskScope: except (TypeError, ValueError) as error: raise TaskConfigError( f"Task '{task_name}' has invalid TASK_SCOPE {raw_scope!r}; " - "expected 'region' or 'target'" + "expected 'configured_target', 'target', or 'region'" ) from error @@ -355,7 +488,8 @@ def _normalize_supported_task_scopes( ) except (TypeError, ValueError) as error: raise TaskConfigError( - "supported_task_scopes must contain only 'region' or 'target'" + "supported_task_scopes must contain only 'configured_target', " + "'target', or 'region'" ) from error @@ -365,13 +499,31 @@ def _build_resolved_execution( return ResolvedExecution( ordered=[ ResolvedTask( + id=task_id, name=name, run=run, depends_on=list(depends_on), - optional=optional, + always_run=always_run, + metadata=json.loads(metadata_json), + dependency_data={ + local_name: { + "task_id": producer_id, + **({"path": path} if path is not None else {}), + } + for local_name, producer_id, path in dependency_data + }, scope=scope, ) - for name, run, depends_on, optional, scope in ordered + for ( + task_id, + name, + run, + depends_on, + always_run, + metadata_json, + dependency_data, + scope, + ) in ordered ], adjacency={name: list(children) for name, children in adjacency}, ) @@ -383,48 +535,39 @@ def _resolve_tasks_cached( task_specs: TaskSpecKey, supported_task_scopes: tuple[TaskScope, ...], ) -> tuple[CachedOrderedTask, CachedAdjacency]: - spec_by_name = {spec.name: spec for spec in task_specs} + spec_by_id = {spec.id: spec for spec in task_specs} - _validate_dependencies(spec_by_name) + _validate_dependencies(spec_by_id) - ordered_names, adjacency = _topological_sort(spec_by_name) + ordered_ids, adjacency = _topological_sort(spec_by_id) loaded_tasks: dict[str, tuple[Callable, TaskScope]] = {} supported_scope_set = frozenset(supported_task_scopes) - for name in ordered_names: - run = _load_provider_task_callable(provider_name=provider_name, task_name=name) - scope = _task_scope(task_name=name, run=run) + for task_id in ordered_ids: + component_name = spec_by_id[task_id].name + run = _load_provider_task_callable( + provider_name=provider_name, task_name=component_name + ) + scope = _task_scope(task_name=component_name, run=run) if scope not in supported_scope_set: raise TaskConfigError( - f"Task '{name}' declares scope '{scope.value}', which provider " - f"'{provider_name}' does not support" - ) - loaded_tasks[name] = (run, scope) - - for name, spec in spec_by_name.items(): - if loaded_tasks[name][1] is not TaskScope.TARGET: - continue - regional_dependencies = [ - dependency - for dependency in spec.depends_on - if loaded_tasks[dependency][1] is TaskScope.REGION - ] - if regional_dependencies: - dependencies = ", ".join(regional_dependencies) - raise TaskConfigError( - f"TARGET task '{name}' cannot depend on REGION task(s): " - f"{dependencies}. TARGET tasks execute before regional fan-out" + f"Task '{task_id}' component '{component_name}' declares scope " + f"'{scope.value}', which provider '{provider_name}' does not support" ) + loaded_tasks[task_id] = (run, scope) ordered: CachedOrderedTask = tuple( ( - name, - loaded_tasks[name][0], - tuple(spec_by_name[name].depends_on), - spec_by_name[name].optional, - loaded_tasks[name][1], + task_id, + spec_by_id[task_id].name, + loaded_tasks[task_id][0], + tuple(spec_by_id[task_id].depends_on), + spec_by_id[task_id].always_run, + spec_by_id[task_id].metadata_json, + spec_by_id[task_id].dependency_data, + loaded_tasks[task_id][1], ) - for name in ordered_names + for task_id in ordered_ids ) frozen_adjacency: CachedAdjacency = tuple( (name, tuple(children)) for name, children in adjacency.items() @@ -433,34 +576,47 @@ def _resolve_tasks_cached( return ordered, frozen_adjacency -def _validate_dependencies(spec_by_name: dict[str, TaskSpec]) -> None: - names = set(spec_by_name.keys()) +def _validate_dependencies(spec_by_id: dict[str, TaskSpec]) -> None: + task_ids = set(spec_by_id) - for task_name, spec in spec_by_name.items(): - for dep in spec.depends_on: - if dep not in names: + for task_id, spec in spec_by_id.items(): + for dependency in spec.depends_on: + if dependency not in task_ids: raise TaskConfigError( - f"Task '{task_name}' depends on unknown task '{dep}'" + f"Task '{task_id}' depends on unknown task ID '{dependency}'" ) - if dep == task_name: - raise TaskConfigError(f"Task '{task_name}' cannot depend on itself") + if dependency == task_id: + raise TaskConfigError(f"Task '{task_id}' cannot depend on itself") + + direct_dependencies = set(spec.depends_on) + for local_name, producer_id, _ in spec.dependency_data: + if producer_id not in task_ids: + raise TaskConfigError( + f"Task '{task_id}' dependency_data.{local_name} references " + f"unknown task ID '{producer_id}'" + ) + if producer_id not in direct_dependencies: + raise TaskConfigError( + f"Task '{task_id}' dependency_data.{local_name} references " + f"'{producer_id}', which must appear directly in depends_on" + ) def _topological_sort( - spec_by_name: dict[str, TaskSpec], + spec_by_id: dict[str, TaskSpec], ) -> tuple[list[str], dict[str, list[str]]]: - names = list(spec_by_name.keys()) + task_ids = list(spec_by_id) graph: dict[str, list[str]] = defaultdict(list) - indegree: dict[str, int] = {name: 0 for name in names} + indegree: dict[str, int] = {task_id: 0 for task_id in task_ids} - for name, spec in spec_by_name.items(): - for dep in spec.depends_on: - graph[dep].append(name) - indegree[name] += 1 + for task_id, spec in spec_by_id.items(): + for dependency in spec.depends_on: + graph[dependency].append(task_id) + indegree[task_id] += 1 - queue = deque(name for name in names if indegree[name] == 0) + queue = deque(task_id for task_id in task_ids if indegree[task_id] == 0) ordered: list[str] = [] while queue: @@ -472,8 +628,11 @@ def _topological_sort( if indegree[child] == 0: queue.append(child) - if len(ordered) != len(names): - raise TaskConfigError("Cycle detected in task dependencies") + if len(ordered) != len(task_ids): + cycle_ids = ", ".join(task_id for task_id in task_ids if indegree[task_id] > 0) + raise TaskConfigError( + f"Cycle detected in task dependencies involving: {cycle_ids}" + ) return ordered, dict(graph) diff --git a/src/anvil/task_planner.py b/src/anvil/task_planner.py new file mode 100644 index 0000000..4f6f9a9 --- /dev/null +++ b/src/anvil/task_planner.py @@ -0,0 +1,271 @@ +"""Pure expansion of resolved tasks into deterministic execution instances.""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import dataclass + +from anvil.providers.base import ExecutionTarget +from anvil.task_loader import ResolvedTask, TaskScope + + +class TaskPlanningError(ValueError): + """Raised when tasks cannot be expanded over an execution topology.""" + + +@dataclass(frozen=True, slots=True) +class TaskInstanceKey: + """Stable identity for one planned task invocation.""" + + task_id: str + scope: TaskScope + execution_target_id: str + region: str + + +@dataclass(frozen=True, slots=True) +class TaskInstance: + """One task invocation and the instances that must settle before it.""" + + key: TaskInstanceKey + task: ResolvedTask + execution_target: ExecutionTarget + region: str + dependencies: tuple[TaskInstanceKey, ...] + + +@dataclass(frozen=True, slots=True) +class TaskInstancePlan: + """Deterministically ordered task instances and their outgoing edges.""" + + instances: tuple[TaskInstance, ...] + adjacency: dict[TaskInstanceKey, tuple[TaskInstanceKey, ...]] + + +_TaskShell = tuple[TaskInstanceKey, ResolvedTask, ExecutionTarget, str] + + +@dataclass(frozen=True, slots=True) +class _TaskShellIndex: + """Dependency lookup indexes for every instance of one resolved task.""" + + scope: TaskScope + all_keys: tuple[TaskInstanceKey, ...] + keys_by_target: dict[str, tuple[TaskInstanceKey, ...]] + keys_by_coordinate: dict[tuple[str, str], tuple[TaskInstanceKey, ...]] + + +def plan_task_instances( + *, + tasks: Sequence[ResolvedTask], + execution_targets: Sequence[ExecutionTarget], + configured_target: ExecutionTarget | None, +) -> TaskInstancePlan: + """Expand task declarations over provider-owned execution topology. + + Args: + tasks: Resolved tasks in dependency order. + execution_targets: Selected targets in provider-defined order. + configured_target: Concrete identity of the provider configuration owner. + + Returns: + A pure, deterministic execution-instance plan. + + Raises: + TaskPlanningError: If task references or target topology are invalid. + """ + + ordered_tasks = tuple(tasks) + ordered_targets = tuple(execution_targets) + _validate_tasks(ordered_tasks) + _validate_execution_targets(ordered_targets) + if configured_target is not None: + _validate_target(configured_target, label="configured target") + + if ( + any(task.scope is TaskScope.CONFIGURED_TARGET for task in ordered_tasks) + and configured_target is None + ): + raise TaskPlanningError( + "configured-target tasks require a concrete configured-target identity" + ) + + shells_by_task_id: dict[str, list[_TaskShell]] = {} + for task in ordered_tasks: + shells_by_task_id[task.id] = _expand_task( + task=task, + execution_targets=ordered_targets, + configured_target=configured_target, + ) + shell_indexes_by_task_id = { + task.id: _index_task_shells(task=task, shells=shells_by_task_id[task.id]) + for task in ordered_tasks + } + + instances: list[TaskInstance] = [] + for task in ordered_tasks: + for key, resolved_task, target, region in shells_by_task_id[task.id]: + dependencies: list[TaskInstanceKey] = [] + for dependency_id in task.depends_on: + matches = _matching_dependency_keys( + producer_index=shell_indexes_by_task_id[dependency_id], consumer=key + ) + if not matches: + raise TaskPlanningError( + f"task '{task.id}' has no matching instance of dependency " + f"'{dependency_id}' for target '{key.execution_target_id}' " + f"and region '{key.region}'" + ) + dependencies.extend(matches) + + instances.append( + TaskInstance( + key=key, + task=resolved_task, + execution_target=target, + region=region, + dependencies=tuple(dependencies), + ) + ) + + adjacency_lists = {instance.key: [] for instance in instances} + for instance in instances: + for dependency in instance.dependencies: + adjacency_lists[dependency].append(instance.key) + + return TaskInstancePlan( + instances=tuple(instances), + adjacency={key: tuple(children) for key, children in adjacency_lists.items()}, + ) + + +def _validate_tasks(tasks: Sequence[ResolvedTask]) -> None: + task_ids: set[str] = set() + for task in tasks: + if not task.id: + raise TaskPlanningError("task IDs must not be empty") + if task.id in task_ids: + raise TaskPlanningError(f"duplicate task ID '{task.id}'") + task_ids.add(task.id) + + for task in tasks: + for dependency_id in task.depends_on: + if dependency_id not in task_ids: + raise TaskPlanningError( + f"task '{task.id}' depends on unknown task ID '{dependency_id}'" + ) + + +def _validate_execution_targets(execution_targets: Sequence[ExecutionTarget]) -> None: + target_ids: set[str] = set() + for target in execution_targets: + _validate_target(target, label="execution target") + if target.id in target_ids: + raise TaskPlanningError(f"duplicate execution target ID '{target.id}'") + target_ids.add(target.id) + + +def _validate_target(target: ExecutionTarget, *, label: str) -> None: + if not target.id: + raise TaskPlanningError(f"{label} ID must not be empty") + if not target.regions: + raise TaskPlanningError( + f"{label} '{target.id}' must define at least one region" + ) + + regions: set[str] = set() + for region in target.regions: + if not region: + raise TaskPlanningError( + f"{label} '{target.id}' region names must not be empty" + ) + if region in regions: + raise TaskPlanningError( + f"{label} '{target.id}' has duplicate region '{region}'" + ) + regions.add(region) + + +def _expand_task( + *, + task: ResolvedTask, + execution_targets: Sequence[ExecutionTarget], + configured_target: ExecutionTarget | None, +) -> list[_TaskShell]: + if task.scope is TaskScope.CONFIGURED_TARGET: + if configured_target is None: + raise TaskPlanningError( + "configured-target tasks require a concrete configured-target identity" + ) + return [_task_shell(task, configured_target, configured_target.regions[0])] + + if task.scope is TaskScope.TARGET: + return [ + _task_shell(task, target, target.regions[0]) for target in execution_targets + ] + + if task.scope is TaskScope.REGION: + return [ + _task_shell(task, target, region) + for target in execution_targets + for region in target.regions + ] + + raise TaskPlanningError(f"task '{task.id}' has unsupported scope {task.scope!r}") + + +def _task_shell(task: ResolvedTask, target: ExecutionTarget, region: str) -> _TaskShell: + return ( + TaskInstanceKey( + task_id=task.id, + scope=task.scope, + execution_target_id=target.id, + region=region, + ), + task, + target, + region, + ) + + +def _index_task_shells( + *, task: ResolvedTask, shells: Sequence[_TaskShell] +) -> _TaskShellIndex: + """Index task instances without changing their deterministic plan order.""" + + all_keys: list[TaskInstanceKey] = [] + keys_by_target: dict[str, list[TaskInstanceKey]] = {} + keys_by_coordinate: dict[tuple[str, str], list[TaskInstanceKey]] = {} + for key, _task, _target, _region in shells: + all_keys.append(key) + keys_by_target.setdefault(key.execution_target_id, []).append(key) + coordinate = (key.execution_target_id, key.region) + keys_by_coordinate.setdefault(coordinate, []).append(key) + + return _TaskShellIndex( + scope=task.scope, + all_keys=tuple(all_keys), + keys_by_target={ + target_id: tuple(keys) for target_id, keys in keys_by_target.items() + }, + keys_by_coordinate={ + coordinate: tuple(keys) for coordinate, keys in keys_by_coordinate.items() + }, + ) + + +def _matching_dependency_keys( + *, producer_index: _TaskShellIndex, consumer: TaskInstanceKey +) -> tuple[TaskInstanceKey, ...]: + """Return applicable producer keys through scope-specific indexed lookup.""" + + if ( + producer_index.scope is TaskScope.CONFIGURED_TARGET + or consumer.scope is TaskScope.CONFIGURED_TARGET + ): + return producer_index.all_keys + if producer_index.scope is TaskScope.TARGET or consumer.scope is TaskScope.TARGET: + return producer_index.keys_by_target.get(consumer.execution_target_id, ()) + return producer_index.keys_by_coordinate.get( + (consumer.execution_target_id, consumer.region), () + ) diff --git a/src/anvil/task_scheduler.py b/src/anvil/task_scheduler.py new file mode 100644 index 0000000..e5bc2f8 --- /dev/null +++ b/src/anvil/task_scheduler.py @@ -0,0 +1,420 @@ +"""Bounded scheduling and settlement for precomputed task-instance graphs.""" + +from __future__ import annotations + +import datetime +import heapq +import threading +from collections.abc import Callable, Mapping, Sequence +from concurrent.futures import FIRST_COMPLETED, Future, ThreadPoolExecutor, wait +from dataclasses import dataclass + +from anvil.results import ExecutionStatus, TaskResult +from anvil.task_errors import TaskExecutionError +from anvil.task_loader import ResolvedTask +from anvil.task_planner import TaskInstance, TaskInstanceKey, TaskInstancePlan + +DependencyResults = Mapping[str, TaskResult | tuple[TaskResult, ...]] +TaskInstanceExecutor = Callable[[TaskInstance, DependencyResults], TaskResult] +TaskInstanceSettled = Callable[[TaskInstance, TaskResult], None] + + +@dataclass(frozen=True, slots=True) +class TaskInstanceEligibility: + """Shared execution and finalizer-activation decision.""" + + should_run: bool + skip_reason: str | None = None + chain_activated: bool = False + + +@dataclass(frozen=True, slots=True) +class ScheduledTaskResult: + """Terminal result associated with its planned invocation identity.""" + + key: TaskInstanceKey + result: TaskResult + + +@dataclass(frozen=True, slots=True) +class TaskInstanceSchedule: + """Terminal task results in deterministic plan order.""" + + results: tuple[ScheduledTaskResult, ...] + + +def task_instance_eligibility( + *, + instance: TaskInstance, + dependency_results: Mapping[TaskInstanceKey, TaskResult], + activated_keys: set[TaskInstanceKey], + stop_reason: str | None, +) -> TaskInstanceEligibility: + """Decide whether one ready instance runs or settles as skipped. + + Args: + instance: Ready planned invocation. + dependency_results: Terminal direct dependency results. + activated_keys: Instances whose dependency chains began. + stop_reason: Scheduler stop reason, if ordinary work must not start. + + Returns: + The shared execution and finalizer-activation decision. + """ + + missing_dependencies = [ + key for key in instance.dependencies if key not in dependency_results + ] + if missing_dependencies: + missing = ", ".join(key.task_id for key in missing_dependencies) + raise RuntimeError( + f"Task instance '{instance.key.task_id}' dependencies have not " + f"settled: {missing}" + ) + + chain_activated = not instance.dependencies or any( + key in activated_keys for key in instance.dependencies + ) + return task_dependency_eligibility( + task=instance.task, + dependency_results=tuple(dependency_results.values()), + chain_activated=chain_activated, + stop_reason=stop_reason, + ) + + +def task_dependency_eligibility( + *, + task: ResolvedTask, + dependency_results: Sequence[TaskResult], + chain_activated: bool, + stop_reason: str | None, +) -> TaskInstanceEligibility: + """Apply the one dependency gate used by schedulers and finalizers. + + Args: + task: Resolved task declaration. + dependency_results: Terminal results of every direct dependency. + chain_activated: Whether this dependency chain began execution. + stop_reason: Scheduler stop reason, if ordinary work must not start. + + Returns: + The execution and activation decision. + """ + + if task.always_run: + if chain_activated: + return TaskInstanceEligibility(should_run=True, chain_activated=True) + return TaskInstanceEligibility( + should_run=False, skip_reason=stop_reason or "cancelled_before_start" + ) + + if stop_reason is not None: + return TaskInstanceEligibility( + should_run=False, + skip_reason=stop_reason, + chain_activated=chain_activated and bool(dependency_results), + ) + + if any( + result.status.is_unsuccessful or result.status.is_skipped + for result in dependency_results + ): + return TaskInstanceEligibility( + should_run=False, + skip_reason="dependency_unsuccessful", + chain_activated=chain_activated, + ) + + return TaskInstanceEligibility(should_run=True, chain_activated=True) + + +def execute_task_instance_plan( + *, + plan: TaskInstancePlan, + execute: TaskInstanceExecutor, + max_workers: int, + max_active_execution_targets: int | None = None, + max_active_coordinates_per_execution_target: int | None = None, + cancel_event: threading.Event, + fail_fast: bool, + external_fail_fast_event: threading.Event | None = None, + on_instance_settled: TaskInstanceSettled | None = None, +) -> TaskInstanceSchedule: + """Execute and settle a precomputed task-instance graph. + + Independent coordinates run concurrently up to ``max_workers``. Invocations + sharing a target-region coordinate remain serial, preserving the existing + per-region task execution boundary. + + Args: + plan: Deterministic task-instance graph. + execute: Provider-neutral callback for one admitted invocation. + max_workers: Maximum concurrently running coordinates. + max_active_execution_targets: Optional cap on distinct concurrently active + execution-target identities. + max_active_coordinates_per_execution_target: Optional cap on concurrently + active target-region coordinates for one execution-target identity. + cancel_event: Graceful cancellation signal. + fail_fast: Whether the first unsuccessful invocation stops ordinary work. + external_fail_fast_event: Optional signal that another execution target + triggered fail-fast. + on_instance_settled: Optional callback invoked as each result settles. + + Returns: + Every planned invocation settled in plan order. + + Raises: + ValueError: If ``max_workers`` is not positive. + RuntimeError: If the supplied graph cannot make progress. + """ + + if max_workers <= 0: + raise ValueError("max_workers must be greater than zero") + if max_active_execution_targets is not None and max_active_execution_targets <= 0: + raise ValueError("max_active_execution_targets must be greater than zero") + if ( + max_active_coordinates_per_execution_target is not None + and max_active_coordinates_per_execution_target <= 0 + ): + raise ValueError( + "max_active_coordinates_per_execution_target must be greater than zero" + ) + + instances_by_key = {instance.key: instance for instance in plan.instances} + plan_order = {instance.key: index for index, instance in enumerate(plan.instances)} + remaining_dependencies = { + instance.key: len(instance.dependencies) for instance in plan.instances + } + results_by_key: dict[TaskInstanceKey, TaskResult] = {} + activated_keys: set[TaskInstanceKey] = set() + ready_by_coordinate: dict[tuple[str, str], list[tuple[int, TaskInstanceKey]]] = {} + available_coordinates: list[tuple[int, str, str]] = [] + active_coordinates: set[tuple[str, str]] = set() + active_futures: dict[Future[TaskResult], tuple[TaskInstance, tuple[str, str]]] = {} + active_target_counts: dict[str, int] = {} + fail_fast_triggered = False + + def enqueue_ready(key: TaskInstanceKey) -> None: + coordinate = (key.execution_target_id, key.region) + coordinate_ready = ready_by_coordinate.setdefault(coordinate, []) + heapq.heappush(coordinate_ready, (plan_order[key], key)) + if coordinate not in active_coordinates: + heapq.heappush( + available_coordinates, + (coordinate_ready[0][0], coordinate[0], coordinate[1]), + ) + + def settle(instance: TaskInstance, result: TaskResult) -> None: + nonlocal fail_fast_triggered + results_by_key[instance.key] = result + if on_instance_settled is not None: + on_instance_settled(instance, result) + if fail_fast and result.status.is_unsuccessful: + fail_fast_triggered = True + for child_key in plan.adjacency[instance.key]: + remaining_dependencies[child_key] -= 1 + if remaining_dependencies[child_key] == 0: + enqueue_ready(child_key) + + for instance in plan.instances: + if remaining_dependencies[instance.key] == 0: + enqueue_ready(instance.key) + + with ThreadPoolExecutor( + max_workers=min(max_workers, max(1, len(plan.instances))) + ) as executor: + while len(results_by_key) < len(plan.instances): + deferred_coordinates: list[tuple[int, str, str]] = [] + while available_coordinates and len(active_futures) < max_workers: + ready_index, target_id, region = heapq.heappop(available_coordinates) + coordinate = (target_id, region) + coordinate_ready = ready_by_coordinate.get(coordinate) + if ( + coordinate in active_coordinates + or not coordinate_ready + or coordinate_ready[0][0] != ready_index + ): + continue + if ( + max_active_execution_targets is not None + and target_id not in active_target_counts + and len(active_target_counts) >= max_active_execution_targets + ) or ( + max_active_coordinates_per_execution_target is not None + and active_target_counts.get(target_id, 0) + >= max_active_coordinates_per_execution_target + ): + deferred_coordinates.append((ready_index, target_id, region)) + continue + + _, key = heapq.heappop(coordinate_ready) + instance = instances_by_key[key] + + stop_reason = ( + "fail_fast" + if fail_fast_triggered + or ( + external_fail_fast_event is not None + and external_fail_fast_event.is_set() + ) + else ("cancelled_before_start" if cancel_event.is_set() else None) + ) + eligibility, grouped_dependency_results = _prepare_dependency_results( + instance=instance, + results_by_key=results_by_key, + activated_keys=activated_keys, + stop_reason=stop_reason, + ) + + if not eligibility.should_run: + skipped_result = _skipped_result( + instance=instance, + skip_reason=eligibility.skip_reason + or "dependency_unsuccessful", + ) + if eligibility.chain_activated: + activated_keys.add(key) + settle(instance, skipped_result) + if coordinate_ready: + heapq.heappush( + available_coordinates, + (coordinate_ready[0][0], target_id, region), + ) + continue + + activated_keys.add(key) + future = executor.submit( + _execute_safely, + instance=instance, + dependency_results=grouped_dependency_results, + execute=execute, + ) + active_futures[future] = (instance, coordinate) + active_coordinates.add(coordinate) + active_target_counts[target_id] = ( + active_target_counts.get(target_id, 0) + 1 + ) + + for deferred_coordinate in deferred_coordinates: + heapq.heappush(available_coordinates, deferred_coordinate) + + if len(results_by_key) == len(plan.instances): + break + + if active_futures: + completed, _ = wait(active_futures, return_when=FIRST_COMPLETED) + for future in sorted( + completed, key=lambda item: plan_order[active_futures[item][0].key] + ): + instance, coordinate = active_futures.pop(future) + active_coordinates.remove(coordinate) + active_target_counts[instance.key.execution_target_id] -= 1 + if active_target_counts[instance.key.execution_target_id] == 0: + del active_target_counts[instance.key.execution_target_id] + result = future.result() + settle(instance, result) + coordinate_ready = ready_by_coordinate.get(coordinate) + if coordinate_ready: + heapq.heappush( + available_coordinates, + (coordinate_ready[0][0], coordinate[0], coordinate[1]), + ) + continue + + pending = ", ".join( + instance.key.task_id + for instance in plan.instances + if instance.key not in results_by_key + ) + raise RuntimeError( + f"Task instance graph stalled with unsettled nodes: {pending}" + ) + + ordered_results = tuple( + ScheduledTaskResult(key=instance.key, result=results_by_key[instance.key]) + for instance in plan.instances + ) + return TaskInstanceSchedule(results=ordered_results) + + +def _prepare_dependency_results( + *, + instance: TaskInstance, + results_by_key: Mapping[TaskInstanceKey, TaskResult], + activated_keys: set[TaskInstanceKey], + stop_reason: str | None, +) -> tuple[TaskInstanceEligibility, dict[str, TaskResult | tuple[TaskResult, ...]]]: + """Prepare eligibility and executor dependency inputs in one traversal.""" + + missing_dependencies: list[TaskInstanceKey] = [] + direct_results: list[TaskResult] = [] + grouped: dict[str, list[TaskResult]] = {} + chain_activated = not instance.dependencies + for dependency in instance.dependencies: + result = results_by_key.get(dependency) + if result is None: + missing_dependencies.append(dependency) + continue + direct_results.append(result) + grouped.setdefault(dependency.task_id, []).append(result) + if dependency in activated_keys: + chain_activated = True + + if missing_dependencies: + missing = ", ".join(key.task_id for key in missing_dependencies) + raise RuntimeError( + f"Task instance '{instance.key.task_id}' dependencies have not " + f"settled: {missing}" + ) + + eligibility = task_dependency_eligibility( + task=instance.task, + dependency_results=direct_results, + chain_activated=chain_activated, + stop_reason=stop_reason, + ) + grouped_results = { + task_id: values[0] if len(values) == 1 else tuple(values) + for task_id, values in grouped.items() + } + return eligibility, grouped_results + + +def _execute_safely( + *, + instance: TaskInstance, + dependency_results: DependencyResults, + execute: TaskInstanceExecutor, +) -> TaskResult: + try: + return execute(instance, dependency_results) + except Exception as error: + now_at = datetime.datetime.now(datetime.UTC).isoformat() + return TaskResult( + task_id=instance.task.id, + task_name=instance.task.name, + region=instance.region, + status=ExecutionStatus.ERROR, + started_at=now_at, + ended_at=now_at, + duration_seconds=0.0, + result=( + error.partial_result if isinstance(error, TaskExecutionError) else None + ), + error=str(error), + ) + + +def _skipped_result(*, instance: TaskInstance, skip_reason: str) -> TaskResult: + now_at = datetime.datetime.now(datetime.UTC).isoformat() + return TaskResult( + task_id=instance.task.id, + task_name=instance.task.name, + region=instance.region, + status=ExecutionStatus.SKIPPED, + started_at=now_at, + ended_at=now_at, + duration_seconds=0.0, + skip_reason=skip_reason, + ) diff --git a/tests/cli/test_cli_smoke.py b/tests/cli/test_cli_smoke.py index 11b0b1f..8a85b4b 100644 --- a/tests/cli/test_cli_smoke.py +++ b/tests/cli/test_cli_smoke.py @@ -87,9 +87,11 @@ def test_write_run_results_uses_config_stem_and_run_id_directories(monkeypatch): target_results=[ SimpleNamespace( target_name="org2", + provider="aws", generated_at="2026-04-30T00:00:00+00:00", dry_run=True, entities=[], + tasks=[], to_dict=lambda: {"target": "org2"}, ) ], @@ -275,7 +277,8 @@ def test_build_rerun_targets_narrows_entities_regions_and_task_dependencies(): "target": "org-a", "entity_id": "111111111111", "region": "us-west-2", - "task": "cleanup", + "task_id": "cleanup", + "task_name": "cleanup", "status": "error", }, { @@ -383,12 +386,14 @@ def test_cmd_results_outputs_jsonl_with_fields_and_limit(capsys): ( '{"record_type":"task","target":"org-a","entity_id":' '"111111111111","entity_name":"dev","region":"us-east-1",' - '"task":"count_vpcs","status":"error","error":"boom"}' + '"task_id":"count_vpcs","task_name":"count_vpcs",' + '"status":"error","error":"boom"}' ), ( '{"record_type":"task","target":"org-a","entity_id":' '"222222222222","entity_name":"prod","region":"us-west-2",' - '"task":"count_vpcs","status":"error","error":"nope"}' + '"task_id":"count_vpcs","task_name":"count_vpcs",' + '"status":"error","error":"nope"}' ), ] ), @@ -435,17 +440,20 @@ def test_cmd_results_limit_stops_after_enough_filtered_records(capsys, tmp_path) ( '{"record_type":"task","target":"org-a","entity_id":' '"000000000000","entity_name":"skip","region":"us-east-1",' - '"task":"count_vpcs","status":"success"}' + '"task_id":"count_vpcs","task_name":"count_vpcs",' + '"status":"success"}' ), ( '{"record_type":"task","target":"org-a","entity_id":' '"111111111111","entity_name":"dev","region":"us-east-1",' - '"task":"count_vpcs","status":"error","error":"boom"}' + '"task_id":"count_vpcs","task_name":"count_vpcs",' + '"status":"error","error":"boom"}' ), ( '{"record_type":"task","target":"org-a","entity_id":' '"222222222222","entity_name":"prod","region":"us-west-2",' - '"task":"count_vpcs","status":"error","error":"nope"}' + '"task_id":"count_vpcs","task_name":"count_vpcs",' + '"status":"error","error":"nope"}' ), "not-json", ] diff --git a/tests/examples/test_examples_pattern.py b/tests/examples/test_examples_pattern.py index b358ce1..5db099f 100644 --- a/tests/examples/test_examples_pattern.py +++ b/tests/examples/test_examples_pattern.py @@ -3,9 +3,20 @@ import pytest EXAMPLES_DIR = Path(__file__).resolve().parents[2] / "examples" +CONFIG_DIR = Path(__file__).resolve().parents[2] / "yaml" +CONFIG_PATHS = sorted([*EXAMPLES_DIR.glob("*.yaml"), *CONFIG_DIR.glob("*.yaml")]) +INVALID_MULTI_ACCOUNT_CONFIG = ( + EXAMPLES_DIR / "invalid" / "aws-configured-target-multiple-accounts.yaml" +) +ADVANCED_CONFIG_PATHS = [ + EXAMPLES_DIR / "04-aws-advanced.yaml", + EXAMPLES_DIR / "08-azure-advanced.yaml", + EXAMPLES_DIR / "12-gcp-advanced.yaml", + EXAMPLES_DIR / "16-github-advanced.yaml", +] -@pytest.mark.parametrize("config_path", sorted(EXAMPLES_DIR.glob("*.yaml"))) +@pytest.mark.parametrize("config_path", CONFIG_PATHS) def test_example_configs_load(config_path: Path) -> None: try: from anvil.cli import _load_targets_from_config_file @@ -14,3 +25,41 @@ def test_example_configs_load(config_path: Path) -> None: loaded_config = _load_targets_from_config_file(config_path) assert loaded_config.targets + + +def test_invalid_multi_account_configured_target_example_fails_offline() -> None: + from anvil.cli import _load_targets_from_config_file + from anvil.providers.aws.provider import AwsProvider + + loaded_config = _load_targets_from_config_file(INVALID_MULTI_ACCOUNT_CONFIG) + + with pytest.raises(ValueError, match="exactly one explicit account"): + AwsProvider().validate_task_configuration( + target=loaded_config.targets[0], + task_scopes={"snapshot_org_config": "configured_target"}, + ) + + +@pytest.mark.parametrize("config_path", ADVANCED_CONFIG_PATHS) +def test_advanced_examples_use_descriptive_task_ids(config_path: Path) -> None: + from anvil.cli import _load_targets_from_config_file + + loaded_config = _load_targets_from_config_file(config_path) + + assert all( + isinstance(task.get("id"), str) and task["id"] + for target in loaded_config.targets + for task in target.tasks + ) + + +def test_aws_advanced_sarif_example_avoids_duplicate_lambda_inventory() -> None: + from anvil.cli import _load_targets_from_config_file + + loaded_config = _load_targets_from_config_file( + EXAMPLES_DIR / "04-aws-advanced.yaml" + ) + task_names = {task["name"] for task in loaded_config.targets[0].tasks} + + assert "detect_deprecated_lambda_runtimes" in task_names + assert "list_lambdas_by_runtime" not in task_names diff --git a/tests/examples/test_result_task_examples.py b/tests/examples/test_result_task_examples.py new file mode 100644 index 0000000..6a96626 --- /dev/null +++ b/tests/examples/test_result_task_examples.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import importlib.util +import inspect +from pathlib import Path + +import pytest + +from anvil.task_context import TaskCallContext + + +RESULT_EXAMPLES_DIR = Path(__file__).resolve().parents[2] / "examples" / "Results" + + +@pytest.mark.parametrize("example_path", sorted(RESULT_EXAMPLES_DIR.glob("*.py"))) +def test_result_task_examples_use_current_runtime_contract(example_path: Path) -> None: + spec = importlib.util.spec_from_file_location( + f"anvil_result_example_{example_path.stem}", example_path + ) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + parameters = inspect.signature(module.run).parameters + + assert frozenset(parameters) == TaskCallContext.keyword_names() + assert all( + parameter.kind is inspect.Parameter.KEYWORD_ONLY + for parameter in parameters.values() + ) diff --git a/tests/processors/test_processors.py b/tests/processors/test_processors.py index 020e86f..431dabb 100644 --- a/tests/processors/test_processors.py +++ b/tests/processors/test_processors.py @@ -14,6 +14,7 @@ ) from anvil.processor_validation import ProcessorValidationError, validate_processors from anvil.processors import html_report +from anvil.processors import sarif_report def _context(tmp_path: Path) -> ProcessorRunContext: @@ -382,3 +383,82 @@ def test_html_report_load_records_keeps_whole_run_context(tmp_path): "111111111111", "222222222222", ] + + +def test_html_report_includes_configured_tasks_and_counts_skips(tmp_path): + context = ProcessorRunContext( + run_dir=tmp_path, + summary_path=tmp_path / "summary.json", + summary={"state": "completed_success"}, + target_result_paths={}, + target_results=[ + { + "target": "production", + "tasks": [ + { + "task_id": "inventory_before", + "task_name": "inventory", + "region": "us-east-1", + "status": "skipped", + } + ], + "entities": [], + } + ], + ) + + records = html_report._load_records(context=context) + cards = html_report._summary_cards(records) + + assert records[0]["entity_type"] == "configured_target" + assert records[0]["task_id"] == "inventory_before" + assert records[0]["task_name"] == "inventory" + assert next(card for card in cards if card["label"] == "Skipped")["value"] == 1 + assert next(card for card in cards if card["label"] == "Failed tasks")["value"] == 0 + + +def test_sarif_report_includes_configured_task_identity(tmp_path): + context = ProcessorRunContext( + run_dir=tmp_path, + summary_path=tmp_path / "summary.json", + summary={"state": "completed_success"}, + target_result_paths={}, + target_results=[ + { + "target": "production", + "tasks": [ + { + "task_id": "detect_public", + "task_name": "detect_resources", + "region": "us-east-1", + "result": { + "sarif_findings": [ + { + "rule": {"id": "ANVIL001"}, + "message": "Public resource", + "locations": [{"uri": "aws://resource"}], + "properties": { + "target": "spoofed-target", + "entity_type": "spoofed-type", + "task_id": "spoofed-id", + "task_name": "spoofed-name", + "resource_name": "public-resource", + }, + } + ] + }, + } + ], + "entities": [], + } + ], + ) + + results, _ = sarif_report._collect_sarif_results(context=context) + + properties = results[0]["properties"] + assert properties["entity_type"] == "configured_target" + assert properties["task_id"] == "detect_public" + assert properties["task_name"] == "detect_resources" + assert properties["target"] == "production" + assert properties["resource_name"] == "public-resource" diff --git a/tests/providers/aws/test_execution_targets.py b/tests/providers/aws/test_execution_targets.py index 954207c..393e4ff 100644 --- a/tests/providers/aws/test_execution_targets.py +++ b/tests/providers/aws/test_execution_targets.py @@ -8,6 +8,7 @@ AwsPreflightData, AwsProvider, ) +from anvil.providers.aws.account import AccountAccessStrategy @dataclass @@ -28,14 +29,16 @@ def _preflight_data( *, session_factory: FakeSessionFactory, base_session: BaseSession | None = None, + base_session_account_id: str = "111111111111", discovered_accounts: dict[str, dict[str, str]] | None = None, + region_statuses: dict[str, str] | None = None, ) -> AwsPreflightData: return AwsPreflightData( session_factory=session_factory, base_session=base_session or BaseSession(), organization_id="o-shared", management_account_id="111111111111", - base_session_account_id="111111111111", + base_session_account_id=base_session_account_id, discovered_accounts=discovered_accounts or { "111111111111": { @@ -47,7 +50,7 @@ def _preflight_data( "account_alias": "member", }, }, - region_statuses={"us-east-1": "ENABLED_BY_DEFAULT"}, + region_statuses=region_statuses or {"us-east-1": "ENABLED_BY_DEFAULT"}, ) @@ -179,6 +182,213 @@ def test_resolve_execution_targets_maps_organization_accounts_and_execution_key( assert session_factory.base_session_calls == [] +def test_organization_configured_target_uses_management_base_session_identity(): + session_factory = FakeSessionFactory() + base_session = BaseSession(profile_name="management") + target = TargetDescriptor( + name="org-a", + provider="aws", + mode="organization", + provider_options={"profile": "management"}, + exclude=["111111111111"], + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + preparation=_preflight_data( + session_factory=session_factory, + base_session=base_session, + base_session_account_id="111111111111", + ), + ) + + assert [execution_target.id for execution_target in plan.execution_targets] == [ + "222222222222" + ] + assert plan.configured_target is not None + assert ( + plan.configured_target.id, + plan.configured_target.name, + plan.configured_target.type, + plan.configured_target.regions, + ) == ("111111111111", "management", "configured_target", ["us-east-1"]) + assert isinstance(plan.configured_target.provider_data, AwsExecutionTargetData) + assert ( + plan.configured_target.provider_data.access_strategy + is AccountAccessStrategy.BASE_SESSION + ) + assert plan.configured_target.provider_data.base_session is base_session + + +def test_organization_configured_target_assumes_management_role_when_needed(): + target = TargetDescriptor( + name="org-a", + provider="aws", + mode="organization", + provider_options={"role_name": "ManagementAccessRole"}, + include=["222222222222"], + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + preparation=_preflight_data( + session_factory=FakeSessionFactory(), base_session_account_id="333333333333" + ), + ) + + assert plan.configured_target is not None + assert isinstance(plan.configured_target.provider_data, AwsExecutionTargetData) + assert ( + plan.configured_target.provider_data.access_strategy + is AccountAccessStrategy.ASSUME_ROLE + ) + assert plan.configured_target.provider_data.role_name == "ManagementAccessRole" + assert plan.configured_target.provider_data.account_id == "111111111111" + + +def test_organization_configured_target_is_stable_when_management_is_selected(): + target = TargetDescriptor( + name="org-a", + provider="aws", + mode="organization", + include=["111111111111", "222222222222"], + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + preparation=_preflight_data(session_factory=FakeSessionFactory()), + ) + + assert [execution_target.id for execution_target in plan.execution_targets] == [ + "111111111111", + "222222222222", + ] + assert plan.configured_target is not None + assert plan.configured_target.id == "111111111111" + assert plan.configured_target.name == "management" + + +def test_organization_management_keyword_selects_management_account(): + target = TargetDescriptor( + name="org-a", provider="aws", mode="organization", include=["management"] + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + preparation=_preflight_data(session_factory=FakeSessionFactory()), + ) + + assert [execution_target.id for execution_target in plan.execution_targets] == [ + "111111111111" + ] + assert plan.execution_targets[0].metadata["is_management"] is True + + +def test_organization_payer_keyword_excludes_management_account(): + target = TargetDescriptor( + name="org-a", provider="aws", mode="organization", exclude=["payer"] + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + preparation=_preflight_data(session_factory=FakeSessionFactory()), + ) + + assert [execution_target.id for execution_target in plan.execution_targets] == [ + "222222222222" + ] + + +def test_organization_configured_target_uses_concrete_resolved_regions(): + target = TargetDescriptor( + name="org-a", provider="aws", mode="organization", regions=["us-*"] + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-*"], + include=target.include, + exclude=target.exclude, + preparation=_preflight_data( + session_factory=FakeSessionFactory(), + region_statuses={"us-east-1": "ENABLED_BY_DEFAULT", "us-west-2": "ENABLED"}, + ), + ) + + assert plan.configured_target is not None + assert plan.configured_target.regions == ["us-east-1", "us-west-2"] + + +def test_single_explicit_account_is_configured_target_identity(monkeypatch): + session_factory = FakeSessionFactory() + monkeypatch.setattr( + "anvil.providers.aws.provider.SessionFactory", lambda: session_factory + ) + target = TargetDescriptor( + name="one-account", + provider="aws", + mode="accounts", + provider_options={"profile": "tooling", "role_name": "SecurityAccessRole"}, + include=["222222222222"], + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + ) + + assert plan.configured_target is not None + assert ( + plan.configured_target.id, + plan.configured_target.name, + plan.configured_target.type, + plan.configured_target.regions, + ) == ("222222222222", "222222222222", "configured_target", ["us-east-1"]) + assert ( + plan.configured_target.provider_data is plan.execution_targets[0].provider_data + ) + + +def test_multiple_explicit_accounts_do_not_select_configured_target(monkeypatch): + session_factory = FakeSessionFactory() + monkeypatch.setattr( + "anvil.providers.aws.provider.SessionFactory", lambda: session_factory + ) + target = TargetDescriptor( + name="many-accounts", + provider="aws", + mode="accounts", + provider_options={"role_name": "SecurityAccessRole"}, + include=["111111111111", "222222222222"], + ) + + plan = AwsProvider().resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + ) + + assert plan.configured_target is None + + def test_resolve_execution_targets_maps_organization_accounts_with_provider_options(): session_factory = FakeSessionFactory() base_session = BaseSession(profile_name="shared") diff --git a/tests/providers/aws/test_runtime.py b/tests/providers/aws/test_runtime.py index 6e61a80..852d7ed 100644 --- a/tests/providers/aws/test_runtime.py +++ b/tests/providers/aws/test_runtime.py @@ -1,6 +1,7 @@ from __future__ import annotations import datetime +import time from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass @@ -12,12 +13,16 @@ ) from anvil.descriptors import TargetDescriptor from anvil.execution_context import ExecutionContext -from anvil.providers.aws.provider import AwsExecutionTargetData, AwsProvider +from anvil.providers.aws.provider import ( + AwsExecutionRuntime, + AwsExecutionTargetData, + AwsProvider, +) from anvil.providers.base import ExecutionTarget from anvil.results import ExecutionStatus -from anvil.runner import _execute_provider_execution_target +from anvil.runner import _execute_provider_execution_target, _execute_provider_targets from anvil.providers.aws.session import AssumedRoleCredentials, CachedClientSession -from anvil.task_loader import ResolvedTask +from anvil.task_loader import ResolvedTask, TaskScope @dataclass @@ -120,11 +125,12 @@ def _execution_target( session_factory: RecordingSessionFactory, access_strategy: AccountAccessStrategy, regions: list[str] | None = None, + target_type: str = "account", ) -> ExecutionTarget: return ExecutionTarget( id="123456789012", name="test-account", - type="account", + type=target_type, provider="aws", regions=regions or ["us-east-1"], provider_data=AwsExecutionTargetData( @@ -297,6 +303,97 @@ def test_runtime_records_region_duration_for_adaptive_refresh_window(): ] == ["access-1", "access-2"] +def test_configured_runtime_assumes_role_for_its_own_concrete_account(): + session_factory = RecordingSessionFactory() + execution_target = _execution_target( + session_factory=session_factory, + access_strategy=AccountAccessStrategy.ASSUME_ROLE, + target_type="configured_target", + ) + + runtime = AwsProvider().prepare_configured_target_runtime( + target=_target(), execution_target=execution_target, context=_context() + ) + runtime.build_session(region="us-east-1") + + assert [call["account_id"] for call in session_factory.assume_role_calls] == [ + "123456789012" + ] + assert len(session_factory.cached_session_calls) == 1 + + +def test_configured_management_runtime_reuses_matching_base_credentials(): + session_factory = RecordingSessionFactory(caller_account_id="123456789012") + execution_target = _execution_target( + session_factory=session_factory, + access_strategy=AccountAccessStrategy.BASE_SESSION, + target_type="configured_target", + ) + + runtime = AwsProvider().prepare_configured_target_runtime( + target=_target(), execution_target=execution_target, context=_context() + ) + runtime.build_session(region="us-east-1") + + assert session_factory.assume_role_calls == [] + assert [call["region_name"] for call in session_factory.worker_session_calls] == [ + "us-east-1", + "us-east-1", + ] + + +def test_configured_callback_identity_matches_direct_profile_session_identity(): + session_factory = RecordingSessionFactory(caller_account_id="123456789012") + configured_target = _execution_target( + session_factory=session_factory, + access_strategy=AccountAccessStrategy.DIRECT_PROFILE, + target_type="configured_target", + ) + callbacks: list[tuple[str, str, str, str, str]] = [] + + def configured(**kwargs): + callbacks.append( + ( + kwargs["execution_target_id"], + kwargs["execution_target_name"], + kwargs["execution_target_type"], + kwargs["region"], + kwargs["session"]._caller_account_id, + ) + ) + + task = ResolvedTask( + id="configured", + name="configured", + run=configured, + depends_on=[], + always_run=False, + metadata={}, + dependency_data={}, + scope=TaskScope.CONFIGURED_TARGET, + ) + context = _context(tasks=[task]) + + _execute_provider_targets( + provider=AwsProvider(), + target=_target(), + context=context, + execution_targets=[], + configured_execution_target=configured_target, + benchmark_data=None, + ) + + assert callbacks == [ + ( + "123456789012", + "test-account", + "configured_target", + "us-east-1", + "123456789012", + ) + ] + + def test_aws_provider_execution_path_preserves_runtime_benchmark_data(): session_factory = RecordingSessionFactory() @@ -305,7 +402,7 @@ def run(**kwargs): context = _context( regions=["us-east-1", "us-west-2"], - tasks=[ResolvedTask("scan", run, depends_on=[], optional=False)], + tasks=[ResolvedTask("scan", run, depends_on=[])], benchmark_enabled=True, ) @@ -337,3 +434,69 @@ def run(**kwargs): assert region_benchmark["task_count"] == 1 assert region_benchmark["interrupted"] is False assert region_benchmark["failed"] is False + + +def test_mixed_graph_records_wall_clock_region_lifecycle_duration(monkeypatch): + session_factory = RecordingSessionFactory(caller_account_id="123456789012") + recorded_durations: list[float] = [] + original_build_session = AwsExecutionRuntime.build_session + original_record_outcome = AwsExecutionRuntime.record_region_outcome + + def delayed_build_session(self, *, region): + time.sleep(0.02) + return original_build_session(self, region=region) + + def capture_outcome(self, *, region, duration_seconds, failed, interrupted): + recorded_durations.append(duration_seconds) + return original_record_outcome( + self, + region=region, + duration_seconds=duration_seconds, + failed=failed, + interrupted=interrupted, + ) + + monkeypatch.setattr(AwsExecutionRuntime, "build_session", delayed_build_session) + monkeypatch.setattr(AwsExecutionRuntime, "record_region_outcome", capture_outcome) + + regional_task = ResolvedTask( + id="regional", + name="regional", + run=lambda **kwargs: {"region": kwargs["region"]}, + depends_on=[], + scope=TaskScope.REGION, + ) + configured_task = ResolvedTask( + id="configured", + name="configured", + run=lambda **kwargs: {"configured": True}, + depends_on=["regional"], + scope=TaskScope.CONFIGURED_TARGET, + ) + context = _context(tasks=[regional_task, configured_task], benchmark_enabled=True) + + result = _execute_provider_targets( + provider=AwsProvider(), + target=_target(), + context=context, + execution_targets=[ + _execution_target( + session_factory=session_factory, + access_strategy=AccountAccessStrategy.BASE_SESSION, + ) + ], + configured_execution_target=_execution_target( + session_factory=session_factory, + access_strategy=AccountAccessStrategy.BASE_SESSION, + target_type="configured_target", + ), + benchmark_data=None, + ) + + assert len(recorded_durations) == 2 + assert all(duration >= 0.015 for duration in recorded_durations) + assert result.entities[0].benchmark is not None + assert ( + result.entities[0].benchmark["regions"]["us-east-1"]["duration_seconds"] + >= 0.015 + ) diff --git a/tests/providers/aws/test_task_implementations.py b/tests/providers/aws/test_task_implementations.py new file mode 100644 index 0000000..9a956b7 --- /dev/null +++ b/tests/providers/aws/test_task_implementations.py @@ -0,0 +1,321 @@ +from __future__ import annotations + +from collections.abc import Sequence +from types import SimpleNamespace + +import pytest +from botocore.exceptions import ClientError + +from anvil.actions import ActionRecorder +from anvil.providers.aws.tasks import compare_asg_to_cluster_instances +from anvil.providers.aws.tasks import remove_iam_user +from anvil.providers.aws.tasks import remove_missing_group_assignments + + +def _client_error(code: str, operation_name: str) -> ClientError: + return ClientError( + {"Error": {"Code": code, "Message": f"{code} from test"}}, operation_name + ) + + +class FakePaginator: + def __init__(self, pages: Sequence[dict[str, object]]) -> None: + self.pages = list(pages) + self.calls: list[dict[str, object]] = [] + + def paginate(self, **kwargs: object): + self.calls.append(kwargs) + yield from self.pages + + +class FakeSession: + def __init__(self, clients: dict[str, object]) -> None: + self.clients = clients + + def client(self, service_name: str, **_kwargs: object) -> object: + return self.clients[service_name] + + +class FakeResourceNotFoundClientError(ClientError): + pass + + +class FakeIdentityStoreClient: + exceptions = SimpleNamespace( + ResourceNotFoundException=FakeResourceNotFoundClientError + ) + + def __init__(self, error: ClientError | None = None) -> None: + self.error = error + + def describe_group(self, **_kwargs: object) -> dict[str, object]: + if self.error is not None: + raise self.error + return {} + + +def test_group_validation_marks_only_resource_not_found_as_missing() -> None: + missing_error = FakeResourceNotFoundClientError( + {"Error": {"Code": "ResourceNotFoundException", "Message": "missing"}}, + "DescribeGroup", + ) + + result = remove_missing_group_assignments._validate_groups( + FakeIdentityStoreClient(missing_error), "store-1", {"group-1"} + ) + + assert result == {"group-1": False} + + +def test_group_validation_surfaces_unexpected_client_errors() -> None: + error = _client_error("ThrottlingException", "DescribeGroup") + + with pytest.raises(ClientError) as raised: + remove_missing_group_assignments._validate_groups( + FakeIdentityStoreClient(error), "store-1", {"group-1"} + ) + + assert raised.value is error + + +class FakeSsoAdminClient: + def __init__(self, *, delete_error: ClientError | None = None) -> None: + self.delete_error = delete_error + self.delete_calls: list[dict[str, object]] = [] + + def list_instances(self) -> dict[str, object]: + return { + "Instances": [ + { + "Status": "ACTIVE", + "InstanceArn": "arn:aws:sso:::instance/ssoins-1", + "IdentityStoreId": "store-1", + "OwnerAccountId": "111111111111", + } + ] + } + + def delete_account_assignment(self, **kwargs: object) -> None: + self.delete_calls.append(kwargs) + if self.delete_error is not None: + raise self.delete_error + + +def _run_remove_missing_group_assignments( + monkeypatch: pytest.MonkeyPatch, + *, + dry_run: bool, + delete_error: ClientError | None = None, +) -> tuple[dict[str, object], list[str], FakeSsoAdminClient]: + assignment = { + "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-1/ps-1", + "PermissionSetName": "ReadOnly", + "AccountId": "222222222222", + "AccountName": "Workload", + "GroupId": "group-1", + } + monkeypatch.setattr( + remove_missing_group_assignments, "_get_account_cache", lambda _client: {} + ) + monkeypatch.setattr( + remove_missing_group_assignments, + "_get_permission_set_name_cache", + lambda _client, _instance_arn: {}, + ) + monkeypatch.setattr( + remove_missing_group_assignments, + "_collect_group_assignments", + lambda *_args: [assignment], + ) + monkeypatch.setattr( + remove_missing_group_assignments, + "_validate_groups", + lambda *_args: {"group-1": False}, + ) + + sso_admin_client = FakeSsoAdminClient(delete_error=delete_error) + session = FakeSession( + { + "sso-admin": sso_admin_client, + "identitystore": object(), + "organizations": object(), + } + ) + actions = ActionRecorder(actions=[]) + result = remove_missing_group_assignments.run( + provider="aws", + execution_target_id="111111111111", + execution_target_name="management", + execution_target_type="account", + region="us-east-1", + session=session, + dry_run=dry_run, + metadata={}, + dependency_data={}, + actions=actions, + ) + return result, actions.actions, sso_admin_client + + +def test_remove_missing_group_assignments_labels_dry_run_and_does_not_delete( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, actions, client = _run_remove_missing_group_assignments( + monkeypatch, dry_run=True + ) + + assert client.delete_calls == [] + assert result["missing_count"] == 1 + assert result["removed_count"] == 0 + assert actions == ["(dry-run) Would remove 1 missing group assignment(s)"] + + +def test_remove_missing_group_assignments_surfaces_delete_failure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + error = _client_error("AccessDeniedException", "DeleteAccountAssignment") + + with pytest.raises(ClientError) as raised: + _run_remove_missing_group_assignments( + monkeypatch, dry_run=False, delete_error=error + ) + + assert raised.value is error + + +class FakeIamClient: + def __init__(self) -> None: + self.paginators = { + "list_groups_for_user": FakePaginator( + [{"Groups": [{"GroupName": "group-a"}]}, {"Groups": []}] + ), + "list_access_keys": FakePaginator( + [ + {"AccessKeyMetadata": [{"AccessKeyId": "key-a"}]}, + {"AccessKeyMetadata": [{"AccessKeyId": "key-b"}]}, + ] + ), + "list_mfa_devices": FakePaginator([{"MFADevices": []}]), + "list_ssh_public_keys": FakePaginator([{"SSHPublicKeys": []}]), + "list_signing_certificates": FakePaginator([{"Certificates": []}]), + "list_attached_user_policies": FakePaginator([{"AttachedPolicies": []}]), + "list_user_policies": FakePaginator([{"PolicyNames": []}]), + "list_user_tags": FakePaginator([{"Tags": []}]), + } + self.mutation_calls: list[tuple[str, dict[str, object]]] = [] + + def get_paginator(self, operation_name: str) -> FakePaginator: + return self.paginators[operation_name] + + def list_service_specific_credentials(self, **_kwargs: object) -> dict[str, object]: + return {"ServiceSpecificCredentials": []} + + def get_login_profile(self, **_kwargs: object) -> None: + raise _client_error("NoSuchEntity", "GetLoginProfile") + + def remove_user_from_group(self, **kwargs: object) -> None: + self.mutation_calls.append(("remove_user_from_group", kwargs)) + + def delete_access_key(self, **kwargs: object) -> None: + self.mutation_calls.append(("delete_access_key", kwargs)) + + +def _run_remove_iam_user(*, dry_run: bool) -> tuple[list[str], FakeIamClient]: + iam_client = FakeIamClient() + actions = ActionRecorder(actions=[]) + remove_iam_user.run( + provider="aws", + execution_target_id="111111111111", + execution_target_name="workload", + execution_target_type="account", + region="us-east-1", + session=FakeSession({"iam": iam_client}), + dry_run=dry_run, + metadata={"user_name": "alice"}, + dependency_data={}, + actions=actions, + ) + return actions.actions, iam_client + + +def test_remove_iam_user_cleans_resources_from_every_page() -> None: + actions, client = _run_remove_iam_user(dry_run=False) + + assert client.mutation_calls == [ + ("remove_user_from_group", {"GroupName": "group-a", "UserName": "alice"}), + ("delete_access_key", {"UserName": "alice", "AccessKeyId": "key-a"}), + ("delete_access_key", {"UserName": "alice", "AccessKeyId": "key-b"}), + ] + assert all( + paginator.calls == [{"UserName": "alice"}] + for paginator in client.paginators.values() + ) + assert actions == ["Removed IAM user resources for alice"] + + +def test_remove_iam_user_labels_dry_run_and_does_not_mutate() -> None: + actions, client = _run_remove_iam_user(dry_run=True) + + assert client.mutation_calls == [] + assert actions == ["(dry-run) Would remove IAM user resources for alice"] + + +class FakeAutoScalingClient: + def describe_auto_scaling_groups(self, **_kwargs: object) -> dict[str, object]: + return { + "AutoScalingGroups": [ + {"Instances": [{"InstanceId": "i-a"}, {"InstanceId": "i-b"}]} + ] + } + + +class FakeEcsClient: + def __init__(self) -> None: + self.paginator = FakePaginator( + [ + {"containerInstanceArns": ["arn:container/a"]}, + {"containerInstanceArns": ["arn:container/b"]}, + ] + ) + self.describe_calls: list[dict[str, object]] = [] + + def get_paginator(self, operation_name: str) -> FakePaginator: + assert operation_name == "list_container_instances" + return self.paginator + + def describe_container_instances(self, **kwargs: object) -> dict[str, object]: + self.describe_calls.append(kwargs) + container_arn = kwargs["containerInstances"][0] + suffix = str(container_arn).rsplit("/", maxsplit=1)[-1] + return { + "containerInstances": [ + {"ec2InstanceId": f"i-{suffix}", "runningTasksCount": 1} + ] + } + + +def test_compare_asg_to_cluster_instances_reads_every_ecs_page() -> None: + ecs_client = FakeEcsClient() + actions = ActionRecorder(actions=[]) + + compare_asg_to_cluster_instances.run( + provider="aws", + execution_target_id="111111111111", + execution_target_name="workload", + execution_target_type="account", + region="us-east-1", + session=FakeSession( + {"autoscaling": FakeAutoScalingClient(), "ecs": ecs_client} + ), + dry_run=False, + metadata={"clusters": ["api"]}, + dependency_data={}, + actions=actions, + ) + + assert ecs_client.paginator.calls == [{"cluster": "api"}] + assert ecs_client.describe_calls == [ + {"cluster": "api", "containerInstances": ["arn:container/a"]}, + {"cluster": "api", "containerInstances": ["arn:container/b"]}, + ] + assert actions.actions == ["Completed ASG vs ECS comparison for 1 clusters"] diff --git a/tests/providers/azure/test_count_resource_groups_task.py b/tests/providers/azure/test_count_resource_groups_task.py index 4d78616..23236f0 100644 --- a/tests/providers/azure/test_count_resource_groups_task.py +++ b/tests/providers/azure/test_count_resource_groups_task.py @@ -77,6 +77,7 @@ def _run_task( session=session, dry_run=dry_run, metadata={}, + dependency_data={}, actions=actions, ) return result, actions.actions @@ -166,5 +167,6 @@ def test_count_resource_groups_requires_azure_subscription_target( session=FakeAzureSession(), dry_run=False, metadata={}, + dependency_data={}, actions=ActionRecorder(actions=[]), ) diff --git a/tests/providers/gcp/test_get_project_info_task.py b/tests/providers/gcp/test_get_project_info_task.py index a767eb5..fb1b580 100644 --- a/tests/providers/gcp/test_get_project_info_task.py +++ b/tests/providers/gcp/test_get_project_info_task.py @@ -85,6 +85,7 @@ def _run_task( session=session, dry_run=dry_run, metadata={}, + dependency_data={}, actions=actions, ) return result, actions.actions @@ -185,6 +186,7 @@ def test_get_project_info_requires_gcp_project_target(fake_gcp_resource_manager_ session=FakeGcpSession(), dry_run=False, metadata={}, + dependency_data={}, actions=ActionRecorder(actions=[]), ) diff --git a/tests/providers/github/test_rest_tasks.py b/tests/providers/github/test_rest_tasks.py index 9d74f9b..ca2040c 100644 --- a/tests/providers/github/test_rest_tasks.py +++ b/tests/providers/github/test_rest_tasks.py @@ -73,6 +73,7 @@ def _run_task( session=FakeSession(client=client), dry_run=False, metadata={} if metadata is None else metadata, + dependency_data={}, actions=actions, ) return result, actions.actions diff --git a/tests/providers/github/test_search_code_task.py b/tests/providers/github/test_search_code_task.py index 9bef3ef..4eae133 100644 --- a/tests/providers/github/test_search_code_task.py +++ b/tests/providers/github/test_search_code_task.py @@ -92,6 +92,7 @@ def _run_task( session=FakeGitHubSession(client=client), dry_run=False, metadata={"query": "secret"} if metadata is None else metadata, + dependency_data={}, actions=actions, ) return result, actions.actions @@ -269,6 +270,7 @@ def test_search_code_requires_task_facing_search_session() -> None: session=object(), dry_run=False, metadata={"query": "secret"}, + dependency_data={}, actions=actions, ) diff --git a/tests/providers/test_provider_contract.py b/tests/providers/test_provider_contract.py index 2869964..d001eb9 100644 --- a/tests/providers/test_provider_contract.py +++ b/tests/providers/test_provider_contract.py @@ -7,6 +7,8 @@ create_provider_instance as create_azure_provider_instance, ) from anvil.providers.base import ( + ExecutionTarget, + ProviderExecutionPlan, ProviderMetadata, configured_or_default_regions, validate_provider_contract, @@ -144,3 +146,52 @@ def resolve_execution_targets( # ty: ignore[invalid-method-override] with pytest.raises(TypeError, match="resolve_execution_targets.*preparation"): validate_provider_contract(BrokenProvider()) + + +def test_configured_target_capability_requires_explicit_provider_hooks(): + class BrokenProvider(_CompleteProvider): + metadata = ProviderMetadata( + name="broken", + display_name="Broken", + supported_task_scopes=frozenset({"configured_target", "region"}), + ) + + with pytest.raises( + TypeError, match="configured_target.*validate_task_configuration" + ): + validate_provider_contract(BrokenProvider()) + + +def test_configured_target_capability_accepts_complete_provider_hooks(): + class ConfiguredProvider(_CompleteProvider): + metadata = ProviderMetadata( + name="configured", + display_name="Configured", + supported_task_scopes=frozenset({"configured_target", "region"}), + ) + + def validate_task_configuration(self, *, target, task_scopes): + return None + + def prepare_configured_target_runtime( + self, *, target, execution_target, context + ): + return None + + validate_provider_contract(ConfiguredProvider()) + + +def test_provider_execution_plan_carries_provider_owned_configured_identity(): + configured_target = ExecutionTarget( + id="provider-owner", + name="Provider Owner", + type="configured_target", + provider="complete", + regions=["home-region"], + ) + + plan = ProviderExecutionPlan( + execution_targets=[], configured_target=configured_target + ) + + assert plan.configured_target is configured_target diff --git a/tests/providers/test_provider_owned_target_validation.py b/tests/providers/test_provider_owned_target_validation.py index 6345db1..a3e0b6b 100644 --- a/tests/providers/test_provider_owned_target_validation.py +++ b/tests/providers/test_provider_owned_target_validation.py @@ -60,6 +60,21 @@ def test_aws_explicit_accounts_do_not_receive_the_organization_default_role() -> assert target.provider_options.get("role_name") is None +@pytest.mark.parametrize("keyword", ["management", "payer", "MANAGEMENT", "PAYER"]) +def test_aws_organization_accepts_management_account_keywords(keyword: str) -> None: + target = _target(provider="aws", mode="organization", include=[keyword]) + + AwsProvider().validate_target(target) + + +@pytest.mark.parametrize("keyword", ["management", "payer"]) +def test_aws_accounts_mode_rejects_management_account_keywords(keyword: str) -> None: + target = _target(provider="aws", mode="accounts", include=[keyword]) + + with pytest.raises(ValueError, match=f"keyword '{keyword}'.*organization mode"): + AwsProvider().validate_target(target) + + def test_provider_owns_cli_filter_semantics() -> None: target = _target( provider="azure", diff --git a/tests/results/test_result_query.py b/tests/results/test_result_query.py index 073b777..efa127d 100644 --- a/tests/results/test_result_query.py +++ b/tests/results/test_result_query.py @@ -25,6 +25,7 @@ def _task_result( *, task_name: str, region: str, status: ExecutionStatus, error: str | None = None ) -> TaskResult: return TaskResult( + task_id=task_name, task_name=task_name, region=region, status=status, @@ -79,7 +80,8 @@ def test_build_jsonl_records_flattens_entities_and_tasks(): assert records[1]["entity_id"] == "111111111111" assert records[1]["entity_name"] == "dev" assert records[1]["entity_type"] == "account" - assert records[1]["task"] == "count_vpcs" + assert records[1]["task_id"] == "count_vpcs" + assert records[1]["task_name"] == "count_vpcs" assert records[1]["region"] == "us-east-1" assert records[1]["status"] == "error" json.dumps(records) @@ -116,11 +118,12 @@ def test_filter_records_supports_failed_status_alias_and_common_fields(): assert matches[0]["record_type"] == "task" -def test_filter_records_failed_status_matches_any_non_success_status(): +def test_filter_records_failed_status_matches_unsuccessful_statuses(): records = [ {"record_type": "entity", "status": "success", "entity_id": "111"}, {"record_type": "entity", "status": "error", "entity_id": "222"}, {"record_type": "entity", "status": "interrupted", "entity_id": "333"}, + {"record_type": "entity", "status": "skipped", "entity_id": "444"}, ] matches = filter_records(records, filters=ResultFilters(status="failed")) @@ -144,16 +147,17 @@ def test_failure_records_include_entity_and_task_failures(): assert [record["record_type"] for record in failures] == ["entity", "task"] -def test_failure_records_include_any_non_success_status(): +def test_failure_records_include_unsuccessful_statuses_only(): records = [ {"record_type": "entity", "status": "success"}, {"record_type": "entity", "status": "interrupted"}, {"record_type": "task", "status": "cancelled"}, + {"record_type": "task", "status": "skipped"}, ] failures = failure_records(records) - assert [record["status"] for record in failures] == ["interrupted", "cancelled"] + assert [record["status"] for record in failures] == ["interrupted"] def test_config_file_for_failure_records_groups_by_config_path(): @@ -209,7 +213,8 @@ def test_build_rerun_targets_includes_interrupted_task_dependencies(): "target": "org-a", "entity_id": "111111111111", "region": "us-west-2", - "task": "cleanup", + "task_id": "cleanup", + "task_name": "cleanup", "status": "interrupted", } ], @@ -224,6 +229,130 @@ def test_build_rerun_targets_includes_interrupted_task_dependencies(): ] +def test_build_rerun_targets_uses_invocation_ids_for_repeated_components(): + from anvil.descriptors import LoadedConfig, TargetDescriptor + + loaded_config = LoadedConfig( + targets=[ + TargetDescriptor( + name="org-a", + provider="aws", + mode="organization", + tasks=[ + {"id": "inventory_before", "name": "inventory"}, + { + "id": "inventory_after", + "name": "inventory", + "depends_on": ["inventory_before"], + }, + {"name": "notify"}, + ], + ) + ] + ) + + targets = build_rerun_targets( + loaded_config=loaded_config, + failures=[ + { + "record_type": "task", + "target": "org-a", + "entity_id": "111111111111", + "region": "us-east-1", + "task_id": "inventory_after", + "task_name": "inventory", + "status": "error", + } + ], + ) + + assert targets[0].tasks == [ + {"id": "inventory_before", "name": "inventory"}, + { + "id": "inventory_after", + "name": "inventory", + "depends_on": ["inventory_before"], + }, + ] + + +def test_build_rerun_targets_keeps_configured_target_identity(): + from anvil.descriptors import LoadedConfig, TargetDescriptor + + target = TargetDescriptor( + name="org-a", + provider="aws", + mode="organization", + include=["111111111111"], + tasks=[ + {"id": "prepare", "name": "inventory"}, + {"id": "finalize", "name": "cleanup", "depends_on": ["prepare"]}, + ], + ) + loaded_config = LoadedConfig(targets=[target]) + + targets = build_rerun_targets( + loaded_config=loaded_config, + failures=[ + { + "record_type": "task", + "target": "org-a", + "entity_id": "org-a", + "entity_type": "configured_target", + "task_id": "finalize", + "task_name": "cleanup", + "status": "error", + } + ], + ) + + assert targets[0].include == ["111111111111"] + assert targets[0].tasks == target.tasks + + +def test_build_rerun_targets_keeps_mixed_configured_and_entity_failures(): + from anvil.descriptors import LoadedConfig, TargetDescriptor + + target = TargetDescriptor( + name="org-a", + provider="aws", + mode="organization", + tasks=[ + {"id": "configured_prepare", "name": "prepare"}, + {"id": "account_mutate", "name": "mutate"}, + {"name": "notify"}, + ], + ) + + targets = build_rerun_targets( + loaded_config=LoadedConfig(targets=[target]), + failures=[ + { + "record_type": "task", + "target": "org-a", + "entity_type": "configured_target", + "task_id": "configured_prepare", + "status": "error", + }, + { + "record_type": "task", + "target": "org-a", + "entity_type": "account", + "entity_id": "111111111111", + "task_id": "account_mutate", + "status": "interrupted", + }, + ], + ) + + assert len(targets) == 1 + assert targets[0].include is None + assert targets[0].tasks == [ + {"id": "configured_prepare", "name": "prepare"}, + {"id": "account_mutate", "name": "mutate"}, + ] + + def test_parse_fields_validates_known_fields(): assert parse_fields("entity_id, entity_metadata,region") == [ "entity_id", @@ -252,14 +381,18 @@ def test_limit_records_applies_after_filtering(): def test_project_records_keeps_requested_fields_in_order(): records = build_jsonl_records_for_target(_target_result()) - projected = project_records(records, fields=["entity_id", "region", "task"]) + projected = project_records(records, fields=["entity_id", "region", "task_id"]) - assert list(projected[0]) == ["entity_id", "region", "task"] - assert projected[0] == {"entity_id": "111111111111", "region": None, "task": None} + assert list(projected[0]) == ["entity_id", "region", "task_id"] + assert projected[0] == { + "entity_id": "111111111111", + "region": None, + "task_id": None, + } assert projected[1] == { "entity_id": "111111111111", "region": "us-east-1", - "task": "count_vpcs", + "task_id": "count_vpcs", } @@ -267,7 +400,7 @@ def test_format_records_table_uses_default_and_selected_fields(): records = build_jsonl_records_for_target(_target_result()) default_table = format_records_table(records) - selected_table = format_records_table(records, fields=["entity_id", "task"]) + selected_table = format_records_table(records, fields=["entity_id", "task_id"]) assert "type" in default_table assert "entity_name" in default_table @@ -279,15 +412,18 @@ def test_format_records_table_uses_default_and_selected_fields(): def test_format_records_jsonl_outputs_one_json_object_per_line(): records = project_records( build_jsonl_records_for_target(_target_result())[:2], - fields=["entity_id", "task"], + fields=["entity_id", "task_id"], ) output = format_records_jsonl(records) lines = output.splitlines() assert len(lines) == 2 - assert json.loads(lines[0]) == {"entity_id": "111111111111", "task": None} - assert json.loads(lines[1]) == {"entity_id": "111111111111", "task": "count_vpcs"} + assert json.loads(lines[0]) == {"entity_id": "111111111111", "task_id": None} + assert json.loads(lines[1]) == { + "entity_id": "111111111111", + "task_id": "count_vpcs", + } def test_load_result_records_discovers_nested_results_jsonl(): diff --git a/tests/runner/test_organization_resolver.py b/tests/runner/test_organization_resolver.py index 88b7bd9..d6099a2 100644 --- a/tests/runner/test_organization_resolver.py +++ b/tests/runner/test_organization_resolver.py @@ -145,15 +145,35 @@ def test_filter_accounts_intersects_include_and_exclude_filters(): included = OrganizationResolver( descriptor=_target(include=["222222222222", "999999999999"]), context=_context() - )._filter_accounts(all_accounts) + )._filter_accounts(all_accounts, management_account_id="111111111111") excluded = OrganizationResolver( descriptor=_target(exclude=["111111111111", "999999999999"]), context=_context() - )._filter_accounts(all_accounts) + )._filter_accounts(all_accounts, management_account_id="111111111111") assert list(included) == ["222222222222"] assert list(excluded) == ["222222222222"] +def test_filter_accounts_expands_management_account_keywords(): + all_accounts = { + "111111111111": { + "account_number": "111111111111", + "account_alias": "management", + }, + "222222222222": {"account_number": "222222222222", "account_alias": "member"}, + } + + included = OrganizationResolver( + descriptor=_target(include=["payer", "MANAGEMENT"]), context=_context() + )._filter_accounts(all_accounts, management_account_id="111111111111") + excluded = OrganizationResolver( + descriptor=_target(exclude=["management"]), context=_context() + )._filter_accounts(all_accounts, management_account_id="111111111111") + + assert list(included) == ["111111111111"] + assert list(excluded) == ["222222222222"] + + def test_resolve_accounts_uses_default_management_account_direct_mode(): discovered_accounts = { "111111111111": { diff --git a/tests/runner/test_provider_execution.py b/tests/runner/test_provider_execution.py index 0ec49f9..b7c54ba 100644 --- a/tests/runner/test_provider_execution.py +++ b/tests/runner/test_provider_execution.py @@ -4,6 +4,8 @@ import time from dataclasses import dataclass +import pytest + from anvil.descriptors import TargetDescriptor from anvil.execution_context import ExecutionContext from anvil.providers.base import ExecutionTarget, ProviderMetadata @@ -58,6 +60,41 @@ def prepare_execution_runtime( return _Runtime(target_id=execution_target.id, calls=self._calls) +class _BenchmarkRuntime(_Runtime): + @property + def benchmark(self) -> dict[str, object]: + return {} + + +class _BenchmarkProvider(_Provider): + def prepare_execution_runtime( + self, + *, + target: TargetDescriptor, + execution_target: ExecutionTarget, + context: ExecutionContext, + ) -> _BenchmarkRuntime: + return _BenchmarkRuntime(target_id=execution_target.id, calls=self._calls) + + +class _TimedLifecycleRuntime(_Runtime): + def close(self) -> None: + super().close() + self._calls["runtime_ended_perf"] = time.perf_counter() + + +class _TimedLifecycleProvider(_Provider): + def prepare_execution_runtime( + self, + *, + target: TargetDescriptor, + execution_target: ExecutionTarget, + context: ExecutionContext, + ) -> _TimedLifecycleRuntime: + self._calls["runtime_started_perf"] = time.perf_counter() + return _TimedLifecycleRuntime(target_id=execution_target.id, calls=self._calls) + + def _target(*, max_workers: int = 1) -> TargetDescriptor: return TargetDescriptor( name="provider-target", @@ -103,12 +140,18 @@ def _task( name: str, run, *, - optional: bool = False, scope: TaskScope = TaskScope.REGION, depends_on: list[str] | None = None, + dependency_data: dict[str, dict[str, str]] | None = None, + always_run: bool = False, ) -> ResolvedTask: return ResolvedTask( - name=name, run=run, depends_on=depends_on or [], optional=optional, scope=scope + name=name, + run=run, + depends_on=depends_on or [], + scope=scope, + dependency_data=dependency_data or {}, + always_run=always_run, ) @@ -138,6 +181,33 @@ def run(**kwargs): assert result.tasks[0].result == {"resources": ["from-all-locations"]} +def test_target_task_preserves_target_benchmark_shape() -> None: + result = _execute_provider_execution_target( + provider=_BenchmarkProvider(calls={}), + target=_target(), + execution_target=_execution_target( + "target-a", regions=["region-a", "region-b"] + ), + context=_context( + tasks=[ + _task( + "inventory", lambda **kwargs: {"ok": True}, scope=TaskScope.TARGET + ) + ] + ), + ) + + assert result.benchmark is not None + assert result.benchmark["target_execution_seconds"] >= 0.0 + assert result.benchmark["target"] == { + "region": "region-a", + "task_count": 1, + "interrupted": False, + "failed": False, + } + assert result.benchmark["regions"] == {} + + def test_mixed_target_and_region_tasks_run_in_separate_phases() -> None: invocations: list[tuple[str, str]] = [] @@ -178,7 +248,88 @@ def region_run(**kwargs): ] -def test_optional_target_failure_blocks_only_dependent_region_tasks() -> None: +def test_region_results_release_target_fan_in_in_configured_region_order() -> None: + received: list[object] = [] + + def regional_run(**kwargs): + if kwargs["region"] == "region-a": + time.sleep(0.02) + return kwargs["region"] + + def target_run(**kwargs): + received.append(kwargs["dependency_data"]["regions"]) + return {"summarized": True} + + context = _context( + tasks=[ + _task("regional", regional_run), + _task( + "summary", + target_run, + scope=TaskScope.TARGET, + depends_on=["regional"], + dependency_data={"regions": {"task_id": "regional", "path": "result"}}, + ), + ], + max_parallel_regions=2, + ) + + result = _execute_provider_execution_target( + provider=_BenchmarkProvider(calls={}), + target=_target(), + execution_target=_execution_target( + "target-a", regions=["region-a", "region-b"] + ), + context=context, + ) + + assert result.status is ExecutionStatus.SUCCESS + assert received == [["region-a", "region-b"]] + assert [(task.task_id, task.region) for task in result.tasks] == [ + ("regional", "region-a"), + ("regional", "region-b"), + ("summary", "region-a"), + ] + assert result.benchmark is not None + assert result.benchmark["target"]["task_count"] == 1 + assert { + region: metrics["task_count"] + for region, metrics in result.benchmark["regions"].items() + } == {"region-a": 1, "region-b": 1} + + +def test_graph_entity_duration_includes_runtime_lifecycle() -> None: + calls: dict[str, object] = {} + context = _context( + tasks=[ + _task("regional", lambda **kwargs: kwargs["region"]), + _task( + "summary", + lambda **kwargs: {"ok": True}, + scope=TaskScope.TARGET, + depends_on=["regional"], + ), + ] + ) + + result = _execute_provider_execution_target( + provider=_TimedLifecycleProvider(calls=calls), + target=_target(), + execution_target=_execution_target("target-a"), + context=context, + ) + + started_perf = calls["runtime_started_perf"] + ended_perf = calls["runtime_ended_perf"] + assert isinstance(started_perf, float) + assert isinstance(ended_perf, float) + observed_lifecycle_seconds = ended_perf - started_perf + assert result.duration_seconds >= observed_lifecycle_seconds + assert result.started_at <= result.tasks[0].started_at + assert result.ended_at >= result.tasks[-1].ended_at + + +def test_task_failure_does_not_suppress_independent_work() -> None: independent_regions: list[str] = [] def target_run(**kwargs): @@ -193,8 +344,8 @@ def independent_run(**kwargs): context = _context( tasks=[ - _task("target", target_run, optional=True, scope=TaskScope.TARGET), - _task("dependent", dependent_run, optional=True, depends_on=["target"]), + _task("target", target_run, scope=TaskScope.TARGET), + _task("dependent", dependent_run, depends_on=["target"]), _task("independent", independent_run), ] ) @@ -208,14 +359,193 @@ def independent_run(**kwargs): context=context, ) - assert result.status is ExecutionStatus.SUCCESS + assert result.status is ExecutionStatus.ERROR assert independent_regions == ["region-a", "region-b"] - assert [task.status for task in result.tasks if task.task_name == "dependent"] == [ - ExecutionStatus.ERROR, - ExecutionStatus.ERROR, + assert [(task.task_name, task.status) for task in result.tasks] == [ + ("target", ExecutionStatus.ERROR), + ("dependent", ExecutionStatus.SKIPPED), + ("independent", ExecutionStatus.SUCCESS), + ("dependent", ExecutionStatus.SKIPPED), + ("independent", ExecutionStatus.SUCCESS), ] +def test_target_failure_with_fail_fast_runs_activated_regional_finalizers() -> None: + cleanup_regions: list[str] = [] + + def fail_target(**kwargs): + raise RuntimeError("target failed") + + tasks = [ + _task("producer", fail_target, scope=TaskScope.TARGET), + _task( + "cleanup", + lambda **kwargs: cleanup_regions.append(kwargs["region"]), + depends_on=["producer"], + always_run=True, + ), + ] + + result = _execute_provider_execution_target( + provider=_Provider(calls={}), + target=_target(), + execution_target=_execution_target( + "target-a", regions=["region-a", "region-b"] + ), + context=_context(tasks=tasks, fail_fast=True), + ) + + assert cleanup_regions == ["region-a", "region-b"] + assert [ + (task.task_id, task.region, task.status, task.skip_reason) + for task in result.tasks + ] == [ + ("producer", "region-a", ExecutionStatus.ERROR, None), + ("cleanup", "region-a", ExecutionStatus.SUCCESS, None), + ("cleanup", "region-b", ExecutionStatus.SUCCESS, None), + ] + assert result.status is ExecutionStatus.ERROR + + +@pytest.mark.parametrize("stop_kind", ["fail_fast", "cancellation"]) +def test_transitive_target_chain_activates_regional_finalizer(stop_kind: str) -> None: + cleanup_regions: list[str] = [] + context_holder: dict[str, ExecutionContext] = {} + + def producer(**kwargs): + if stop_kind == "fail_fast": + raise RuntimeError("target failed") + context_holder["context"].cancel_event.set() + return {"started": True} + + tasks = [ + _task("producer", producer, scope=TaskScope.TARGET), + _task( + "blocked", + lambda **kwargs: (_ for _ in ()).throw( + AssertionError("blocked task must not run") + ), + scope=TaskScope.TARGET, + depends_on=["producer"], + ), + _task( + "cleanup", + lambda **kwargs: cleanup_regions.append(kwargs["region"]), + depends_on=["blocked"], + always_run=True, + ), + ] + context = _context(tasks=tasks, fail_fast=stop_kind == "fail_fast") + context_holder["context"] = context + + result = _execute_provider_execution_target( + provider=_Provider(calls={}), + target=_target(), + execution_target=_execution_target( + "target-a", regions=["region-a", "region-b"] + ), + context=context, + ) + + assert cleanup_regions == ["region-a", "region-b"] + assert [(task.task_id, task.status, task.skip_reason) for task in result.tasks] == [ + ( + "producer", + ( + ExecutionStatus.ERROR + if stop_kind == "fail_fast" + else ExecutionStatus.SUCCESS + ), + None, + ), + ( + "blocked", + ExecutionStatus.SKIPPED, + "fail_fast" if stop_kind == "fail_fast" else "cancelled_before_start", + ), + ("cleanup", ExecutionStatus.SUCCESS, None), + ("cleanup", ExecutionStatus.SUCCESS, None), + ] + assert result.status is ( + ExecutionStatus.ERROR + if stop_kind == "fail_fast" + else ExecutionStatus.INTERRUPTED + ) + + +def test_target_fail_fast_settles_regions_without_unused_sessions() -> None: + calls: dict[str, object] = {} + tasks = [ + _task( + "producer", + lambda **kwargs: (_ for _ in ()).throw(RuntimeError("target failed")), + scope=TaskScope.TARGET, + ), + _task("ordinary", lambda **kwargs: {"unexpected": True}), + ] + + result = _execute_provider_execution_target( + provider=_Provider(calls=calls), + target=_target(), + execution_target=_execution_target( + "target-a", regions=["region-a", "region-b"] + ), + context=_context(tasks=tasks, fail_fast=True), + ) + + assert [ + (task.task_id, task.region, task.status, task.skip_reason) + for task in result.tasks + ] == [ + ("producer", "region-a", ExecutionStatus.ERROR, None), + ("ordinary", "region-a", ExecutionStatus.SKIPPED, "fail_fast"), + ("ordinary", "region-b", ExecutionStatus.SKIPPED, "fail_fast"), + ] + assert calls["build_sessions"] == [("target-a", "region-a")] + + +@pytest.mark.parametrize("stop_kind", ["fail_fast", "cancellation"]) +def test_unactivated_regional_finalizer_does_not_build_unused_sessions( + stop_kind: str, +) -> None: + calls: dict[str, object] = {} + tasks = [ + _task("producer", lambda **kwargs: {"unexpected": True}), + _task( + "cleanup", + lambda **kwargs: {"unexpected": True}, + depends_on=["producer"], + always_run=True, + ), + ] + context = _context(tasks=tasks, fail_fast=stop_kind == "fail_fast") + stop_event = ( + context.fail_fast_event if stop_kind == "fail_fast" else context.cancel_event + ) + stop_event.set() + skip_reason = "fail_fast" if stop_kind == "fail_fast" else "cancelled_before_start" + + result = _execute_provider_execution_target( + provider=_Provider(calls=calls), + target=_target(), + execution_target=_execution_target( + "target-a", regions=["region-a", "region-b"] + ), + context=context, + ) + + assert [ + (task.task_id, task.region, task.status, task.skip_reason) + for task in result.tasks + ] == [ + ("producer", "region-a", ExecutionStatus.SKIPPED, skip_reason), + ("cleanup", "region-a", ExecutionStatus.SKIPPED, skip_reason), + ("producer", "region-b", ExecutionStatus.SKIPPED, skip_reason), + ("cleanup", "region-b", ExecutionStatus.SKIPPED, skip_reason), + ] + assert calls.get("build_sessions", []) == [] + + def test_provider_execution_respects_max_parallel_regions() -> None: active_regions = 0 max_active_regions = 0 @@ -252,7 +582,7 @@ def run(**kwargs): ] -def test_provider_execution_stops_launching_regions_after_required_failure() -> None: +def test_provider_execution_continues_independent_regions_without_fail_fast() -> None: def run(**kwargs): raise RuntimeError(f"failed {kwargs['region']}") @@ -269,9 +599,10 @@ def run(**kwargs): ) assert result.status is ExecutionStatus.ERROR - assert [call[1] for call in calls["build_sessions"]] == ["region-a"] + assert [call[1] for call in calls["build_sessions"]] == ["region-a", "region-b"] assert [call[1:] for call in calls["region_outcomes"]] == [ - ("region-a", True, False) + ("region-a", True, False), + ("region-b", True, False), ] @@ -323,7 +654,8 @@ def run(**kwargs): benchmark_data=None, ) - assert context.cancel_event.is_set() + assert context.fail_fast_event.is_set() + assert not context.cancel_event.is_set() assert [entity.id for entity in result.entities] == ["target-a"] assert result.entities[0].status is ExecutionStatus.ERROR assert [call[0] for call in calls["build_sessions"]] == ["target-a"] diff --git a/tests/runner/test_provider_lifecycle.py b/tests/runner/test_provider_lifecycle.py new file mode 100644 index 0000000..a174560 --- /dev/null +++ b/tests/runner/test_provider_lifecycle.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +import pytest + +from anvil.provider_lifecycle import CoordinateLifecycleState +from anvil.results import ExecutionStatus, TaskResult + + +def _result(status: ExecutionStatus, *, skip_reason: str | None = None) -> TaskResult: + return TaskResult( + task_id="task", + task_name="task", + region="region-a", + status=status, + started_at="2026-01-01T00:00:00+00:00", + ended_at="2026-01-01T00:00:00+00:00", + duration_seconds=0.0, + skip_reason=skip_reason, + ) + + +def test_coordinate_lifecycle_state_uses_constant_size_aggregation() -> None: + instance_count = 10_000 + state = CoordinateLifecycleState(remaining_instances=instance_count) + success = _result(ExecutionStatus.SUCCESS) + + for index in range(instance_count): + settled = state.record_settlement( + result=success, region_scoped=True, ended_perf=float(index) + ) + + assert settled is True + assert state.remaining_instances == 0 + assert state.failed is False + assert state.interrupted is False + assert state.region_ended_perf == instance_count - 1 + assert not hasattr(state, "results") + + +@pytest.mark.parametrize( + ("result", "failed", "interrupted"), + [ + (_result(ExecutionStatus.ERROR), True, False), + (_result(ExecutionStatus.INTERRUPTED), False, True), + ( + _result(ExecutionStatus.SKIPPED, skip_reason="cancelled_before_start"), + False, + True, + ), + ( + _result(ExecutionStatus.SKIPPED, skip_reason="dependency_unsuccessful"), + False, + False, + ), + ], +) +def test_coordinate_lifecycle_state_preserves_outcome_semantics( + result: TaskResult, failed: bool, interrupted: bool +) -> None: + state = CoordinateLifecycleState(remaining_instances=1) + + assert state.record_settlement(result=result, region_scoped=False, ended_perf=1.0) + assert state.failed is failed + assert state.interrupted is interrupted diff --git a/tests/runner/test_runner_flow.py b/tests/runner/test_runner_flow.py index 4a90df0..6dce630 100644 --- a/tests/runner/test_runner_flow.py +++ b/tests/runner/test_runner_flow.py @@ -179,6 +179,71 @@ def test_provider_session_failure_becomes_entity_error(monkeypatch): assert entity.error == "session failed" +def test_fail_fast_task_error_remains_engine_failure(monkeypatch): + provider = _Provider() + _patch_provider(monkeypatch, provider) + + def fail(**kwargs): + raise RuntimeError("task failed") + + monkeypatch.setattr( + "anvil.runner.resolve_tasks", + lambda **kwargs: ResolvedExecution( + ordered=[ResolvedTask("failing", fail, depends_on=[])], adjacency={} + ), + ) + + result = run_multiple_targets( + targets=[_target(tasks=[{"name": "failing"}], fail_fast=True)], + max_parallel_targets=1, + cli_dry_run=None, + cli_include=None, + cli_exclude=None, + ) + + assert result.state is EngineState.COMPLETED_WITH_FAILURES + assert result.target_results[0].entities[0].status is ExecutionStatus.ERROR + + +def test_task_error_outweighs_concurrent_external_cancellation(monkeypatch): + provider = _Provider() + _patch_provider(monkeypatch, provider) + context_holder: dict[str, ExecutionContext] = {} + + def fail_during_cancellation(**kwargs): + context_holder["context"].cancel_event.set() + raise RuntimeError("task failed during cancellation") + + monkeypatch.setattr( + "anvil.runner.resolve_tasks", + lambda **kwargs: ResolvedExecution( + ordered=[ResolvedTask("failing", fail_during_cancellation, depends_on=[])], + adjacency={}, + ), + ) + from anvil import runner as runner_module + + original_build_context = runner_module._build_execution_context + + def capture_context(**kwargs): + context = original_build_context(**kwargs) + context_holder["context"] = context + return context + + monkeypatch.setattr("anvil.runner._build_execution_context", capture_context) + + result = run_multiple_targets( + targets=[_target(tasks=[{"name": "failing"}])], + max_parallel_targets=1, + cli_dry_run=None, + cli_include=None, + cli_exclude=None, + ) + + assert result.state is EngineState.COMPLETED_WITH_FAILURES + assert result.target_results[0].entities[0].status is ExecutionStatus.ERROR + + def test_universal_task_receives_provider_neutral_kwargs_and_records_actions( monkeypatch, ): @@ -196,6 +261,7 @@ def task( session, dry_run, metadata, + dependency_data, actions, ): seen.update(locals()) @@ -205,8 +271,7 @@ def task( monkeypatch.setattr( "anvil.runner.resolve_tasks", lambda **kwargs: ResolvedExecution( - ordered=[ResolvedTask("neutral", task, depends_on=[], optional=False)], - adjacency={}, + ordered=[ResolvedTask("neutral", task, depends_on=[])], adjacency={} ), ) @@ -229,6 +294,7 @@ def task( assert seen["session"].region_name == "global" assert seen["dry_run"] is False assert seen["metadata"] == {"team": "security"} + assert seen["dependency_data"] == {} def test_prepare_target_carries_provider_preflight_and_execution_controls(monkeypatch): diff --git a/tests/task_redesign/test_declaration_contract.py b/tests/task_redesign/test_declaration_contract.py new file mode 100644 index 0000000..9394844 --- /dev/null +++ b/tests/task_redesign/test_declaration_contract.py @@ -0,0 +1,444 @@ +from __future__ import annotations + +import sys +from types import ModuleType + +import pytest + +from anvil.task_loader import TaskConfigError, TaskScope, resolve_tasks +from anvil.validators import validate_config_schema + + +def _config(tasks: list[dict[str, object]]) -> dict[str, object]: + return { + "schema_version": 2, + "targets": [ + { + "name": "contract-target", + "provider": {"name": "aws", "mode": "accounts"}, + "include": ["111111111111"], + "regions": ["us-east-1"], + "tasks": tasks, + } + ], + } + + +def _install_task_modules( + monkeypatch: pytest.MonkeyPatch, scopes: dict[str, str] +) -> None: + runs = {} + for task_name, scope in scopes.items(): + module_name = f"tests.task_redesign.fake_{task_name}" + module = ModuleType(module_name) + module.TASK_SCOPE = scope + + def run(**kwargs): + return kwargs + + run.__module__ = module_name + module.run = run + monkeypatch.setitem(sys.modules, module_name, module) + runs[task_name] = run + + monkeypatch.setattr( + "anvil.task_loader._load_provider_task_callable", + lambda *, provider_name, task_name: runs[task_name], + ) + resolve_tasks.__globals__["_resolve_tasks_cached"].cache_clear() + + +def test_schema_v2_accepts_complete_redesigned_task_declaration() -> None: + validate_config_schema( + config=_config( + [ + { + "id": "restore_guardrails", + "name": "reconcile_config_guardrails", + "depends_on": ["detach_guardrails"], + "always_run": True, + "metadata": {"attachment_state": "present"}, + "dependency_data": { + "attachments": { + "task_id": "detach_guardrails", + "path": "result.attachments", + } + }, + }, + { + "id": "detach_guardrails", + "name": "reconcile_config_guardrails", + "metadata": {"attachment_state": "absent"}, + }, + ] + ) + ) + + +@pytest.mark.parametrize( + ("invalid_id", "expected_detail"), [("", "non-empty"), (None, "string")] +) +def test_schema_v2_rejects_empty_or_null_explicit_id( + invalid_id: object, expected_detail: str +) -> None: + with pytest.raises(ValueError) as error: + validate_config_schema(config=_config([{"id": invalid_id, "name": "noop"}])) + + assert "id" in str(error.value) + assert expected_detail in str(error.value) + + +def test_schema_documents_every_task_property_with_examples() -> None: + import json + from importlib.resources import files + + schema = json.loads( + files("anvil.schemas") + .joinpath("common.schema.v2.json") + .read_text(encoding="utf-8") + ) + properties = schema["$defs"]["taskEntry"]["properties"] + + assert set(properties) == { + "id", + "name", + "metadata", + "depends_on", + "always_run", + "dependency_data", + } + for property_name, declaration in properties.items(): + assert declaration.get("description"), property_name + assert declaration.get("examples"), property_name + + nested_declarations = { + "depends_on.items": properties["depends_on"]["items"], + "dependency_data.propertyNames": properties["dependency_data"]["propertyNames"], + "dependencyDataReference": schema["$defs"]["dependencyDataReference"], + **{ + f"dependencyDataReference.{property_name}": declaration + for property_name, declaration in schema["$defs"][ + "dependencyDataReference" + ]["properties"].items() + }, + } + for property_name, declaration in nested_declarations.items(): + assert declaration.get("description"), property_name + assert declaration.get("examples"), property_name + + +def test_schema_rejects_yaml_task_scope_override() -> None: + with pytest.raises(ValueError, match=r"scope"): + validate_config_schema( + config=_config([{"name": "noop", "scope": "configured_target"}]) + ) + + +def test_repeated_component_names_resolve_by_explicit_invocation_id( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"reconcile": "region"}) + + execution = resolve_tasks( + task_specs=[ + {"id": "detach", "name": "reconcile"}, + {"id": "restore", "name": "reconcile", "depends_on": ["detach"]}, + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + assert [(task.id, task.name) for task in execution.ordered] == [ + ("detach", "reconcile"), + ("restore", "reconcile"), + ] + assert execution.ordered[1].depends_on == ["detach"] + + +def test_repeated_component_names_require_explicit_ids( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"reconcile": "region"}) + + with pytest.raises(TaskConfigError, match=r"explicit.*unique.*ID"): + resolve_tasks( + task_specs=[{"name": "reconcile"}, {"name": "reconcile"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +def test_omitted_id_defaults_to_component_name(monkeypatch: pytest.MonkeyPatch) -> None: + _install_task_modules(monkeypatch, {"inventory": "region"}) + + execution = resolve_tasks( + task_specs=[{"name": "inventory"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + assert execution.ordered[0].id == "inventory" + assert execution.ordered[0].name == "inventory" + + +def test_effective_task_ids_must_be_unique(monkeypatch: pytest.MonkeyPatch) -> None: + _install_task_modules(monkeypatch, {"inventory": "region", "cleanup": "region"}) + + with pytest.raises(TaskConfigError, match=r"Duplicate task ID.*shared"): + resolve_tasks( + task_specs=[ + {"id": "shared", "name": "inventory"}, + {"id": "shared", "name": "cleanup"}, + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +def test_every_repeated_component_occurrence_requires_explicit_id( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"reconcile": "region"}) + + with pytest.raises(TaskConfigError, match=r"every occurrence.*explicit.*ID"): + resolve_tasks( + task_specs=[{"name": "reconcile"}, {"id": "second", "name": "reconcile"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +def test_dependencies_do_not_fall_back_to_component_names( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"producer": "region", "consumer": "region"}) + + with pytest.raises(TaskConfigError, match=r"unknown task ID.*producer"): + resolve_tasks( + task_specs=[ + {"id": "producer_invocation", "name": "producer"}, + {"name": "consumer", "depends_on": ["producer"]}, + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +def test_dependency_cycles_are_reported_by_invocation_id( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules( + monkeypatch, {"component_a": "region", "component_b": "region"} + ) + + with pytest.raises(TaskConfigError, match=r"Cycle.*invocation_a.*invocation_b"): + resolve_tasks( + task_specs=[ + { + "id": "invocation_a", + "name": "component_a", + "depends_on": ["invocation_b"], + }, + { + "id": "invocation_b", + "name": "component_b", + "depends_on": ["invocation_a"], + }, + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +def test_dependency_data_requires_direct_dependency( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"producer": "region", "consumer": "region"}) + + with pytest.raises(TaskConfigError, match=r"direct.*depends_on"): + resolve_tasks( + task_specs=[ + {"name": "producer"}, + { + "name": "consumer", + "dependency_data": { + "payload": {"task_id": "producer", "path": "result.value"} + }, + }, + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +@pytest.mark.parametrize( + "dependency_data", + [ + {"": {"task_id": "producer"}}, + {"payload": {"task_id": ""}}, + {"payload": {"task_id": "producer", "path": ""}}, + {"payload": {"task_id": "producer", "path": ".result"}}, + {"payload": {"task_id": "producer", "path": "result..value"}}, + {"payload": {"task_id": "producer", "path": "result[0]"}}, + {"payload": {"task_id": "producer", "path": "result", "unknown": True}}, + ], +) +def test_schema_rejects_invalid_dependency_data_references( + dependency_data: dict[str, object], +) -> None: + with pytest.raises(ValueError, match=r"dependency_data|path|task_id"): + validate_config_schema( + config=_config( + [ + {"name": "producer"}, + { + "name": "consumer", + "depends_on": ["producer"], + "dependency_data": dependency_data, + }, + ] + ) + ) + + +def test_always_run_requires_at_least_one_dependency( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"cleanup": "region"}) + + with pytest.raises(TaskConfigError, match=r"always_run.*depend"): + resolve_tasks( + task_specs=[{"name": "cleanup", "always_run": True}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + +def test_configured_target_is_a_module_declared_scope( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"organization_task": "configured_target"}) + + execution = resolve_tasks( + task_specs=[{"name": "organization_task"}], + provider_name="aws", + supported_task_scopes=frozenset({"configured_target", "target", "region"}), + ) + + assert execution.ordered[0].scope is TaskScope.CONFIGURED_TARGET + + +def test_task_metadata_and_dependency_data_are_preserved_and_isolated( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"producer": "region", "consumer": "region"}) + specs = [ + {"name": "producer"}, + { + "name": "consumer", + "depends_on": ["producer"], + "always_run": True, + "metadata": {"nested": {"items": ["configured"]}}, + "dependency_data": { + "payload": {"task_id": "producer", "path": "result.value"} + }, + }, + ] + + first = resolve_tasks( + task_specs=specs, + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + first.ordered[1].metadata["nested"]["items"].append("mutated") + first.ordered[1].dependency_data["payload"]["path"] = "result.changed" + + second = resolve_tasks( + task_specs=specs, + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + + assert second.ordered[1].always_run + assert second.ordered[1].metadata == {"nested": {"items": ["configured"]}} + assert second.ordered[1].dependency_data == { + "payload": {"task_id": "producer", "path": "result.value"} + } + + +def test_resolution_cache_distinguishes_invocation_ids( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"inventory": "region"}) + cached_resolver = resolve_tasks.__globals__["_resolve_tasks_cached"] + + resolve_tasks( + task_specs=[{"id": "first", "name": "inventory"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + after_first = cached_resolver.cache_info() + resolve_tasks( + task_specs=[{"id": "second", "name": "inventory"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + after_second = cached_resolver.cache_info() + + assert after_second.misses == after_first.misses + 1 + + +def test_resolution_cache_normalizes_mapping_order( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"inventory": "region"}) + cached_resolver = resolve_tasks.__globals__["_resolve_tasks_cached"] + + first = resolve_tasks( + task_specs=[ + { + "name": "inventory", + "metadata": {"alpha": 1, "nested": {"left": True, "right": False}}, + } + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + after_first = cached_resolver.cache_info() + second = resolve_tasks( + task_specs=[ + { + "name": "inventory", + "metadata": {"nested": {"right": False, "left": True}, "alpha": 1}, + } + ], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + after_second = cached_resolver.cache_info() + + assert after_second.hits == after_first.hits + 1 + assert first.ordered[0].metadata == second.ordered[0].metadata + + +def test_resolution_cache_uses_effective_id_for_implicit_and_explicit_forms( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _install_task_modules(monkeypatch, {"inventory": "region"}) + cached_resolver = resolve_tasks.__globals__["_resolve_tasks_cached"] + + implicit = resolve_tasks( + task_specs=[{"name": "inventory"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + after_implicit = cached_resolver.cache_info() + explicit = resolve_tasks( + task_specs=[{"id": "inventory", "name": "inventory"}], + provider_name="aws", + supported_task_scopes=frozenset({"region"}), + ) + after_explicit = cached_resolver.cache_info() + + assert after_explicit.hits == after_implicit.hits + 1 + assert implicit.ordered[0].id == explicit.ordered[0].id == "inventory" diff --git a/tests/task_redesign/test_execution_and_result_contract.py b/tests/task_redesign/test_execution_and_result_contract.py new file mode 100644 index 0000000..85048d2 --- /dev/null +++ b/tests/task_redesign/test_execution_and_result_contract.py @@ -0,0 +1,258 @@ +from __future__ import annotations + +import inspect + +from anvil.result_query import ( + ResultFilters, + build_jsonl_records_for_target, + failure_records, + filter_records, +) +from anvil.results import ( + EngineResult, + EngineState, + EntityResult, + ExecutionStatus, + TargetResult, + TaskResult, +) +from anvil.task_context import TaskCallContext + + +def _task_result( + *, + task_id: str, + task_name: str, + status: ExecutionStatus, + result: object | None = None, + error: str | None = None, + skip_reason: str | None = None, +) -> TaskResult: + required_fields = inspect.signature(TaskResult).parameters + assert "task_id" in required_fields + assert "skip_reason" in required_fields + kwargs = { + "task_id": task_id, + "task_name": task_name, + "region": "us-east-1", + "status": status, + "started_at": "2026-07-28T00:00:00+00:00", + "ended_at": "2026-07-28T00:00:01+00:00", + "duration_seconds": 1.0, + "result": result, + "error": error, + "skip_reason": skip_reason, + } + return TaskResult(**kwargs) + + +def _entity(status: ExecutionStatus, tasks: list[TaskResult]) -> EntityResult: + return EntityResult( + id="111111111111", + name="account", + type="account", + provider="aws", + metadata={}, + status=status, + tasks=tasks, + started_at="2026-07-28T00:00:00+00:00", + ended_at="2026-07-28T00:00:01+00:00", + duration_seconds=1.0, + ) + + +def test_task_call_contract_has_separate_dependency_data() -> None: + assert TaskCallContext.keyword_names() == frozenset( + { + "provider", + "execution_target_id", + "execution_target_name", + "execution_target_type", + "region", + "session", + "dry_run", + "metadata", + "dependency_data", + "actions", + } + ) + + +def test_task_context_deeply_isolates_metadata_and_dependency_data() -> None: + parameters = inspect.signature(TaskCallContext).parameters + assert "dependency_data" in parameters + + metadata = {"nested": {"items": ["original"]}} + dependency_data = {"payload": {"items": ["original"]}} + common_kwargs = { + "provider": "aws", + "execution_target_type": "account", + "session": object(), + "dry_run": False, + "metadata": metadata, + "dependency_data": dependency_data, + "actions": object(), + } + first = TaskCallContext( + **common_kwargs, + execution_target_id="111111111111", + execution_target_name="account", + region="us-east-1", + ).to_kwargs() + second = TaskCallContext( + **common_kwargs, + execution_target_id="222222222222", + execution_target_name="other", + region="us-west-2", + ).to_kwargs() + + first["metadata"]["nested"]["items"].append("mutated") + first["dependency_data"]["payload"]["items"].append("mutated") + + assert second["metadata"] == {"nested": {"items": ["original"]}} + assert second["dependency_data"] == {"payload": {"items": ["original"]}} + assert metadata == {"nested": {"items": ["original"]}} + assert dependency_data == {"payload": {"items": ["original"]}} + + +def test_skipped_is_neutral_and_not_unsuccessful() -> None: + skipped = ExecutionStatus("skipped") + + assert skipped.is_skipped + assert not skipped.is_error + assert not skipped.is_interrupted + assert not skipped.is_unsuccessful + + +def test_task_result_serializes_invocation_id_and_component_name() -> None: + payload = _task_result( + task_id="detach_guardrails", + task_name="reconcile_config_guardrails", + status=ExecutionStatus.SUCCESS, + ).to_dict() + + assert payload["task_id"] == "detach_guardrails" + assert payload["task_name"] == "reconcile_config_guardrails" + assert "task" not in payload + + +def test_failed_queries_exclude_skipped_tasks() -> None: + records = [ + {"record_type": "task", "status": "skipped", "task_id": "blocked"}, + {"record_type": "task", "status": "error", "task_id": "failed"}, + ] + + assert failure_records(records) == [records[1]] + assert filter_records(records, filters=ResultFilters(status="failed")) == [ + records[1] + ] + + +def test_success_plus_skipped_aggregates_to_success_and_counts_skip() -> None: + skipped = ExecutionStatus("skipped") + tasks = [ + _task_result( + task_id="successful", task_name="noop", status=ExecutionStatus.SUCCESS + ), + _task_result( + task_id="blocked", + task_name="noop", + status=skipped, + skip_reason="dependency_unsuccessful", + ), + ] + target = TargetResult.create( + target_name="target", + provider="aws", + dry_run=False, + entities=[_entity(ExecutionStatus.SUCCESS, tasks)], + ) + engine = EngineResult.create( + state=EngineState.COMPLETED_SUCCESS, auth_results=[], target_results=[target] + ) + + assert not target.has_failures + summary = engine.build_summary() + assert summary["total_skipped_tasks"] == 1 + assert summary["total_interrupted_tasks"] == 0 + assert engine.state is EngineState.COMPLETED_SUCCESS + + +def test_successful_finalizer_does_not_erase_upstream_failure() -> None: + tasks = [ + _task_result( + task_id="mutate", + task_name="mutate", + status=ExecutionStatus.ERROR, + result={"changed": ["resource-a"]}, + error="mutation failed", + ), + _task_result( + task_id="restore", task_name="restore", status=ExecutionStatus.SUCCESS + ), + ] + target = TargetResult.create( + target_name="target", + provider="aws", + dry_run=False, + entities=[_entity(ExecutionStatus.ERROR, tasks)], + ) + + assert target.has_failures + assert target.entities[0].status is ExecutionStatus.ERROR + assert target.entities[0].tasks[0].result == {"changed": ["resource-a"]} + + +def test_configured_target_results_are_stored_directly_on_target() -> None: + parameters = inspect.signature(TargetResult).parameters + assert "tasks" in parameters + configured_task = _task_result( + task_id="configured_inventory", + task_name="inventory", + status=ExecutionStatus.SUCCESS, + ) + target = TargetResult.create( + **{ + "target_name": "target", + "provider": "aws", + "dry_run": False, + "entities": [], + "tasks": [configured_task], + } + ) + + assert target.tasks == [configured_task] + assert target.to_dict()["tasks"][0]["task_id"] == "configured_inventory" + + +def test_configured_target_results_are_present_in_jsonl_and_summary() -> None: + parameters = inspect.signature(TargetResult).parameters + assert "tasks" in parameters + configured_task = _task_result( + task_id="configured_inventory", + task_name="inventory", + status=ExecutionStatus.SUCCESS, + ) + target = TargetResult.create( + **{ + "target_name": "target", + "provider": "aws", + "dry_run": False, + "entities": [], + "tasks": [configured_task], + } + ) + engine = EngineResult.create( + state=EngineState.COMPLETED_SUCCESS, auth_results=[], target_results=[target] + ) + + task_records = [ + record + for record in build_jsonl_records_for_target(target) + if record["record_type"] == "task" + ] + assert len(task_records) == 1 + assert task_records[0]["task_id"] == "configured_inventory" + assert task_records[0]["task_name"] == "inventory" + assert task_records[0]["entity_type"] == "configured_target" + assert engine.build_summary()["targets"][0]["total_tasks"] == 1 diff --git a/tests/task_redesign/test_provider_and_preservation_contract.py b/tests/task_redesign/test_provider_and_preservation_contract.py new file mode 100644 index 0000000..37e4255 --- /dev/null +++ b/tests/task_redesign/test_provider_and_preservation_contract.py @@ -0,0 +1,1317 @@ +from __future__ import annotations + +import inspect +import importlib +import threading +import time +from dataclasses import dataclass +from types import SimpleNamespace + +import pytest + +from anvil.descriptors import TargetDescriptor +from anvil.execution_context import ExecutionContext +from anvil.providers.aws.provider import AwsPreflightData, AwsProvider +from anvil.providers.base import ( + ExecutionTarget, + ProviderAuthResult, + ProviderExecutionPlan, + ProviderMetadata, + ProviderPreparation, +) +from anvil.results import AuthResult, ExecutionStatus +from anvil.runner import ( + AuthCheckCache, + PreparedTarget, + _SingleFlightCache, + _execute_provider_execution_target, + _execute_provider_targets, + prepare_target, + run_prepared_target, +) +from anvil.task_loader import ResolvedTask, TaskScope, discover_tasks + + +@dataclass +class _BaseSession: + profile_name: str | None = "management" + + +class _SessionFactory: + pass + + +def _aws_preflight() -> AwsPreflightData: + return AwsPreflightData( + session_factory=_SessionFactory(), + base_session=_BaseSession(), + organization_id="o-contract", + management_account_id="111111111111", + base_session_account_id="111111111111", + discovered_accounts={ + "111111111111": { + "account_number": "111111111111", + "account_alias": "management", + }, + "222222222222": { + "account_number": "222222222222", + "account_alias": "member", + }, + }, + region_statuses={"us-east-1": "ENABLED_BY_DEFAULT"}, + ) + + +def test_aws_configured_target_uses_management_identity_when_excluded() -> None: + provider = AwsProvider() + target = TargetDescriptor( + name="organization", + provider="aws", + mode="organization", + exclude=["111111111111"], + regions=["us-east-1"], + ) + + plan = provider.resolve_execution_targets( + target=target, + regions=["us-east-1"], + include=target.include, + exclude=target.exclude, + preparation=_aws_preflight(), + ) + + assert [entity.id for entity in plan.execution_targets] == ["222222222222"] + assert plan.configured_target is not None + assert plan.configured_target.id == "111111111111" + assert plan.configured_target.name == "management" + assert plan.configured_target.type == "configured_target" + assert plan.configured_target.regions == ["us-east-1"] + + +def test_aws_rejects_ambiguous_configured_target_before_runtime() -> None: + provider = AwsProvider() + target = TargetDescriptor( + name="ambiguous", + provider="aws", + mode="accounts", + provider_options={"role_name": "OrganizationAccountAccessRole"}, + include=["111111111111", "222222222222"], + tasks=[{"name": "organization_task"}], + ) + + with pytest.raises(ValueError, match=r"ambiguous.*organization_task|no single"): + provider.validate_task_configuration( + target=target, task_scopes={"organization_task": "configured_target"} + ) + + +def test_aws_declares_configured_target_capability() -> None: + assert AwsProvider.metadata.supported_task_scopes == frozenset( + {"configured_target", "region"} + ) + + +def test_aws_ambiguous_configured_target_stops_before_auth_and_preflight( + monkeypatch: pytest.MonkeyPatch, +) -> None: + events: list[str] = [] + provider = AwsProvider() + monkeypatch.setattr("anvil.runner._load_provider", lambda provider_name: provider) + monkeypatch.setattr( + "anvil.runner.resolve_tasks", + lambda **kwargs: SimpleNamespace( + ordered=[ + SimpleNamespace( + id="organization_task", + name="organization_task", + scope=TaskScope.CONFIGURED_TARGET, + ) + ] + ), + ) + monkeypatch.setattr(provider, "auth_check", lambda target: events.append("auth")) + monkeypatch.setattr( + provider, "prepare_target", lambda **kwargs: events.append("prepare") + ) + target = TargetDescriptor( + name="ambiguous", + provider="aws", + mode="accounts", + provider_options={"role_name": "OrganizationAccountAccessRole"}, + include=["111111111111", "222222222222"], + regions=["us-east-1"], + tasks=[{"name": "organization_task"}], + ) + + prepared = prepare_target( + index=0, + target=target, + cli_dry_run=None, + cli_include=None, + cli_exclude=None, + preparation_cache=_SingleFlightCache(), + auth_cache=AuthCheckCache(), + ) + + assert events == [] + assert prepared.context is None + assert prepared.auth_result.status is ExecutionStatus.ERROR + assert "ambiguous" in (prepared.auth_result.message or "") + assert "organization_task" in (prepared.auth_result.message or "") + + +class _OfflineValidationProvider: + metadata = ProviderMetadata( + name="fake", + display_name="Fake", + supported_task_scopes=frozenset({"configured_target", "region"}), + default_regions=("region-a",), + ) + + def __init__(self, events: list[str]) -> None: + self.events = events + + def validate_target(self, target) -> None: + self.events.append("validate_target") + + def validate_task_configuration(self, *, target, task_scopes) -> None: + self.events.append("validate_task_configuration") + raise ValueError("configured-target identity is ambiguous") + + def resolve_target_filters(self, *, target, include_override, exclude_override): + return target.include, target.exclude + + def auth_cache_key(self, target): + return None + + def auth_check(self, target) -> ProviderAuthResult: + self.events.append("auth") + return ProviderAuthResult(status=ExecutionStatus.SUCCESS, source="fake") + + def prepare_target(self, **kwargs) -> ProviderPreparation: + self.events.append("prepare") + return ProviderPreparation() + + +def test_configured_target_ambiguity_is_rejected_before_authentication( + monkeypatch: pytest.MonkeyPatch, +) -> None: + events: list[str] = [] + provider = _OfflineValidationProvider(events) + monkeypatch.setattr("anvil.runner._load_provider", lambda provider_name: provider) + monkeypatch.setattr( + "anvil.runner.resolve_tasks", + lambda **kwargs: SimpleNamespace( + ordered=[ + SimpleNamespace( + id="configured_task", + name="configured_task", + scope=TaskScope.CONFIGURED_TARGET, + ) + ] + ), + ) + target = TargetDescriptor( + name="ambiguous", + provider="fake", + mode="resources", + regions=["region-a"], + tasks=[{"name": "configured_task"}], + ) + + prepared = prepare_target( + index=0, + target=target, + cli_dry_run=None, + cli_include=None, + cli_exclude=None, + preparation_cache=_SingleFlightCache(), + auth_cache=AuthCheckCache(), + ) + + assert events == ["validate_target", "validate_task_configuration"] + assert prepared.context is None + assert prepared.auth_result.status is ExecutionStatus.ERROR + assert "ambiguous" in (prepared.auth_result.message or "") + + +def test_unsupported_scope_resolution_fails_before_authentication( + monkeypatch: pytest.MonkeyPatch, +) -> None: + events: list[str] = [] + provider = _OfflineValidationProvider(events) + monkeypatch.setattr("anvil.runner._load_provider", lambda provider_name: provider) + + def reject_unsupported_scope(**kwargs): + from anvil.task_loader import TaskConfigError + + raise TaskConfigError( + "Provider 'fake' does not support task scope 'configured_target'" + ) + + monkeypatch.setattr("anvil.runner.resolve_tasks", reject_unsupported_scope) + target = TargetDescriptor( + name="unsupported", + provider="fake", + mode="resources", + regions=["region-a"], + tasks=[{"name": "configured_task"}], + ) + + prepared = prepare_target( + index=0, + target=target, + cli_dry_run=None, + cli_include=None, + cli_exclude=None, + preparation_cache=_SingleFlightCache(), + auth_cache=AuthCheckCache(), + ) + + assert events == ["validate_target"] + assert prepared.context is None + assert prepared.auth_result.status is ExecutionStatus.ERROR + assert "does not support" in (prepared.auth_result.message or "") + + +def test_ordinary_configuration_does_not_call_configured_validation_hook( + monkeypatch: pytest.MonkeyPatch, +) -> None: + events: list[str] = [] + provider = _OfflineValidationProvider(events) + monkeypatch.setattr("anvil.runner._load_provider", lambda provider_name: provider) + monkeypatch.setattr( + "anvil.runner.resolve_tasks", + lambda **kwargs: SimpleNamespace( + ordered=[ + SimpleNamespace( + id="ordinary_task", name="ordinary_task", scope=TaskScope.REGION + ) + ] + ), + ) + target = TargetDescriptor( + name="ordinary", + provider="fake", + mode="resources", + regions=["region-a"], + tasks=[{"name": "ordinary_task"}], + ) + + prepared = prepare_target( + index=0, + target=target, + cli_dry_run=None, + cli_include=None, + cli_exclude=None, + preparation_cache=_SingleFlightCache(), + auth_cache=AuthCheckCache(), + ) + + assert events == ["validate_target", "auth", "prepare"] + assert prepared.context is not None + + +class _LifecycleRuntime: + def __init__(self, calls: list[tuple[str, str]], target_id: str) -> None: + self.calls = calls + self.target_id = target_id + self.closed = False + + def build_session(self, *, region: str) -> object: + self.calls.append(("session", region)) + return {"target_id": self.target_id, "region": region} + + def record_region_outcome( + self, *, region: str, duration_seconds: float, failed: bool, interrupted: bool + ) -> None: + self.calls.append(("outcome", region)) + + def close(self) -> None: + self.calls.append(("close", "")) + self.closed = True + + @property + def benchmark(self) -> dict[str, object]: + if self.closed: + raise RuntimeError("benchmark accessed after runtime close") + return {"access_strategy": f"runtime-{self.target_id}"} + + +class _LifecycleProvider: + metadata = ProviderMetadata( + name="fake", + display_name="Fake", + supported_task_scopes=frozenset({"configured_target", "region", "target"}), + ) + + def __init__(self, calls: list[tuple[str, str]]) -> None: + self.calls = calls + + def prepare_execution_runtime(self, **kwargs) -> _LifecycleRuntime: + self.calls.append(("runtime", kwargs["execution_target"].id)) + return _LifecycleRuntime(self.calls, kwargs["execution_target"].id) + + def prepare_configured_target_runtime(self, **kwargs) -> _LifecycleRuntime: + self.calls.append(("configured_runtime", kwargs["execution_target"].id)) + return _LifecycleRuntime(self.calls, kwargs["execution_target"].id) + + +class _PlanLifecycleProvider(_LifecycleProvider): + def __init__( + self, + calls: list[tuple[str, str]], + *, + execution_targets: list[ExecutionTarget], + configured_target: ExecutionTarget, + ) -> None: + super().__init__(calls) + self.execution_targets = execution_targets + self.configured_target = configured_target + + def resolve_execution_targets(self, **kwargs) -> ProviderExecutionPlan: + return ProviderExecutionPlan( + execution_targets=self.execution_targets, + configured_target=self.configured_target, + ) + + +def _execution_target(regions: list[str]) -> ExecutionTarget: + return ExecutionTarget( + id="entity-a", + name="Entity A", + type="resource", + provider="fake", + regions=regions, + ) + + +def _context( + tasks: list[ResolvedTask], *, max_parallel_regions: int = 1 +) -> ExecutionContext: + return ExecutionContext( + regions=["region-a", "region-b"], + dry_run=False, + tasks=tasks, + metadata={}, + max_parallel_regions=max_parallel_regions, + ) + + +def _target( + tasks: list[dict[str, object]], *, max_workers: int = 10 +) -> TargetDescriptor: + return TargetDescriptor( + name="lifecycle", + provider="fake", + mode="resources", + regions=["region-a", "region-b"], + tasks=tasks, + max_workers=max_workers, + ) + + +def _baseline_task(*, name: str, run) -> ResolvedTask: + return ResolvedTask(name=name, run=run, depends_on=[], scope=TaskScope.REGION) + + +def _contract_task( + *, + task_id: str, + name: str, + run, + depends_on: list[str] | None = None, + always_run: bool = False, + dependency_data: dict[str, dict[str, str]] | None = None, + scope: str = "region", +) -> ResolvedTask: + parameters = inspect.signature(ResolvedTask).parameters + for field_name in ("id", "always_run", "metadata", "dependency_data"): + assert field_name in parameters + + kwargs = { + "id": task_id, + "name": name, + "run": run, + "depends_on": depends_on or [], + "always_run": always_run, + "metadata": {}, + "dependency_data": dependency_data or {}, + "scope": TaskScope(scope), + } + return ResolvedTask(**kwargs) + + +def test_preserves_existing_empty_task_lifecycle() -> None: + calls: list[tuple[str, str]] = [] + + result = _execute_provider_execution_target( + provider=_LifecycleProvider(calls), + target=_target([]), + execution_target=_execution_target(["region-a", "region-b"]), + context=_context([]), + ) + + assert result.status is ExecutionStatus.SUCCESS + assert calls == [ + ("runtime", "entity-a"), + ("session", "region-a"), + ("outcome", "region-a"), + ("session", "region-b"), + ("outcome", "region-b"), + ("close", ""), + ] + + +def test_preserves_runtime_session_reuse_boundary_and_action_isolation() -> None: + calls: list[tuple[str, str]] = [] + seen_actions: list[list[str]] = [] + + def run(**kwargs): + seen_actions.append(list(kwargs["actions"].actions)) + kwargs["actions"].record(kwargs["region"]) + return {"region": kwargs["region"]} + + task = _baseline_task(name="scan", run=run) + result = _execute_provider_execution_target( + provider=_LifecycleProvider(calls), + target=_target([{"name": "scan"}]), + execution_target=_execution_target(["region-a", "region-b"]), + context=_context([task]), + ) + + assert result.status is ExecutionStatus.SUCCESS + assert [call for call in calls if call[0] == "runtime"] == [("runtime", "entity-a")] + assert [call for call in calls if call[0] == "session"] == [ + ("session", "region-a"), + ("session", "region-b"), + ] + assert seen_actions == [[], []] + assert [task_result.actions for task_result in result.tasks] == [ + ["region-a"], + ["region-b"], + ] + assert calls == [ + ("runtime", "entity-a"), + ("session", "region-a"), + ("outcome", "region-a"), + ("session", "region-b"), + ("outcome", "region-b"), + ("close", ""), + ] + + +def test_preserves_task_discovery_without_import_time_execution() -> None: + discovery = discover_tasks() + + assert any(task.name == "noop" for task in discovery.tasks) + assert any(task.name == "count_vpc" for task in discovery.tasks) + assert not discovery.issues + + +def test_fail_fast_settles_unstarted_nodes_and_preserves_root_error() -> None: + def fail(**kwargs): + raise RuntimeError(f"failed {kwargs['region']}") + + context = _context([_contract_task(task_id="scan", name="scan", run=fail)]) + object.__setattr__(context, "fail_fast", True) + result = _execute_provider_execution_target( + provider=_LifecycleProvider([]), + target=_target([{"name": "scan"}]), + execution_target=_execution_target(["region-a", "region-b"]), + context=context, + ) + + assert [(task.status.value, task.skip_reason) for task in result.tasks] == [ + ("error", None), + ("skipped", "fail_fast"), + ] + assert result.status is ExecutionStatus.ERROR + + +def test_finalizer_activation_uses_same_transitive_dependency_gating() -> None: + finalizer_calls: list[str] = [] + + def fail(**kwargs): + raise RuntimeError("root failure") + + def blocked(**kwargs): + raise AssertionError("blocked task must not execute") + + def finalize(**kwargs): + finalizer_calls.append(kwargs["execution_target_id"]) + return {"restored": True} + + tasks = [ + _contract_task(task_id="producer", name="producer", run=fail), + _contract_task( + task_id="blocked", name="blocked", run=blocked, depends_on=["producer"] + ), + _contract_task( + task_id="finalizer", + name="finalizer", + run=finalize, + depends_on=["blocked"], + always_run=True, + ), + ] + result = _execute_provider_execution_target( + provider=_LifecycleProvider([]), + target=_target([]), + execution_target=_execution_target(["region-a"]), + context=_context(tasks), + ) + + assert finalizer_calls == ["entity-a"] + assert [(task.task_id, task.status.value) for task in result.tasks] == [ + ("producer", "error"), + ("blocked", "skipped"), + ("finalizer", "success"), + ] + assert result.status is ExecutionStatus.ERROR + + +def test_cancellation_before_chain_start_does_not_activate_finalizer() -> None: + finalizer_ran = False + + def finalize(**kwargs): + nonlocal finalizer_ran + finalizer_ran = True + + tasks = [ + _contract_task(task_id="work", name="work", run=lambda **kwargs: {}), + _contract_task( + task_id="finalizer", + name="finalizer", + run=finalize, + depends_on=["work"], + always_run=True, + ), + ] + context = _context(tasks) + context.cancel_event.set() + result = _execute_provider_execution_target( + provider=_LifecycleProvider([]), + target=_target([]), + execution_target=_execution_target(["region-a"]), + context=context, + ) + + assert not finalizer_ran + assert [(task.status.value, task.skip_reason) for task in result.tasks] == [ + ("skipped", "cancelled_before_start"), + ("skipped", "cancelled_before_start"), + ] + + +def test_missing_dependency_path_is_task_error_before_consumer_call() -> None: + consumer_ran = False + + def consume(**kwargs): + nonlocal consumer_ran + consumer_ran = True + + tasks = [ + _contract_task( + task_id="producer", name="producer", run=lambda **kwargs: {"present": True} + ), + _contract_task( + task_id="consumer", + name="consumer", + run=consume, + depends_on=["producer"], + dependency_data={ + "missing": {"task_id": "producer", "path": "result.missing"} + }, + ), + ] + result = _execute_provider_execution_target( + provider=_LifecycleProvider([]), + target=_target([]), + execution_target=_execution_target(["region-a"]), + context=_context(tasks), + ) + + assert not consumer_ran + assert result.tasks[-1].status is ExecutionStatus.ERROR + assert "missing" in (result.tasks[-1].error or "") + + +def test_failed_producer_partial_result_is_available_to_always_run_cleanup() -> None: + received: list[object] = [] + + def produce(**kwargs): + task_error_module = importlib.import_module("anvil.task_errors") + task_execution_error = task_error_module.TaskExecutionError + raise task_execution_error( + "mutation failed", partial_result={"attachments": ["partial"]} + ) + + def cleanup(**kwargs): + received.append(kwargs["dependency_data"]["attachments"]) + return {"restored": True} + + tasks = [ + _contract_task(task_id="producer", name="producer", run=produce), + _contract_task( + task_id="cleanup", + name="cleanup", + run=cleanup, + depends_on=["producer"], + always_run=True, + dependency_data={ + "attachments": {"task_id": "producer", "path": "result.attachments"} + }, + ), + ] + result = _execute_provider_execution_target( + provider=_LifecycleProvider([]), + target=_target([]), + execution_target=_execution_target(["region-a"]), + context=_context(tasks), + ) + + assert received == [["partial"]] + assert [task.status.value for task in result.tasks] == ["error", "success"] + assert result.status is ExecutionStatus.ERROR + + +def test_configured_only_execution_does_not_prepare_ordinary_runtimes() -> None: + calls: list[tuple[str, str]] = [] + callbacks: list[dict[str, object]] = [] + + def configured(**kwargs): + callbacks.append( + { + "id": kwargs["execution_target_id"], + "name": kwargs["execution_target_name"], + "type": kwargs["execution_target_type"], + "region": kwargs["region"], + "session_identity": kwargs["session"]["target_id"], + } + ) + return {"configured": True} + + configured_task = _contract_task( + task_id="configured", + name="configured", + run=configured, + scope="configured_target", + ) + _execute_provider_targets( + **{ + "provider": _LifecycleProvider(calls), + "target": _target([{"name": "configured"}]), + "context": _context([configured_task]), + "execution_targets": [ + _execution_target(["region-a"]), + ExecutionTarget( + id="entity-b", + name="Entity B", + type="resource", + provider="fake", + regions=["region-a"], + ), + ], + "configured_execution_target": ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + "benchmark_data": None, + } + ) + + assert [call for call in calls if "runtime" in call[0]] == [ + ("configured_runtime", "configured-owner") + ] + assert [call for call in calls if call[0] == "session"] == [("session", "region-a")] + assert [call for call in calls if call[0] == "close"] == [("close", "")] + assert callbacks == [ + { + "id": "configured-owner", + "name": "Configured Owner", + "type": "configured_target", + "region": "region-a", + "session_identity": "configured-owner", + } + ] + + +def test_runner_uses_configured_identity_from_provider_execution_plan() -> None: + calls: list[tuple[str, str]] = [] + callback_ids: list[str] = [] + configured_task = _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: callback_ids.append(kwargs["execution_target_id"]), + scope="configured_target", + ) + execution_target = _execution_target(["region-a"]) + configured_target = ExecutionTarget( + id="provider-owner", + name="Provider Owner", + type="configured_target", + provider="fake", + regions=["home-region"], + ) + provider = _PlanLifecycleProvider( + calls, execution_targets=[execution_target], configured_target=configured_target + ) + target = _target([{"name": "configured"}]) + now_at = "2026-01-01T00:00:00+00:00" + + outcome = run_prepared_target( + prepared_target=PreparedTarget( + index=0, + provider=provider, + effective_target=target, + auth_result=AuthResult( + target_name=target.name, + status=ExecutionStatus.SUCCESS, + source="fake", + started_at=now_at, + ended_at=now_at, + duration_seconds=0.0, + ), + context=_context([configured_task]), + ) + ) + + assert callback_ids == ["provider-owner"] + assert [call for call in calls if "runtime" in call[0]] == [ + ("configured_runtime", "provider-owner") + ] + assert outcome.target_result.error is None + + +def test_ordinary_only_execution_does_not_prepare_configured_runtime() -> None: + calls: list[tuple[str, str]] = [] + ordinary_task = _contract_task( + task_id="ordinary", + name="ordinary", + run=lambda **kwargs: {"entity": kwargs["execution_target_id"]}, + ) + + _execute_provider_targets( + provider=_LifecycleProvider(calls), + target=_target([{"name": "ordinary"}]), + context=_context([ordinary_task]), + execution_targets=[_execution_target(["region-a"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert [call for call in calls if "runtime" in call[0]] == [("runtime", "entity-a")] + + +def test_empty_tasks_keep_ordinary_lifecycle_with_configured_identity() -> None: + calls: list[tuple[str, str]] = [] + + _execute_provider_targets( + provider=_LifecycleProvider(calls), + target=_target([]), + context=_context([]), + execution_targets=[_execution_target(["region-a", "region-b"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert calls == [ + ("runtime", "entity-a"), + ("session", "region-a"), + ("outcome", "region-a"), + ("session", "region-b"), + ("outcome", "region-b"), + ("close", ""), + ] + + +def test_mixed_scope_runtime_and_session_construction_is_bounded() -> None: + calls: list[tuple[str, str]] = [] + tasks = [ + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + scope="configured_target", + ), + _contract_task( + task_id="target_wide", + name="target_wide", + run=lambda **kwargs: {}, + scope="target", + ), + _contract_task(task_id="regional", name="regional", run=lambda **kwargs: {}), + ] + + result = _execute_provider_targets( + provider=_LifecycleProvider(calls), + target=_target([], max_workers=2), + context=_context(tasks, max_parallel_regions=2), + execution_targets=[ + _execution_target(["region-a", "region-b"]), + ExecutionTarget( + id="entity-b", + name="Entity B", + type="resource", + provider="fake", + regions=["region-a", "region-b"], + ), + ], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert sorted(call for call in calls if "runtime" in call[0]) == [ + ("configured_runtime", "configured-owner"), + ("runtime", "entity-a"), + ("runtime", "entity-b"), + ] + assert len([call for call in calls if call[0] == "session"]) == 5 + assert len([call for call in calls if call[0] == "outcome"]) == 5 + assert len([call for call in calls if call[0] == "close"]) == 3 + assert [entity.id for entity in result.entities] == ["entity-a", "entity-b"] + assert [task.task_id for task in result.tasks] == ["configured"] + + +def test_mixed_graph_records_outcome_before_next_sequential_region_session() -> None: + calls: list[tuple[str, str]] = [] + tasks = [ + _contract_task( + task_id="regional_first", + name="regional_first", + run=lambda **kwargs: {"region": kwargs["region"]}, + ), + _contract_task( + task_id="regional_second", + name="regional_second", + run=lambda **kwargs: {"region": kwargs["region"]}, + depends_on=["regional_first"], + ), + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + depends_on=["regional_second"], + scope="configured_target", + ), + ] + + result = _execute_provider_targets( + provider=_LifecycleProvider(calls), + target=_target([], max_workers=1), + context=_context(tasks, max_parallel_regions=1), + execution_targets=[_execution_target(["region-a", "region-b"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + first_outcome_index = calls.index(("outcome", "region-a")) + second_region_session_index = calls.index(("session", "region-b")) + assert first_outcome_index < second_region_session_index + assert result.entities[0].benchmark is not None + assert result.entities[0].benchmark["access_strategy"] == "runtime-entity-a" + assert set(result.entities[0].benchmark["regions"]) == {"region-a", "region-b"} + + +def test_dependent_configured_task_waits_for_upstream_runtime_outcome() -> None: + calls: list[tuple[str, str]] = [] + ordinary_outcome_completed = threading.Event() + configured_saw_completed_outcome: list[bool] = [] + + class DelayedOutcomeRuntime(_LifecycleRuntime): + def record_region_outcome( + self, + *, + region: str, + duration_seconds: float, + failed: bool, + interrupted: bool, + ) -> None: + time.sleep(0.03) + super().record_region_outcome( + region=region, + duration_seconds=duration_seconds, + failed=failed, + interrupted=interrupted, + ) + ordinary_outcome_completed.set() + + class DelayedOutcomeProvider(_LifecycleProvider): + def prepare_execution_runtime(self, **kwargs) -> _LifecycleRuntime: + execution_target = kwargs["execution_target"] + self.calls.append(("runtime", execution_target.id)) + return DelayedOutcomeRuntime(self.calls, execution_target.id) + + def configured(**kwargs) -> dict[str, object]: + configured_saw_completed_outcome.append(ordinary_outcome_completed.is_set()) + return {} + + tasks = [ + _contract_task(task_id="regional", name="regional", run=lambda **kwargs: {}), + _contract_task( + task_id="configured", + name="configured", + run=configured, + depends_on=["regional"], + scope="configured_target", + ), + ] + + _execute_provider_targets( + provider=DelayedOutcomeProvider(calls), + target=_target([], max_workers=1), + context=_context(tasks, max_parallel_regions=1), + execution_targets=[_execution_target(["region-a"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert configured_saw_completed_outcome == [True] + + +def test_mixed_graph_surfaces_outcome_hook_errors_and_closes_runtimes() -> None: + calls: list[tuple[str, str]] = [] + + class FailingOutcomeRuntime(_LifecycleRuntime): + def record_region_outcome( + self, + *, + region: str, + duration_seconds: float, + failed: bool, + interrupted: bool, + ) -> None: + raise RuntimeError(f"outcome failed for {self.target_id}") + + class FailingOutcomeProvider(_LifecycleProvider): + def prepare_execution_runtime(self, **kwargs) -> _LifecycleRuntime: + execution_target = kwargs["execution_target"] + self.calls.append(("runtime", execution_target.id)) + return FailingOutcomeRuntime(self.calls, execution_target.id) + + tasks = [ + _contract_task(task_id="regional", name="regional", run=lambda **kwargs: {}), + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + depends_on=["regional"], + scope="configured_target", + ), + ] + + with pytest.raises(RuntimeError, match="outcome failed for entity-a"): + _execute_provider_targets( + provider=FailingOutcomeProvider(calls), + target=_target([], max_workers=1), + context=_context(tasks, max_parallel_regions=1), + execution_targets=[_execution_target(["region-a"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert ("close", "") in calls + + +def test_mixed_target_only_benchmark_does_not_report_regional_execution() -> None: + tasks = [ + _contract_task( + task_id="target_wide", + name="target_wide", + run=lambda **kwargs: {"target": kwargs["execution_target_id"]}, + scope="target", + ), + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + depends_on=["target_wide"], + scope="configured_target", + ), + ] + + result = _execute_provider_targets( + provider=_LifecycleProvider([]), + target=_target([], max_workers=1), + context=_context(tasks, max_parallel_regions=1), + execution_targets=[_execution_target(["region-a", "region-b"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + benchmark = result.entities[0].benchmark + assert benchmark is not None + assert benchmark["target"]["task_count"] == 1 + assert benchmark["regions"] == {} + assert benchmark["region_execution_seconds"] == 0.0 + + +def test_repeated_component_result_order_uses_invocation_ids() -> None: + tasks = [ + _contract_task( + task_id="inventory_before", + name="inventory", + run=lambda **kwargs: {"stage": "before"}, + ), + _contract_task( + task_id="inventory_after", + name="inventory", + run=lambda **kwargs: {"stage": "after"}, + depends_on=["inventory_before"], + ), + ] + + result = _execute_provider_targets( + provider=_LifecycleProvider([]), + target=_target([]), + context=_context(tasks, max_parallel_regions=2), + execution_targets=[_execution_target(["region-a", "region-b"])], + benchmark_data=None, + ) + + assert [(task.region, task.task_id) for task in result.entities[0].tasks] == [ + ("region-a", "inventory_before"), + ("region-a", "inventory_after"), + ("region-b", "inventory_before"), + ("region-b", "inventory_after"), + ] + + +def test_configured_graph_preserves_bounded_target_concurrency() -> None: + lock = threading.Lock() + active_targets: set[str] = set() + max_active_targets = 0 + + def regional(**kwargs): + nonlocal max_active_targets + target_id = kwargs["execution_target_id"] + with lock: + active_targets.add(target_id) + max_active_targets = max(max_active_targets, len(active_targets)) + time.sleep(0.03) + with lock: + active_targets.remove(target_id) + + tasks = [ + _contract_task(task_id="regional", name="regional", run=regional), + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + scope="configured_target", + ), + ] + _execute_provider_targets( + provider=_LifecycleProvider([]), + target=_target([], max_workers=1), + context=_context(tasks, max_parallel_regions=2), + execution_targets=[ + _execution_target(["region-a"]), + ExecutionTarget( + id="entity-b", + name="Entity B", + type="resource", + provider="fake", + regions=["region-a"], + ), + ], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert max_active_targets == 1 + + +def test_configured_graph_preserves_parallel_regions_within_target_limit() -> None: + lock = threading.Lock() + active_regions = 0 + max_active_regions = 0 + + def regional(**kwargs): + nonlocal active_regions, max_active_regions + with lock: + active_regions += 1 + max_active_regions = max(max_active_regions, active_regions) + time.sleep(0.03) + with lock: + active_regions -= 1 + + tasks = [ + _contract_task(task_id="regional", name="regional", run=regional), + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + scope="configured_target", + ), + ] + _execute_provider_targets( + provider=_LifecycleProvider([]), + target=_target([], max_workers=1), + context=_context(tasks, max_parallel_regions=2), + execution_targets=[_execution_target(["region-a", "region-b"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert max_active_regions == 2 + + +def test_configured_graph_honors_single_region_limit_per_target() -> None: + lock = threading.Lock() + active_regions = 0 + max_active_regions = 0 + + def regional(**kwargs): + nonlocal active_regions, max_active_regions + with lock: + active_regions += 1 + max_active_regions = max(max_active_regions, active_regions) + time.sleep(0.03) + with lock: + active_regions -= 1 + + tasks = [ + _contract_task(task_id="regional", name="regional", run=regional), + _contract_task( + task_id="configured", + name="configured", + run=lambda **kwargs: {}, + scope="configured_target", + ), + ] + _execute_provider_targets( + provider=_LifecycleProvider([]), + target=_target([], max_workers=2), + context=_context(tasks, max_parallel_regions=1), + execution_targets=[_execution_target(["region-a", "region-b"])], + configured_execution_target=ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + benchmark_data=None, + ) + + assert max_active_regions == 1 + + +def test_configured_fan_in_dependency_order_uses_target_then_region_order() -> None: + received: list[object] = [] + + def regional(**kwargs): + if ( + kwargs["execution_target_id"] == "entity-a" + and kwargs["region"] == "region-a" + ): + time.sleep(0.03) + return f"{kwargs['execution_target_id']}:{kwargs['region']}" + + def configured(**kwargs): + received.append(kwargs["dependency_data"]["values"]) + + tasks = [ + _contract_task(task_id="regional", name="regional", run=regional), + _contract_task( + task_id="configured", + name="configured", + run=configured, + depends_on=["regional"], + dependency_data={"values": {"task_id": "regional", "path": "result"}}, + scope="configured_target", + ), + ] + _execute_provider_targets( + **{ + "provider": _LifecycleProvider([]), + "target": _target([]), + "context": _context(tasks, max_parallel_regions=2), + "execution_targets": [ + _execution_target(["region-a", "region-b"]), + ExecutionTarget( + id="entity-b", + name="Entity B", + type="resource", + provider="fake", + regions=["region-a", "region-b"], + ), + ], + "configured_execution_target": ExecutionTarget( + id="configured-owner", + name="Configured Owner", + type="configured_target", + provider="fake", + regions=["region-a"], + ), + "benchmark_data": None, + } + ) + + assert received == [ + [ + "entity-a:region-a", + "entity-a:region-b", + "entity-b:region-a", + "entity-b:region-b", + ] + ] diff --git a/tests/task_redesign/test_single_scope_dag.py b/tests/task_redesign/test_single_scope_dag.py new file mode 100644 index 0000000..4b7ccb1 --- /dev/null +++ b/tests/task_redesign/test_single_scope_dag.py @@ -0,0 +1,335 @@ +from __future__ import annotations + +import datetime +import inspect +import threading + +import pytest + +from anvil.execution_context import ExecutionContext +from anvil.providers.base import ExecutionTarget +from anvil.results import ExecutionStatus, TaskResult +from anvil.runner import _execute_provider_region +from anvil.task_loader import ResolvedTask, TaskScope + + +class _Runtime: + def build_session(self, *, region: str) -> object: + return {"region": region} + + def record_region_outcome( + self, *, region: str, duration_seconds: float, failed: bool, interrupted: bool + ) -> None: + pass + + +def _task( + task_id: str, run, *, depends_on: list[str] | None = None, always_run: bool = False +) -> ResolvedTask: + return ResolvedTask( + id=task_id, + name=task_id, + run=run, + depends_on=depends_on or [], + always_run=always_run, + scope=TaskScope.REGION, + ) + + +def _terminal_result( + task_id: str, status: ExecutionStatus, *, skip_reason: str | None = None +) -> TaskResult: + parameters = inspect.signature(TaskResult).parameters + assert "skip_reason" in parameters + now = datetime.datetime.now(datetime.UTC).isoformat() + return TaskResult( + task_id=task_id, + task_name=task_id, + region="region-a", + status=status, + started_at=now, + ended_at=now, + duration_seconds=0.0, + error="upstream failed" if status.is_error else None, + skip_reason=skip_reason, + ) + + +def _execute( + tasks: list[ResolvedTask], + *, + dependency_results: dict[str, TaskResult] | None = None, + fail_fast: bool = False, + cancel_event: threading.Event | None = None, +): + context = ExecutionContext( + regions=["region-a"], + dry_run=False, + tasks=tasks, + metadata={}, + fail_fast=fail_fast, + cancel_event=cancel_event or threading.Event(), + ) + outcome = _execute_provider_region( + execution_target=ExecutionTarget( + id="entity-a", + name="Entity A", + type="resource", + provider="fake", + regions=["region-a"], + ), + runtime=_Runtime(), + context=context, + region="region-a", + target_cancel_event=threading.Event(), + dependency_results=dependency_results, + ) + return context, outcome + + +def _status_rows(outcome) -> list[tuple[str, str, str | None]]: + return [ + (result.task_name, result.status.value, result.skip_reason) + for result in outcome.task_results + ] + + +def test_normal_dependency_failure_skips_consumer_but_not_independent_branch() -> None: + ran: list[str] = [] + + def fail(**kwargs): + ran.append("producer") + raise RuntimeError("root failure") + + tasks = [ + _task("producer", fail), + _task( + "consumer", lambda **kwargs: ran.append("consumer"), depends_on=["producer"] + ), + _task("independent", lambda **kwargs: ran.append("independent")), + ] + + _context, outcome = _execute(tasks) + + assert ran == ["producer", "independent"] + assert _status_rows(outcome) == [ + ("producer", "error", None), + ("consumer", "skipped", "dependency_unsuccessful"), + ("independent", "success", None), + ] + assert outcome.failed + + +@pytest.mark.parametrize( + ("dependency_status_value", "skip_reason"), + [("error", None), ("interrupted", None), ("skipped", "dependency_unsuccessful")], +) +def test_normal_task_is_skipped_after_any_unsuccessful_dependency( + dependency_status_value: str, skip_reason: str | None +) -> None: + ran = False + + def consumer(**kwargs): + nonlocal ran + ran = True + + _context, outcome = _execute( + [_task("consumer", consumer, depends_on=["producer"])], + dependency_results={ + "producer": _terminal_result( + "producer", + ExecutionStatus(dependency_status_value), + skip_reason=skip_reason, + ) + }, + ) + + assert not ran + assert _status_rows(outcome) == [("consumer", "skipped", "dependency_unsuccessful")] + + +@pytest.mark.parametrize( + ("dependency_status_value", "skip_reason"), + [ + ("success", None), + ("error", None), + ("interrupted", None), + ("skipped", "dependency_unsuccessful"), + ], +) +def test_always_run_executes_after_every_activated_terminal_dependency( + dependency_status_value: str, skip_reason: str | None +) -> None: + ran: list[str] = [] + _context, outcome = _execute( + [ + _task( + "cleanup", + lambda **kwargs: ran.append("cleanup"), + depends_on=["producer"], + always_run=True, + ) + ], + dependency_results={ + "producer": _terminal_result( + "producer", + ExecutionStatus(dependency_status_value), + skip_reason=skip_reason, + ) + }, + ) + + assert ran == ["cleanup"] + assert _status_rows(outcome) == [("cleanup", "success", None)] + + +def test_always_run_failure_remains_an_error_and_root_error_is_preserved() -> None: + tasks = [ + _task( + "producer", + lambda **kwargs: (_ for _ in ()).throw(RuntimeError("root failure")), + ), + _task( + "cleanup", + lambda **kwargs: (_ for _ in ()).throw(RuntimeError("cleanup failure")), + depends_on=["producer"], + always_run=True, + ), + ] + + _context, outcome = _execute(tasks) + + assert [result.status for result in outcome.task_results] == [ + ExecutionStatus.ERROR, + ExecutionStatus.ERROR, + ] + assert outcome.task_results[0].error == "root failure" + assert outcome.task_results[1].error == "cleanup failure" + assert outcome.failed + + +def test_successful_finalizer_does_not_clear_upstream_failure() -> None: + tasks = [ + _task( + "producer", + lambda **kwargs: (_ for _ in ()).throw(RuntimeError("root failure")), + ), + _task( + "cleanup", + lambda **kwargs: {"restored": True}, + depends_on=["producer"], + always_run=True, + ), + ] + + _context, outcome = _execute(tasks) + + assert _status_rows(outcome) == [ + ("producer", "error", None), + ("cleanup", "success", None), + ] + assert outcome.task_results[0].error == "root failure" + assert outcome.failed + + +def test_fail_fast_settles_pending_nodes_and_runs_activated_finalizer() -> None: + ran: list[str] = [] + tasks = [ + _task( + "producer", + lambda **kwargs: (_ for _ in ()).throw(RuntimeError("root failure")), + ), + _task("pending", lambda **kwargs: ran.append("pending")), + _task( + "cleanup", + lambda **kwargs: ran.append("cleanup"), + depends_on=["producer"], + always_run=True, + ), + ] + + _context, outcome = _execute(tasks, fail_fast=True) + + assert ran == ["cleanup"] + assert _status_rows(outcome) == [ + ("producer", "error", None), + ("pending", "skipped", "fail_fast"), + ("cleanup", "success", None), + ] + assert outcome.task_results[0].error == "root failure" + + +def test_graceful_cancellation_runs_only_activated_finalizers() -> None: + cancel_event = threading.Event() + ran: list[str] = [] + + def producer(**kwargs): + ran.append("producer") + cancel_event.set() + return {"changed": True} + + tasks = [ + _task("producer", producer), + _task("ordinary", lambda **kwargs: ran.append("ordinary")), + _task( + "cleanup", + lambda **kwargs: ran.append("cleanup"), + depends_on=["producer"], + always_run=True, + ), + _task( + "never_activated", + lambda **kwargs: ran.append("never_activated"), + depends_on=["ordinary"], + always_run=True, + ), + ] + + _context, outcome = _execute(tasks, cancel_event=cancel_event) + + assert ran == ["producer", "cleanup"] + assert _status_rows(outcome) == [ + ("producer", "success", None), + ("ordinary", "skipped", "cancelled_before_start"), + ("cleanup", "success", None), + ("never_activated", "skipped", "cancelled_before_start"), + ] + + +def test_cancellation_before_chain_start_settles_every_task_without_cleanup() -> None: + cancel_event = threading.Event() + cancel_event.set() + ran: list[str] = [] + tasks = [ + _task("producer", lambda **kwargs: ran.append("producer")), + _task( + "cleanup", + lambda **kwargs: ran.append("cleanup"), + depends_on=["producer"], + always_run=True, + ), + ] + + _context, outcome = _execute(tasks, cancel_event=cancel_event) + + assert not ran + assert _status_rows(outcome) == [ + ("producer", "skipped", "cancelled_before_start"), + ("cleanup", "skipped", "cancelled_before_start"), + ] + + +def test_success_plus_skipped_aggregates_to_success() -> None: + results_module = __import__( + "anvil.results", fromlist=["aggregate_execution_statuses"] + ) + aggregate = getattr(results_module, "aggregate_execution_statuses", None) + assert callable(aggregate) + + skipped = ExecutionStatus("skipped") + assert aggregate([ExecutionStatus.SUCCESS, skipped]) is ExecutionStatus.SUCCESS + assert aggregate([skipped]) is ExecutionStatus.SUCCESS + assert ( + aggregate([ExecutionStatus.ERROR, ExecutionStatus.INTERRUPTED, skipped]) + is ExecutionStatus.ERROR + ) diff --git a/tests/task_redesign/test_task_inputs_contract.py b/tests/task_redesign/test_task_inputs_contract.py new file mode 100644 index 0000000..9030265 --- /dev/null +++ b/tests/task_redesign/test_task_inputs_contract.py @@ -0,0 +1,295 @@ +from __future__ import annotations + +import importlib +import inspect +import threading + +import pytest + +from anvil.execution_context import ExecutionContext +from anvil.providers.base import ExecutionTarget +from anvil.results import ExecutionStatus, TaskResult +from anvil.runner import _execute_provider_region +from anvil.task_context import TaskCallContext +from anvil.task_loader import ResolvedTask, TaskScope, list_tasks +from anvil.task_validation import validate_tasks + + +class _Runtime: + def build_session(self, *, region: str) -> object: + return {"region": region} + + def record_region_outcome( + self, *, region: str, duration_seconds: float, failed: bool, interrupted: bool + ) -> None: + pass + + +def _result( + *, + result: object | None = None, + status: ExecutionStatus = ExecutionStatus.SUCCESS, + error: str | None = None, +) -> TaskResult: + return TaskResult( + task_id="producer", + task_name="producer", + region="us-east-1", + status=status, + started_at="2026-07-28T00:00:00+00:00", + ended_at="2026-07-28T00:00:01+00:00", + duration_seconds=1.0, + result=result, + error=error, + actions=["recorded"], + ) + + +def _run_region( + *, tasks: list[ResolvedTask], metadata: dict[str, object] | None = None +): + return _execute_provider_region( + execution_target=ExecutionTarget( + id="111111111111", + name="account", + type="account", + provider="aws", + regions=["us-east-1"], + ), + runtime=_Runtime(), + context=ExecutionContext( + regions=["us-east-1"], dry_run=False, tasks=tasks, metadata=metadata or {} + ), + region="us-east-1", + target_cancel_event=threading.Event(), + ) + + +def test_all_discovered_tasks_use_the_phase2_call_signature() -> None: + validate_tasks(list_tasks()) + + expected = TaskCallContext.keyword_names() + assert "dependency_data" in expected + for task in list_tasks(): + run = task.load() + assert set(inspect.signature(run).parameters) == expected + + +def test_task_metadata_is_merged_recursively_without_mutating_sources() -> None: + received: list[dict[str, object]] = [] + target_metadata = { + "nested": {"kept": "target", "replaced": "target"}, + "list": ["target"], + "scalar": "target", + } + task_metadata = { + "nested": {"replaced": "task", "added": "task"}, + "list": ["task"], + "scalar": {"now": "mapping"}, + } + task = ResolvedTask( + id="consumer", + name="consumer", + run=lambda **kwargs: received.append(kwargs["metadata"]), + depends_on=[], + scope=TaskScope.REGION, + metadata=task_metadata, + ) + + _run_region(tasks=[task], metadata=target_metadata) + + assert received == [ + { + "nested": {"kept": "target", "replaced": "task", "added": "task"}, + "list": ["task"], + "scalar": {"now": "mapping"}, + } + ] + assert target_metadata == { + "nested": {"kept": "target", "replaced": "target"}, + "list": ["target"], + "scalar": "target", + } + assert task_metadata == { + "nested": {"replaced": "task", "added": "task"}, + "list": ["task"], + "scalar": {"now": "mapping"}, + } + + +def test_dependency_input_resolution_supports_complete_results_and_paths() -> None: + task_context_module = importlib.import_module("anvil.task_context") + resolver = getattr(task_context_module, "resolve_dependency_data", None) + assert callable(resolver) + producer = _result( + result={"attachments": {"items": ["a"]}, "nullable": None}, + error="retained error detail", + ) + + resolved = resolver( + references={ + "complete": {"task_id": "producer"}, + "payload": {"task_id": "producer", "path": "result"}, + "nested": {"task_id": "producer", "path": "result.attachments.items"}, + "nullable": {"task_id": "producer", "path": "result.nullable"}, + "status": {"task_id": "producer", "path": "status"}, + "error": {"task_id": "producer", "path": "error"}, + "actions": {"task_id": "producer", "path": "actions"}, + }, + dependency_results={"producer": producer}, + ) + + assert resolved == { + "complete": producer, + "payload": {"attachments": {"items": ["a"]}, "nullable": None}, + "nested": ["a"], + "nullable": None, + "status": ExecutionStatus.SUCCESS, + "error": "retained error detail", + "actions": ["recorded"], + } + + +def test_dependency_input_resolution_applies_paths_to_all_ordered_results() -> None: + task_context_module = importlib.import_module("anvil.task_context") + resolver = getattr(task_context_module, "resolve_dependency_data", None) + assert callable(resolver) + + resolved = resolver( + references={"values": {"task_id": "producer", "path": "result.value"}}, + dependency_results={ + "producer": [ + _result(result={"value": "first"}), + _result(result={"value": "second"}), + ] + }, + ) + + assert resolved == {"values": ["first", "second"]} + + +def test_multi_result_resolution_fails_if_any_result_lacks_the_path() -> None: + task_context_module = importlib.import_module("anvil.task_context") + resolver = getattr(task_context_module, "resolve_dependency_data", None) + error_type = getattr(task_context_module, "TaskInputResolutionError", None) + assert callable(resolver) + assert isinstance(error_type, type) + + with pytest.raises(error_type, match=r"values.*result\.value"): + resolver( + references={"values": {"task_id": "producer", "path": "result.value"}}, + dependency_results={ + "producer": [ + _result(result={"value": "first"}), + _result(result={"other": "second"}), + ] + }, + ) + + +def test_missing_dependency_path_is_a_clear_input_error() -> None: + task_context_module = importlib.import_module("anvil.task_context") + resolver = getattr(task_context_module, "resolve_dependency_data", None) + error_type = getattr(task_context_module, "TaskInputResolutionError", None) + assert callable(resolver) + assert isinstance(error_type, type) + + with pytest.raises(error_type, match=r"consumer_input.*result\.missing"): + resolver( + references={ + "consumer_input": {"task_id": "producer", "path": "result.missing"} + }, + dependency_results={"producer": _result(result={"present": None})}, + ) + + +def test_task_context_deep_copies_complete_dependency_results() -> None: + producer = _result(result={"items": ["original"]}) + context = TaskCallContext( + provider="aws", + execution_target_id="111111111111", + execution_target_name="account", + execution_target_type="account", + region="us-east-1", + session=object(), + dry_run=False, + metadata={}, + dependency_data={"complete": producer}, + actions=object(), + ) + + first = context.to_kwargs() + second = context.to_kwargs() + first["dependency_data"]["complete"].result["items"].append("mutated") + + assert second["dependency_data"]["complete"].result == {"items": ["original"]} + assert producer.result == {"items": ["original"]} + + +def test_runner_passes_resolved_dependency_data_to_the_consumer() -> None: + received: list[dict[str, object]] = [] + producer = ResolvedTask( + id="producer_invocation", + name="shared_component", + run=lambda **kwargs: {"value": {"items": ["resolved"]}}, + depends_on=[], + scope=TaskScope.REGION, + ) + consumer = ResolvedTask( + id="consumer", + name="consumer", + run=lambda **kwargs: received.append(kwargs["dependency_data"]), + depends_on=["producer_invocation"], + scope=TaskScope.REGION, + dependency_data={ + "payload": {"task_id": "producer_invocation", "path": "result.value"} + }, + ) + + outcome = _run_region(tasks=[producer, consumer]) + + assert [result.status for result in outcome.task_results] == [ + ExecutionStatus.SUCCESS, + ExecutionStatus.SUCCESS, + ] + assert received == [{"payload": {"items": ["resolved"]}}] + + +def test_task_execution_error_carries_validated_partial_result() -> None: + task_error_module = importlib.import_module("anvil.task_errors") + task_execution_error = getattr(task_error_module, "TaskExecutionError") + + error = task_execution_error( + "mutation failed", partial_result={"attachments": ["partial"]} + ) + + assert str(error) == "mutation failed" + assert error.partial_result == {"attachments": ["partial"]} + with pytest.raises(TypeError, match="JSON-serializable"): + task_execution_error("invalid", partial_result={"bad": object()}) + + +def test_failed_task_result_preserves_partial_execution_data() -> None: + task_error_module = importlib.import_module("anvil.task_errors") + task_execution_error = getattr(task_error_module, "TaskExecutionError") + + def run(**kwargs): + raise task_execution_error( + "mutation failed", partial_result={"attachments": ["partial"]} + ) + + outcome = _run_region( + tasks=[ + ResolvedTask( + id="producer", + name="producer", + run=run, + depends_on=[], + scope=TaskScope.REGION, + ) + ] + ) + + assert outcome.task_results[0].status is ExecutionStatus.ERROR + assert outcome.task_results[0].error == "mutation failed" + assert outcome.task_results[0].result == {"attachments": ["partial"]} diff --git a/tests/task_redesign/test_task_instance_planner.py b/tests/task_redesign/test_task_instance_planner.py new file mode 100644 index 0000000..9bb969f --- /dev/null +++ b/tests/task_redesign/test_task_instance_planner.py @@ -0,0 +1,313 @@ +from __future__ import annotations + +import importlib + +import pytest + +from anvil.providers.base import ExecutionTarget +from anvil.task_loader import ResolvedTask, TaskScope + + +def _task( + task_id: str, + scope: TaskScope, + *, + depends_on: list[str] | None = None, + calls: list[str] | None = None, +) -> ResolvedTask: + def run(**kwargs): + if calls is not None: + calls.append(task_id) + + return ResolvedTask( + id=task_id, name=task_id, run=run, depends_on=depends_on or [], scope=scope + ) + + +def _target(target_id: str, regions: list[str]) -> ExecutionTarget: + return ExecutionTarget( + id=target_id, name=target_id, type="resource", provider="fake", regions=regions + ) + + +def _planner_api(): + module = importlib.import_module("anvil.task_planner") + planner = getattr(module, "plan_task_instances", None) + error_type = getattr(module, "TaskPlanningError", None) + assert callable(planner) + assert isinstance(error_type, type) + return planner, error_type + + +def _coordinates(instance) -> tuple[str, str]: + return instance.key.execution_target_id, instance.key.region + + +def _dependency_coordinates(instance) -> list[tuple[str, str]]: + return [ + (dependency.execution_target_id, dependency.region) + for dependency in instance.dependencies + ] + + +@pytest.mark.parametrize( + ("producer_scope", "consumer_scope", "expected"), + [ + ( + TaskScope.REGION, + TaskScope.REGION, + { + ("target-b", "region-b2"): [("target-b", "region-b2")], + ("target-b", "region-b1"): [("target-b", "region-b1")], + ("target-a", "region-a1"): [("target-a", "region-a1")], + }, + ), + ( + TaskScope.TARGET, + TaskScope.REGION, + { + ("target-b", "region-b2"): [("target-b", "region-b2")], + ("target-b", "region-b1"): [("target-b", "region-b2")], + ("target-a", "region-a1"): [("target-a", "region-a1")], + }, + ), + ( + TaskScope.CONFIGURED_TARGET, + TaskScope.REGION, + { + ("target-b", "region-b2"): [("configured", "home-region")], + ("target-b", "region-b1"): [("configured", "home-region")], + ("target-a", "region-a1"): [("configured", "home-region")], + }, + ), + ( + TaskScope.REGION, + TaskScope.TARGET, + { + ("target-b", "region-b2"): [ + ("target-b", "region-b2"), + ("target-b", "region-b1"), + ], + ("target-a", "region-a1"): [("target-a", "region-a1")], + }, + ), + ( + TaskScope.TARGET, + TaskScope.TARGET, + { + ("target-b", "region-b2"): [("target-b", "region-b2")], + ("target-a", "region-a1"): [("target-a", "region-a1")], + }, + ), + ( + TaskScope.CONFIGURED_TARGET, + TaskScope.TARGET, + { + ("target-b", "region-b2"): [("configured", "home-region")], + ("target-a", "region-a1"): [("configured", "home-region")], + }, + ), + ( + TaskScope.REGION, + TaskScope.CONFIGURED_TARGET, + { + ("configured", "home-region"): [ + ("target-b", "region-b2"), + ("target-b", "region-b1"), + ("target-a", "region-a1"), + ] + }, + ), + ( + TaskScope.TARGET, + TaskScope.CONFIGURED_TARGET, + { + ("configured", "home-region"): [ + ("target-b", "region-b2"), + ("target-a", "region-a1"), + ] + }, + ), + ( + TaskScope.CONFIGURED_TARGET, + TaskScope.CONFIGURED_TARGET, + {("configured", "home-region"): [("configured", "home-region")]}, + ), + ], +) +def test_scope_relationship_matrix( + producer_scope: TaskScope, + consumer_scope: TaskScope, + expected: dict[tuple[str, str], list[tuple[str, str]]], +) -> None: + planner, _error_type = _planner_api() + plan = planner( + tasks=[ + _task("producer", producer_scope), + _task("consumer", consumer_scope, depends_on=["producer"]), + ], + execution_targets=[ + _target("target-b", ["region-b2", "region-b1"]), + _target("target-a", ["region-a1"]), + ], + configured_target=_target("configured", ["home-region"]), + ) + + consumers = [ + instance for instance in plan.instances if instance.task.id == "consumer" + ] + + assert { + _coordinates(instance): _dependency_coordinates(instance) + for instance in consumers + } == expected + + +def test_regional_dependency_matching_does_not_scan_unrelated_targets() -> None: + planner, _error_type = _planner_api() + + class CountingTargetId(str): + comparisons = 0 + __hash__ = str.__hash__ + + def __eq__(self, other: object) -> bool: + type(self).comparisons += 1 + return super().__eq__(other) + + def __ne__(self, other: object) -> bool: + type(self).comparisons += 1 + return super().__ne__(other) + + targets = [ + _target( + CountingTargetId(f"target-{target_index}"), + [f"region-{region_index}" for region_index in range(10)], + ) + for target_index in range(100) + ] + CountingTargetId.comparisons = 0 + + plan = planner( + tasks=[ + _task("producer", TaskScope.REGION), + _task("consumer", TaskScope.REGION, depends_on=["producer"]), + ], + execution_targets=targets, + configured_target=None, + ) + + assert len(plan.instances) == 2_000 + assert CountingTargetId.comparisons < len(plan.instances) + + +def test_planner_is_deterministic_and_never_executes_tasks() -> None: + planner, _error_type = _planner_api() + calls: list[str] = [] + tasks = [ + _task("inventory", TaskScope.REGION, calls=calls), + _task( + "summarize", + TaskScope.CONFIGURED_TARGET, + depends_on=["inventory"], + calls=calls, + ), + ] + targets = [ + _target("target-b", ["region-b2", "region-b1"]), + _target("target-a", ["region-a1"]), + ] + configured_target = _target("configured", ["home-region"]) + + first = planner( + tasks=tasks, execution_targets=targets, configured_target=configured_target + ) + second = planner( + tasks=tasks, execution_targets=targets, configured_target=configured_target + ) + + assert first == second + assert not calls + assert [ + (instance.task.id, *_coordinates(instance)) for instance in first.instances + ] == [ + ("inventory", "target-b", "region-b2"), + ("inventory", "target-b", "region-b1"), + ("inventory", "target-a", "region-a1"), + ("summarize", "configured", "home-region"), + ] + + +def test_planner_builds_stable_fan_out_adjacency() -> None: + planner, _error_type = _planner_api() + plan = planner( + tasks=[ + _task("configured", TaskScope.CONFIGURED_TARGET), + _task("regional", TaskScope.REGION, depends_on=["configured"]), + ], + execution_targets=[ + _target("target-b", ["region-b2", "region-b1"]), + _target("target-a", ["region-a1"]), + ], + configured_target=_target("configured-owner", ["home-region"]), + ) + producer = next( + instance for instance in plan.instances if instance.task.id == "configured" + ) + + assert [ + (child.task_id, child.execution_target_id, child.region) + for child in plan.adjacency[producer.key] + ] == [ + ("regional", "target-b", "region-b2"), + ("regional", "target-b", "region-b1"), + ("regional", "target-a", "region-a1"), + ] + + +@pytest.mark.parametrize( + ("targets", "configured_target", "match"), + [ + ( + [_target("duplicate", ["region-a"]), _target("duplicate", ["region-b"])], + None, + "duplicate execution target", + ), + ([_target("target", [])], None, "at least one region"), + ([_target("target", ["region", "region"])], None, "duplicate region"), + ], +) +def test_planner_rejects_invalid_execution_topology( + targets: list[ExecutionTarget], + configured_target: ExecutionTarget | None, + match: str, +) -> None: + planner, error_type = _planner_api() + + with pytest.raises(error_type, match=match): + planner( + tasks=[_task("regional", TaskScope.REGION)], + execution_targets=targets, + configured_target=configured_target, + ) + + +def test_planner_requires_concrete_configured_target_identity() -> None: + planner, error_type = _planner_api() + + with pytest.raises(error_type, match="configured-target.*identity"): + planner( + tasks=[_task("configured", TaskScope.CONFIGURED_TARGET)], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + + +def test_planner_rejects_unknown_dependency_ids_without_implicit_edges() -> None: + planner, error_type = _planner_api() + + with pytest.raises(error_type, match="unknown task ID 'missing'"): + planner( + tasks=[_task("consumer", TaskScope.REGION, depends_on=["missing"])], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) diff --git a/tests/task_redesign/test_task_instance_scheduler.py b/tests/task_redesign/test_task_instance_scheduler.py new file mode 100644 index 0000000..7f809e2 --- /dev/null +++ b/tests/task_redesign/test_task_instance_scheduler.py @@ -0,0 +1,514 @@ +from __future__ import annotations + +import datetime +import importlib +import threading +import time +from dataclasses import replace + +import pytest + +from anvil.providers.base import ExecutionTarget +from anvil.results import ExecutionStatus, TaskResult +from anvil.task_loader import ResolvedTask, TaskScope +from anvil.task_planner import plan_task_instances + + +def _task( + task_id: str, + scope: TaskScope, + *, + depends_on: list[str] | None = None, + always_run: bool = False, +) -> ResolvedTask: + return ResolvedTask( + id=task_id, + name=task_id, + run=lambda **kwargs: None, + depends_on=depends_on or [], + always_run=always_run, + scope=scope, + ) + + +def _target(target_id: str, regions: list[str]) -> ExecutionTarget: + return ExecutionTarget( + id=target_id, name=target_id, type="resource", provider="fake", regions=regions + ) + + +def _result( + instance, + status: ExecutionStatus = ExecutionStatus.SUCCESS, + *, + error: str | None = None, +) -> TaskResult: + now_at = datetime.datetime.now(datetime.UTC).isoformat() + return TaskResult( + task_id=instance.task.id, + task_name=instance.task.name, + region=instance.region, + status=status, + started_at=now_at, + ended_at=now_at, + duration_seconds=0.0, + error=error, + ) + + +def _scheduler_api(): + module = importlib.import_module("anvil.task_scheduler") + scheduler = getattr(module, "execute_task_instance_plan", None) + assert callable(scheduler) + return scheduler + + +def test_scheduler_releases_fan_in_barrier_only_after_every_region_settles() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("regional", TaskScope.REGION), + _task("summary", TaskScope.TARGET, depends_on=["regional"]), + ], + execution_targets=[_target("target", ["region-a", "region-b"])], + configured_target=None, + ) + completed_regions: list[str] = [] + summary_inputs: list[list[str]] = [] + + def execute(instance, dependency_results): + if instance.task.id == "regional": + if instance.region == "region-a": + time.sleep(0.02) + completed_regions.append(instance.region) + else: + summary_inputs.append( + [result.region for result in dependency_results["regional"]] + ) + assert set(completed_regions) == {"region-a", "region-b"} + return _result(instance) + + scheduler( + plan=plan, + execute=execute, + max_workers=2, + cancel_event=threading.Event(), + fail_fast=False, + ) + + assert summary_inputs == [["region-a", "region-b"]] + + +def test_dependency_preparation_reads_each_fan_in_result_once() -> None: + scheduler_module = importlib.import_module("anvil.task_scheduler") + prepare_dependency_results = getattr( + scheduler_module, "_prepare_dependency_results", None + ) + assert callable(prepare_dependency_results) + + plan = plan_task_instances( + tasks=[ + _task("regional", TaskScope.REGION), + _task("summary", TaskScope.TARGET, depends_on=["regional"]), + ], + execution_targets=[_target("target", ["region-a", "region-b", "region-c"])], + configured_target=None, + ) + instances_by_key = {instance.key: instance for instance in plan.instances} + summary = next( + instance for instance in plan.instances if instance.task.id == "summary" + ) + + class CountingResults(dict): + lookup_count = 0 + + def get(self, key, default=None): + self.lookup_count += 1 + return super().get(key, default) + + results_by_key = CountingResults( + { + dependency: _result(instances_by_key[dependency]) + for dependency in summary.dependencies + } + ) + eligibility, grouped = prepare_dependency_results( + instance=summary, + results_by_key=results_by_key, + activated_keys=set(summary.dependencies), + stop_reason=None, + ) + + assert eligibility.should_run + assert results_by_key.lookup_count == len(summary.dependencies) + assert [result.region for result in grouped["regional"]] == [ + "region-a", + "region-b", + "region-c", + ] + + +def test_scheduler_bounds_concurrency_and_serializes_one_region_coordinate() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[_task("first", TaskScope.REGION), _task("second", TaskScope.REGION)], + execution_targets=[_target("target", ["region-a", "region-b", "region-c"])], + configured_target=None, + ) + active = 0 + maximum_active = 0 + active_coordinates: set[tuple[str, str]] = set() + lock = threading.Lock() + + def execute(instance, dependency_results): + nonlocal active, maximum_active + coordinate = (instance.key.execution_target_id, instance.region) + with lock: + assert coordinate not in active_coordinates + active_coordinates.add(coordinate) + active += 1 + maximum_active = max(maximum_active, active) + time.sleep(0.01) + with lock: + active -= 1 + active_coordinates.remove(coordinate) + return _result(instance) + + scheduler( + plan=plan, + execute=execute, + max_workers=2, + cancel_event=threading.Event(), + fail_fast=False, + ) + + assert maximum_active == 2 + + +def test_scheduler_releases_large_chain_through_plan_adjacency() -> None: + scheduler = _scheduler_api() + instance_count = 500 + tasks = [ + _task( + f"task-{index}", + TaskScope.REGION, + depends_on=[f"task-{index - 1}"] if index else None, + ) + for index in range(instance_count) + ] + plan = plan_task_instances( + tasks=tasks, + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + + class CountingAdjacency(dict): + lookup_count = 0 + + def __getitem__(self, key): + self.lookup_count += 1 + return super().__getitem__(key) + + adjacency = CountingAdjacency(plan.adjacency) + plan = replace(plan, adjacency=adjacency) + schedule = scheduler( + plan=plan, + execute=lambda instance, dependencies: _result(instance), + max_workers=1, + cancel_event=threading.Event(), + fail_fast=False, + ) + + assert len(schedule.results) == instance_count + assert adjacency.lookup_count == instance_count + + +def test_fail_fast_settles_unstarted_nodes_and_runs_activated_finalizer() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("scan", TaskScope.REGION), + _task( + "cleanup", + TaskScope.CONFIGURED_TARGET, + depends_on=["scan"], + always_run=True, + ), + ], + execution_targets=[_target("target", ["region-a", "region-b", "region-c"])], + configured_target=_target("configured", ["home-region"]), + ) + cleanup_inputs: list[list[str]] = [] + + def execute(instance, dependency_results): + if instance.task.id == "scan": + return _result(instance, ExecutionStatus.ERROR, error="regional failure") + cleanup_inputs.append( + [result.status.value for result in dependency_results["scan"]] + ) + return _result(instance) + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=1, + cancel_event=threading.Event(), + fail_fast=True, + ) + + assert [ + (item.result.status.value, item.result.skip_reason) for item in schedule.results + ] == [ + ("error", None), + ("skipped", "fail_fast"), + ("skipped", "fail_fast"), + ("success", None), + ] + assert cleanup_inputs == [["error", "skipped", "skipped"]] + + +def test_cancellation_before_chain_start_does_not_activate_finalizer() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("work", TaskScope.REGION), + _task("cleanup", TaskScope.REGION, depends_on=["work"], always_run=True), + ], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + cancel_event = threading.Event() + cancel_event.set() + + schedule = scheduler( + plan=plan, + execute=lambda instance, dependencies: _result(instance), + max_workers=1, + cancel_event=cancel_event, + fail_fast=False, + ) + + assert [ + (item.result.status.value, item.result.skip_reason) for item in schedule.results + ] == [("skipped", "cancelled_before_start"), ("skipped", "cancelled_before_start")] + + +def test_graceful_cancellation_runs_finalizer_for_started_chain() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("work", TaskScope.REGION), + _task("pending", TaskScope.REGION), + _task("cleanup", TaskScope.REGION, depends_on=["work"], always_run=True), + ], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + cancel_event = threading.Event() + calls: list[str] = [] + + def execute(instance, dependency_results): + calls.append(instance.task.id) + if instance.task.id == "work": + cancel_event.set() + return _result(instance) + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=1, + cancel_event=cancel_event, + fail_fast=False, + ) + + assert calls == ["work", "cleanup"] + assert [ + (item.result.status.value, item.result.skip_reason) for item in schedule.results + ] == [("success", None), ("skipped", "cancelled_before_start"), ("success", None)] + + +def test_fail_fast_runs_finalizer_after_transitively_skipped_dependency() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("producer", TaskScope.REGION), + _task("blocked", TaskScope.REGION, depends_on=["producer"]), + _task("cleanup", TaskScope.REGION, depends_on=["blocked"], always_run=True), + ], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + calls: list[str] = [] + + def execute(instance, dependency_results): + calls.append(instance.task.id) + if instance.task.id == "producer": + return _result(instance, ExecutionStatus.ERROR, error="producer failed") + return _result(instance) + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=1, + cancel_event=threading.Event(), + fail_fast=True, + ) + + assert calls == ["producer", "cleanup"] + assert [ + (item.result.status.value, item.result.skip_reason) for item in schedule.results + ] == [("error", None), ("skipped", "fail_fast"), ("success", None)] + + +def test_cancellation_runs_finalizer_after_transitively_skipped_dependency() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("producer", TaskScope.REGION), + _task("blocked", TaskScope.REGION, depends_on=["producer"]), + _task("cleanup", TaskScope.REGION, depends_on=["blocked"], always_run=True), + ], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + cancel_event = threading.Event() + calls: list[str] = [] + + def execute(instance, dependency_results): + calls.append(instance.task.id) + if instance.task.id == "producer": + cancel_event.set() + return _result(instance) + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=1, + cancel_event=cancel_event, + fail_fast=False, + ) + + assert calls == ["producer", "cleanup"] + assert [ + (item.result.status.value, item.result.skip_reason) for item in schedule.results + ] == [("success", None), ("skipped", "cancelled_before_start"), ("success", None)] + + +@pytest.mark.parametrize( + ("producer_scope", "finalizer_scope"), + [ + (TaskScope.REGION, TaskScope.TARGET), + (TaskScope.TARGET, TaskScope.REGION), + (TaskScope.REGION, TaskScope.CONFIGURED_TARGET), + (TaskScope.CONFIGURED_TARGET, TaskScope.REGION), + (TaskScope.TARGET, TaskScope.CONFIGURED_TARGET), + (TaskScope.CONFIGURED_TARGET, TaskScope.TARGET), + ], +) +@pytest.mark.parametrize("run_state", ["success", "error", "fail_fast", "cancellation"]) +def test_finalizer_state_matrix_across_scope_boundaries( + producer_scope: TaskScope, finalizer_scope: TaskScope, run_state: str +) -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("producer", producer_scope), + _task( + "finalizer", finalizer_scope, depends_on=["producer"], always_run=True + ), + ], + execution_targets=[_target("target", ["region-a", "region-b"])], + configured_target=_target("configured", ["home-region"]), + ) + cancel_event = threading.Event() + finalizer_calls: list[tuple[str, str]] = [] + + def execute(instance, dependency_results): + if instance.task.id == "producer": + if run_state == "cancellation": + cancel_event.set() + if run_state in {"error", "fail_fast"}: + return _result(instance, ExecutionStatus.ERROR, error="producer failed") + else: + finalizer_calls.append((instance.key.execution_target_id, instance.region)) + return _result(instance) + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=1, + cancel_event=cancel_event, + fail_fast=run_state == "fail_fast", + ) + + expected_finalizer_count = 2 if finalizer_scope is TaskScope.REGION else 1 + assert len(finalizer_calls) == expected_finalizer_count + assert all( + item.result.status is ExecutionStatus.SUCCESS + for item in schedule.results + if item.key.task_id == "finalizer" + ) + + +def test_executor_exception_becomes_error_and_blocks_normal_dependent() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[ + _task("runtime", TaskScope.REGION), + _task("consumer", TaskScope.REGION, depends_on=["runtime"]), + ], + execution_targets=[_target("target", ["region"])], + configured_target=None, + ) + + def execute(instance, dependency_results): + raise RuntimeError("runtime construction failed") + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=1, + cancel_event=threading.Event(), + fail_fast=False, + ) + + assert [ + (item.result.status.value, item.result.error, item.result.skip_reason) + for item in schedule.results + ] == [ + ("error", "runtime construction failed", None), + ("skipped", None, "dependency_unsuccessful"), + ] + + +def test_output_order_follows_plan_not_completion_order() -> None: + scheduler = _scheduler_api() + plan = plan_task_instances( + tasks=[_task("scan", TaskScope.REGION)], + execution_targets=[ + _target("target-b", ["region-b2", "region-b1"]), + _target("target-a", ["region-a1"]), + ], + configured_target=None, + ) + + def execute(instance, dependency_results): + if instance.region == "region-b2": + time.sleep(0.03) + return _result(instance) + + schedule = scheduler( + plan=plan, + execute=execute, + max_workers=3, + cancel_event=threading.Event(), + fail_fast=False, + ) + + assert [ + (item.key.execution_target_id, item.key.region) for item in schedule.results + ] == [ + ("target-b", "region-b2"), + ("target-b", "region-b1"), + ("target-a", "region-a1"), + ] diff --git a/tests/tasks/test_task_context.py b/tests/tasks/test_task_context.py index 074d58c..84267d1 100644 --- a/tests/tasks/test_task_context.py +++ b/tests/tasks/test_task_context.py @@ -14,6 +14,7 @@ def _context(metadata: dict[str, object]) -> TaskCallContext: session=object(), dry_run=False, metadata=metadata, + dependency_data={}, actions=ActionRecorder(actions=[]), ) diff --git a/tests/tasks/test_task_loader.py b/tests/tasks/test_task_loader.py index 13a58ea..2af138d 100644 --- a/tests/tasks/test_task_loader.py +++ b/tests/tasks/test_task_loader.py @@ -144,18 +144,19 @@ def test_region_task_may_depend_on_target_task(monkeypatch): assert [task.name for task in execution.ordered] == ["target_wide", "regional"] -def test_target_task_cannot_depend_on_region_task(monkeypatch): +def test_target_task_may_depend_on_region_task(monkeypatch): _mock_scoped_tasks(monkeypatch, {"target_wide": "target", "regional": "region"}) - with pytest.raises(TaskConfigError, match="execute before regional fan-out"): - resolve_tasks( - task_specs=[ - {"name": "target_wide", "depends_on": ["regional"]}, - {"name": "regional"}, - ], - provider_name="azure", - supported_task_scopes=frozenset({"region", "target"}), - ) + execution = resolve_tasks( + task_specs=[ + {"name": "target_wide", "depends_on": ["regional"]}, + {"name": "regional"}, + ], + provider_name="azure", + supported_task_scopes=frozenset({"region", "target"}), + ) + + assert [task.name for task in execution.ordered] == ["regional", "target_wide"] def test_resolve_tasks_dependency_order(monkeypatch): diff --git a/tests/tasks/test_task_validation.py b/tests/tasks/test_task_validation.py index 4feaf1f..8e48700 100644 --- a/tests/tasks/test_task_validation.py +++ b/tests/tasks/test_task_validation.py @@ -45,6 +45,7 @@ def run( session, dry_run, metadata, + dependency_data, actions, ): """Run a valid provider-neutral task.""" @@ -65,6 +66,27 @@ def test_validate_tasks_accepts_real_provider_tasks(name, run): validate_tasks([_task(name, run)]) +def test_validate_tasks_rejects_legacy_signature_without_dependency_data(): + def run( + *, + provider, + execution_target_id, + execution_target_name, + execution_target_type, + region, + session, + dry_run, + metadata, + actions, + ): + """Run a task using the removed pre-Phase-2 signature.""" + + pass + + with pytest.raises(TaskValidationError, match="dependency_data"): + validate_tasks([_task("legacy-signature", run)]) + + def test_validate_tasks_rejects_task_missing_actions(): def run( *, @@ -76,6 +98,7 @@ def run( session, dry_run, metadata, + dependency_data, ): """Run an invalid task.""" @@ -106,6 +129,7 @@ def run( session, dry_run, metadata, + dependency_data, actions, extra, ): @@ -128,6 +152,7 @@ def run( session, dry_run, metadata, + dependency_data, actions, ): """Run an invalid task with a positional-or-keyword parameter.""" @@ -149,6 +174,7 @@ def run( session, dry_run, metadata, + dependency_data, actions, ): """Run a duplicate test task.""" @@ -184,6 +210,7 @@ def run( session, dry_run, metadata, + dependency_data, actions, ): """Run a provider-specific task.""" @@ -212,6 +239,7 @@ def run( session, dry_run, metadata, + dependency_data, actions, ): pass diff --git a/uv.lock b/uv.lock index 589ae3a..6b672d5 100644 --- a/uv.lock +++ b/uv.lock @@ -89,11 +89,11 @@ provides-extras = ["azure", "gcp", "github"] [package.metadata.requires-dev] dev = [ { name = "check-jsonschema", specifier = ">=0.37.3,<0.40.0" }, - { name = "prek", specifier = ">=0.4.5,<0.5.0" }, + { name = "prek", specifier = ">=0.4.12,<0.5.0" }, { name = "pytest", specifier = ">=9.1.1,<10.0.0" }, { name = "pytest-cov", specifier = ">=7.1.0,<8.0.0" }, - { name = "ruff", specifier = ">=0.15.19,<0.20.0" }, - { name = "ty", specifier = ">=0.0.53,<0.20.0" }, + { name = "ruff", specifier = ">=0.16.0,<0.20.0" }, + { name = "ty", specifier = ">=0.0.60,<0.20.0" }, ] [[package]] @@ -1046,26 +1046,26 @@ wheels = [ [[package]] name = "prek" -version = "0.4.11" +version = "0.4.12" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/c4/1a/73b6dae5ce7e997cb35a69bfe1d25a798e85fa3d2eabf95324563f461b30/prek-0.4.11.tar.gz", hash = "sha256:4a14cb9bbae850605ae3904fbdbb12f0e00c12455efaa2266da8fb8e5c0350d7", size = 516254, upload-time = "2026-07-24T17:05:35.107Z" } +sdist = { url = "https://files.pythonhosted.org/packages/fc/5c/cb6e63f7e5a58a5313ddb70409174f4dc004e4b0910b8a8d3f59b2225a95/prek-0.4.12.tar.gz", hash = "sha256:04beeba7f40437cd2f36804b84101bd7f3c9fb40b52da46a25604642ab2bfb09", size = 519080, upload-time = "2026-08-03T11:28:33.147Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/8e/d7/a00b2de492a80e99b1698e72c1d196ac3ed544dc7ee0ada261ac066e78e0/prek-0.4.11-py3-none-linux_armv6l.whl", hash = "sha256:3830cb7cc47e837888b8b464ecb21355a69235cfacef0fd89101e17f09345d63", size = 5770511, upload-time = "2026-07-24T17:05:12.829Z" }, - { url = "https://files.pythonhosted.org/packages/c1/1e/f97c74defcd5d5645888cb99fcdd9b8b48cc7247cf31e64414d106d56d66/prek-0.4.11-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:45facadf9c2332b28e6ab2744312ae0275a93ab5a437da8bcc53a8c7260cb4b0", size = 6118049, upload-time = "2026-07-24T17:05:14.451Z" }, - { url = "https://files.pythonhosted.org/packages/d9/92/8367d26421ee6fe6019a63928fb0ed31179cd0d6199879c524f89ef4c95c/prek-0.4.11-py3-none-macosx_11_0_arm64.whl", hash = "sha256:2f8a194b1d00d24dff8baff691c99e2668339da2da3482b4ee99c1a0f2409378", size = 5601478, upload-time = "2026-07-24T17:05:15.81Z" }, - { url = "https://files.pythonhosted.org/packages/e1/6f/7617c9b87afaadede4167720aae7ef12d7e51167db903bc8fbac0cadef7b/prek-0.4.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:21d93e6d76bf3d7a9bb70c6ac86ef372adaee50068c45749e6b9e1c2ac4ec939", size = 5932071, upload-time = "2026-07-24T17:05:17.217Z" }, - { url = "https://files.pythonhosted.org/packages/ef/41/6796a4011b04212333259064aa885a71d03d9581ba9bff52db3ce58d1f06/prek-0.4.11-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7fb07cde2d2156efa6980122b3f13dc88f20c84b933c6205675d0f1e7be2cde8", size = 5677617, upload-time = "2026-07-24T17:05:18.658Z" }, - { url = "https://files.pythonhosted.org/packages/03/5f/3e339901f8460b6073313619b4fd9bf4135e48430695c53640b83ace88ea/prek-0.4.11-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6b7f5e446d2aca739bd18b380578e9c78bb4c086299abc3e167d51c47840c56f", size = 6106370, upload-time = "2026-07-24T17:05:20.219Z" }, - { url = "https://files.pythonhosted.org/packages/8e/4e/94e24b5c1910ec15692ecaf33d0d8ef0d02a02a8b5e40d3c976396880cba/prek-0.4.11-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b7059d640e595d098600e2d97af961f46292b4112670edbe632de84f6828389e", size = 6884342, upload-time = "2026-07-24T17:05:21.587Z" }, - { url = "https://files.pythonhosted.org/packages/a2/7c/fc0daa033dcafe74990c00af2da1e16922790b9c1b8da7e8eebf1123838b/prek-0.4.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:85a4df33998fcac878bce3b2c624e1caeb9609f3d562c640702c1234ed815daf", size = 6331365, upload-time = "2026-07-24T17:05:22.87Z" }, - { url = "https://files.pythonhosted.org/packages/d7/36/49f152b8f539930e9685cff0509695ce4054abcecc22f4c16c4e1e5c23d0/prek-0.4.11-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:866991f387527c5f880ce2cc3ddea9b33cc5b986881f8c7f92524cf0969c1350", size = 5939075, upload-time = "2026-07-24T17:05:24.249Z" }, - { url = "https://files.pythonhosted.org/packages/4f/fe/7b097af9161edae7ddedcc9f5cda4a5f31346a492ce3f92583d96c46f628/prek-0.4.11-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:247e5d8740e137ebdf24fa96f01a95b2d2f1892e956a1ab00c7b1474a59ebcc0", size = 5799029, upload-time = "2026-07-24T17:05:25.652Z" }, - { url = "https://files.pythonhosted.org/packages/49/63/dc955ff99e1002d3cd375b21467c87046bc41deddfce86d898240757b151/prek-0.4.11-py3-none-musllinux_1_1_armv7l.whl", hash = "sha256:603ba9f2fd9d666dddb3ae190a25a5c55091b843cde90ed52e0a1116f50d4062", size = 5651211, upload-time = "2026-07-24T17:05:27.027Z" }, - { url = "https://files.pythonhosted.org/packages/8f/b8/bf2139ec25eefb5afef43afac7620c5181f2c2661f220598beacb1771207/prek-0.4.11-py3-none-musllinux_1_1_i686.whl", hash = "sha256:f2682ece3c5fc7201106c4fdd84b0587ccea4b8a2ecefa3e94d3074d2841f1df", size = 5954784, upload-time = "2026-07-24T17:05:28.391Z" }, - { url = "https://files.pythonhosted.org/packages/fc/29/3fe5990aee1bd7c4d50a03358ad867c5e912d9510aafb83a359c7347e5fa/prek-0.4.11-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:22721d30394192931fecc80d6fc6dd47e8ddf8db8b9693805aba8ec0f50087ec", size = 6448916, upload-time = "2026-07-24T17:05:29.837Z" }, - { url = "https://files.pythonhosted.org/packages/0f/fb/abddacf43738302242ecd237236c7c80cd7c1d27545e16e803770aee76e9/prek-0.4.11-py3-none-win32.whl", hash = "sha256:8b093e7624522146049e994d5cf283d01d71632b638620b79ae0f96afeaa2624", size = 5483539, upload-time = "2026-07-24T17:05:31.228Z" }, - { url = "https://files.pythonhosted.org/packages/00/1e/c293f7a15cb93963c4be36a02e144b93f43906bc02644a3f04e5708e7453/prek-0.4.11-py3-none-win_amd64.whl", hash = "sha256:5a3d7c80b970b456e5f1bcec8382008ee1ae6a3f324a6b9bb4ff7e666ab0f3c4", size = 5861119, upload-time = "2026-07-24T17:05:32.479Z" }, - { url = "https://files.pythonhosted.org/packages/cd/0c/05fe6eb9d6a54d0e02dfa8cc5ad6f86869bf953419ec15909a32466a28ee/prek-0.4.11-py3-none-win_arm64.whl", hash = "sha256:e7b0df37ce05e45a14a9da39ab104691474d72f139bf4f6c860f754763a322cb", size = 5626386, upload-time = "2026-07-24T17:05:33.813Z" }, + { url = "https://files.pythonhosted.org/packages/f3/23/5811a3161e072e5f93e4da01af611ee30c32922507b8ab4d9873df6affd3/prek-0.4.12-py3-none-linux_armv6l.whl", hash = "sha256:cd92000b051e433f26340821cf1cc8e6e3960f1275f3d516ca01f05905abba64", size = 5793226, upload-time = "2026-08-03T11:28:09.534Z" }, + { url = "https://files.pythonhosted.org/packages/a3/88/8607845d94eb1482e1bd335dadf098618f077a15775f7e98de99669052b4/prek-0.4.12-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:5904fe6c6ab26e7d8792a3c7f1e3fc8d94fcfb63ad33b247c35f004b62cb6275", size = 6132269, upload-time = "2026-08-03T11:28:11.147Z" }, + { url = "https://files.pythonhosted.org/packages/ac/28/571d79ba457fbd9ecf40ae879c91952e12f5fa475306218c91139b86db7a/prek-0.4.12-py3-none-macosx_11_0_arm64.whl", hash = "sha256:df3eff1db9c24dc293010a07bc7a0ae0c541d55af828f5586405dedc28c4920d", size = 5614964, upload-time = "2026-08-03T11:28:12.983Z" }, + { url = "https://files.pythonhosted.org/packages/b0/a9/3f5cb79a73c764a8ac38d5bcd51e0df57239856eca7949b09bdac4338bf3/prek-0.4.12-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:c7733b44ca772ea32ec6a8bee669d0358bdf45873e79767afed196065084f31c", size = 5941047, upload-time = "2026-08-03T11:28:14.45Z" }, + { url = "https://files.pythonhosted.org/packages/8c/00/1dfed0ef8af10c5c32aa903486dccd33d2df171f3d945a037c5692f10760/prek-0.4.12-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:87f170cf1ffd6e3a196f947b83dff1f6c2cd68635f8d49740278bebe7b682262", size = 5707994, upload-time = "2026-08-03T11:28:15.914Z" }, + { url = "https://files.pythonhosted.org/packages/c0/bd/5f388f6cbdc0445b850e7c1a160d0be67fcef8bf221e3c8141a1feccef17/prek-0.4.12-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:57dad513831f060cf73808df8edec29d46ec311435aa69f21c80edebf23dc5e1", size = 6133784, upload-time = "2026-08-03T11:28:17.184Z" }, + { url = "https://files.pythonhosted.org/packages/ba/47/342091a987bf68a74acec6d226a40ce7d51faf0019aa4126cc7bc952f8a7/prek-0.4.12-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b204844abc7ded983471f576ae8dc13b99e9b8d022e4d4b46176c6654769c9d8", size = 6901589, upload-time = "2026-08-03T11:28:18.545Z" }, + { url = "https://files.pythonhosted.org/packages/5b/8a/3ef7bdc3c3441649ebc040b9e164a13163e1e5fabae23e7bbb901992f3de/prek-0.4.12-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:43b0a5a9d3f2f77871fdcb7893bfc5c8fe7e44f4e603ce6e4712bfec96b2d6f2", size = 6342189, upload-time = "2026-08-03T11:28:20Z" }, + { url = "https://files.pythonhosted.org/packages/c4/da/6277908442301b1b92a2879f6b04aaa03accb900f80e42776fc28b8197ef/prek-0.4.12-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:0d188e572c306cc44b96e1bae5647e25b7bd311113f3f3f4a67320c257ee64a3", size = 5951250, upload-time = "2026-08-03T11:28:21.339Z" }, + { url = "https://files.pythonhosted.org/packages/a3/68/bff51a7332837edb1ecbe017325adb7fafd69b9c7828ddc81a1334b884af/prek-0.4.12-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:986f52d104b7066190f0f32aebe3467710356de265e9bfd892101ba99371db4d", size = 5804147, upload-time = "2026-08-03T11:28:22.656Z" }, + { url = "https://files.pythonhosted.org/packages/aa/de/b7f544971072ed7814125145dfeb1f7c15cce6b78ccea65a96298ff37838/prek-0.4.12-py3-none-musllinux_1_1_armv7l.whl", hash = "sha256:13e34d9e09bafcbf1f25a01cf86985e2c5e486591d3f45b2786ba3de82e5153a", size = 5680104, upload-time = "2026-08-03T11:28:24.271Z" }, + { url = "https://files.pythonhosted.org/packages/68/94/95942bcc20a6a91ec2989aa30fdeb00ad095be736ec48b4bbcf0376166b1/prek-0.4.12-py3-none-musllinux_1_1_i686.whl", hash = "sha256:3d0208370da73e8b5bc97f2492dc3975f8dd2c22f4bf6e1f2cf3342503764b52", size = 5975030, upload-time = "2026-08-03T11:28:25.683Z" }, + { url = "https://files.pythonhosted.org/packages/ef/6d/26e6497198d81cf9aa82495400aef46adea8df3e4a4efc5f00e3b6ab3292/prek-0.4.12-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:b1005f42920111bec1403c25e8f2f12ec7af0be06686cc3b8dcf85429af908a8", size = 6458532, upload-time = "2026-08-03T11:28:27.121Z" }, + { url = "https://files.pythonhosted.org/packages/44/02/ee140c2eb4701bd194db429d84630733492be94897d5f72b61d6f11e6619/prek-0.4.12-py3-none-win32.whl", hash = "sha256:afee229488dcceaea282288e4d7096a93da5a8b85649d9ef506dbdbcd78f38a7", size = 5502213, upload-time = "2026-08-03T11:28:28.691Z" }, + { url = "https://files.pythonhosted.org/packages/e5/7b/744cff84def48c1ce38c0b4f643a3553c66976c5bb7869ab7317044870e4/prek-0.4.12-py3-none-win_amd64.whl", hash = "sha256:fdd27bad8adafea8fe77606950ca09200d59296a47ab131cfb88718d460949d7", size = 5868065, upload-time = "2026-08-03T11:28:30.377Z" }, + { url = "https://files.pythonhosted.org/packages/46/1d/e2c0fc222904ef73df1739b11a83edc29e38bc4bc61259f2ca6d2f15abb0/prek-0.4.12-py3-none-win_arm64.whl", hash = "sha256:45e34a24fba4a4e4568682477158591698efc2375b8d1d418ae424691c4bd01b", size = 5632819, upload-time = "2026-08-03T11:28:31.743Z" }, ] [[package]] @@ -1391,27 +1391,27 @@ wheels = [ [[package]] name = "ruff" -version = "0.16.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/4d/94/1e5e4967626faf12fa56999cd6222dff6992ceb086ad7945756baf70c7a7/ruff-0.16.0.tar.gz", hash = "sha256:e460aafd5495ec89efaa6ced2e4a9a581116451e1c88b9d37ef497e0f8e93982", size = 4790557, upload-time = "2026-07-23T19:11:30.981Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/4b/81/1c8818fee7ce1a04cd7d1b3172e0a8f8e4f1dc4feb7fc390e16daa8af323/ruff-0.16.0-py3-none-linux_armv6l.whl", hash = "sha256:e5115729eb08c585e5121978ba5d5b60caeae394ce21b9fb5e6cd33a1c6c9b1e", size = 10754633, upload-time = "2026-07-23T19:10:46.415Z" }, - { url = "https://files.pythonhosted.org/packages/23/df/beaf59c09d68db84304d555f188b276a77132a5d5b0b67a5c762aa143628/ruff-0.16.0-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:3c954b1d580bfa035b41654f7858cc7e71d5fc3ac5b723dd62bd9133830ed522", size = 10969164, upload-time = "2026-07-23T19:10:50.271Z" }, - { url = "https://files.pythonhosted.org/packages/42/ce/741cd197496a1abbf51352710fd15ed995d2a2be87189c1da26a450d6e83/ruff-0.16.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:e01c21d10eb1b29f47b7454e1f4056db9a3f0260c646aa88457c610291db9f81", size = 10488846, upload-time = "2026-07-23T19:10:52.639Z" }, - { url = "https://files.pythonhosted.org/packages/52/2a/a2db8e88cade358f5cdcb05674a917751074109315d014eb6352d9a893f7/ruff-0.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6e364e5ed22ed8dc05082fd78e35308618260907ac2d3c1d637b2e682415b6c9", size = 10889729, upload-time = "2026-07-23T19:10:54.89Z" }, - { url = "https://files.pythonhosted.org/packages/42/65/62a771694ebd63029dc953e27dbad40e1588bd4860ff9fe881018fddaa49/ruff-0.16.0-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d327b8fc113a1d4421a04f3839d3752057c8dd1ee320223a6f3f52d04ada462a", size = 10568275, upload-time = "2026-07-23T19:10:56.993Z" }, - { url = "https://files.pythonhosted.org/packages/3f/e2/ced249fe8af5f086c5c58cc21cc3356d50f32f7401c5df87050c999620a7/ruff-0.16.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a9b50c55e263103586b3dcf5f73d479eb8cb5fdb6098fec59a62891dab653717", size = 11385112, upload-time = "2026-07-23T19:10:59.615Z" }, - { url = "https://files.pythonhosted.org/packages/87/0b/05154977a8fd69eeb6c103271f55403bfd8711f5c0f8ed07489d95a504e7/ruff-0.16.0-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0ff4a79ce3ec0172f3241943835de1c4cb4e2dcd07f0f8c2d02603dbbbee4b17", size = 12207008, upload-time = "2026-07-23T19:11:02.154Z" }, - { url = "https://files.pythonhosted.org/packages/fb/29/98225831a3a1eab0e02f4acc6ca6559a98611dcc68b6965ff4b7234627c1/ruff-0.16.0-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e95c448fca1fb2a18372a9440926c5a6ee789639bb975c72e7ae6d0b04218ab4", size = 11650842, upload-time = "2026-07-23T19:11:04.557Z" }, - { url = "https://files.pythonhosted.org/packages/91/66/6bd3cf90500653d55dc0ffc8507aa8300bd49d0214b2e8cb4d3fef2943ba/ruff-0.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f11a8d11010301d0a398a2fdef67691feca7294da6aef55e2150e8fa2cd520b", size = 11400718, upload-time = "2026-07-23T19:11:09.233Z" }, - { url = "https://files.pythonhosted.org/packages/8e/a2/a54eb4eae05d66364050a5d3b8a9c5ef88196531b3cbe7109d873f87f819/ruff-0.16.0-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:48044c678e9cb8698246c99b14aaccfa6601dea7379eb48a6f8f73f7a6d86cd0", size = 11426177, upload-time = "2026-07-23T19:11:11.994Z" }, - { url = "https://files.pythonhosted.org/packages/1a/be/16e3eea4b2a478a496919f5e36f17c4559e54620bd3bbac5d6affa068006/ruff-0.16.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:7aa0959bad8eb8bef50340154fc9b58678dae31fa4293afa38b44b6e552c0213", size = 10856126, upload-time = "2026-07-23T19:11:14.221Z" }, - { url = "https://files.pythonhosted.org/packages/a2/84/252eb8b868a16eec7257c14f504f77537e734b2d69c762e639e588e304a3/ruff-0.16.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:28ea2b7df8ebf7f9da6b7d47b230ab48f387c0a29be3b474c4d0740e197bb9af", size = 10571208, upload-time = "2026-07-23T19:11:16.378Z" }, - { url = "https://files.pythonhosted.org/packages/21/09/817a482f542f7570cbb4554b26e896610c7114f539b1d9e2d2145bf6bef6/ruff-0.16.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:33a3dfac8c35f81498dea9181bccc2f4c4bc8f1521a1dd9406e77643e0f0fb09", size = 11063329, upload-time = "2026-07-23T19:11:19.173Z" }, - { url = "https://files.pythonhosted.org/packages/2e/23/9403c180ca1cb9b1f7335f5c3e5305c09d49ea5b345196682a36028bde4a/ruff-0.16.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:a5237a0bda500d30d81b8e07a6973a5cbc772864cbf746ae2f4e8a2e01c9f4ed", size = 11489751, upload-time = "2026-07-23T19:11:21.74Z" }, - { url = "https://files.pythonhosted.org/packages/b2/1d/1b2ef7bcde851c78d7f17f1cca13fd6dc695fc4b3d6197941e72cae5b132/ruff-0.16.0-py3-none-win32.whl", hash = "sha256:7fab76fa065c873f41ff744347c6e77bcc3dfec4bcc754dc26b63d23c0f7f5fb", size = 10785885, upload-time = "2026-07-23T19:11:23.947Z" }, - { url = "https://files.pythonhosted.org/packages/b2/a3/d5e4ef7a56be3f928ffb90b94c25ba7d3cb9c7fe0736aeaaedf361770712/ruff-0.16.0-py3-none-win_amd64.whl", hash = "sha256:429c117f022bf481fabd9d551e7a3952b24c65e6ef44337ea09d90bebef14472", size = 11923141, upload-time = "2026-07-23T19:11:26.409Z" }, - { url = "https://files.pythonhosted.org/packages/cb/9a/8415f2657cbe200f41a4531ccededf135505a92d4a012229121f885b26f9/ruff-0.16.0-py3-none-win_arm64.whl", hash = "sha256:14296fedcd2705c77ab8235439278bbb38f285cf7da5528b00b3e330c3d4872d", size = 11273407, upload-time = "2026-07-23T19:11:28.705Z" }, +version = "0.16.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/70/25/7113f6d5498888c5fb7db34081cba7d5971c4cb1bfb26819966eee68f003/ruff-0.16.1.tar.gz", hash = "sha256:fedad7c801dabd3fb9741d76aca39246e6ddd9ca446a015875207bf19f1e6bc7", size = 4877500, upload-time = "2026-07-30T19:37:01.379Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1b/bd/694da69368e0973de65df2ddc73ab18d43c469d5963d9b150911de6bc513/ruff-0.16.1-py3-none-linux_armv6l.whl", hash = "sha256:58edb313b88f0c5460a26adf5f39a37a3be789494a15e3e411e35fa78b89f9a0", size = 10839126, upload-time = "2026-07-30T19:36:13.697Z" }, + { url = "https://files.pythonhosted.org/packages/3f/f0/b626e5d5bd0dd9576263658ef12885e2288afd1029a48e26ffed65ec1ac1/ruff-0.16.1-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fde5a99e2f97479af66edd6622c6d5a2a7592c77cf4153d9e4428f5eeb55b60c", size = 11070253, upload-time = "2026-07-30T19:36:17.14Z" }, + { url = "https://files.pythonhosted.org/packages/83/63/f40acfb6b35b88623e71684942b552c3edd96035f5d98f313815f7b277de/ruff-0.16.1-py3-none-macosx_11_0_arm64.whl", hash = "sha256:e0d4c20532fca4f7fa609369161d968dd28f65d83dabbd61d8e9c7edbf7001f6", size = 10561425, upload-time = "2026-07-30T19:36:20.04Z" }, + { url = "https://files.pythonhosted.org/packages/aa/dd/14ec0e9c2b4d315547dd38765004b4863e354e1b52cb308272215d9f6f6d/ruff-0.16.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:30affbcedf59ad5703d9c91f82266e02b47739f797e1a7b6e158e5526a6dae38", size = 10948879, upload-time = "2026-07-30T19:36:22.476Z" }, + { url = "https://files.pythonhosted.org/packages/33/e9/9d870cbae575030fdef595f04b4b97573c525b5497cce4f4498cf2f85446/ruff-0.16.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:24e9c631573cbca9d20f1283f8f479b2afa4a8503504822bd71a293889f16743", size = 10643691, upload-time = "2026-07-30T19:36:24.914Z" }, + { url = "https://files.pythonhosted.org/packages/c4/09/12743d544e2173f53ecd27217c65f90d2bc0f8424a66a60339e56bbc0457/ruff-0.16.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b41bdd48fb420987a9b5212e4957c26ad4abce401fa9ea9d4d85843727945f4f", size = 11435354, upload-time = "2026-07-30T19:36:28.447Z" }, + { url = "https://files.pythonhosted.org/packages/7f/89/a1652b2daee52083c9554a6333b678a8b01d0400f976827bb87857f9449a/ruff-0.16.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b0d1e1393b7648079e13669de1c1f4fde06d4583e84d8fd5c1551e0a77a2aa75", size = 12259033, upload-time = "2026-07-30T19:36:31.326Z" }, + { url = "https://files.pythonhosted.org/packages/16/96/ecdcb8c54ee7b123b487f807eb014e6e019155a0b81dfb669acd52f28ce3/ruff-0.16.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:07bf434b1c95f4e093be4532068ef4fcf00924eb2ade8796075980902d6fd54a", size = 11667981, upload-time = "2026-07-30T19:36:34.394Z" }, + { url = "https://files.pythonhosted.org/packages/cd/90/c52e12e0d862e9572f2a33aa227409143520abe53111e9a6babbac7b4af8/ruff-0.16.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39897739f112253ee4fdd2e8aa9a4f9ded99fb2be367d5f31dfa4ded6025584c", size = 11468183, upload-time = "2026-07-30T19:36:37.339Z" }, + { url = "https://files.pythonhosted.org/packages/2c/6b/4ffb7ad1d83eb16cf8cbb3c8815d3f11c88460fd162d4b372a2059be1c2a/ruff-0.16.1-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:82ae3c0c0d74daf17b968a10b7b3bb3ef297ab7de0c1f749646b25e690ccb150", size = 11470071, upload-time = "2026-07-30T19:36:39.91Z" }, + { url = "https://files.pythonhosted.org/packages/9c/72/32ae7db4c0b5e32ab611787caa19d1546800676d79f7483b7100a3561bf4/ruff-0.16.1-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:4d5f2ed10f8242d83fc08d521301089364e3375375705356f20c0e31606ef3ef", size = 10919503, upload-time = "2026-07-30T19:36:42.65Z" }, + { url = "https://files.pythonhosted.org/packages/f7/ca/3d901ba6ad6fc38da39c3448fc6c59ac945679293a17c3ceb6d6c1cba13e/ruff-0.16.1-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:a4665b309891f83f3e3c25447935f1213e9abbd4b5640af7a1f2def9f8d413c1", size = 10649861, upload-time = "2026-07-30T19:36:45.18Z" }, + { url = "https://files.pythonhosted.org/packages/92/79/894ef1ced26552d5f8c9cf6d85b0687840e1128c55aeab7b9c2d54a0d880/ruff-0.16.1-py3-none-musllinux_1_2_i686.whl", hash = "sha256:26e9ca5c9bc3971f20d3cf18a957f52ffd6a5f6564ff15c4912a144dcac22494", size = 11148137, upload-time = "2026-07-30T19:36:47.936Z" }, + { url = "https://files.pythonhosted.org/packages/2d/69/3609a09fa1cb46cc28b762363e440a354204e5dff01bd0c8d7437874d6b9/ruff-0.16.1-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:67e1e1e3fa4f0c82f0e36d4cd61e661f6e7a6196cb1aa92fe0828fa7b8f257cd", size = 11559211, upload-time = "2026-07-30T19:36:50.448Z" }, + { url = "https://files.pythonhosted.org/packages/fc/8a/fb22af2fd78a736e241fabf67e30ce1799a64244026377a49e133af90762/ruff-0.16.1-py3-none-win32.whl", hash = "sha256:d31765e131295b8445caf301e3e8a85b34d1b9b211b4109b7ba457888b051806", size = 10838258, upload-time = "2026-07-30T19:36:53.298Z" }, + { url = "https://files.pythonhosted.org/packages/d4/35/e57fd9fb5d423961df087a00b12d42c0a830288dc2f3b45ecca299158b4f/ruff-0.16.1-py3-none-win_amd64.whl", hash = "sha256:09b05e8b90c2cb06ad63464350e7a45e8e44a2dfe52072ebfba6666ca8d3f596", size = 11961111, upload-time = "2026-07-30T19:36:56.107Z" }, + { url = "https://files.pythonhosted.org/packages/cb/46/240ea004bf6dc4feb40e9832f2205a476a47dd5b8a3f8211a5fc5f95e20e/ruff-0.16.1-py3-none-win_arm64.whl", hash = "sha256:dbaadaac38c70239f056d306b7476f246b0bf000fa6b3876402acbf5b227eaf8", size = 11309414, upload-time = "2026-07-30T19:36:58.79Z" }, ] [[package]] @@ -1437,27 +1437,27 @@ wheels = [ [[package]] name = "ty" -version = "0.0.64" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/8e/aa/14c9965d3b173105692473897cc89c34cd91241368b2044e43167e1c17ff/ty-0.0.64.tar.gz", hash = "sha256:d12ddbb05f15158bb518af619378b385486450def95fb06f8ab98037febe9f2c", size = 6350966, upload-time = "2026-07-27T18:32:45.403Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ae/4c/c54937e4ff3fa7b34a99ea3387ec766bf0ad98dc8df8d792e89b388e658e/ty-0.0.64-py3-none-linux_armv6l.whl", hash = "sha256:3830a6675ab43635ced1c4c557f380ac4a49e9414e03975e4e4e8db644c64944", size = 12118357, upload-time = "2026-07-27T18:32:08.702Z" }, - { url = "https://files.pythonhosted.org/packages/ce/aa/a839ee2bc78e943d079e6abe199a97b4eeffb7e5c9a57326d69de452186a/ty-0.0.64-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:3ff07d7bc32a2135f58afe57393789a32ea2fed1a66216129a8e535e76043903", size = 11790882, upload-time = "2026-07-27T18:32:10.997Z" }, - { url = "https://files.pythonhosted.org/packages/4c/de/19f14357888a7198438926303753cf749428e3d62e8980ff1e9a72a78402/ty-0.0.64-py3-none-macosx_11_0_arm64.whl", hash = "sha256:4f6d1c7f897cca05d12bacbf1435150d5ffa496099515aa6ed303c8b29e1d0bb", size = 11317394, upload-time = "2026-07-27T18:32:13.162Z" }, - { url = "https://files.pythonhosted.org/packages/08/2f/f54462300535ab99b551eda733177be2eef5dbc2997d3fdb357c4ddd760a/ty-0.0.64-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:138d6c37ad4bf8583aa7a9b29d90954151d7856f9910a03eae3eb34b34c57215", size = 11863042, upload-time = "2026-07-27T18:32:15.307Z" }, - { url = "https://files.pythonhosted.org/packages/5e/95/dbecf745520ebe8bd7b02fc55eee6441c9be312ebf6addce605eb52740dd/ty-0.0.64-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1ed9719d1b7b66fb8efe073d860208a44c40af1f6cd5c2364aa9b323a1e579b4", size = 11910730, upload-time = "2026-07-27T18:32:17.467Z" }, - { url = "https://files.pythonhosted.org/packages/3b/26/12cfd40028e51ceed7b3cb645281c61c02eb64ff9fb0c09231d65c30ff25/ty-0.0.64-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d68b23e5169e2137b5f1de7169ab0cebecab6c8eda374c34c1f6394308f58242", size = 12631936, upload-time = "2026-07-27T18:32:19.533Z" }, - { url = "https://files.pythonhosted.org/packages/a0/d0/65ffc2b0a686347193c6f98e9421a7fc2a96fc3cd0b1001cf7cf284baff9/ty-0.0.64-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f41cb07d89d32626fcaf3ed4d262778fcb28b2628b6ed1e172cd7b18820668d8", size = 13171049, upload-time = "2026-07-27T18:32:22.026Z" }, - { url = "https://files.pythonhosted.org/packages/f6/a4/975a5961842dcd6fa60f0770a102c0bba7509da909ab652919bfcdcd4fc7/ty-0.0.64-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:5f24e1504ab9e212f92356b82fe088fdeb3a39f9a2f4ff25e505d2e1d0db9056", size = 12826438, upload-time = "2026-07-27T18:32:24.178Z" }, - { url = "https://files.pythonhosted.org/packages/af/ef/dfb9b7f9bcc032d3b540b0d1f55f532a336e2fb41b1bd539c05ae81a151a/ty-0.0.64-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:86db830cb914bb33bb8247b66ccea58de4496c2391bd42658aba744b439f3290", size = 12440880, upload-time = "2026-07-27T18:32:26.341Z" }, - { url = "https://files.pythonhosted.org/packages/ee/d0/bedac20505e8a8f5501ad73d7d15d8e421a563fef59993909a23036929a4/ty-0.0.64-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:652ee3d6d03bea76cd2fe8949c78bb5970394ebd7c5fd270e9518c0dee1b931d", size = 12782439, upload-time = "2026-07-27T18:32:28.642Z" }, - { url = "https://files.pythonhosted.org/packages/79/d5/795733f13ceff1378f08b3de0c49d0f518df220ed856b0dfac869f3b7c81/ty-0.0.64-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:4838768295774a86e95f9ec5633e739d016adbbb69dcbdc62f3549578a11f624", size = 11814821, upload-time = "2026-07-27T18:32:30.632Z" }, - { url = "https://files.pythonhosted.org/packages/52/3e/9d99cd1e1831003434f508ed9f258a56543194afc3bbe051eba2545fa676/ty-0.0.64-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:5a3d700669868599edf39ce5125196682f15a8880cc8c669bc2a83b99984d99b", size = 11928678, upload-time = "2026-07-27T18:32:32.888Z" }, - { url = "https://files.pythonhosted.org/packages/b8/3d/448f49a3503fb119a34348a5714bb92001f252fadbbb12d645f2e744b557/ty-0.0.64-py3-none-musllinux_1_2_i686.whl", hash = "sha256:b161f0a82a8e2f2432db3bf7702b4d3924fa9486ba0014f6710a160fc157df0d", size = 12202249, upload-time = "2026-07-27T18:32:34.905Z" }, - { url = "https://files.pythonhosted.org/packages/dd/9b/75768e562cec990d189dc05807ae72890b20ffbd1e1f43597bb98db63c60/ty-0.0.64-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:39b9dd42908df47c2dc57dda87e656fab97097ffd2618474bbdea986af0d6a9d", size = 12548817, upload-time = "2026-07-27T18:32:36.995Z" }, - { url = "https://files.pythonhosted.org/packages/34/89/44cc276ea6ca0245495014758ffeadde1798fb0b5840c9cf36d8d2ed3250/ty-0.0.64-py3-none-win32.whl", hash = "sha256:d0676ab0e0935795e5843baa28dd5e366dc343d7c0945a996df9eab8e0644885", size = 11545474, upload-time = "2026-07-27T18:32:39.389Z" }, - { url = "https://files.pythonhosted.org/packages/01/7e/d1c8a871a38d17c8f168b9a6975f6247f7660f8334e517656a5e4b4a4858/ty-0.0.64-py3-none-win_amd64.whl", hash = "sha256:dcb9bd31f54097e362b776c26ab4564d4564cdd1355cb883481167a26c03cc3f", size = 12542987, upload-time = "2026-07-27T18:32:41.412Z" }, - { url = "https://files.pythonhosted.org/packages/35/4d/6d18640d0204cacd69abbaca95ad6a34c6d7e9169e9051d0117b17b827ec/ty-0.0.64-py3-none-win_arm64.whl", hash = "sha256:82cc34c1ad9a8feb6059aef193bebcecc656e548f6fab3d518bcd8b57d198d39", size = 11899263, upload-time = "2026-07-27T18:32:43.366Z" }, +version = "0.0.66" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/58/4f6ab2a86589e422a3cf840bcf6114c565e4c39ddf4d0b7cd328af5b52b4/ty-0.0.66.tar.gz", hash = "sha256:24bddd4479ce445b51ac015410dd2d34af1cadd62a77f5b3cb269149ed83f9b5", size = 6520402, upload-time = "2026-08-04T01:09:47.714Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/45/4d/bbc28310d6d887ef73e5800f062c4bf54caa35a1b47d70c7b03d0515ecf1/ty-0.0.66-py3-none-linux_armv6l.whl", hash = "sha256:8b46450438b54b732338e4d7a78a7d2f5e1a012a13d77d121aacaca20fb814e2", size = 12409743, upload-time = "2026-08-04T01:09:01.229Z" }, + { url = "https://files.pythonhosted.org/packages/73/4e/c3d2eb2242fa0a2ef445ca2c7009dc9118e5b3dcb8b8a8bec70d58c8e4bc/ty-0.0.66-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:8e4adbe662bc3c62b52b83d46b07f703fdc3c123bb72601606c58be5ea017ed3", size = 12078362, upload-time = "2026-08-04T01:09:04.233Z" }, + { url = "https://files.pythonhosted.org/packages/fa/ba/7a4b5e45d701a8de6b714dacf6ffca91b0411baaceaa781d494037623a18/ty-0.0.66-py3-none-macosx_11_0_arm64.whl", hash = "sha256:776814351735847eb934f9a3cbea21d2278ba14aa0fc099f683da11bc2d5c90a", size = 11583289, upload-time = "2026-08-04T01:09:06.973Z" }, + { url = "https://files.pythonhosted.org/packages/4a/4a/fbb1f71ee2999f981f8c4b3b139231e4bcff4ede0c3b37e858fd1334ed26/ty-0.0.66-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cca1da877f613965b954bfd22d495386e260ec767c5536c8022cca98a260ea5d", size = 12137274, upload-time = "2026-08-04T01:09:09.592Z" }, + { url = "https://files.pythonhosted.org/packages/d5/07/9e0662f6603a5ac171ae6b314396e677ead4b45695fc948efb0a4837051f/ty-0.0.66-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:aa523e777bc36c0fbf8ded5844096f46cbfc3712fe5a003351a94259b7e86cf4", size = 12207047, upload-time = "2026-08-04T01:09:12.533Z" }, + { url = "https://files.pythonhosted.org/packages/bb/cc/12e01bc2ea47fa7bf20fc6cdd3249d6a340be9be4edc52b1d77256245dd0/ty-0.0.66-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3f7993c1f95f80a4e44056e6aaf8e46e608d778db131ae5ce59262ab59358f9a", size = 12919304, upload-time = "2026-08-04T01:09:15.175Z" }, + { url = "https://files.pythonhosted.org/packages/75/88/c6c0d3a8e71c9cdb560cc5c627a4bba6c47b5eec460b2f8c449b57c6d03a/ty-0.0.66-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f9b29354bcbac9f53b6952d8f46789bd81eeb9bdd7a68df7d26e654ca7498c3c", size = 13470963, upload-time = "2026-08-04T01:09:17.849Z" }, + { url = "https://files.pythonhosted.org/packages/27/c2/2f8c18063412ad80e1b3f87afce7bd50982b4a048166607b67f80660a9fb/ty-0.0.66-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:fbf4e5325f7f584d9c346946e7c415b2e3cd3b8f1119d468082a90a6afd020ce", size = 13244773, upload-time = "2026-08-04T01:09:20.59Z" }, + { url = "https://files.pythonhosted.org/packages/ed/f6/fdae2b95831116dffc055ad53a99a8f21437263651047b3ad46def4950a3/ty-0.0.66-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bd304363764bd723c22fb20b17035c345420fdf66fee856934b158ebde08a91", size = 12751343, upload-time = "2026-08-04T01:09:23.225Z" }, + { url = "https://files.pythonhosted.org/packages/14/1e/7e75f0371d11463a256dc2bee98b0df401e0fa06021e200b8b601d21949d/ty-0.0.66-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:205d8589bd957ea9718d488b731b2fbdd0d1b1cefd37c79f52c0deb7cafddfef", size = 13068057, upload-time = "2026-08-04T01:09:26.052Z" }, + { url = "https://files.pythonhosted.org/packages/b4/84/f20f24518f6f0bea936e2e668ede250b9ce0774624559931599bd1f42772/ty-0.0.66-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:7304a1df54741a2343801354f41d7ad87974acb541ee3e93c26cb6a1a0b863af", size = 12082318, upload-time = "2026-08-04T01:09:28.877Z" }, + { url = "https://files.pythonhosted.org/packages/08/20/70ca0eac2427d4a58a81a3a9426b20e46fb4a5a13aefc9edc2e5172e1243/ty-0.0.66-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:cf2c062863e5da0f588b0b120fec0da2e81c0913ee4cd07b50d77aa60ffd8deb", size = 12228978, upload-time = "2026-08-04T01:09:31.905Z" }, + { url = "https://files.pythonhosted.org/packages/7c/01/5a461ab34456d788248780830aed673c9e0e796f6e9dd92d3dbf02e04d7f/ty-0.0.66-py3-none-musllinux_1_2_i686.whl", hash = "sha256:a58d32c879d86428978332adf21e85009ec269314a23d330c3483c65b57aedc7", size = 12471917, upload-time = "2026-08-04T01:09:34.418Z" }, + { url = "https://files.pythonhosted.org/packages/46/ed/f8eb5eff7c9ee644490c6d2626425935c097acc54871adf659ea70624a2c/ty-0.0.66-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:b2f810fa3516c977630d78dbe9161592b3c27029fa9bb81074366624d9b5e4b6", size = 12858086, upload-time = "2026-08-04T01:09:37.602Z" }, + { url = "https://files.pythonhosted.org/packages/02/7f/8a78361f752274a550a7f6f07f127b246ece7d7fdde31121d22074e46eab/ty-0.0.66-py3-none-win32.whl", hash = "sha256:d28c3df565a387c1c5ea359aa452a46d19163616ef71be819058a424a62f1ae1", size = 11782494, upload-time = "2026-08-04T01:09:40.136Z" }, + { url = "https://files.pythonhosted.org/packages/7d/e2/deed22b823ce309b8410d50414fd15afe9e90aee8b8ca04789ba55c21231/ty-0.0.66-py3-none-win_amd64.whl", hash = "sha256:e3a457f3312c078f24c47d0da6e4f73de34d0a77ed2de22571c066c80b2fd5e7", size = 12893338, upload-time = "2026-08-04T01:09:42.825Z" }, + { url = "https://files.pythonhosted.org/packages/f9/ce/6c828f42ef1ed39f53f57f9c0cdcdf03e23666fa7a29d799042a2c34bcb4/ty-0.0.66-py3-none-win_arm64.whl", hash = "sha256:2f62ae247b9c75674fcc060635f9f00210357da7681de59234d97b50fb9e9e94", size = 12227381, upload-time = "2026-08-04T01:09:45.365Z" }, ] [[package]] diff --git a/yaml/orgs.yaml b/yaml/orgs.yaml index c5d8e71..ce4a837 100644 --- a/yaml/orgs.yaml +++ b/yaml/orgs.yaml @@ -12,7 +12,6 @@ targets: - name: discover - name: remove_iam_user depends_on: ["discover"] - optional: true dry_run: true max_workers: 5 metadata: