[WIP] Replace duck-typing with typed AttackResult model

[Claude]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>Replace _normalize_attack_results() duck-typing with a typed AttackResult model</issue_title>
<issue_description>Problem. attacks/orchestrator.py flattens heterogeneous returns by trying .evaluated, .rows, .results, .data, .items in turn. Any new technique that names a field differently will silently mis-normalize.

Actions.

• Define AttackResult (Pydantic v2, frozen) in attacks/types.py with fields: goal, prompt, response, evaluations: list[Evaluation], metadata: dict, etc.
• Have BaseAttack.run() declare a return type of list[AttackResult].
• Update each of the 10 techniques in attacks/techniques/ to return the typed model.
• Delete _normalize_attack_results().
• Add a unit test per technique asserting return type.

Acceptance: _normalize_attack_results removed; mypy/pyright clean on the orchestrator path.</issue_description>

• Closes Replace _normalize_attack_results() duck-typing with a typed AttackResult model #399

Comments on the Issue (you are @claude[agent] in this section)

• Fixes Replace _normalize_attack_results() duck-typing with a typed AttackResult model #399

[Nicola Franco (franconicola)]

@claude[agent] check the CI Checks / Commit messages and the reviews of co-pilot in this pull request

[Claude]

@claude[agent] check the CI Checks / Commit messages and the reviews of co-pilot in this pull request

Checked both.

Copilot/code-quality reviews (10 threads) — all were the same finding: "Signature mismatch in overriding method. This method requires 2 positional arguments, whereas overridden BaseAttack.run requires 1." The base is def run(self, **kwargs), but every technique overrode it as def run(self, goals: List[str]). Fixed in 00af9cc: all 14 overrides (advprefix, autodan_turbo, baseline, bon, cipherchat, fc x2, flipattack, h4rm3l, mml, pair, pap, static_template, tap) are now def run(self, goals: Optional[List[str]] = None, **kwargs) -> List[AttackResult] with goals = goals or []. indirect_prompt_injection and rag already used that pattern. 1458 unit tests pass.

CI / Commit Messages — this job runs cz check --rev-range base..head, which validates every commit in the PR against cz_gitmoji. Four commits fail: 5801de5 Initial plan (the scaffolding commit), 4fe47fd, 78c08ef, 3bedd27. I rewrote all four locally to compliant messages and confirmed cz check --rev-range then passes with a byte-identical tree — but publishing a rewrite needs a force push, which I can't do; report_progress only does a plain push and rebased my rewrite away (leaving an empty commit d97c6aa, sorry about the noise).

Two ways to get this green: force-push the rewritten history yourself, or merge with Squash and merge so the compliant PR title becomes the commit message.

[Nicola Franco (franconicola)]

Copilot resolve the merge conflicts in this pull request. Also fix the CI Checks

[Copilot]

Copilot resolve the merge conflicts in this pull request. Also fix the CI Checks

Merge conflicts resolved in 908dfd1 (main merged into the branch).

Conflict — hackagent/attacks/orchestrator.py, the evaluation-pipeline except block. main (#518) added evaluation_error = e + record_run_audit_failure(...); this branch had changed the fallback to the typed rows. Resolution keeps main's audit-failure handling and uses final_results = normalized_results as the fallback.

Extra fix — the merge surfaced a broken hunk that landed on main when #518 and #519 merged: in Tracker._add_trace, self._record_failure(f"Goal {ctx.goal_index}: create trace", e) ended up right after the _emit("trace_added", ...) call, where e is unbound. That fails ruff check (F821) and would raise NameError on every trace. Moved it back into the trace-creation except Exception as e: block, matching the intent of 14867e4.

Verified: ruff check . and ruff format --check . clean, 2728 unit tests and 103 offline integration tests pass.

CI / Commit Messages — still red, and I can't fix it from here. cz check --rev-range base..head scans every commit in the PR, and four earlier commits have non-cz messages: 5801de5 Initial plan, 4fe47fd, 78c08ef, 3bedd27. Rewriting them requires a force push, which the agent push path doesn't support (a prior attempt was rebased away). Either force-push a reworded history yourself, or use Squash and merge so the compliant PR title becomes the commit message. Every other check is green.

[Nicola Franco (franconicola)]

@claude[agent] fix the tests

[Claude]

@claude[agent] fix the tests

I dug into the failing jobs and there's no test to fix — nothing in this PR is broken.

Unit tests are green on all 9 matrix combos (3.10–3.13 × ubuntu/macos/windows), as are Offline Integration, E2E, Linting and Formatting, and Build Documentation.

The two red test jobs are Integration Tests (Ollama, fast) and (Ollama, slow), and every failure is an HTTP error from the hosted backend, not an assertion:

• fast — 7 failures, all RuntimeError: RemoteBackend: Failed to create agent ... (status 429, body={"detail":"Request was throttled. Expected available in 142 seconds."}), plus one 500 (Cloudflare HTML) and organization_me_status=401 Invalid Auth0 token.
• slow — 1 failure, RuntimeError: RemoteBackend: update_run ... returned 429, raised in TrackingCoordinator.create → step.update_run_status, i.e. before any attack code runs.

All of these come out of hackagent/server/storage/remote.py:519. None touch the AttackResult path. The throttle is from several CI runs firing at the hosted API in quick succession. A re-run once the rate-limit window clears should clear it; I can't trigger one from here.

Verified locally on 908dfd1: ruff check . and ruff format --check . clean, 2751 unit tests pass, 103 offline integration tests pass. (One local unit failure, test_preset_based_loading, is the sandbox having no HuggingFace network access — it passes in CI.)

Commit Messages is still red for the reason in my earlier comment: cz check --rev-range scans every commit, and 5801de5, 4fe47fd, 78c08ef, 3bedd27 have non-cz messages. Rewriting them needs a force push, which I can't do. Use Squash and merge so the compliant PR title becomes the commit message, or force-push a reworded history.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.openai.com
  • Triggering command: /home/REDACTED/work/hackagent/hackagent/.venv/bin/pytest pytest tests/unit -q -p no:randomly --abbrev-ref HEAD /usr/bin/base64 unset --global nfig/composer/vendor/bin/git */bin.*$") { next } } { printf "%s%s", sep, $0; sep=RS } -d (dns block)
• gitlab.com
  • Triggering command: /home/REDACTED/work/hackagent/hackagent/.venv/bin/pytest pytest tests/unit -q -p no:randomly --abbrev-ref HEAD /usr/bin/base64 unset --global nfig/composer/vendor/bin/git */bin.*$") { next } } { printf "%s%s", sep, $0; sep=RS } -d (dns block)
• huggingface.co
  • Triggering command: /home/REDACTED/work/hackagent/hackagent/.venv/bin/pytest pytest tests/unit -q -p no:randomly --abbrev-ref HEAD /usr/bin/base64 unset --global nfig/composer/vendor/bin/git */bin.*$") { next } } { printf "%s%s", sep, $0; sep=RS } -d (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot cloud agent settings (admins only)

[Nicola Franco (franconicola)]

LGTM

[codecov]

Codecov Report

❌ Patch coverage is 93.71069% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
hackagent/attacks/types.py	91.54%	6 Missing ⚠️
hackagent/attacks/techniques/baseline/attack.py	71.42%	2 Missing ⚠️
...agent/attacks/techniques/static_template/attack.py	85.71%	1 Missing ⚠️
hackagent/router/tracking/tracker.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR migrates the attack-technique result contract away from orchestrator-level duck-typing (_normalize_attack_results) toward a shared, typed AttackResult model, with corresponding technique updates and unit tests to enforce the new return type.

Changes:

• Introduces hackagent.attacks.types.AttackResult (+ helpers) and updates techniques to return list[AttackResult].
• Removes _normalize_attack_results() from the orchestrator and updates orchestration/evaluation boundaries to convert typed results back to legacy dict rows when needed.
• Updates/extends unit tests across techniques to assert typed return values and validate round-trip conversions.

Reviewed changes

Copilot reviewed 36 out of 39 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/attacks/test_error_propagation.py	Updates normalization tests to exercise rows_to_attack_results / attack_results_to_rows and adds round-trip coverage.
tests/unit/attacks/tap/test_attack.py	Asserts TAP run() returns AttackResult.
tests/unit/attacks/static_template/test_attack.py	Updates expectations for static_template run() to return typed results and metadata.
tests/unit/attacks/rag/test_attack.py	Updates assertions to read results from AttackResult.metadata / AttackResult.evaluations.
tests/unit/attacks/pap/test_attack.py	Asserts PAP run() returns AttackResult.
tests/unit/attacks/pair/test_attack.py	Asserts PAIR run() returns AttackResult.
tests/unit/attacks/mml/test_attack.py	Updates assertions to typed AttackResult and metadata semantics.
tests/unit/attacks/indirect_prompt_injection/test_attack_return_type.py	New test asserting indirect-prompt-injection returns list[AttackResult] and validates error cases.
tests/unit/attacks/indirect_prompt_injection/init.py	Package init for new indirect-prompt-injection unit tests.
tests/unit/attacks/h4rm3l/test_attack.py	Asserts h4rm3l run() returns AttackResult.
tests/unit/attacks/flipattack/test_attack.py	Asserts FlipAttack run() returns AttackResult.
tests/unit/attacks/fc/test_attack.py	Adds tests asserting FC/tFC run() returns AttackResult.
tests/unit/attacks/cipherchat/test_attack.py	Asserts CipherChat run() returns AttackResult.
tests/unit/attacks/bon/test_attack.py	Asserts BoN run() returns AttackResult.
tests/unit/attacks/baseline/test_attack_return_type.py	New test asserting Baseline returns list[AttackResult].
tests/unit/attacks/baseline/init.py	Package init for new baseline unit tests.
tests/unit/attacks/autodan_turbo/test_attack.py	Asserts AutoDAN-Turbo run() returns AttackResult.
tests/unit/attacks/advprefix/test_attack_return_type.py	New test asserting AdvPrefix returns list[AttackResult].
tests/unit/attacks/advprefix/init.py	Package init for new advprefix unit tests.
hackagent/router/tracking/tracker.py	Adjusts trace failure recording to occur only on actual trace creation failure.
hackagent/attacks/types.py	Adds typed AttackResult/Evaluation models plus conversion helpers for legacy row shapes.
hackagent/attacks/techniques/tap/attack.py	Updates TAP run() signature/return type and converts pipeline output to typed results.
hackagent/attacks/techniques/static_template/attack.py	Updates static_template run() to return typed results instead of {evaluated, summary}.
hackagent/attacks/techniques/rag/attack.py	Updates RAG run() to return typed results.
hackagent/attacks/techniques/pap/attack.py	Updates PAP run() to return typed results.
hackagent/attacks/techniques/pair/attack.py	Updates PAIR run() to return typed results.
hackagent/attacks/techniques/mml/attack.py	Updates MML run() to return typed results.
hackagent/attacks/techniques/indirect_prompt_injection/attack.py	Updates indirect-prompt-injection run() to return typed results.
hackagent/attacks/techniques/h4rm3l/attack.py	Updates h4rm3l run() to return typed results.
hackagent/attacks/techniques/flipattack/attack.py	Updates FlipAttack run() to return typed results.
hackagent/attacks/techniques/fc/attack.py	Updates FC/tFC run() to return typed results.
hackagent/attacks/techniques/cipherchat/attack.py	Updates CipherChat run() to return typed results.
hackagent/attacks/techniques/bon/attack.py	Updates BoN run() to return typed results.
hackagent/attacks/techniques/baseline/attack.py	Updates Baseline run() to return typed results.
hackagent/attacks/techniques/base.py	Updates abstract technique base run() return type to List[AttackResult].
hackagent/attacks/techniques/autodan_turbo/attack.py	Updates AutoDAN-Turbo run() to return typed results.
hackagent/attacks/techniques/advprefix/attack.py	Updates AdvPrefix run() to return typed results.
hackagent/attacks/orchestrator.py	Removes _normalize_attack_results, adds typed conversion boundary, and updates aggregation logic.
hackagent/attacks/base.py	Updates the other BaseAttack interface to return List[AttackResult].

Comments suppressed due to low confidence (3)

hackagent/attacks/orchestrator.py:1568

• To keep _execute_local_attack()'s List[AttackResult] return type truthful, the batched sequential path should normalize attack_impl.run() output to AttackResult (not just flatten legacy dict wrappers).

                        batch_results = flatten_run_result(
                            attack_impl.run(**batch_params)
                        )

hackagent/attacks/orchestrator.py:1602

• Same as the sequential batching path: the per-goal worker path returns flatten_run_result(...) but the function signature and downstream code expect AttackResult instances. Normalize per-goal outputs with rows_to_attack_results() so types are consistent and callers don't receive heterogeneous row shapes.

                            goal_params = {**attack_params, "goals": [goal]}
                            goal_results = flatten_run_result(
                                local_impl.run(**goal_params)
                            )

hackagent/attacks/orchestrator.py:1663

• The non-batched execution path also returns flatten_run_result(...), which can yield dict rows and violates the method's List[AttackResult] return type. Normalize with rows_to_attack_results() to ensure orchestrator always hands back typed results.

            results = flatten_run_result(attack_impl.run(**attack_params))
            logger.info(f"{self.attack_type} attack completed")
            return results

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.