fix(audit): surface tracking and evaluation failures

[Nicola Franco (franconicola)]

Summary

Closes #389.

Audit-bearing code (tracking decorators, StepTracker, Tracker, orchestrator, evaluators, RemoteBackend) could previously swallow exceptions silently, letting a run be reported as COMPLETED/"no findings" even though tracking or evaluation had partially failed.

This PR makes those failures visible:

• Adds hackagent/router/tracking/audit.py (record_run_audit_failure, AuditPersistenceError) to persist a structured {"step", "status": "failed", "error"} entry on the run record and mark the run FAILED when a tracking/evaluation failure occurs. Raises AuditPersistenceError if even that write fails, so failures can never be reported as trustworthy successes.
• Wires this helper into router/tracking/tracker.py, step.py, and decorators.py.
• AttackOrchestrator.execute: records evaluation-pipeline failures, re-checks the persisted run status (flushing queued remote writes via get_run) before deciding between COMPLETED and FAILED, and can raise via fail_on_run_error.
• RemoteBackend: raises instead of returning a fake RunRecord/logging a warning on failed update_run/update_result/create_trace writes and on failed background writer flush.
• Adds exc_info=True to previously silent except Exception logging across evaluator/sync.py, evaluation_step.py, inline_step_judge.py.
• Documents the "never silently swallow in audit-bearing code" policy in CONTRIBUTING.md.
• Adds/updates unit tests asserting a tracking-side exception surfaces in the run record.

Verification

• pytest tests/unit → 2711 passed, 171 subtests passed
• ruff check . → all checks passed
• No except Exception: pass/continue swallowers remain in router/tracking/, attacks/orchestrator.py, attacks/evaluator/ (acceptance criteria from Audit and tighten broad except Exception: blocks in the audit path #389)

[Copilot]

Pull request overview

This PR hardens the audit/tracking/evaluation pipeline so that persistence and evaluation failures cannot be silently swallowed and misreported as a successful “COMPLETED / no findings” run.

Changes:

• Introduces record_run_audit_failure + AuditPersistenceError to persist structured audit failures onto the run record and force FAILED status when audit-bearing operations break.
• Makes RemoteBackend persistence fail fast (raise on update_run/update_result/create_trace errors; raise on failed background flush; flush before get_run).
• Updates orchestrator/tracking/evaluator paths to log exceptions with exc_info=True, record audit failures, and adds unit tests to assert these failures surface.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/server/storage/test_remote_backend.py	Adds tests asserting RemoteBackend raises on failed run updates and surfaces background write failures on flush().
tests/unit/router/tracking/test_tracker.py	Adds tests asserting tracking-side exceptions are recorded on the run and audit persistence failures raise.
tests/unit/router/tracking/test_decorators.py	Asserts decorator extraction failures are recorded as structured audit failures on the run.
tests/unit/attacks/test_orchestrator_extended.py	Adds tests to ensure tracking/evaluation failures keep the run FAILED (not overwritten as COMPLETED).
hackagent/server/storage/remote.py	Raises on failed writes, tracks background writer failures, raises on flush(), and flushes before get_run().
hackagent/router/tracking/tracker.py	Records goal-tracking failures to the run record via the new audit helper.
hackagent/router/tracking/step.py	Records step-tracking failures to the run record; stops returning False for run-status update failures (now raises).
hackagent/router/tracking/decorators.py	Stops silently swallowing extractor/default-extractor exceptions; logs with exc_info=True and records audit failures.
hackagent/router/tracking/audit.py	New helper for persisting structured audit failures and raising if the failure cannot be persisted.
hackagent/attacks/orchestrator.py	Records evaluation/flush/final-status verification failures, re-checks persisted run status before COMPLETED, improves exception logging.
hackagent/attacks/evaluator/sync.py	Adds exc_info=True logging for previously silent evaluator update paths.
hackagent/attacks/evaluator/inline_step_judge.py	Adds exc_info=True logging for judge init and per-candidate judge failures.
hackagent/attacks/evaluator/evaluation_step.py	Adds exc_info=True logging for structured metrics sync failure paths.
CONTRIBUTING.md	Documents “never silently swallow in audit-bearing code” policy and approved handling patterns.

Comments suppressed due to low confidence (2)

hackagent/router/tracking/step.py:431

• update_run_status now raises on backend failures, but the docstring still states it returns False otherwise. This is misleading for callers (e.g., coordinator) and contradicts the class docs about failing gracefully. Update the docstring to reflect the new contract: return True on success, False only when tracking is disabled/invalid UUID, and raise on backend errors.

    def update_run_status(self, status: StatusEnum) -> bool:
        """
        Update the run status on the backend.

        Args:
            status: New status to set

        Returns:
            True if update was successful, False otherwise
        """

hackagent/attacks/orchestrator.py:2051

• Same issue as the success path: the FAILED-status update assumes run_id is always a UUID via UUID(run_id). This contradicts the earlier non-UUID handling comment and breaks custom/test backends that use opaque run identifiers.

                self.hackagent_agent.backend.update_run(
                    UUID(run_id),
                    status=StatusEnum.FAILED.value,
                    run_notes=f"Execution failed: {str(e)}",

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 74.35897% with 40 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
hackagent/attacks/orchestrator.py	77.27%	15 Missing ⚠️
hackagent/router/tracking/decorators.py	55.55%	8 Missing ⚠️
hackagent/router/tracking/step.py	64.28%	5 Missing ⚠️
hackagent/router/tracking/tracker.py	66.66%	5 Missing ⚠️
hackagent/router/tracking/audit.py	85.00%	3 Missing ⚠️
hackagent/server/storage/remote.py	81.81%	2 Missing ⚠️
hackagent/attacks/evaluator/evaluation_step.py	80.00%	1 Missing ⚠️
hackagent/attacks/evaluator/sync.py	66.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

hackagent/attacks/orchestrator.py:1819

• This update_run call always forces UUID(run_id), but earlier in this method you explicitly handle non-UUID run identifiers. If non-UUID IDs are truly supported (per the comment), this conversion prevents status updates from working on those backends. Either remove the non-UUID branch (if unsupported) or pass through the raw run_id when UUID parsing fails.

        try:
            logger.info(f"Updating run {run_id} status to RUNNING")
            self.hackagent_agent.backend.update_run(
                UUID(run_id),
                status=StatusEnum.RUNNING.value,
            )

hackagent/attacks/orchestrator.py:2030

• Same as the RUNNING status update: this final status write unconditionally wraps run_id in UUID(...), which contradicts the earlier non-UUID handling. If non-UUID IDs are supported, route the raw run_id through to the backend when UUID parsing fails so the final status isn’t silently skipped.

                self.hackagent_agent.backend.update_run(
                    UUID(run_id),
                    status=final_status.value,
                )

hackagent/attacks/orchestrator.py:2052

• In the FAILED path, UUID(run_id) is still forced even though the method comments mention non-UUID run identifiers. If you intend to support non-UUID backends, use the same “parse UUID or pass-through” approach here; otherwise the error handler can itself raise before recording failure status.

                self.hackagent_agent.backend.update_run(
                    UUID(run_id),
                    status=StatusEnum.FAILED.value,
                    run_notes=f"Execution failed: {str(e)}",
                )