✨ feat: make evaluations and tracing async

[Marco Russo (marcorusso97)]

Summary

This PR converts the judge-evaluation hot path from thread-per-request to a true asyncio fan-out, adds an async dispatch path to the request router so those judge calls are actually non-blocking I/O, overlaps Attack/Run record creation with independent config prep in the orchestrator, parallelizes per-goal Result-record initialization, and cleans up a dead/confusing config parameter (batch_size_judge → judge_concurrency).

Changes by area

1. hackagent/attacks/evaluator/base.py — async judge fan-out

• Replaced ThreadPoolExecutor(max_workers=batch_size) + threading.Lock() with asyncio.gather() over an asyncio.Semaphore(judge_concurrency).
• New _process_row_async, _request_with_assertions_async, _route_request_async (async twins of the existing sync methods).
• Now imports the shared run_coroutine_blocking() bridge helper from hackagent/async_utils.py (see below) instead of keeping a private copy.

2. hackagent/async_utils.py — new shared module (extracted this session)

• run_coroutine_blocking(coro_factory): runs a coroutine from synchronous code — asyncio.run() directly when no loop is running, or a dedicated bridge thread when called from inside an already-running loop (e.g. notebooks).
• Previously duplicated only inside base.py; now shared so other synchronous call sites (see bump(deps-dev): bump ruff from 0.11.9 to 0.11.10 #4 below) can reuse the same async fan-out pattern instead of re-implementing it.

3. hackagent/router/router.py — async dispatch path

• New route_request_async(): an async twin of route_request(), preserving identical before/after-guardrail semantics and error envelopes.
• Refactored LiteLLM dispatch into shared _prepare_litellm_dispatch() / _finalize_litellm_dispatch() helpers, reused by both a new sync _dispatch_via_litellm() (litellm.completion) and the new _dispatch_via_litellm_async() (litellm.acompletion). Old implementation kept as _dispatch_via_litellm_legacy for reference/fallback.

4. hackagent/router/tracking/coordinator.py — parallel goal-result initialization (new this session)

• TrackingCoordinator.initialize_goals() used to create one Result record per goal via Tracker.create_goal_result() in a plain sequential for loop — each call is an independent network round trip (or, for the local backend, a locked SQLite write), so N goals meant N blocking round trips before the attack itself could start (observed ~30s for 9 goals in a live run).
• Now fans these calls out concurrently with asyncio.gather() + asyncio.Semaphore(_GOAL_INIT_CONCURRENCY=8), wrapping each blocking create_goal_result() call in asyncio.to_thread(), bridged back to the synchronous caller via run_coroutine_blocking().
• Does not change the goal category classification, which was already batched before this PR (a single/few LLM calls via GoalCategoryClassifier.classify_goals() instead of one per goal) — this change only parallelizes the remaining per-goal Result-creation network calls.

5. hackagent/attacks/orchestrator.py — overlapped run creation

• Attack creation → Run creation → update_run(RUNNING) (3 sequential server round-trips) now run on a background thread (ThreadPoolExecutor(max_workers=1)), submitted immediately, overlapping with independent config preparation instead of blocking on it first.
• run_future.result() is only awaited right before attack execution starts — error propagation semantics are unchanged.

6. judge_concurrency rename (config layer, all technique attack.py files, TUI, docs, tests)

• Removed batch_size_judge / judge_batch_size, which had become dead: they only fed EvaluatorConfig.batch_size, a field the new async code never reads. Every judge run was silently always using the hardcoded default of 10, regardless of user config.
• Now a single, correctly-wired judge_concurrency field flows from user config → EvaluatorConfig → the asyncio.Semaphore.
• judge_parallelism (controls how many different judges run concurrently with each other, not row-level concurrency within one judge) was left untouched.

7. Misc: flask>=3.1.3 added to pyproject.toml (unrelated dependency bump, not part of the async work).

Functional impact: local vs. remote models

	Local model (e.g. Ollama on localhost)	Remote/hosted API (OpenRouter, OpenAI, etc.)
Bottleneck	GPU/CPU compute, single-model queue	Network round-trip latency
Effect of raising judge_concurrency	Little to none — requests queue at the same local inference server regardless of how many are "in flight"	Large — concurrent requests overlap network latency instead of paying for it serially
Effect of goal-init parallelism (#4)	Speeds up the local SQLite backend a little (writes are serialized by a lock anyway, but no longer block on Python-level sequencing)	Speeds up meaningfully — up to 8 create_result HTTP round trips overlap instead of running one at a time
Measured benchmark	N/A (compute-bound)	~9.8x wall-clock speedup on judge evaluation at judge_concurrency=10 vs 1
Risk of raising concurrency too high	Can overload/starve a local GPU server (OOM, thrashing)	Can hit provider rate limits (429s)

In short: this PR doesn't change what the SDK does with local vs. remote-tracked runs (dashboard sync, tracing, result persistence are conceptually untouched). It changes how fast two previously-sequential phases complete — judge evaluation and goal-result initialization — and both speedups are largest when the bottleneck is network latency (remote APIs / remote backend), and negligible-to-nothing when the bottleneck is local compute or a locked local resource.

[Copilot]

Pull request overview

This PR migrates judge evaluation and related tracking/router flows from thread-per-request to true asyncio concurrency, so LLM and backend calls can overlap as non-blocking I/O. It also standardizes the config surface by renaming the judge batching knob to judge_concurrency, and updates tests/docs/examples accordingly.

Changes:

• Add an async request-routing path (route_request_async) and async LiteLLM dispatch (litellm.acompletion) while preserving the existing response envelope semantics.
• Convert the judge evaluator hot path to async fan-out with a semaphore (judge_concurrency) and add a shared run_coroutine_blocking() bridge utility.
• Parallelize tracking goal Result-record initialization and overlap Attack/Run record creation with config preparation in the orchestrator.

Reviewed changes

Copilot reviewed 47 out of 48 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
uv.lock	Locks the new Flask dependency addition.
pyproject.toml	Adds flask>=3.1.3 to runtime dependencies.
hackagent/async_utils.py	Introduces shared sync→async bridge helper (run_coroutine_blocking).
hackagent/attacks/evaluator/base.py	Reworks judge row evaluation to async fan-out; introduces async router/request methods.
hackagent/router/router.py	Adds route_request_async and async LiteLLM dispatch path; refactors LiteLLM request preparation/finalization.
hackagent/router/tracking/coordinator.py	Parallelizes per-goal Result initialization via async fan-out + to_thread.
hackagent/attacks/orchestrator.py	Overlaps Attack/Run record creation with independent configuration prep using a background thread.
hackagent/attacks/evaluator/evaluation_step.py	Renames config key mapping to judge_concurrency and adjusts judge-parallelism defaults.
hackagent/attacks/evaluator/inline_step_judge.py	Updates inline judge base config to emit judge_concurrency.
hackagent/attacks/techniques/config.py	Renames batch_size_judge → judge_concurrency in technique config model and exported default constant.
hackagent/attacks/techniques/advprefix/config.py	Wires judge_concurrency into AdvPrefix evaluation config models.
hackagent/attacks/techniques/advprefix/attack.py	Updates AdvPrefix pipeline config key lists to judge_concurrency.
hackagent/attacks/techniques/baseline/attack.py	Updates pipeline config key list to judge_concurrency.
hackagent/attacks/techniques/bon/attack.py	Updates pipeline config key lists to judge_concurrency.
hackagent/attacks/techniques/cipherchat/attack.py	Updates pipeline config key list to judge_concurrency.
hackagent/attacks/techniques/fc/attack.py	Updates pipeline config key lists to judge_concurrency.
hackagent/attacks/techniques/flipattack/attack.py	Updates pipeline config key list to judge_concurrency.
hackagent/attacks/techniques/h4rm3l/attack.py	Updates pipeline config key list to judge_concurrency.
hackagent/attacks/techniques/mml/attack.py	Updates pipeline config key list to judge_concurrency.
hackagent/attacks/techniques/pap/attack.py	Updates pipeline config key lists to judge_concurrency.
hackagent/attacks/techniques/static_template/attack.py	Updates pipeline config key list to judge_concurrency.
hackagent/attacks/techniques/tap/attack.py	Updates pipeline config key lists to judge_concurrency.
hackagent/cli/tui/attack_specs.py	Renames the TUI config field to judge_concurrency and updates labels/descriptions.
hackagent/examples/vllm/hack.py	Updates example config constant/key to judge_concurrency.
hackagent/examples/openai_sdk/quick_evaluation/run_h4rm3l.py	Updates example config key to judge_concurrency.
hackagent/examples/openai_sdk/multi_judge/run_flipattack_multi_judge.py	Updates example config key to judge_concurrency.
hackagent/examples/litellm_multi_provider/demo.py	Updates demo config key to judge_concurrency.
docs/docs/sdk/python-quickstart.md	Updates docs default config key to judge_concurrency.
docs/docs/cli/attack.mdx	Updates CLI docs to reference judge_concurrency.
docs/docs/attacks/advprefix.md	Updates attack docs to reference judge_concurrency.
docs/docs/attacks/bon.md	Updates attack docs to reference judge_concurrency.
docs/docs/attacks/cipherchat.md	Updates attack docs to reference judge_concurrency.
docs/docs/attacks/flipattack.md	Updates attack docs to reference judge_concurrency and related narrative.
docs/docs/attacks/h4rm3l.md	Updates attack docs to reference judge_concurrency.
docs/docs/attacks/mml.md	Updates attack docs to reference judge_concurrency.
docs/docs/attacks/pap.md	Updates attack docs to reference judge_concurrency.
tests/unit/router/test_dispatch.py	Adds coverage for route_request_async (LiteLLM + ADK paths).
tests/unit/attacks/test_evaluator_base.py	Adds regression test ensuring async router path preserves input ordering.
tests/unit/attacks/test_evaluation_step.py	Updates evaluation-step config expectations for judge_concurrency.
tests/unit/attacks/evaluator/test_inline_step_judge.py	Updates inline judge base-config tests to judge_concurrency.
tests/unit/attacks/advprefix/test_advprefix_evaluation_extended.py	Updates AdvPrefix evaluation tests to judge_concurrency.
tests/unit/attacks/bon/test_config.py	Updates required config key expectations to judge_concurrency.
tests/unit/attacks/cipherchat/test_config.py	Updates required config key expectations to judge_concurrency.
tests/unit/attacks/flipattack/test_flipattack_config.py	Updates FlipAttack config tests to judge_concurrency.
tests/unit/attacks/flipattack/test_flipattack_attack.py	Updates FlipAttack evaluation-step config key expectations.
tests/unit/attacks/h4rm3l/test_config.py	Updates required config key expectations to judge_concurrency.
tests/unit/attacks/mml/test_attack.py	Updates MML evaluation-step config key expectations.
tests/unit/attacks/pap/test_config.py	Updates required config key expectations to judge_concurrency.

Comments suppressed due to low confidence (1)

hackagent/attacks/evaluator/base.py:536

• If the coroutine fan-out is executed in a bridge thread (when a loop is already running), progress updates are skipped inside _fan_out_one(); without compensating updates here, the progress bar will stay at 0 until the context exits. Update the progress bar from the caller thread while iterating over the results returned by run_coroutine_blocking(_fan_out) when in_running_loop is true.

                results_map[idx] = (
                    original_index,
                    current_eval,
                    current_expl,
                    current_raw_response,

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 50 out of 51 changed files in this pull request and generated 3 comments.

[codecov]

Codecov Report

❌ Patch coverage is 80.69164% with 67 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
hackagent/router/router.py	76.76%	46 Missing ⚠️
hackagent/attacks/evaluator/base.py	85.39%	13 Missing ⚠️
hackagent/attacks/orchestrator.py	84.00%	4 Missing ⚠️
hackagent/async_utils.py	72.72%	3 Missing ⚠️
hackagent/attacks/techniques/h4rm3l/attack.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Nicola Franco (franconicola)]

LGTM