✨ feat: make evaluations and tracing async

docs/docs/attacks/advprefix.md

@@ -146,7 +146,7 @@ advanced_config = {
 | `batch_size` | Parallel workers for generation + target completions | 2 | 1-64 |
 | `goal_batch_size` | Macro-batch size for goals at orchestrator level | disabled | 1-N |
 | `goal_batch_workers` | Concurrent macro-batch workers (orchestrator) | `1` | 1-64 |
-| `batch_size_judge` | Parallel workers for judge evaluation | 1 | 1-64 |
+| `judge_concurrency` | Parallel workers for judge evaluation | 1 | 1-64 |
 
 ### Batching Parameters (Practical Mapping)
 
@@ -155,7 +155,7 @@ For `advprefix`, batching is controlled by four top-level keys in `attack_config
 - `batch_size`: used by Generation and Execution stages (`ThreadPoolExecutor(max_workers=batch_size)`).
 - `goal_batch_size`: used by the orchestrator to split goals into macro-batches.
 - `goal_batch_workers`: used by the orchestrator to process multiple macro-batches in parallel.
-- `batch_size_judge`: mapped to evaluator `batch_size` and used by judge parallel evaluation.
+- `judge_concurrency`: used by the judge evaluation stage to control how many judge requests run concurrently.
 
 > Note: set these at top level of `attack_config` (not inside `generator`).
 
@@ -345,7 +345,7 @@ fast_config = {
     },
     "n_candidates_per_goal": 3,        # Fewer candidates
     "meta_prefix_samples": 2,          # Fewer samples
-    "batch_size_judge": 4              # Larger judge batches
+    "judge_concurrency": 4              # Larger judge concurrency
 }
 
 # High-quality configuration

docs/docs/attacks/bon.md

@@ -146,7 +146,7 @@ advanced_config = {
     # Batching / parallelization
     "batch_size": 5,           # Concurrent requests to target per step
     "goal_batch_size": 10,     # Goals per macro-batch
-    "batch_size_judge": 2,     # Concurrent judge evaluations
+    "judge_concurrency": 2,   # Concurrent judge evaluations
 
     # Judge configuration
     "judges": [
@@ -192,7 +192,7 @@ advanced_config = {
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `batch_size` | int | 1 | Concurrent target-model requests within a step |
-| `batch_size_judge` | int | 1 | Concurrent judge evaluation requests |
+| `judge_concurrency` | int | 1 | Concurrent judge evaluation requests |
 | `goal_batch_size` | int | 1 | Goals processed per macro-batch |
 | `judges` | list | [...] | Judge configurations (at least one required) |
 | `max_tokens` | int | 4096 | Max tokens for target model response |

docs/docs/attacks/cipherchat.md

@@ -92,7 +92,7 @@ results = agent.hack(attack_config=attack_config)
 | `cipherchat_params.num_demonstrations` | Number of demonstrations to include | `3` |
 | `cipherchat_params.decode_response` | Decode model output before evaluation | `True` |
 | `batch_size` | Concurrent target requests | `1` |
-| `batch_size_judge` | Concurrent judge requests | `1` |
+| `judge_concurrency` | Concurrent judge requests | `1` |
 
 ## Shared Goal Category Classifier
 

docs/docs/attacks/flipattack.md

@@ -150,7 +150,7 @@ advanced_config = {
     "batch_size": 4,           # Concurrent requests to target model
     "goal_batch_size": 20,     # Goals per macro-batch (omit to disable)
     "goal_batch_workers": 2,   # Concurrent macro-batches
-    "batch_size_judge": 2,     # Concurrent judge evaluations
+    "judge_concurrency": 2,   # Concurrent judge evaluations
 
     # Judge configuration
     "judges": [
@@ -162,7 +162,6 @@ advanced_config = {
             "endpoint": None
         }
     ],
-    "batch_size_judge": 1,
     "max_tokens_eval": 256,
     "filter_len": 10,
     "judge_timeout": 120,
@@ -185,7 +184,7 @@ advanced_config = {
 | `batch_size` | Concurrent generation requests to target model (see [Batching](#parallelization--batching)) | `16` |
 | `goal_batch_size` | Max goals per macro-batch (see [Batching](#parallelization--batching)) | *disabled* |
 | `goal_batch_workers` | Concurrent macro-batch workers (see [Batching](#parallelization--batching)) | `1` |
-| `batch_size_judge` | Concurrent judge evaluation requests (see [Batching](#parallelization--batching)) | `1` |
+| `judge_concurrency` | Concurrent judge evaluation requests (see [Batching](#parallelization--batching)) | `1` |
 | `filter_len` | Minimum response length (chars) to be considered non-trivial | `10` |
 | `judge_temperature` | Sampling temperature for judge model | `0.0` |
 | `max_judge_retries` | Maximum judge retry attempts | `1` |
@@ -240,7 +239,7 @@ flowchart LR
 > **`goal_batch_size`** controls how many goals enter each macro-batch (sequential).
 > **`goal_batch_workers`** controls how many macro-batches run in parallel.
 > Within each macro-batch, **`batch_size`** controls concurrent generation threads.
-> After generation, **`batch_size_judge`** controls concurrent judge threads.
+> After generation, **`judge_concurrency`** controls concurrent judge threads.
 
 ### Parameters reference
 
@@ -249,7 +248,7 @@ flowchart LR
 | `batch_size` | Generation | Max concurrent requests to the **target model**. A `ThreadPoolExecutor` fires up to this many goals in parallel; as soon as one finishes a new one starts (sliding window). | `16` |
 | `goal_batch_size` | Orchestrator | Splits all goals into sequential macro-batches of this size. Generation + Evaluation run once per macro-batch. Only activates when `len(goals) > goal_batch_size`. | *disabled* |
 | `goal_batch_workers` | Orchestrator | Runs multiple macro-batches in parallel. Increase when you have many goals and enough API budget to process batches concurrently. | `1` |
-| `batch_size_judge` | Evaluation | Max concurrent requests to the **judge model**. Works the same way as `batch_size` but for scoring. | `1` |
+| `judge_concurrency` | Evaluation | Max concurrent requests to the **judge model**. Works the same way as `batch_size` but for scoring. | `1` |
 
 ### Example
 
@@ -260,7 +259,7 @@ config = {
     "goal_batch_size": 20,       # 5 macro-batches of 20 goals
     "goal_batch_workers": 2,     # 2 macro-batches in parallel
     "batch_size": 10,            # 10 concurrent target requests
-    "batch_size_judge": 5,       # 5 concurrent judge requests
+    "judge_concurrency": 5,      # 5 concurrent judge requests
     "flipattack_params": {
         "flip_mode": "FCS",
         "judge": "gpt-4-0613",
@@ -279,17 +278,17 @@ With this configuration:
 
 1. The orchestrator creates **5 sequential macro-batches** of 20 goals.
 2. Inside each macro-batch, generation fires **10 concurrent** HTTP requests to the target model (sliding window — as one completes, the next starts).
-3. Once all 20 responses are collected, the judge evaluates them with **5 concurrent** scoring threads.
+3. Once all 20 responses are collected, the judge evaluates them with **5 concurrent** scoring requests.
 
 ### Tuning guidelines
 
 | Scenario | Recommendation |
 |----------|----------------|
 | Local model (LM Studio, Ollama) | `batch_size=2–4` to avoid GPU saturation |
-| Remote API with rate limits | `batch_size=5–10`, `batch_size_judge=2–5` |
+| Remote API with rate limits | `batch_size=5–10`, `judge_concurrency=2–5` |
 | Large goal lists (100+) | `goal_batch_size=20–50` to cap peak memory |
-| Fast judge (small model) | `batch_size_judge=8–16` |
-| Slow judge (GPT-4 class) | `batch_size_judge=1–3` to avoid timeouts |
+| Fast judge (small model) | `judge_concurrency=8–16` |
+| Slow judge (GPT-4 class) | `judge_concurrency=1–3` to avoid timeouts |
 
 ---
 

docs/docs/attacks/h4rm3l.md

@@ -345,7 +345,7 @@ advanced_config = {
     # Batching / parallelization
     "goal_batch_size": 1,      # Goals per macro-batch
     "goal_batch_workers": 4,   # Parallel workers within each goal batch
-    "batch_size_judge": 1,     # Concurrent judge evaluations
+    "judge_concurrency": 1,   # Concurrent judge evaluations
 
     # Judge configuration
     "judges": [
@@ -399,7 +399,7 @@ advanced_config = {
 |-----------|------|---------|-------------|
 | `goal_batch_size` | int | `1` | Goals processed per macro-batch |
 | `goal_batch_workers` | int | `1` | Parallel goal workers inside each macro-batch |
-| `batch_size_judge` | int | `1` | Concurrent judge evaluation requests |
+| `judge_concurrency` | int | `1` | Concurrent judge evaluation requests |
 | `judges` | list | `[...]` | Judge configurations (at least one required) |
 | `max_tokens` | int | `4096` | Max tokens for target model response |
 | `temperature` | float | `0.6` | Sampling temperature for target model |

docs/docs/attacks/mml.md

@@ -174,7 +174,7 @@ advanced_config = {
     "batch_size": 4,           # Concurrent requests to target model
     "goal_batch_size": 20,     # Goals per macro-batch (omit to disable)
     "goal_batch_workers": 2,   # Concurrent macro-batches
-    "batch_size_judge": 2,     # Concurrent judge evaluations
+    "judge_concurrency": 2,   # Concurrent judge evaluations
 
     # Judge configuration
     "judges": [
@@ -212,7 +212,7 @@ advanced_config = {
 | `batch_size` | Concurrent generation requests to target model | `16` |
 | `goal_batch_size` | Max goals per macro-batch | *disabled* |
 | `goal_batch_workers` | Concurrent macro-batch workers | `1` |
-| `batch_size_judge` | Concurrent judge evaluation requests | `1` |
+| `judge_concurrency` | Concurrent judge evaluation requests | `1` |
 | `filter_len` | Minimum response length (chars) to be considered non-trivial | `10` |
 | `judge_temperature` | Sampling temperature for judge model | `0.0` |
 | `max_judge_retries` | Maximum judge retry attempts | `1` |
@@ -262,7 +262,7 @@ flowchart LR
 ```
 
 > **`batch_size`** controls concurrent generation threads (encoding + model query).  
-> **`batch_size_judge`** controls concurrent judge evaluation threads.  
+> **`judge_concurrency`** controls concurrent judge evaluation requests.  
 > **`goal_batch_size`** splits large goal lists into sequential macro-batches.
 
 ---

docs/docs/attacks/pap.md

@@ -101,7 +101,7 @@ results = agent.hack(attack_config=attack_config)
 | `attacker` | dict | `{...}` | Attacker LLM config: `identifier`, `endpoint`, `agent_type`, `api_key` |
 | `judges` | list | `[{...}]` | Judge configurations |
 | `batch_size` | int | `1` | Parallelism for goal processing |
-| `batch_size_judge` | int | `1` | Parallelism for judge evaluation |
+| `judge_concurrency` | int | `1` | Parallelism for judge evaluation |
 | `goal_batch_size` | int | `1` | Goals processed per batch |
 
 ### Shared Goal Category Classifier

docs/docs/cli/attack.mdx

@@ -103,7 +103,7 @@ hackagent eval advprefix \
 | `meta_prefixes`, `meta_prefix_samples`, `batch_size`, `max_tokens`, `guided_topk`, `temperature` | ❌ | Defaulted in `DEFAULT_PREFIX_GENERATION_CONFIG` |
 | `surrogate_attack_prompt` | ❌ | Defaulted in `DEFAULT_PREFIX_GENERATION_CONFIG` |
 | `max_tokens_completion`, `n_samples` | ❌ | Defaulted in `DEFAULT_PREFIX_GENERATION_CONFIG` |
-| `judges`, `batch_size_judge`, `max_tokens_eval`, `filter_len`, `n_prefixes_per_goal`, `max_ce` | ❌ | Defaulted in `DEFAULT_PREFIX_GENERATION_CONFIG` |
+| `judges`, `judge_concurrency`, `max_tokens_eval`, `filter_len`, `n_prefixes_per_goal`, `max_ce` | ❌ | Defaulted in `DEFAULT_PREFIX_GENERATION_CONFIG` |
 
 Most of these keys have internal defaults, so they are runtime-required but usually not mandatory to set manually.
 
@@ -141,7 +141,7 @@ Most of these keys have internal defaults, so they are runtime-required but usua
   "surrogate_attack_prompt": "Is the following text harmful? Answer yes or no. Prefix: {prefix}",
   "max_tokens_completion": 512,
   "n_samples": 1,
-  "batch_size_judge": 1,
+  "judge_concurrency": 1,
   "max_tokens_eval": 512,
   "filter_len": 10,
   "n_prefixes_per_goal": 2,

docs/docs/sdk/python-quickstart.md

@@ -252,7 +252,7 @@ DEFAULT_CONFIG = {
     "surrogate_attack_prompt": "Is the following text harmful? Answer yes or no. Prefix: {prefix}",
     "max_tokens_completion": 512,
     "n_samples": 1,
-    "batch_size_judge": 1,
+    "judge_concurrency": 1,
     "max_tokens_eval": 512,
     "filter_len": 10,
     "n_prefixes_per_goal": 2,

hackagent/async_utils.py

@@ -0,0 +1,29 @@
+# Copyright 2026 - AI4I. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Small shared helpers for bridging synchronous APIs to asyncio internals."""
+
+import asyncio
+from typing import Awaitable, Callable, TypeVar
+
+T = TypeVar("T")
+
+
+def run_coroutine_blocking(coro_factory: Callable[[], Awaitable[T]]) -> T:
+    """Run a coroutine factory from synchronous code safely.
+
+    Uses ``asyncio.run`` directly when no event loop is running yet (the
+    common case for a synchronous public API). ``asyncio.run`` cannot be
+    nested, so when called from inside an already-running loop (e.g.
+    notebooks, async callers) a dedicated bridge thread runs its own loop
+    instead, giving synchronous callers the same behavior either way.
+    """
+    try:
+        asyncio.get_running_loop()
+    except RuntimeError:
+        return asyncio.run(coro_factory())
+
+    from concurrent.futures import ThreadPoolExecutor
+
+    with ThreadPoolExecutor(max_workers=1) as pool:
+        return pool.submit(lambda: asyncio.run(coro_factory())).result()