Skip to content

Route messages to all mentioned agents so tasks run in parallel - #575

Open
QuanCheng-QC wants to merge 3 commits into
developfrom
bugfix/parallel-mention-routing
Open

Route messages to all mentioned agents so tasks run in parallel#575
QuanCheng-QC wants to merge 3 commits into
developfrom
bugfix/parallel-mention-routing

Conversation

@QuanCheng-QC

@QuanCheng-QC QuanCheng-QC commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

The problem, and where it came from

Reported during manual multi-agent testing: assigning a different task to each
agent in the same channel did not run them in parallel.
The UI showed a
"message queued" badge and results came back one after another. The channel
was in dynamic orchestration mode with tasks assigned by explicit human
@mention.

Two independent routing defects produce that symptom. They have different
repro steps, so they are described separately below.

Defect 1 — explicit assignments collapsed to a single target

Repro: in a dynamic channel with two agents, send one message naming
both:

@agent-a do X, @agent-b do Y

Before: only one agent was targeted. Three separate causes, all
single-target:

  • _fallback_targets returned mentions[0] and discarded the rest;
  • in dynamic mode a human message with explicit @mentions still went
    through the LLM router, so the router could override a target the user had
    already chosen;
  • the router truncated its own answer with .split(",")[0], so even a correct
    multi-name decision collapsed to one name.

Both tasks then landed on one agent, whose per-channel adapter queue
(BaseAdapter._channelBusy) serialized them. That queue is what surfaces in
the UI as the "message queued" badge.

Defect 2 — peer progress messages woke agents that were already working

Repro: in a dynamic channel with three agents, send three messages
back-to-back, one ~60s task each:

@agent-a  task PARALLEL-A … <60s command> … report START/END
@agent-b  task PARALLEL-B … <60s command> … report START/END
@agent-c  task PARALLEL-C … <60s command> … report START/END

Each command was equivalent to:

python -c "import time,datetime; print('START', datetime.datetime.now().isoformat()); time.sleep(60); print('END', datetime.datetime.now().isoformat())"

Before: each agent posts a progress message when it starts ("Starting the
command now…", "Executing command…"). Those messages @mention nobody, so they
fell through to the LLM router, which cannot distinguish progress narration
from a handoff and picked a bystander agent as the next target.

Observed with claude-test-001 / codex-test-001 / kimi-test-001
(A→claude, B→codex, C→kimi): claude-test-001 answered PARALLEL-A three
times
— once legitimately, then twice more after being handed codex's and
kimi's progress messages. Both spurious turns queued behind its own running
job, so they surfaced ~60s late as duplicate answers.

In a second run one of those spurious replies opened with "you seem to be
describing the PARALLEL-B run"
— the agent was visibly reacting to a message
addressed to someone else.

This is systematic, not flaky: N agents working in parallel produce up to N−1
spurious turns per round, so the symptom gets worse the more parallelism works.

After

Re-tested in a dynamic channel with three claude-adapter agents (all three
can actually execute shell commands, unlike the original mixed set):
claude-test-001, claude-dp-002, claude-dp-003. Three separate messages,
one task each.

Agent Task START END Duration
claude-test-001 PARALLEL-A 17:17:23.247326 17:18:23.248152 60.00s
claude-dp-002 PARALLEL-B 17:17:48.765432 17:18:48.766026 60.00s
claude-dp-003 PARALLEL-C 17:18:04.496977 17:19:04.497673 60.00s

(2026-08-04, times UTC as reported by each agent.)

The decisive number: PARALLEL-B starts at 17:17:48, while PARALLEL-A is
still running until 17:18:23.
Under the old behaviour B could not have
started until A's turn had finished. Same for C, which starts at 17:18:04.

All three intervals overlap between 17:18:04.496977 and 17:18:23.248152
18.75s with three agents executing concurrently. Pairwise overlap is larger
(A∩B = 34.5s, B∩C = 44.3s).

The staggered starts are the human send interval plus per-agent startup, not
queueing.

Also confirmed in the same run:

  • each agent returned exactly one result, for its own task only;
  • no agent answered another agent's assignment;
  • no duplicate results;
  • no queue badge.

The fix

Routing — workspace_mod.py

  • _fallback_targets returns all explicit @mentions (deduplicated,
    sender's self-mention filtered) instead of mentions[0].
  • A human message with explicit @mentions skips the LLM router — the user has
    already chosen the targets.
  • The router accepts multi-name answers (next:agent-a,agent-b); each name is
    validated against the candidate set, and unknown names and self-loops are
    dropped. _ROUTER_PROMPT documents the format and max_tokens goes 30 → 64
    so a multi-name answer is not truncated.
  • Master-mode delegation mentions are deduplicated at the source.

Direct-assignment hold — workspace_mod.py

New _human_assignment_holds(). When all of these hold:

  • the message came from an agent,
  • that agent was directly addressed by a human,
  • the message @mentions no peer,
  • the channel is not in workflow mode,

the message targets nobody and skips the router entirely. An assigned agent can
post progress notes and status without creating work for a peer.

The assignment window ends when a human posts a message with no @mention
at all. A newer assignment naming a different agent does not end it —
parallel assignments arrive as separate messages (@a …, @b …, @c …), so
ending the window on any later assignment would defeat the rule for exactly the
case it exists to handle.

Explicit agent-to-agent handoffs still route normally: a message that mentions
a peer is not progress narration and goes through the router as before.

_ROUTER_PROMPT additionally treats progress narration as stop — a second
line of defence for the paths the deterministic rule does not cover
(workflow mode, free-form chat).

Cloud agents — cloud_agent.py

  • Targeted cloud agents are invoked concurrently instead of in a sequential
    await loop.
  • Fan-out bounded by a semaphore; new CLOUD_AGENT_MAX_CONCURRENCY config,
    default 4.
  • Targets deduplicated before the gather, so a duplicated mention cannot invoke
    — or bill — the same agent twice. Sentinel targets are filtered before
    invocation; depth-limit behaviour is unchanged.
  • DB work is split into short-lived sessions released before every provider API
    call, so a large mention list can no longer pin pool connections for the
    duration of multiple external round-trips. Image/audio file records and their
    response message still commit in one transaction.

Blast radius

Area Effect
Single-agent channels Unchanged — the len(real_participants) < 2 path is untouched.
master mode Unchanged except delegation-mention dedupe; star topology intact.
dynamic mode Where the fix lives: explicit-human-mention bypass + direct-assignment hold.
workflow mode Behaviour change — an explicit human @mention now bypasses the plan-steered router. The agent-side hold is deliberately scoped to mode != "workflow", so the plan still drives agent→agent hops.
Router cost max_tokens 30 → 64 per call, but two paths now skip the call entirely, so net router calls go down.
Legacy clients ["__no_response__"] sentinel semantics unchanged.
Local agents One agent still processes one task at a time per channel — parallelism is across agents, not within one.
Cloud agents Concurrent invocation + shorter DB session lifetime. Highest-risk part of the diff.

Worth a reviewer's attention

  1. workflow-mode ordering. Explicit human mentions now take precedence
    over the plan-steered router. Deliberate — the human picked the target — but
    it is a routing-policy decision, not a mechanical fix.

  2. _ASSIGNMENT_LOOKBACK = 20. _human_assignment_holds() scans at most 20
    recent human messages. How far back a direct assignment stays relevant is a
    judgement call.

  3. The termination rule. An unaddressed human message reopens routing; a
    later assignment for another agent does not. Required for back-to-back
    parallel assignments, but still a policy choice.

  4. The cloud_agent.py session split. Correctness depends on provider-call
    code never touching unloaded attributes on a detached ORM object. This is the
    part that most deserves a close read — it combines concurrent provider calls,
    transaction boundaries, file records and response-message creation.

  5. CLOUD_AGENT_MAX_CONCURRENCY = 4. An operational default; may need
    tuning per deployment.

Intentionally unchanged

One agent still processes one task at a time within a channel. The per-channel
adapter queue stays, because a single agent's session context must stay
ordered. This PR makes different agents run concurrently; it does not make one
agent run two tasks at once. The fix stops unrelated assignments from being
misrouted into the same queue — it does not remove the queue.

Tests

Router suite: 25 → 39 tests (+14) covering explicit-mention fan-out,
dedupe, self-mention filtering, human-mention router bypass, master-mode
preservation, multi-name router answers, unknown-target and self-loop
filtering, and the six direct-assignment cases (progress note stops; assignment
survives a later task for another agent; explicit handoff still routes;
unaddressed human message reopens routing; never-assigned agent unaffected;
master mode unaffected).

New test_cloud_agent.py (5 tests): duplicate targets invoked once;
sentinel-only produces no invocation; sentinel mixed with real targets is
dropped; concurrency cap enforced; depth limit skips all.

Full backend suite, rebased onto origin/develop c3a18260:

Revision Result
Base c3a18260 64 failed, 529 passed, 16 skipped
Branch head c027149c 64 failed, 548 passed, 16 skipped

548 − 529 = 19 — exactly the tests added here. The 64 failures are
byte-identical between the two revisions: pre-existing environmental
failures (filesystem-permission errors) in test_skill_install,
test_knowledge, test_browser_contexts, test_events,
test_login_session_auth and test_migration_schema.

CI's other gate, test_onm_addressing / test_onm_events /
test_onm_pipeline: 65 passed.

Known CI noise — pre-existing, not from this PR

CI lints only changed Python files, and the files this PR touches were never
ruff-clean, so the lint job will go red. On origin/develop those same files
already produce 17 ruff check errors; on this branch they produce 14
a strict subset (rewriting cloud_agent.py removed three F841s). The new
test_cloud_agent.py is clean. ruff format --check already fails on those
files on develop too.

No new violation is introduced here. Auto-fixing would add ~660 lines of pure
formatting churn on top of a 623-line functional diff, so it is deliberately
left for a separate formatting PR.

@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openagents-workspace Ready Ready Preview Aug 4, 2026 9:34am

Request Review

Assigning tasks to different agents in one channel used to funnel every
message through single-target routing — the fallback kept only the first
@mention, the LLM router truncated multi-name answers to one agent, and
in dynamic mode explicit mentions never bypassed the router at all. All
tasks landed on one agent whose per-channel queue then serialized them,
showing the 'message queued' status in the UI instead of parallel work.

- honor ALL explicit @mentions (deduped, self-mentions filtered) in
  _fallback_targets instead of just the first one
- human messages with explicit mentions now skip the LLM router and
  target the mentioned agents directly in dynamic/workflow modes
  (master mode keeps its star topology)
- the LLM router now accepts multi-name answers ('next; a, b') and
  targets every valid name, dropping only unknowns and self-loops
- cloud agents targeted by one message are invoked concurrently with
  per-task DB sessions instead of a sequential await loop
- bound cloud agent fan-out with a semaphore (new
  CLOUD_AGENT_MAX_CONCURRENCY config, default 4) and split DB access
  into short-lived sessions released before every provider API call,
  so a large mention list can no longer pin pool connections for the
  duration of external model round-trips
- dedupe cloud agent targets before the gather and dedupe master-mode
  delegation mentions at the source, so duplicate mentions can't invoke
  the same agent twice with duplicate billable requests
- teach the router prompt the multi-target output format ('next' with
  comma-separated names) for independent tasks, matching the parser;
  raise the router max_tokens from 30 to 64 so multi-name answers
  don't get truncated
- new tests for fan-out dedupe, sentinel filtering, concurrency cap,
  depth limit, and master-mode delegation dedupe
While several agents worked in parallel on tasks a human handed them by
name, each worker's own output ("running the command now…") had no
@mention, so it fell through to the LLM router — which cannot tell a
progress note from a handoff and picked a bystander agent. That agent
then answered a task it was never given, and only after its own 60s job
drained off the per-channel queue, so it read as duplicate, late replies.
With N agents working at once this produced N-1 spurious turns every run.

- new _human_assignment_holds(): walks the channel's recent human
  messages newest → oldest — one that @mentions the sender means the
  assignment stands, one with no @mention at all means free-form chat
  reopened routing, one that @mentions only others keeps looking back
  (parallel assignments arrive as separate messages, one per agent, so
  a newer task for someone else must not cancel this agent's own)
- dynamic mode: an agent message with no peer @mention whose assignment
  still holds targets nobody and skips the router entirely — the mirror
  image of the human-mention bypass. Explicit handoffs (@peer) still go
  through the router, and master mode's star topology is untouched
- router prompt: progress narration about work in flight is "stop",
  covering workflow mode and free-form chat where the rule above
  does not apply
- 6 tests for the new class
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant