Summary
Data Machine hand-rolls its own fan-out (ExecuteStepAbility, mode => fanout) in its child-job batch scheduler. agents-api now provides a proven, concurrent, reliable parallel-branch fan-out primitive (concurrent Action Scheduler branch execution: concurrent runner trigger + concurrent_batches/batch_size policy + loopback dispatch, on MySQL FOR UPDATE SKIP LOCKED). DM should converge onto that substrate instead of maintaining a second ad-hoc fan-out engine, so there is ONE concurrent-fan-out engine across consumers (Data Machine, Site Forge, wp-codebox, Studio Native) rather than several.
The two fan-out shapes are not the same primitive today
- agents-api fan-out — a
parallel MAP step runs N branch STEPS concurrently within ONE workflow run, then reconciles/collects them back into that run. Unit of parallelism: an Action Scheduler branch action inside a single run.
- Data Machine native fan-out (
inc/Abilities/Engine/ExecuteStepAbility.php, mode => fanout) — a step that produces N DataPackets spawns one CHILD JOB per packet; each child job continues the PIPELINE forward (fetch → AI → publish …) on its own packet. Unit of parallelism: a DM child job in DM's own job/queue system.
DM's agents-api bridge (inc/Abilities/Job/ExecuteAgentWorkflowAbility.php, docs agents-api-workflow-bridge.md) currently and deliberately does NOT bridge parallel/branching — the doc lists foreach, cron/action triggers, branching, parallel execution, nested workflows, and consumer-defined step types as unsupported. That exclusion was written when agents-api parallel fan-out wasn't solid, which is now fixed.
Proposed move (two stages)
- Converge the concurrency ENGINE first. DM's child-job scheduler should not hand-roll Action Scheduler concurrency (batch/claim behavior) now that agents-api owns it as a substrate primitive. This is the lower-risk, high-value part.
- Then, as a modeling exercise, express DM's per-packet child-job fan-out as an agents-api
parallel MAP branch that runs the downstream pipeline for its packet. The bridge already hosts DM step execution (ExecuteStepAbility), so lifting the bridge's parallel exclusion is now viable. This is the larger piece — the job-vs-branch modeling is where the real work is; the concurrency primitive is the free part.
Why now
agents-api parallel fan-out is proven concurrent and reliable on a live MySQL native-PHP runtime (Site Forge multi-page generation completes end to end; the concurrency policy resolves correctly). The reason DM excluded parallel from its bridge no longer holds.
Not a swap
This is a real project, not a drop-in. DM's fan-out has pipeline-continuation semantics (child jobs carry the pipeline forward per packet) that must be modeled onto branches. Scope the engine convergence and the modeling separately.
References
- agents-api branch:
feat/concurrent-branch-fanout (Automattic/agents-api) — the concurrent branch execution + completion hook.
- DM code:
inc/Abilities/Engine/ExecuteStepAbility.php (mode => fanout, filterPacketsForFanOut), inc/Abilities/Job/ExecuteAgentWorkflowAbility.php, docs/agents-api-workflow-bridge.md.
Filed with AI assistance (Claude Opus 4.8 via Claude Code) based on a code-grounded review of both fan-out paths.
Summary
Data Machine hand-rolls its own fan-out (
ExecuteStepAbility,mode => fanout) in its child-job batch scheduler. agents-api now provides a proven, concurrent, reliable parallel-branch fan-out primitive (concurrent Action Scheduler branch execution: concurrent runner trigger +concurrent_batches/batch_sizepolicy + loopback dispatch, on MySQLFOR UPDATE SKIP LOCKED). DM should converge onto that substrate instead of maintaining a second ad-hoc fan-out engine, so there is ONE concurrent-fan-out engine across consumers (Data Machine, Site Forge, wp-codebox, Studio Native) rather than several.The two fan-out shapes are not the same primitive today
parallelMAP step runs N branch STEPS concurrently within ONE workflow run, then reconciles/collects them back into that run. Unit of parallelism: an Action Scheduler branch action inside a single run.inc/Abilities/Engine/ExecuteStepAbility.php,mode => fanout) — a step that produces N DataPackets spawns one CHILD JOB per packet; each child job continues the PIPELINE forward (fetch → AI → publish …) on its own packet. Unit of parallelism: a DM child job in DM's own job/queue system.DM's agents-api bridge (
inc/Abilities/Job/ExecuteAgentWorkflowAbility.php, docsagents-api-workflow-bridge.md) currently and deliberately does NOT bridge parallel/branching — the doc listsforeach, cron/action triggers, branching, parallel execution, nested workflows, and consumer-defined step types as unsupported. That exclusion was written when agents-api parallel fan-out wasn't solid, which is now fixed.Proposed move (two stages)
parallelMAP branch that runs the downstream pipeline for its packet. The bridge already hosts DM step execution (ExecuteStepAbility), so lifting the bridge's parallel exclusion is now viable. This is the larger piece — the job-vs-branch modeling is where the real work is; the concurrency primitive is the free part.Why now
agents-api parallel fan-out is proven concurrent and reliable on a live MySQL native-PHP runtime (Site Forge multi-page generation completes end to end; the concurrency policy resolves correctly). The reason DM excluded parallel from its bridge no longer holds.
Not a swap
This is a real project, not a drop-in. DM's fan-out has pipeline-continuation semantics (child jobs carry the pipeline forward per packet) that must be modeled onto branches. Scope the engine convergence and the modeling separately.
References
feat/concurrent-branch-fanout(Automattic/agents-api) — the concurrent branch execution + completion hook.inc/Abilities/Engine/ExecuteStepAbility.php(mode => fanout,filterPacketsForFanOut),inc/Abilities/Job/ExecuteAgentWorkflowAbility.php,docs/agents-api-workflow-bridge.md.Filed with AI assistance (Claude Opus 4.8 via Claude Code) based on a code-grounded review of both fan-out paths.