Declarative workflow authoring: the Abacus DSL - #6
Merged
Conversation
…thoring Adds a second authoring path: a workflow as a JSON document, validated against a published schema and interpreted onto the existing runtime. The governing rule is that the DSL composes but never computes. A document declares which nodes exist, how they connect, and when an edge is taken; it carries no behaviour. Every unit of work is a capability the host already shipped, reached either as a built-in kind or as a custom node registered by name. Three decisions follow from putting JSON in front of a generic, delegate- shaped API: - One envelope type. Every node is HostExecutor<DslMessage, DslMessage>, so the graph is uniformly typed, checkpoints serialize for free, and the null-return park path keeps working. The envelope carries the start context alongside the current value, restoring the ambient scope a document otherwise lacks. - A closed expression language: total, pure, statically checkable, with a fixed function set. Non-deterministic functions are refused in edge conditions and gate predicates, because a resumed run must retrace the routing its checkpoint recorded. - A named node catalog with a registration seam, and no delegate kind. Validation is two phases: JSON Schema for shape, then a semantic validator for what a schema cannot express - id uniqueness, reachability, cycles without a delay, expression parsing, catalog resolution. Every diagnostic carries a JSON Pointer. The schema is published rather than described. It checks as legal Draft 2020-12, accepts the worked example, and rejects 17 malformed variants covering each conditional branch it declares. The plan keeps the core change to one additive opt-in interface, IContextValidatingWorkflow, consulted by the registry after the type bind. Runtime publication, sub-workflows and iteration are named as deferred, with reasons.
Slots the DSL artifacts in alongside the existing docs/implementation sequence as 06 and 07, and repoints the relative links the move broke. The schema stays at docs/schema/. It is a published product artifact - embedded as a resource, served from GET /v2/dsl/schema, and consumed by editors - not an implementation note, so it does not belong in the series.
Adds Abacus.Run.Dsl with the two pieces the interpreter is built on, both
testable without a host.
DslMessage is the single envelope every DSL node sends and receives. It
carries the frozen start context alongside the current value, which is what
lets an expression eleven nodes deep still read $ctx - a compiled node
closes over C# scope, and a document has none. Immutable, so a message a
checkpoint captured cannot be mutated by a node that runs later.
AbEx is the expression language: a hand-written lexer and recursive-descent
parser producing an immutable AST, a total evaluator, and static analysis.
Total is the load-bearing property. Absence is a value, not an exception, so
no expression over any document shape can throw - a workflow must take the
other branch, not fail. Comparisons involving an absent operand are false
including !=, because a document asking whether a field it never set differs
from a value must not be told yes; has() is how presence is asked about.
Conditions are strictly boolean, with no truthiness ladder. Arithmetic is
decimal, because these documents price orders.
The function set is closed. An unknown name is a validation error with a
nearest-match suggestion, and the pattern argument to matches() must be a
string literal so every regex in a document is reviewable by reading the
document. Regex matching carries a 200ms timeout; a timed-out match is a
non-match rather than a way for an author to stall a dispatcher.
Two deviations from the grammar as written, both recorded in the plan:
unary ! and - bind tightest rather than sitting between && and comparison,
and bare-identifier path roots are dropped in favour of requiring $, $ctx or
$run. The first matches what an author expects; the second removes a real
ambiguity between a path and a function name.
One additive change to Abacus.Run: ITemplateBindingSource, which lets a
message resolve its own {{ }} placeholders. The default dotted-path walk
suits one POCO root and cannot address an envelope carrying two objects.
Opt-in - a type that does not implement it resolves exactly as before.
207 tests.
Text in, typed model and pointer-accurate diagnostics out. Validation runs in two phases because one cannot do the job. JSON Schema checks shape - required properties, kind-discriminated variants, id and SemVer patterns. It cannot compare two array items, follow a reference, walk a graph, or parse a sub-language, so the semantic validator handles id uniqueness, edge endpoints, reachability, cycles, expressions, gates, catalog resolution and limits. Twenty-two stable codes, each with a test asserting its code, pointer and severity. The phases stop where continuing would be noise: a document that fails the schema is not read into the model, because reporting forty type errors from a half-understood document buries the one that matters. Two rules are worth calling out. A cycle is refused only when nothing on it yields - polling and wait-and-recheck are legitimate, but a cycle of pure compute nodes is a hot spin that occupies a dispatcher until the lifetime cap. And a duplicate unconditional edge is refused while two conditional edges between the same pair are fine, because that is exactly how a branch with a fallback is written. Environment-dependent checks - custom node registration, parameter schemas, egress hosts, hash conflicts - are reported as skipped rather than passed when there is no host to check against. A check that silently did not run is worse than one that openly did not, because only the second can be acted on. Document identity is a canonical SHA-256 (RFC 8785 JCS). Reformatting and property reordering do not change it; one byte of behaviour does. That makes the immutability rule enforceable and answers the operational question directly: is this instance running the document I am looking at? The schema is embedded from docs/schema/ rather than copied, with a test asserting the embedded resource matches the published file - otherwise an editor validates against one document and the host enforces another. One bug found by its own tests: the validator crashed on duplicate node ids, which is one of the things it exists to report. Building the kind lookup with ToDictionary threw before the diagnostic could be produced. Now built tolerantly, with robustness tests over pathological documents. 317 tests (+110).
A DSL document now registers and runs as an ordinary workflow definition. Every kind maps onto an executor the host already ships. Nothing here reimplements HTTP, prompting, egress control, idempotency keys, durable waits or cost accounting: DslHostedExecutor calls ExecuteTerminalAsync on the real executor and projects the result back into the envelope, skipping the inner gate and pipeline because the outer node has already run both. A front end that forked the execution path would stop being one. Custom nodes go through the same wrapper, so a factory author writes ExecuteCoreAsync and gets the bound expression roots, the declared notification and the result projection for free. The graph gained an entry and an exit node, neither of them planned. The runner sends the deserialized context as the first message, typed JsonElement, and the engine routes by type - so without a node typed to receive it, the first DSL node never runs and the workflow completes having done nothing at all. The exit node is the mirror: YieldOutputAsync is checked against the executor's declared output type, so a DSL node cannot yield anything but an envelope, and the caller would otherwise be handed the start context back as though it were a result. Registration is deferred until the container is built, so AddDslWorkflow and AddDslNode can be written in either order - a document is always validated against the complete node catalog. An invalid document fails startup with every diagnostic, not the first: three broken documents should take one startup to fix. Two defects surfaced through their own tests. AbExValue.FromNode probed CLR types in turn, and a JsonValue created from an int will not hand back a decimal, so an HTTP status of 200 fell through to the string branch and never equalled 200. And the semantic validator crashed on duplicate node ids, which is one of the things it exists to report. Two pre-existing issues are recorded in the plan rather than fixed here: FanInExecutor<TItem,TOut> cannot work with AddFanInBarrierEdge, because the engine type-checks a barrier target against the individual message and not the list; and no ITimerService is registered anywhere, so DelayExecutor cannot run on a stock host. Suites: 723 unit (unchanged), 358 DSL unit, 201 integration (+51), 7 chaos.
Documents the DSL where the compiled path is documented, so the two are legible side by side - the honest reason to pick one over the other is what a reader most needs. The wiki chapter sits beside "Authoring a workflow" and covers the envelope, the expression language and its semantics, every node kind and what it puts on the envelope, the custom-node seam, registration, the two validation phases with their diagnostic codes, version immutability, the routes, the limits, and - stated plainly - what the DSL does not do. Ships example-order.workflow.json beside the compiled example, with tests asserting it validates and registers. Documentation people copy should fail here rather than in their editor. A parity fixture runs the same work authored both ways on one host and asserts identical results, including on values where binary floating point would diverge from the compiled path. That is the clearest available statement that the DSL is a front end and not a fork. One deviation, recorded in the plan: the parity pair is not ExampleOrderWorkflow. That workflow sums an array of order lines and the DSL has no iteration, which is exactly the limitation the design records - found by trying to hit it. Suites: 723 unit, 358 DSL unit, 209 integration (+59), 7 chaos.
…on failures
Three items from 07-workflow-dsl-implementation-plan.md were unimplemented, and
writing the tests for one of them found a defect.
Catalog provenance (plan 4.3). GET /v2/workflows/{name} now reports source and
documentHash. Abacus.Run cannot name the DSL, so a definition answers for itself
through IDocumentAuthoredWorkflow and the catalog reports "compiled" with a null
hash for anything that does not implement it. Same probe-by-is pattern the
runtime already uses for INotifyingWorkflow.
Architecture tests. Nothing enforced the DSL project's layering: it now asserts
the DSL references Abacus.Run and neither the host nor infrastructure, holds no
internals access, that the framework references neither the DSL nor
JsonSchema.Net, and that the schema ships as exactly one embedded resource.
Redaction and checkpoint size. Both were listed as tested mitigations and were
not tested. Under a restrictive policy a node reads a secret from ctx, no event
in the run carries it, and the instance's own state still does. A 128 KB context
through a four-hop document shows the checkpoint holds about one context rather
than one per hop.
Defect: a custom node of the wrong executor shape was refused correctly and then
retried, burning the whole attempt budget re-deriving an error that cannot
change. Interpretation failures now throw DslInterpretationException and classify
as a dead stop ahead of the document's own onFailure rules. Found because
WrongShapeNodeFactory had been registered in the fixture but never exercised.
Also covers egress enforcement at startup in both directions and compares the
/dsl/* routes' authorization metadata against /workflows, so locking down the
catalog forces the DSL routes to keep pace.
723 unit, 361 DSL, 232 integration, 7 chaos, all green.
The wiki carried the whole compiled-authoring surface plus a 16-recipe appendix, and the DSL had a full reference beside it. That left the two paths documented at different depths and in different places. Adds docs/workflow-authoring-guide.md: the complete C# reference, mirroring the DSL guide section for section. Every built-in executor with its real constructor and behaviour, the full gate, notification, trigger and audit surfaces, both middleware seams, registration, versioning, a section on choosing between the two front ends, and an options appendix with every default. Written from the source, which corrected a few things the wiki had wrong: the exception type is ApiCallFailureException, HumanApprovalExecutor is identity work and the gate is what pauses, FanInExecutor cannot be a barrier target, and there is no SubWorkflow API. A "sharp edges" section states those and the rest rather than leaving them to be discovered. The wiki's authoring chapter now opens with the fork: two ways in, what each reaches, and where to read next for either. Its appendix moved into the two guides so a recipe sits beside the field reference it uses; the heading stays as a signpost, so existing deep links still land. Also commits docs/dsl-authoring-guide.md, which the wiki has linked to since phase 5 but which .gitignore's docs/* rule kept out of the repository.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second front end onto the runtime that already exists: a workflow authored as a JSON document
instead of C#. Nothing about how a compiled workflow behaves changes.
A document registers as an ordinary
IWorkflowDefinition. It appears in the same catalog, startsthrough the same route, checkpoints through the same store, traverses the same middleware, and is
controlled by the same endpoints. Nothing downstream of registration knows the difference.
{ "dsl": "abacus.workflow/1.0", "name": "order-settlement", "version": "1.2.0", "start": "price", "output": ["settle"], "nodes": [ { "id": "price", "kind": "transform", "set": { "total": "$ctx.amount * 1.2" } }, { "id": "settle", "kind": "http", "url": "https://ledger.internal/v1/settle", "allowedHosts": ["ledger.internal"], "gate": { "mode": "conditional", "when": "$.total > 25000", "assignTo": ["group:finance"] } } ], "edges": [ { "from": "price", "to": "settle" } ] }The governing rule
The DSL composes, it never computes. A document declares which nodes exist, how they connect and
when an edge is taken; it never carries behaviour. Every unit of work is a capability the host
already shipped — a built-in kind, or a custom node registered by name. So there is no
delegatekind and never will be, a document's ceiling is the host's node catalog rather than the JSON syntax,
and a document is safe to accept from outside the build: it cannot execute, reach the filesystem,
open a socket the host has not allow-listed, or loop unboundedly.
What lands in
Abacus.RunDeliberately small — three opt-in interfaces, ~95 lines. A definition that implements none behaves
exactly as today.
IContextValidatingWorkflowITemplateBindingSourceTemplateBindingsresolves dotted paths by reflection over one root, which cannot address an envelope carrying both a context and a payload. Lets{{ $ctx.orderId }}work inside the existingApiCallExecutorandLlmExecutorrather than forking eitherIDocumentAuthoredWorkflowGET /v2/workflows/{name}reportssourceanddocumentHash. The framework cannot name the DSL, so the definition answers for itself and anything that says nothing reports"compiled"Everything else is a new project,
src/Abacus.Run.Dsl, referencing the public surface only. It pullsin a JSON Schema validator and an expression parser; a host that authors every workflow in C# should
carry neither.
What a document reaches
Nine node kinds (
transform,http,llm,delay,approval,publish,wait-event,fan-in,custom), every edge shape, the full approval-gate surface includinglocked, failure rules,notifications, domain-event triggers, and an audit record shape. AbEx — the expression language — is
closed, total and statically checked: absence is a value rather than an exception, conditions are
strictly boolean, arithmetic is decimal.
Not expressible, deliberately: arbitrary code, iteration, sub-workflows, raw/agent bindings, runtime
publication. Each is stated in the guide and in §19 rather than left to be discovered, and the answer
to "the DSL cannot express this" is always register a node.
Validation is two-phase — JSON Schema, then 22 semantic checks — and every diagnostic carries a JSON
Pointer. An invalid document fails startup with every diagnostic logged, and a published
(name, version)is immutable, enforced by an RFC 8785 canonical hash.Decisions worth reviewing
$entry,$exit), neither planned. The runner sendsthe context typed as
JsonElementand the engine routes by type, so without a node typed toreceive it nothing runs;
YieldOutputAsyncis checked against the declared output type, so withoutthe exit node the caller gets the start context back as though it were a result.
AddFanInBarrierEdgedoes not deliver a list — the edgerunner type-checks each released message individually — so the node holds arrivals and emits when
the expected count lands.
correlationKeyis a literal, not an expression. A subscription is registered beforeany message exists. The validator warns when one is written to look like an expression rather than
silently evaluating or dropping it.
ExampleOrderWorkflow. It sums an array of order lines, which the DSLcannot express — exactly the limitation the design records, found by trying to hit it. The parity
pair is a purpose-built workflow authored both ways, which also pins decimal arithmetic across the
two.
Defects
Fixed here:
AbExValue.FromNodemisread numbers (an HTTP status of 200 compared as"200"); thesemantic validator crashed on duplicate node ids, one of the things it exists to report; and a custom
node of the wrong executor shape was refused correctly and then retried, burning the whole attempt
budget re-deriving an error that cannot change.
Found and not fixed, being pre-existing in the compiled surface:
FanInExecutor<TItem, TOut>cannot work withAddFanInBarrierEdge— it declaresHostExecutor<List<TItem>, TOut>and the barrier filters by the individual message type, sodeliveries are dropped. Nothing in the repository exercised it, and the wiki's "the barrier
delivers a list" was wrong.
ITimerServiceis registered anywhere, so adelaynode — and a compiled workflow usingDelayExecutor— cannot run on a stock host.Documentation
Authoring is now forked into two references of equal depth, with the wiki as the orientation that
introduces both:
docs/workflow-authoring-guide.md(C#) anddocs/dsl-authoring-guide.md(JSON).The wiki's 16-recipe appendix moved into them so a recipe sits beside the field reference it uses.
Writing the C# guide from the source corrected several wiki claims — the exception type is
ApiCallFailureException,HumanApprovalExecutoris identity work and the gate is what pauses, andthere is no
SubWorkflowAPI.Note for the reviewer:
.gitignorehasdocs/*with only!docs/wiki.mdnegated, so the DSL guidehad been linked from the wiki since it was written but never committed. Both guides are force-added
here. Any future file under
docs/will be ignored the same way.Testing
dotnet build Abacus.Run.slnxthendotnet test, all green:Integration covers startup and egress enforcement, the catalog and provenance, every route and its
authorization, an end-to-end run of every node kind, gates parking and resuming, redaction of a
secret carried in
ctx, checkpoint size under a 128 KB context, and parity between the two frontends. Architecture tests pin the layering: the DSL depends on the public surface only, and the
framework references neither the DSL nor its schema validator.