Workflows

Agent Swarm includes a workflow automation engine that lets you define multi-step processes as directed acyclic graphs (DAGs). Workflows connect triggers, conditions, and actions into pipelines that execute automatically.

Bundled workflows declare the parameters and integrations they require. If one is enabled before setup is complete, it reports needs_setup with exactly what is missing in the dashboard and refuses to trigger until setup is complete. Operator-created workflows remain unaffected unless they opt into the same params, requiredParams, and requires metadata.

Core Concepts

A workflow consists of:

• Nodes — Individual steps with a next field that defines routing (no explicit edges needed)
• Executors — Class-based step implementations with Zod-typed config and output schemas
• Triggers — Entry points: webhook (HMAC-SHA256, timestamped HMAC, shared token), schedule (ref), or manual
• Checkpoint durability — Atomic DB write after every step; resume from last checkpoint on crash
• Fan-out / convergence — Parallel execution via array next, or item-driven fan-out via foreach (any executor as the body, with an optional concurrency cap), with automatic convergence gating

Executor Types

The engine includes 15 built-in executor types:

Each executor has typed config and output schemas available via GET /api/executor-types (returns JSON Schemas via Zod v4 z.toJSONSchema()).

Inline script environment

An inline script node can pass an optional config.env map to its Bash, Python, or TypeScript process. Values are strings and support workflow interpolation, just like args:

{
  "id": "gitlab-check",
  "type": "script",
  "config": {
    "runtime": "python",
    "env": {
      "GITLAB_TOKEN": "{{input.gitlabToken}}",
      "PROJECT_ID": "{{input.projectId}}"
    },
    "script": "import os; token = os.environ['GITLAB_TOKEN']; project = os.environ['PROJECT_ID']"
  }
}

Read variables with $NAME in Bash, os.environ["NAME"] in Python, or process.env.NAME in TypeScript. Names must match [A-Za-z_][A-Za-z0-9_]*, and values cannot contain NUL bytes. The names PATH, HOME, LANG, LC_ALL, TMPDIR, PWD, SHLVL, and OLDPWD are reserved and cannot be set through env (case-insensitive). Native loader controls (LD_*, DYLD_*, GCONV_PATH, GLIBC_TUNABLES, LOCPATH, NLSPATH, GETCONF_DIR) and shell controls / special variables (including BASH_*, BASH, BASHOPTS, BASHPID, ENV, SHELLOPTS, IFS, OPTIND, OPTERR, PS0–PS4, EPOCHSECONDS, and EPOCHREALTIME) are also rejected before spawning: they can execute code before resource limits or be rewritten by the shell. See the complete reserved-name list. Use application names such as GITLAB_TOKEN, PROJECT_ID, and MY_SETTING. The process receives only the built-in environment plus the explicitly supplied variables, never all of the server's environment. Omitting env keeps existing behavior; args remains supported.

Use secret input references rather than literal tokens. Secret workflow inputs stay masked as ***REDACTED*** in saved step inputs while the script receives the real values. Values passed through env are not inserted into the script or intermediate shell's command-line arguments. This does not isolate secrets from other processes with permission to read their environment, and environment size limits still apply. Avoid printing credentials or embedding them in executable source. The separate swarm-script node does not use this field.

Catalog scripts

swarm-script is the durable, reusable counterpart to inline script nodes. In workflow definitions it accepts scriptName, optional scope (agent or global), optional pinHash, JSON args, and timeoutMs. The timeout defaults to 30000 ms, accepts integers from 1000 through 300000 (5 minutes), and raises the script runtime's wall-clock budget for I/O-bound work while still respecting the separate 60-second CPU-time ceiling.

For config.args, values written as an exact single token such as {{input.payload}} preserve the resolved JSON type, so objects, arrays, numbers, and booleans reach the script as native values. If you wrap a token with surrounding text, such as "prefix-{{input.id}}", interpolation falls back to string output. Raw-token resolution is scoped to the node's local interpolation context, including any explicit inputs mapping on that node.

Choosing a decider

When a node has to decide, use the cheapest node that can decide. Agents authoring workflows apply this rule by default, so you do not need to ask for system-one-decision.

Jev is weak at counting. For a hybrid, count in a script and pass the number to system-one-decision as part of state.

SystemOne Decision

A system-one-decision node (display name "SystemOne Decision") sends one state and a map of questions to a typed-decisions model and returns one validated answer per question. Jev (TypeSafe) and Clef (Cloudflare Workers AI) are peer models for it; the node name is not model-specific. It reaches the model through a provider: typesafe or openrouter (Jev), cloudflare (Clef or Clef-flash on Workers AI), laya, or openai (the OpenAI Decisions API, gpt-6-luna). A node that names no provider uses the swarm's SYSTEM_ONE_DEFAULT_PROVIDER setting (Settings → Configuration, Workflows group), which is typesafe when unset; set it to cloudflare to run every such node on Clef with no workflow edit. Each provider needs its own global secret, TYPESAFE_API_KEY, OPENROUTER_API_KEY, LAYA_API_KEY, OPENAI_DECISIONS_API_KEY, or CLOUDFLARE_API_TOKEN (CLOUDFLARE_API_KEY is read when it is unset); laya also needs LAYA_URL, its server's base URL, and cloudflare needs CLOUDFLARE_ACCOUNT_ID, as global config values. The key is read on the server and never appears in a definition, a step output, or an error.

{
  "id": "qualify",
  "type": "system-one-decision",
  "inputs": { "lead": "trigger.lead" },
  "config": {
    "provider": "typesafe",
    "model": "jev-1.13.0",
    "state": "{{lead}}",
    "questions": {
      "fit": {
        "type": "noul",
        "instructions": "Is this a real engineering team seeking agent workflow automation?",
        "criteria": { "true": "A matching team with a concrete use case", "false": "Spam or no matching team" }
      },
      "authority": {
        "type": "choice",
        "instructions": "What purchasing authority does the message support?",
        "criteria": { "buyer": "Can approve the purchase", "champion": "Influences the decision", "unknown": "Not established" }
      }
    },
    "returns": { "fit": { "type": "noul" }, "authority": { "type": "choice" } }
  },
  "next": "gate"
}

• provider is typesafe, openrouter, laya, openai, or cloudflare, and must be a literal. Unset means SYSTEM_ONE_DEFAULT_PROVIDER, read at run time. model is that provider's own id (jev-1.13.0 on typesafe, typesafe/jev-1.13 on openrouter, gpt-6-luna on openai, clef or clef-flash on cloudflare); unset means the provider's default. laya has no default and sends no model, because it silently ignores one it does not know. Every provider returns the same output, so everything below holds on each, except that laya's confidence is its answer_confidence, the top probability. openai speaks its own request format, which the node translates; an OpenAI refusal of any question fails the step. cloudflare takes at most 64 questions per node. There is no fallback: a node does not switch to another provider when its own key is missing.
• Question types are noul (probability that a claim is true), choice (2 to 255 named options), and score (2 to 10 ordered levels). returns repeats each question id and type, and a mismatch is rejected when the workflow is saved.
• The output is { model, answers, usage }, plus routing: { model } on laya (the checkpoint that answered). A downstream node reads <alias>.answers.<question>.<field> through an inputs mapping. A noul answer is { type, noul } with no confidence. choice and score answers carry probabilities and confidence.
• A low-confidence answer is still a success. To have a person review the uncertain ones, set humanReview (below). To route on a threshold yourself, put it in the next node, such as property-match or code-match, and route the uncertain band to human-in-the-loop.
• The node retries connection errors, 408, 429, and 5xx itself, up to maxRetries (0 to 3, default 2) within timeoutMs (default 30000). It never retries a 401, a 422, or an answer that fails validation. Do not set retry or validation.retry on a system-one-decision node.
• An unresolved {{token}} fails the step before the request is sent.

Human review

Set humanReview to define a confidence band in the same node. An answer whose confidence is inside the band goes to a person instead of passing straight through.

"humanReview": {
  "band": { "min": 0.5, "max": 0.8 },
  "approvers": { "users": ["head-of-sales"], "policy": "any" },
  "timeout": { "seconds": 86400, "action": "reject" }
},
"next": { "approved": "route", "rejected": "discard", "timeout": "escalate" }

• band is min and max, each 0 to 1, and both ends count as inside. choice and score answers are tested on the confidence they report. A noul answer reports none, so it is tested on max(noul, 1 - noul). The number is the provider's own, so a band tuned on one model does not carry to another. On laya it is answer_confidence, the top probability.
• approvers, timeout, and notifications are the human-in-the-loop settings. The node raises the request through that executor: same approvers, waiting run state, Slack notification, and approval card. Nothing is built twice.
• If any answer is in the band, the run waits. The card asks the person to accept or reject the decision, and asks one question for each answer in the band. Leave it empty to confirm what the model said, or pick another option to replace it. Answers outside the band stay the model's.
• The output is the same { model, answers, usage }, plus review. review.status is not_required, approved, rejected, or timeout. review.questions.<id> records the confidence tested, whether it was in the band, and decidedBy (model or human). When a person decided, modelAnswer keeps what the model said. Rejected and timed-out decisions leave answers as the model's.
• next must be a port map with approved (every accepted answer, whether the model's or a person's) and optionally rejected and timeout. A string or list next is refused, because it would run the same nodes after a rejection.

Keys and preflight

The swarm checks a system-one-decision node's key before first use. Set TYPESAFE_API_KEY, OPENROUTER_API_KEY, LAYA_API_KEY, OPENAI_DECISIONS_API_KEY, or CLOUDFLARE_API_TOKEN as a global secret on the Secrets page (Settings > Secrets). For laya, also set LAYA_URL, and for cloudflare, CLOUDFLARE_ACCOUNT_ID (the 32-character account id), as a global config value (set-config, scope global). openai never uses OPENAI_API_KEY. Every message names the key or config value for the node's provider.

• Saving a workflow with a system-one-decision node whose key is missing succeeds and returns a warning: in the tool result message and data.warnings over MCP, and as warnings on the workflow body over HTTP. A save is not rejected, because definitions are also saved before a human has supplied the key and the key can change later.
• Starting a run fails before any node executes when the key is missing. The run is recorded as failed with no steps and no request sent. Retrying a failed run is refused, and the run stays failed, while a node still to run needs a missing key.
• A key the provider refuses (HTTP 401 or 403) fails the step with <KEY> was rejected by <provider> and is not retried.

Example: a laya decision end to end

This workflow classifies a text with provider: laya, sends borderline answers to a person, and branches on the result. It is the definition of the system-one-decision-demo workflow that was run live against laya.

1. Run your own laya server. Calls do not go through any desplega API: your swarm's API server calls your laya server directly, so it must be able to reach it. laya is @desplega.ai/laya-server from laya-js, on npm or built from its Dockerfile (no image is published). The deploy guide covers Docker, Kubernetes, and hosted platforms. It runs on CPU; plan for about 8 GB of RAM for now, since the Docker image has been measured peaking at 7 GB under load. Give the server a LAYA_API_KEY and wait for GET /health to return status: "ok". Without a laya server, use the default typesafe provider instead.
2. Point the swarm at it. Set LAYA_URL to the server's base URL as a global config value (set-config, scope global, not secret). Set LAYA_API_KEY to the same token you gave the server as a global secret on the Secrets page (Settings > Secrets). The URL must be https; plain http is accepted for localhost only.
3. Create the workflow with a manual trigger:

{
  "name": "system-one-decision-demo",
  "triggerSchema": { "type": "object", "required": ["text"], "properties": { "text": { "type": "string" } } },
  "definition": {
    "onNodeFailure": "fail",
    "nodes": [
      {
        "id": "decide",
        "type": "system-one-decision",
        "inputs": { "text": "trigger.text" },
        "config": {
          "provider": "laya",
          "state": "{{text}}",
          "questions": {
            "sentiment": {
              "type": "choice",
              "instructions": "What is the sentiment of this customer review?",
              "criteria": { "positive": "satisfied", "negative": "unhappy", "mixed": "both good and bad" }
            }
          },
          "returns": { "sentiment": { "type": "choice" } },
          "timeoutMs": 30000,
          "maxRetries": 1,
          "humanReview": {
            "band": { "min": 0.5, "max": 0.8 },
            "approvers": { "roles": ["lead"], "policy": "any" },
            "title": "Borderline laya answer",
            "timeout": { "seconds": 900, "action": "reject" }
          }
        },
        "next": { "approved": "gate", "rejected": "rejected-end", "timeout": "timeout-end" }
      },
      {
        "id": "gate",
        "type": "property-match",
        "inputs": { "s": "decide.answers.sentiment.choice" },
        "config": { "conditions": [{ "field": "s", "op": "eq", "value": "positive" }], "mode": "all" },
        "next": { "true": "pass-end", "false": "fail-end" }
      },
      { "id": "pass-end", "type": "script", "config": { "runtime": "bash", "script": "echo pass" } },
      { "id": "fail-end", "type": "script", "config": { "runtime": "bash", "script": "echo fail" } },
      { "id": "rejected-end", "type": "script", "config": { "runtime": "bash", "script": "echo rejected" } },
      { "id": "timeout-end", "type": "script", "config": { "runtime": "bash", "script": "echo timeout" } }
    ]
  }
}

Set approvers to whoever reviews the borderline answers. Replace the script end nodes with the real work for each branch.

4. Trigger it with trigger-workflow (triggerData: { "text": "..." }), the dashboard, or POST /api/workflows/<id>/trigger with the body { "text": "..." }.
5. Read the result on the decide step. answers.sentiment.choice is the label, answers.sentiment.confidence is laya's top probability, and routing.model is the checkpoint that answered (english for English text; laya picks it from the text because the node sends no model). When the confidence falls in the band, the run waits on an approval; after it resolves, review.status says how and review.questions.sentiment.decidedBy says whether the model or a person produced the answer.

The live runs of this definition:

Each laya call took under 0.7 s.

Prefer choice questions on laya for now. On laya-server 0.1.0, a noul question asking whether the review was positive returned close to 0 for every input, including clearly positive ones, while the same judgment as a choice answered correctly. Check a noul question against a few known inputs before you rely on it.

SystemOne Extraction

A system-one-extraction node (display name "SystemOne Extraction") sends one state text, and optionally up to 4 images, to a typed-extraction model and returns one typed answer per declared field. The model is Cloudflare's Clairic on Workers AI (provider: cloudflare, the default and only provider today). It shares the decision node's provider registry, credential, preflight, transport, and error messages: it needs the global secret CLOUDFLARE_API_TOKEN (or CLOUDFLARE_API_KEY) and the global config value CLOUDFLARE_ACCOUNT_ID.

{
  "id": "claim",
  "type": "system-one-extraction",
  "inputs": { "email": "trigger.email" },
  "config": {
    "state": "{{email}}",
    "fields": {
      "order_id": { "type": "text", "instructions": "Order number" },
      "delivered_on": { "type": "date", "instructions": "Delivery date" },
      "amount_paid": { "type": "number", "decimals": 2, "unit": "USD", "instructions": "Total paid including tax" },
      "days_since_delivery": { "type": "duration", "unit": "days", "integer": true },
      "damage": { "type": "list", "items": "text" },
      "repair_cost": { "type": "estimate", "unit": "USD", "min": 10, "max": 5000, "scale": "log", "step": 1 }
    }
  },
  "next": "decide"
}

Field types and their options, checked when the workflow is saved:

Each answer is at <node>.answers.<field>. value is the extracted value, or null when the input does not say. guess is what the model would have said anyway, p_null how likely "not in the input" is, confidence how sure it is of the value, and span (or spans for a list) the character offsets in state. An estimate adds median, p10, and p90; a forecast carries one entry per step in value, median, p10, and p90. Branch on claim.answers.amount_paid.value, or on confidence, with a property-match or a system-one-decision node.

• state is a string; a structured value is sent as JSON text. It may be empty only when images are sent. images are base64 data: URLs or { content_type, base64 } (PNG, JPEG, WebP; 4 MiB each, 8 MiB total). referenceDate (YYYY-MM-DD) anchors dates written without a year.
• Field ids double as output paths, so they use letters, digits, _, and - (no dots), up to 64 characters.
• The node has no humanReview. A wrong value is a typed value a person would have to type back in, so route a low confidence to a human-in-the-loop node instead.
• Retries, time budget, and the ban on retry and validation.retry are the same as for system-one-decision.

Foreach fan-out

A foreach node resolves config.over to an array, runs its body (any executor except foreach; sub-workflow runs one child workflow per item) once for each item, waits for all children to finish, and exposes an aggregate result to its successor. Use an exact interpolation token for over so the value remains an array; itemKey must identify a unique, non-empty string property on every item.

- id: review-by-agent
  type: foreach
  inputs: { agents: "discover.result.agents" }
  config:
    over: "{{agents}}"
    itemKey: id
    body:
      type: agent-task
      config:
        agentId: "{{item.id}}"
        template: "Review the change as {{item.name}} (index {{index}})"
  next: summarize

Set concurrency (an integer, 1 or more) to cap how many items are in flight; without it every item starts at once. The cap holds across restarts. Child steps use synthetic IDs such as review-by-agent#agent-id.

failurePolicy is "fail-fast" or "collect-outcomes", and defaults from onNodeFailure ("continue" collects outcomes, the default "fail" is fail-fast). Fail-fast cancels the queued items, lets in-flight items finish, and fails the run. Collect-outcomes lets every item finish and records failed entries in the aggregate alongside okCount and failedCount. Retrying a failed fan-out with retry-workflow-run reruns only the failed and cancelled items, each as a new attempt; completed items keep their output.

Defining a Workflow

Workflows use a nodes-with-next schema. Each node declares its next field for routing — edges are auto-generated for the UI graph visualization.

The next field supports three formats:

{
  "name": "pr-review-pipeline",
  "description": "Auto-assign PR reviews on webhook",
  "triggers": [
    { "type": "webhook", "hmacSecret": "secret.GITHUB_WEBHOOK_SECRET" }
  ],
  "definition": {
    "nodes": [
      {
        "id": "check-type",
        "type": "property-match",
        "label": "Check Event Type",
        "inputs": { "event": "trigger.event" },
        "config": {
          "conditions": [{ "field": "event", "op": "eq", "value": "pull_request.opened" }],
          "mode": "all"
        },
        "next": { "true": "create-review" }
      },
      {
        "id": "create-review",
        "type": "agent-task",
        "label": "Create Review Task",
        "inputs": { "pr": "trigger.pull_request" },
        "config": {
          "template": "Review PR #{{pr.number}}: {{pr.title}}",
          "tags": ["review", "automated"],
          "priority": 60
        }
      }
    ]
  }
}

The webhook payload supplies event and pull_request. A nonmatching event ends the branch because no false target is declared.

Triggers

An event trigger uses { "type": "event", "eventName": "slack.message" }. The Slack message payload becomes the workflow's triggerData and contains channel, text, user, ts, and threadTs; any configured triggerSchema is applied before a run starts.

Workflow runs preserve the human who actually triggered them in createdBy. Authenticated HTTP and MCP triggers use the trusted caller context, schedules use their creator when present, and Kapso routing uses its resolved sender. Generic ownerless webhooks and creatorless schedules remain unattributed rather than inheriting the workflow author's identity. Retry, resume, and recovery paths rehydrate this requester from the run so agent tasks created later keep the same attribution.

Webhook verification formats

Webhook triggers verify signatures only when hmacSecret is set. The secret can be a literal, an environment reference such as ${WEBHOOK_SECRET}, or a swarm secret reference such as secret.SUPERAGENT_WEBHOOK_SECRET.

Omitting verification keeps the legacy behavior: HMAC-SHA256 over the raw request body, accepting either sha256=<hex> or bare hex. The verifier checks hmacHeader first and then the legacy fallback headers.

{
  "type": "webhook",
  "hmacSecret": "secret.GITHUB_WEBHOOK_SECRET",
  "hmacHeader": "X-Hub-Signature-256"
}

Set verification.format to make the expected format explicit. Explicit formats read only the configured header.

{
  "type": "webhook",
  "hmacSecret": "secret.GITHUB_WEBHOOK_SECRET",
  "verification": {
    "format": "hmac-sha256",
    "header": "X-Hub-Signature-256"
  }
}

Use timestamped-hmac-sha256 for Stripe/Superagent-style headers such as t=<timestamp>,v1=<hex>. The signed payload is <timestamp>.<raw body>, multiple v1 entries are accepted for secret rotation, and timestamps outside the tolerance window are rejected.

{
  "type": "webhook",
  "hmacSecret": "secret.SUPERAGENT_WEBHOOK_SECRET",
  "verification": {
    "format": "timestamped-hmac-sha256",
    "header": "X-Superagent-Signature",
    "timestampKey": "t",
    "signatureKey": "v1",
    "toleranceSeconds": 300
  }
}

Use token-equality for shared-token providers such as GitLab. The configured hmacSecret stores the expected token and the request header must match it.

{
  "type": "webhook",
  "hmacSecret": "secret.GITLAB_WEBHOOK_TOKEN",
  "verification": {
    "format": "token-equality",
    "header": "X-Gitlab-Token"
  }
}

Use standard-webhooks for any sender that follows the Standard Webhooks specification (for example Granola, or services built on Svix-compatible signing). The headers are fixed by the spec, so there is no header field:

• webhook-id, webhook-timestamp (unix seconds), and webhook-signature must all be present.
• The signed content is <webhook-id>.<webhook-timestamp>.<raw body>, HMAC-SHA256 keyed with the base64-decoded secret. Store the sender's signing secret as-is in hmacSecret; a whsec_ prefix is stripped automatically.
• webhook-signature is a space-delimited list of v1,<base64> entries. Any matching v1 entry is accepted, so senders can rotate secrets without downtime. Other versions (such as asymmetric v1a) are ignored.
• Timestamps more than toleranceSeconds (default 300) in the past or future are rejected.

{
  "type": "webhook",
  "hmacSecret": "secret.GRANOLA_WEBHOOK_SECRET",
  "verification": {
    "format": "standard-webhooks",
    "toleranceSeconds": 300
  }
}

Cooldown

Workflows support a cooldown period to prevent rapid re-triggering:

{
  "cooldown": { "hours": 0, "minutes": 5, "seconds": 0 }
}

A second trigger within the cooldown window results in a run with skipped status. The window runs from the last run start (any status but skipped) and, by default, also from the last successful run's finish. Set "from": "start" to count from starts only, so a long run never pushes out the next trigger of a fixed-time schedule. Simultaneous triggers start at most one run.

Safe retries (Idempotency-Key)

POST /api/workflows and POST /api/workflows/{id}/trigger accept an optional Idempotency-Key header. Generate one key per logical operation (a UUID works) and send the same key with the same body on every retry of it, including after a network timeout. The server accepts the operation once: a retry gets the original status and body, with Idempotent-Replayed: true, and creates no second workflow or run. A concurrent duplicate, a lost response and a server restart are all covered.

# One key per logical trigger; resend the same $KEY on every retry of it.
KEY=$(uuidgen)
curl -X POST "$MCP_BASE_URL/api/workflows/$WORKFLOW_ID/trigger" \
  -H "Authorization: Bearer $AGENT_SWARM_API_KEY" \
  -H "Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"pr": {"number": 42}}'

• The key is scoped to the authenticated caller, the operation and (for trigger) the workflow. Another caller's identical key is a separate operation. It is not a credential.
• The same key with a different body returns 409. A request rejected before it is accepted (validation, trigger schema, disabled workflow) does not use up the key.
• A retry of a trigger returns the original runId and skipped, even if the workflow was disabled or edited since. It never starts, resumes or waits for the run.
• If the workflow is deleted, a retry returns 410. The key is never reused, and nothing is created again.
• Accepted keys do not expire.

This guarantees the server accepts a request once. It does not make the nodes of a run exactly-once: a run's external side effects follow the usual retry and recovery rules. It is separate from cooldown, which skips runs by time for every caller. See runbooks/workflows.md § Idempotent create and trigger.

Trigger payload validation (triggerSchema)

Workflows can attach an optional JSON Schema to validate the triggerData payload across every trigger path — manual /trigger, webhooks, schedules, and the trigger-workflow MCP tool. When set, mismatched payloads are rejected with HTTP 400 (or an MCP error) before the workflow starts: no run is created, no nodes execute. When unset, any payload is accepted (current default).

Set or update via create-workflow / update-workflow / patch-workflow (MCP) or POST / PUT / PATCH /api/workflows/{id} (HTTP). The PUT / PATCH paths accept null to clear an existing schema.

The validator supports a deliberate JSON-Schema subset: type, required, properties, enum, const, items. Other keywords (oneOf, anyOf, $ref, pattern, format, additionalProperties, …) are silently ignored. On validation failure, the response echoes the workflow's current triggerSchema so callers can self-correct without a follow-up get-workflow.

See runbooks/workflows.md § Trigger schema for the full reference and authoring examples.

Execution Model

When a workflow triggers:

1. The trigger source (webhook, schedule, or manual) creates a new workflow run
2. The engine's walkGraph() finds ready nodes and executes them in parallel
3. Each node's executor runs and produces output that determines port-based routing via next
4. Checkpoint durability — after every step, an atomic DB write saves the step result and execution context
5. On crash or restart, execution resumes from the last checkpoint

Fan-Out and Convergence

Workflows support fan-out by specifying an array of node IDs in the next field. All target nodes execute in parallel, and downstream convergence nodes automatically wait for all predecessors to complete before executing.

{
  "definition": {
    "nodes": [
      {
        "id": "start",
        "type": "agent-task",
        "next": ["task-a", "task-b", "task-c"]
      },
      { "id": "task-a", "type": "agent-task", "next": "merge" },
      { "id": "task-b", "type": "agent-task", "next": "merge" },
      { "id": "task-c", "type": "agent-task", "next": "merge" },
      {
        "id": "merge",
        "type": "agent-task",
        "inputs": { "a": "task-a", "b": "task-b", "c": "task-c" },
        "config": { "template": "Combine results from {{a.taskOutput}}, {{b.taskOutput}}, {{c.taskOutput}}" }
      }
    ]
  }
}

The engine detects convergence nodes (nodes with multiple predecessors) and gates execution until all predecessors complete. This works correctly with async executors like agent-task — the resume logic uses convergence-aware node detection.

Node Failure Behavior (onNodeFailure)

By default, if any node's task fails or is cancelled, the entire workflow run is marked as failed. You can change this with the onNodeFailure field on the workflow definition:

{
  "definition": {
    "onNodeFailure": "continue",
    "nodes": [...]
  }
}

This is useful for fan-out patterns where some branches may fail but you still want to collect results from the successful ones.

Per-Step Retry

Each node can define a retry policy on the node itself (not inside config):

{
  "retry": {
    "maxRetries": 3,
    "strategy": "exponential",
    "baseDelayMs": 1000,
    "maxDelayMs": 60000
  }
}

Backoff strategies: exponential, linear, static. The poller detects failed steps and re-executes them according to the policy.

For an agent-task node, retry also covers a task that fails after dispatch, for example one the heartbeat fails with superseded_workflow_task after a worker restart. The engine detaches the failed task and the poller dispatches a fresh task for the same step. onNodeFailure applies only once maxRetries is spent. A cancelled task is never retried. Before a retry, the engine restores completed upstream outputs from step checkpoints, so declared inputs resolve to the same values as the original execution. Retrying a failed run also reconstructs the branch selected by each completed step and never revives a branch that was not taken.

A deploy or heartbeat remediation that cuts an in-flight agent-task step fails its task with superseded_workflow_task. When the node has no retry, the engine applies an implicit policy for that reason only: one retry, static, after 30 seconds. A second cut, or any other failure, follows onNodeFailure. An explicit retry always wins, so set "retry": { "maxRetries": 0 } on a node with external side effects (posting, paying, merging) that must not run twice. Steps inside a foreach body do not get the implicit retry yet; a cut item follows its fan-out's failure policy, and retry-workflow-run reruns only the failed items.

Per-Node Timeout

Timeouts live in each executor's config so the executor and the workflow watchdog use the same budget. For a swarm-script node:

{
  "id": "run-report",
  "type": "swarm-script",
  "config": {
    "scriptName": "build-report",
    "timeoutMs": 300000
  }
}

For swarm-script nodes, set config.timeoutMs to raise or lower both the workflow watchdog and the spawned script's wall-clock budget within the allowed 1000-300000 ms range. Create, update, bulk-patch, and single-node patch operations reject an out-of-range executor config before saving the workflow.

Inline script nodes use config.timeout instead. That value is the wall-clock budget for both the inline executor and the workflow step watchdog, so a long-running script is not stopped by the default 30-second watchdog before its configured timeout. It accepts 1000 through 300000 ms and defaults to 30000 ms.

For orchestration that needs more than 5 minutes, launch a durable one-off script workflow with launch-script-run and split the operation into bounded, journaled ctx.step.swarmScript (or other durable) steps. Durable runs resume across process restarts, but an individual script step still uses its own bounded scripts-runtime window.

Cancelling a Workflow Run

Running or waiting workflow runs can be cancelled using the cancel-workflow-run MCP tool, the API (POST /api/workflow-runs/{id}/cancel) or the dashboard. An optional reason can be provided for audit purposes.

The cancel covers the run and every child run below it, including children of earlier retry attempts. In one transaction it cancels their non-terminal steps, tasks, pending approvals and queued retries, and marks every unfinished run cancelled. A child run that already completed or failed keeps its status and result. After the cancel, no step, task, child run or retry starts anywhere in that tree, and a late task result or approval does not resume it. Cancelling an already cancelled run again finishes any child runs still open.

Cancellation does not stop work that is already executing outside the database, such as a running script or an agent session before its worker sees the task cancelled, and it does not undo external side effects.

Per-Step Validation

The validate executor runs after a step to check output quality. It can:

• Pass — continue to the next step
• Halt — stop the workflow run
• Retry — re-execute the previous step

Cross-Node Data Access (inputs Mapping)

By default, upstream step outputs are not available for interpolation. Built-in trigger, input, workflow, swarm, and run context remains available for ordinary config values. To access another node's output, declare an inputs mapping on the node:

{
  "id": "summarize",
  "type": "agent-task",
  "inputs": { "cityData": "generate-city" },
  "config": {
    "template": "The city is {{cityData.taskOutput.city}} in {{cityData.taskOutput.country}}"
  }
}

• Keys are local names used in {{interpolation}}, values are context paths (usually a node ID)
• Agent-task output shape is { taskId, taskOutput } — access fields via localName.taskOutput.field
• For trigger data: { "pr": "trigger.pullRequest" } → {{pr.number}}
• Without inputs, ordinary templates referencing upstream nodes resolve to empty strings and report the token in run diagnostics

Executable source has a stricter boundary. Inline script source may interpolate only input, workflow, swarm, and run; pass trigger or upstream values through config.args or config.env so they arrive as data instead of executable text. Named swarm-script source is never workflow-interpolated, so pass all dynamic values through its args object. Disallowed or unresolved source tokens fail the node before execution, while unrelated mustache strings remain literal data.

Structured Output (outputSchema)

Agent-task nodes can require structured JSON output from the agent via config.outputSchema:

{
  "id": "generate-city",
  "type": "agent-task",
  "config": {
    "template": "Pick a random city and return it as JSON",
    "outputSchema": {
      "type": "object",
      "required": ["city", "country"],
      "properties": {
        "city": { "type": "string" },
        "country": { "type": "string" }
      }
    }
  }
}

There are two different outputSchema locations — they validate at different layers:

For agent-task nodes, put your schema in config.outputSchema. The agent is prompted to produce JSON matching this schema, and store-progress validates it inline.

Workspace Scoping

Workspace scoping can be set at two levels:

Workflow level — Set dir and vcsRepo on the workflow itself. All agent-task nodes that don't explicitly set these fields inherit the workflow-level defaults. These values are also available for interpolation:

{
  "name": "my-workflow",
  "dir": "/workspace/myrepo",
  "vcsRepo": "https://github.com/org/repo",
  "definition": {
    "nodes": [
      {
        "id": "build",
        "type": "agent-task",
        "config": {
          "template": "Build and test in {{workflow.dir}}"
        }
      }
    ]
  }
}

Node level — Set config.dir or config.vcsRepo on individual agent-task nodes. Node-level settings override workflow-level defaults.

Interpolation sources available in node config:

• {{workflow.dir}} — Workflow-level dir
• {{workflow.vcsRepo}} — Workflow-level vcsRepo
• {{trigger.*}} — Trigger payload data
• {{input.*}} — Workflow-level resolved inputs

Version History

Every workflow update creates a snapshot in the version history table, recording the previous definition. This provides an audit trail and enables rollback.

MCP Tools

list-workflow-runs returns bounded rows by default: run identifiers, status, timestamps, error, and a trigger-data summary capped at 400 characters. It omits full context and trigger data to keep routine history queries small. Follow page.hasMore / page.nextOffset for pagination, call get-workflow-run to retrieve one run with its steps and context, or pass includeContext: true for an explicitly full page.

get-workflow-run (and GET /api/workflow-runs/{id}) returns the full run by default: the run context plus each step's input, a snapshot of the context when the step started. In runs with loops (polling, wait, iterations) that grows with steps × context size. Pass fields: "slim" (?fields=slim over HTTP) to drop both and keep statuses, outputs, errors, diagnostics, retry fields and the parent link, which is enough for monitoring and deciding what to do next.

Dashboard UI

The dashboard includes a Workflows section (under Operations) with:

• Workflows list — View all workflows with enable/disable status
• Workflow detail — Tabbed view with Definition (node inspector showing config/input/output schemas) and Runs tabs
• Run detail — Split layout with interactive graph and collapsible steps panel, JsonTree viewer, bidirectional node selection, expand/collapse all, millisecond duration display

Human-in-the-Loop (HITL) Node

The human-in-the-loop executor pauses a workflow until a human responds via the dashboard. This enables approval gates, manual input steps, and review checkpoints within automated pipelines.

{
  "id": "approve-deploy",
  "type": "human-in-the-loop",
  "label": "Approve Deployment",
  "config": {
    "title": "Deploy to production?",
    "questions": [
      { "type": "approval", "label": "Approve this deployment?" },
      { "type": "text", "label": "Any notes?" }
    ]
  },
  "next": { "approved": "deploy", "rejected": "notify-rejected" }
}

Question types supported: approval (yes/no), text, single-select, multi-select, boolean, and config (set a swarm config key or secret; see below).

When the node executes, it creates an approval request accessible at /approval-requests/{id} in the dashboard. The workflow run pauses until a human responds. Upon resolution, a follow-up task is created for the requesting agent and the workflow resumes via the appropriate next port. Cancelling the run or its human-in-the-loop step marks the pending request as cancelled, rejects later responses, and updates any linked Slack approval thread so reviewers do not act on a stale gate.

The dashboard validates every required response before submission. Missing or invalid approval, text, select, and boolean answers are marked inline and keep Submit Response disabled; if the server rejects a response, its error is shown directly so the user can correct it.

Config questions

A config question asks a person to set one swarm config key from the dashboard, for example a token a new integration needs. Its settings sit in a nested config object:

{
  "id": "bitbucket_token",
  "type": "config",
  "config": { "key": "BITBUCKET_TOKEN", "scope": "global" }
}

label is optional and defaults to the key; the dashboard shows the integration's field label when it knows the key. The card shows whether the key is already set and when it was last updated, never a secret value. The person can keep the current value, replace it, or skip an optional question.

Rules:

• A request with a config question needs approvers.policy: "any".
• The value is written to swarm_config (encrypted for secrets) only when the request resolves approved. A rejection skips every config question.
• Answering needs the same permission as PUT /api/config.
• The request, the workflow step output and the follow-up task store only the result, never a secret:

{ "key": "BITBUCKET_TOKEN", "scope": "global", "scopeId": null, "isSecret": true, "status": "set", "updatedAt": "2026-10-23T16:40:12.000Z" }

status is set, kept or skipped. Non-secret results also carry value.

questions can also be built at run time: set it to one exact token such as "{{plan.questions}}" (with a matching inputs mapping) and the upstream array becomes the card's questions, for example one single-select per item. The array is validated when the node runs; a missing, empty, or malformed value (or more than 100 questions) fails the node without creating a card.

Loop Support

Workflows support cycles (loops) in the DAG. A node can route back to a previous node via its next field, enabling iterative patterns like retry-until-success or iterative refinement.

The engine uses iteration-aware idempotency keys to distinguish between loop iterations. Each time a node re-executes in a new iteration, it gets a unique checkpoint, allowing the engine to track progress correctly across multiple passes through the same node.

{
  "definition": {
    "nodes": [
      {
        "id": "start",
        "type": "agent-task",
        "config": { "template": "Prepare the draft requirements" },
        "next": "generate"
      },
      {
        "id": "generate",
        "type": "agent-task",
        "config": { "template": "Generate a draft" },
        "next": "validate"
      },
      {
        "id": "validate",
        "type": "validate",
        "inputs": { "draft": "generate" },
        "config": {
          "targetNodeId": "draft",
          "prompt": "Does the draft in taskOutput meet the requirements and read clearly?"
        },
        "next": { "pass": "publish", "fail": "generate" }
      },
      {
        "id": "publish",
        "type": "agent-task",
        "config": { "template": "Publish the final draft" }
      }
    ]
  }
}

The start node provides an entry outside the cycle. The draft input maps the generator output into the validator context. In this example, the validate node routes back to generate on failure, creating a loop that continues until validation passes.

Related

• Scheduled Tasks — Time-based task automation (complementary to event-driven workflows)
• Task Lifecycle — How tasks created by workflows flow through the system
• MCP Tools Reference — Full tool documentation
• Workflows API Reference — REST API endpoints for creating and managing workflows