The Agent Reasons, the Engine Remembers: Designing an Agentic Workflow Engine

🗓️ Last updated: July 2026

Picture a workflow that began an hour ago with a customer email. Step 1 classified intent. Step 2 retrieved order history. Step 3, an LLM agent, read both, decided the customer wanted a refund, and called the refund API. The refund succeeded. Step 4 was supposed to send a confirmation email. It crashed. The worker process restarted. Now the engine has a decision to make.

What is the state of this workflow? Has the refund happened? Yes, there is an event in the log that says so. Has the customer been notified? No, that step never completed. Should the engine retry step 4? Yes, after a delay. Should the engine rerun step 3? No, the agent already made the decision, the side effect is committed, and rerunning risks a double refund. Should the engine retry the agent's reasoning if a newer model has been deployed since? That depends on whether the operator asked for strict replay or fresh inference, and the engine has to know the difference.

Every one of those questions has a classical workflow engine answer that has been worked out over twenty years of distributed systems literature. And every one of them gets harder the moment an LLM agent is one of the steps. This article is about the engine that answers them honestly.

A note on framing: the previous article in this series, Designing an LLM Inference Platform, designed the system that serves a single LLM call cheaply and reliably at scale. This article is the layer above it. The inference platform owns GPUs and serves tokens; the workflow engine owns a state machine, an event log, a retry policy, and a schema enforced contract for every step. A workflow engine calls into an inference platform on every agentic step. The two systems are often built and operated by different teams, sometimes by different companies. The unit of work for the inference platform is the call. The unit of work for the workflow engine is the graph. Confuse the two and you build the wrong system; the failure modes of one masquerade as the failure modes of the other.

This is also the closing article of System Design, Reimagined, the place where this entire series joins back up with Architecting Agents. Six classical system design problems redrawn for the probabilistic era ends here, with the realisation that what you have actually been building all along is one thing: a discipline for putting probabilistic computation inside deterministic scaffolding.

The thesis: an agentic workflow engine is a graph, not a caller. Its job is to put probabilistic steps inside deterministic scaffolding: contracts, retries, budgets, replay semantics, and observability, so that the agents inside it can be operated, audited, bounded, and trusted. The hard part is not the agent. The hard part is the graph that contains the agent.

Requirements

Five nonfunctional requirements drive every later decision.

1. Durability across process restarts. A workflow may run for seconds, hours, or weeks. If a worker process crashes mid step, the workflow must resume from the last committed checkpoint without losing work or reexecuting committed side effects. This is the core property of every serious workflow engine. Agentic steps do not get an exemption.

2. Replay determinism, to the extent possible. Operators must be able to take a failed or suspicious run and reexecute it from a known state to understand what happened. For deterministic steps, replay is exact. For LLM steps, replay is approximate at best, and the architecture has to be explicit about which is which.

3. End to end observability of every step. Every step's inputs, outputs, tool calls, model used, prompt, token spend, and elapsed time must be queryable as structured data, not log greps. This requirement is borrowed wholesale from Observability for Agentic Systems in the Architecting Agents series; it is the only way to operate something this stateful.

4. Bounded cost per workflow. A workflow that can spend an unbounded number of tokens or invoke an unbounded number of tool calls is not a workflow; it is a budget incident waiting to be filed. The engine must enforce per workflow ceilings on tokens, tool calls, wall clock time, and dollars.

5. Safe interaction with external systems. When a workflow step writes to a database, calls a payment API, or sends an email, the engine has to guarantee that retries do not double execute and that compensation actions undo correctly if a later step fails. This is classical workflow semantics, but now the decision to call the tool may come from a probabilistic step.

The PM trap requirements, "the agent should figure out the whole workflow from a goal" and "the agent should be able to call any tool it wants," sound powerful and are operational disasters. Both have to be bounded at the engine level even when the contract language does not say so.

The Classical Baseline

The classical workflow engine playbook is well developed. A handful of patterns dominate the field:

• Airflow (2014) and Argo Workflows model workflows as static DAGs defined ahead of time. Excellent for batch and ETL; rigid for dynamic flow.
• AWS Step Functions models workflows as state machines defined declaratively. Good cloud native integration; bounded expressiveness.
• Temporal (and its Uber predecessor Cadence) takes a different approach: workflows are normal code, executed by a deterministic replay engine that rederives state from an event log. This buys arbitrary control flow with strong durability guarantees, at the cost of strict determinism rules on workflow code.
• Prefect and Dagster sit closer to data engineering use cases with strong observability primitives.

What all of these systems share is a small set of assumptions: a step is a function whose behaviour depends only on its inputs; the workflow graph is either static or grown by deterministic logic; and replay is exact because the event log captures every nondeterministic decision.

LLM steps break the first and third of those assumptions. They sometimes break the second one too: when the agent's output decides the next branch, the graph is being co authored by the workflow code and the model. The interesting design work is preserving as much of the classical guarantee set as possible while making honest concessions where the probabilistic steps genuinely change the contract.

Four Cascading Decisions

Decision 1: The step boundary. Define what counts as a single step. The choice has dramatic consequences for durability, retries, and observability. Three common settings:

• Step equals one model call. Each LLM invocation is its own durable step; tool calls are also their own steps. Fine grained, easy to replay, but expensive in checkpoint overhead.
• Step equals one model plus tool turn. The model proposes a tool call, the engine executes it, the result is fed back to the model; the whole exchange is one step. Less checkpoint overhead, but a partial failure inside the step is harder to reason about.
• Step equals a whole agent reasoning block. The agent reasons over multiple internal turns before emitting a result; only the block boundary is durable. Useful for short interactive agents; dangerous for long autonomous ones because a crash mid block loses unbounded work.

There is no universal right answer, but there is a universal wrong answer: leaving this implicit. If your engine does not have a documented step boundary, every team building on it picks a different one and the resulting failure modes are impossible to reason about across the system.

Decision 2: State persistence model. Decide how workflow state is captured and restored. Two patterns dominate.

The event sourced model (Temporal's approach) replays a log of past events to reconstruct workflow state. Pure functions of the event log are deterministic by construction; nondeterministic operations have to be wrapped in primitives that record their outcome to the log.

The snapshot model (Step Functions and most home grown engines) checkpoints the entire workflow state at every step boundary. Simpler to reason about, but the state has to be serialisable and the snapshot has to be cheap to take and load.

Event sourcing is more powerful, especially when paired with a true replay debugger. Snapshots are more pragmatic and play better with LLM steps that emit large free form outputs. The right pick is the one your team can operate; both models are battle tested in production at scale.

Decision 3: Branching model, deterministic graph versus runtime decided graph. Classical workflows have a graph determined at definition time. Agentic workflows often have steps where the model decides what happens next: call this tool, give up and escalate, ask the user, route to a different agent.

A clean architecture makes the agent's branching explicit at the engine level: the agent step returns a structured decision (an enum or a typed branch token), the engine matches it against the workflow definition, and the workflow engine, not the agent, moves the cursor. The antipattern is letting the agent emit free form next step instructions that the engine then has to parse. Anywhere the agent's natural language output becomes the workflow's control flow, the system is fragile.

Decision 4: Workflow level bounds. Decide how cost, time, and risk ceilings are enforced. Per step bounds are necessary but not sufficient; they do not catch loops where the workflow keeps reaching new steps within each step's budget. Workflow level bounds (total tokens, total tool calls, total wall clock, total dollars) must be tracked centrally and enforced by the engine, not by step code.

These four decisions cascade. The step boundary determines what state has to be persisted. The state model constrains what kinds of branching can be replayed. The branching model determines how budgets are enforced (per branch versus globally). Resolve them in order, or watch each one get relitigated three times as the implementation reveals their interactions.

Step Contracts and Schemas

A workflow step has two contracts. The input contract describes what state and arguments the step expects. The output contract describes what it produces and which branches downstream code can take based on that output. Classical workflow engines enforce both with strong types, schemas, and registry checks at deploy time.

Agentic steps strain this. An LLM emits free form text by default. To make it a workflow step, the output has to be coerced into a structured shape. The standard tools are structured output (JSON schema constrained decoding, available in most major model APIs), tool call schemas (OpenAI function calling, Anthropic tool use, and MCP defined tools), and post hoc validators that reject and retry malformed outputs.

The discipline this article inherits from earlier in the series, and from Architecting Safe Tool Calling Agents in particular, is that the schema is the contract. If the agent produces output that does not validate, the engine should not pass it downstream and hope. It should retry the step, fall back to a human in the loop branch, or fail the workflow with a clear schema violation error. Trying to repair invalid output downstream is how silent corruption enters the system.

A practical pattern: every agentic step has a wrapping decorator that handles structured output prompting, JSON schema validation, bounded retries on validation failure, and an explicit fall through branch in the workflow graph. The agent does the reasoning; the wrapper enforces the contract. The workflow engine treats the wrapped step exactly like a deterministic step.

Retry, Idempotency, and Compensation

Three classical workflow primitives apply unchanged in the agentic case. Their importance, if anything, goes up.

Retries need to distinguish between three failure classes: transient (network timeout, rate limit), deterministic (schema violation, validation failure), and content (the model produced a plausible answer that turns out to be wrong). Classical engines collapse this into "retryable or not." Agentic engines have to handle a fourth class, semantically incorrect, which usually is not retryable in the same way and instead needs escalation or human review.

Idempotency is the property that retrying a step does not produce duplicate side effects. The standard technique is an idempotency key passed to the downstream service: the same key submitted twice produces one effect. For agentic steps, the idempotency key has to be derived deterministically from the workflow context, not from the model's output, which may vary across retries.

Compensation is the Saga pattern (Garcia Molina and Salem, 1987): if step N+1 fails after step N has committed a side effect, step N's compensating action is invoked to undo it. The discipline here does not change for agentic workflows, but the surface area does, because agentic workflows tend to have more steps and more side effecting tools than classical ones, and every irreversible side effect needs a compensating action defined alongside it.

The forbidden combination is the one that appears most often in early agentic systems: an LLM step decides to call a tool, the tool succeeds, the next step fails, and there is no compensation defined because the tool was "just something the agent decided to do." If the agent can decide to call the tool, the tool needs a compensation defined at the workflow level. The agent's freedom does not exempt the engine from the saga discipline.

Observability and Replay

Workflow engines have the strongest observability story in distributed systems: every step's input, output, retry count, and elapsed time is captured by construction. Agentic workflows need that plus three more things, and the architecture has to make all three first class.

Prompt and model capture. Every LLM step records the exact prompt, model identifier, temperature, max tokens setting, and any tool definitions in scope. Without this, debugging "why did the agent decide that?" is impossible. With it, post mortems become precise.

Cost capture. Every LLM call records its input tokens, output tokens, cached tokens (if prefix caching applies), and a derived cost in dollars. This rolls up to a per step and per workflow total. Without it, the cost bounded agent discipline from earlier in the series has nothing to enforce against.

Replay semantics, honestly stated. Replay of an agentic workflow has three modes, and the engine should support all three:

• Strict deterministic replay. The engine replays the recorded event log, including the recorded LLM outputs. The workflow takes the same path it took originally. This is what most operators want most of the time.
• Fresh inference replay. The engine reruns every LLM call against the live model. The workflow may take a different path. This is for evaluating whether a fix to the prompt or the model would change the outcome.
• Counterfactual replay. The engine replays the recorded events up to a chosen step, then runs fresh inference from there. This is the most useful debugging mode and the hardest to implement cleanly.

The architectural commitment is naming these three modes and exposing them as first class operations on every workflow. This is Observability for Agentic Systems applied to workflows: structured capture, structured queries, and structured replay. Trace IDs are not enough.

Cost Bounds and Coordination Patterns

Two more concerns pull in callbacks from earlier in the series.

Cost bounds at the workflow level. Cost Bounded Agent Design in Architecting Agents argued for explicit token, tool call, time, and dollar ceilings on every agent. In a workflow engine, those ceilings sit at the workflow level, not the step level. A workflow's budget descends to its steps; a step that exceeds its share returns a budget exceeded signal to the engine; the engine decides whether to retry, escalate, or fail the workflow.

The nonobvious design point is when to charge. Token costs are known after the LLM call returns, but a step that loops internally can run up costs before the engine sees a checkpoint. The pattern that works: every LLM client wrapper increments a workflow level counter on every call and checks against the budget before each call. The check is cheap. The alternative, discovering an overrun after the workflow has completed, does not constitute cost control.

Coordination patterns. Coordinating Multi Agent Systems (the closing article of Architecting Agents) walked the three coordination patterns: hierarchical (planner and worker), peer to peer (debate, voting), and pipeline (sequential specialists). Every one of these is a graph pattern. A workflow engine is the natural home for them.

The architectural consequence is that "multi agent system" is not a separate kind of system from "agentic workflow." They are the same system, viewed at different scales. A multi agent product is a workflow where multiple steps happen to be LLM agents and the workflow defines how they pass work and information between each other. Treating the two as separate frameworks, a multi agent library on top of an unrelated workflow library, is a common architectural mistake that produces two competing state stores, two competing observability pipelines, and two competing budget models. Pick the workflow engine. Run the agents inside it.

Failure Modes

Seven failure modes recur. Ordered by severity:

1. Loop blow up. An agent step decides to call a tool, the tool result feeds back to the agent, the agent decides to call another tool, and the cycle continues without converging. Per step budgets do not catch it because each iteration stays within its share. Fix: hard caps on agent step iteration count, workflow level budget enforcement, and a circuit breaker pattern that bails out after N steps without progress.

2. Schema drift between steps. Step N's output schema changes between deploys; step N+1 still expects the old shape; the workflow fails midway in production. Fix: schema versioning, downstream compatible changes, and contract tests in CI that exercise every step boundary with a snapshot of the prior step's output shape.

3. Orphaned compensations. A side effecting step committed; a later step failed; the compensation did not run because it was not defined or it errored. Fix: every irreversible tool call has a compensation defined and tested; compensations themselves are retried; the engine tracks "compensation owed" state explicitly and surfaces it as a metric.

4. Replay nondeterminism. An operator replays a failed workflow expecting to reproduce the issue; the agent makes a different decision on replay; the bug does not reproduce. Fix: the engine names the three replay modes explicitly, defaults strict replay to using recorded LLM outputs, and offers fresh inference replay only as an explicit operation.

5. Coordination deadlock. Two agent steps wait on each other to produce output; neither does; the workflow stalls. Fix: every agent step has a wall clock timeout, the workflow engine enforces it independently of any per call timeout, and the deadlock surfaces as a timeout exception with diagnostic context.

6. Lost context at step boundaries. Useful state in the agent's reasoning is discarded at the step checkpoint; the next step has to rederive it; the model picks something different the second time. Fix: explicit working memory state passed between steps (a callback to Stateful Memory in Agentic Systems), not relying on the agent to remember across boundaries.

7. Double execution under retry. A tool call succeeded but its acknowledgement was lost; the engine retries the step; the tool runs twice. Fix: idempotency keys derived from workflow context, enforced by the downstream tool or by an idempotency layer the engine owns.

The fashionable failure mode, "the agent went rogue," is downstream of all seven of these. An agent that exceeds its budget, drifts off schema, double executes a tool, or deadlocks looks rogue from the outside. The architectural truth is that it was operating in a system that did not bound it. The engine is the bound.

Cost Economics

Workflow engine cost has three layers that need separate accounting.

The engine layer, workers, schedulers, event log storage, dashboards, is classical infrastructure. Costs are proportional to workflow volume and history retention, well understood from twenty years of workflow engine deployment experience.

The LLM layer dominates everything else. Token spend at every agentic step rolls up to a per workflow cost. For workflows with many agentic steps or long contexts, this number can dwarf the engine layer's cost by a wide margin. The unit metric that matters is dollars per completed workflow, not dollars per step. A workflow that completes for ten cents looks healthy; a workflow that fails halfway after spending ten cents is pure burn.

The tool layer covers external API calls invoked from agentic steps: payment gateways, search APIs, code execution sandboxes, third party data services. Each has its own cost model and its own rate limit story. These costs are easy to underbudget because they sit at the leaves of the graph; the engine has to surface them as a first class line in the per workflow cost breakdown.

The single biggest cost lever is workflow completion rate. A workflow that completes 95 percent of the time at one dollar costs the same per success as a workflow that completes 50 percent of the time at fifty cents, and the second one has half the customer experience. Engineering for completion (good schemas, sound retries, calibrated escalation) is the same as engineering for cost.

Architect's Checklist

A reusable list for designing or reviewing an agentic workflow engine:

1. Define the step boundary explicitly; document it; do not let teams pick per step.
2. Pick the state persistence model, event sourced or snapshot, and stay with it across the codebase.
3. Express agentic branching as structured decisions matched against the workflow graph, never as free form text the engine parses.
4. Enforce workflow level bounds on tokens, tool calls, wall clock time, and dollars; per step bounds are necessary but insufficient.
5. Treat the schema as the contract; reject malformed agentic output at the step boundary, do not repair it downstream.
6. Distinguish four failure classes: transient, deterministic, semantically wrong, and content wrong; handle each one explicitly.
7. Derive idempotency keys from workflow context, not from model output.
8. Define a compensating action for every irreversible side effect; test compensations regularly.
9. Capture every LLM call's prompt, model, settings, tokens, and cost as structured data; first class, not log lines.
10. Support three replay modes, strict, fresh inference, and counterfactual, and name them in the operator interface.
11. Pass working memory between steps explicitly; do not rely on agents to remember across boundaries.
12. Measure dollars per completed workflow; engineer for completion as the primary cost lever.

Architect's Mental Model

A workflow engine is a graph that durably executes steps. An agentic workflow engine is the same thing, with a class of steps whose behaviour is partly chosen at runtime by an LLM. Almost everything that makes the classical version work continues to work. The interesting design is in the seams.

The seams are the same five concerns the Architecting Agents series spent six articles on. Failure modes have to be designed for; they are not optional. State must persist between steps; it is the workflow's memory. Tool calls must be safe under retry; saga discipline applies whether a human or an agent decided to invoke the tool. Observability must be structured down to the prompt and the token; otherwise the system cannot be operated. Costs must be bounded at the workflow level; per step ceilings catch the wrong failure mode. Multi agent coordination is a workflow shape; it is not a separate kind of system.

That symmetry is the point of this series of series. Architecting Agents covered the discipline of building a sound agent. System Design, Reimagined covered the discipline of building sound systems around them. The two pieces compose into one body of work, and the workflow engine is where they touch most cleanly. Everything the first series demanded of an agent, the second series demands of the system that contains agents. The graph is what makes that demand enforceable.

The agent reasons. The workflow remembers, retries, bounds, and proves. Without the second, there is no system.