Reasoning Is Cheap, Side Effects Are Forever

🗓️ Last updated: June 2026. Reflecting current tool use patterns and saga based agent architectures.

There's a quiet failure mode buried in every agentic system that touches the real world:

The agent's reasoning can be retried. The world cannot.

If your agent books the wrong flight, you can't unbook it cleanly. If it sends the wrong email, the email is already in someone's inbox. If it charges a card twice during a retry storm, refunds become a customer service problem, and customer service rarely thanks engineering for the extra volume.

Everything works until the network blinks. Then it doesn't.

The reframe that makes everything easier, and the reason this article exists, is that we've solved this problem before. Distributed systems engineers have spent decades building patterns for safe side effects in unreliable environments. The agentic AI era doesn't need new fundamentals. It needs to remember the ones we already have.

The Two Categories of Tool Calls

Most agent frameworks treat tool calls as a uniform abstraction. That's the root design error.

There are two fundamentally different categories:

Read only calls: querying a database, fetching a document, listing items. Safe to retry. Safe to replay. Safe to call concurrently.
Mutating calls: charging a card, sending an email, booking a flight, deleting a record, posting to an external API. Every retry is a potential bug. Every concurrent invocation is a potential race condition.

A multi turn agent can issue many tool calls per session, often dozens, sometimes more. If you don't separate these two categories at the architecture level, you've made the entire system unsafe by default.

Idempotency: The First Principle of Safe Mutations

The simplest and most useful guarantee in the distributed systems toolkit is idempotency.

A tool call is idempotent if running it twice has the same effect as running it once. Formally: f(f(x)) = f(x).

Why this matters for agents:

Agent frameworks often retry tool calls automatically when a response fails to parse, times out, or returns an error
Network failures cause clients to retry without knowing whether the server succeeded
Multi agent systems may invoke the same action through different paths
Failover and replay during recovery may re issue calls

Without idempotency, every one of these creates a duplicate side effect.

Patterns that make a tool idempotent

Idempotency keys. The client sends a unique key with the request. The server stores recent keys and dedupes. Stripe's payments API documented this pattern at industry scale, and it has since become a standard convention across SaaS APIs.
Conditional writes (CAS). "Update X only if its current value is Y." Most modern databases (DynamoDB, etcd, Postgres with row versioning) support this directly.
Deterministic IDs. Generate the resource ID from a hash of the request fingerprint. Re issuing the same request hits the same ID, and the create becomes an upsert.
Dedupe windows. Store recent request fingerprints with a TTL. Drop duplicates within the window.

Idempotency doesn't mean "safe"; it means "retry safe." A retry safe destructive action is still destructive. But retry safety eliminates the entire class of failure then duplicate bugs.

If you're building agentic AI today and your tool layer doesn't enforce idempotency, start there.

Where the idempotency key must live

There's a subtle trap here that defeats naive implementations: the scope and ownership of the key.

"One key per logical operation" sounds simple, but the hard question is who mints the key and where it lives. If the key is generated inside the LLM's context, embedded in the reasoning trace, then a context window summarization or truncation can silently drop it. The next attempt mints a fresh key, the dedupe layer sees a new operation, and you've doubled the side effect. This is exactly the "phantom retry" failure mode described below, and it's the most common way idempotency layers fail in practice.

The rule: the idempotency key must be minted and persisted outside the model, by the tool invocation middleware keyed to a stable logical operation ID, so that it survives context summarization, session replay, and failover. The model may reference the operation; it must never be the sole keeper of the key.

The Saga Pattern: When You Can't Roll Back

Idempotency handles individual mutations. But agents rarely act in isolation; they execute multi step workflows where every step has side effects.

Consider an agent booking travel:

Search flights
Reserve seat on flight
Charge customer card
Reserve hotel room
Send confirmation email

If step 4 fails, what happens to steps 2 and 3? You can't roll back the card charge with a database ROLLBACK. You can't unreserve the seat with a transaction abort. The world doesn't have ACID, and it will not be acquiring it for you.

This problem isn't new. Hector Garcia Molina and Kenneth Salem named it in 1987 in a paper titled Sagas, proposing that a longrunning workflow be broken into a sequence of local transactions, each with a compensating action that semantically undoes it.

Release the seat reservation
Refund the card
Cancel the hotel hold
Send a cancellation email instead of a confirmation

That's not rollback. That's forward recovery: executing a sequence of compensating actions to bring the system back to a consistent state.

Agentic multi step workflows are sagas. Most teams writing them today don't frame them that way. The pattern, the literature, and the failure modes have been studied for nearly four decades. Use them.

Choreography vs orchestration

Sagas come in two flavors:

Choreography: each step emits an event; the next step listens. Decentralized. Fits naturally into eventdriven architectures.
Orchestration: a central coordinator sequences each step explicitly. Easier to reason about, easier to debug.

For agents, orchestration is almost always the right default, but be precise about who the coordinator is. It is tempting to say "the LLM is already the coordinator, so let it drive the saga." That's a mistake, and it contradicts the core thesis of this article: the model is a probabilistic component that may issue duplicate or outoforder calls. You do not want a stochastic component owning compensation ordering.

The correct division of labor: the LLM emits intent ("execute this saga plan"), and a deterministic saga executor owns sequencing, persistence, and the order in which compensations run. The model proposes; the deterministic envelope disposes. Give the LLM visibility into the saga's state and compensations so it can reason about them, but never let it be the thing that remembers what compensation is owed.

When compensations themselves fail

There's a gap most saga writeups skip, and it's the one that bites hardest in incident response: compensating actions can fail too. Refunds time out. Cancellation APIs return 500. The seatrelease endpoint is down for maintenance at the exact moment you need it.

Design for this upfront:

Make compensations idempotent and retryable. A compensation is itself a mutating tool call, so it needs the same idempotency contract as the forward action, so retrying a refund doesn't doublerefund.
Persist "compensation owed" state durably. The obligation to compensate must outlive the saga executor process, the agent session, and any single retry attempt. Write it to the event log before attempting the compensation, not after.
Provide a deadletter / escalation path. When a compensation exhausts its retries, the saga doesn't get to silently give up. It routes to a durable failure queue and escalates to a human. An inconsistent system that knows it's inconsistent is recoverable; one that has forgotten is not.

A saga that only handles the happypath failure (forward step fails, compensation succeeds) is a demo. A saga that handles compensationfailure is production.

The Compensation Catalog

For every mutating tool in your agent's catalog, you need three things documented upfront:

The forward action: what the tool does
The compensating action: how to undo or neutralize its effect
The idempotency contract: how the tool handles repeat invocations (for both the forward action and its compensation)

If a tool has no compensating action (an irreversible operation like sending an email or firing a webhook), that must be explicitly declared in the catalog. It changes how the agent is allowed to use it.

This catalog isn't optional metadata. It's the safety contract that makes the agent's actions auditable, recoverable, and reasonedabout. Treat it like a database schema: versioned, reviewed, and tested.

In the previous article in this series, I called this procedural memory, the tier of agent memory that stores tool definitions and capabilities. The compensation catalog lives there.

Failure Modes That Show Up in Agent Side Effects

LLM agents inherit most of distributed systems' classical failure modes, and amplify them, since the orchestrator is now a probabilistic component that may issue duplicate or outoforder calls. Worth naming the ones you'll meet most often:

Partial execution

The agent calls a tool. The network drops before the response. Did the action happen? If you don't know, you must assume it might have, and design the next step to be safe whether or not it did.

Phantom retries

The model decides on its own to retry a tool call because it didn't "see" the result (helpfulness being, in this one case, a liability). Without an idempotency key shared across both attempts, and crucially one minted outside the model so a context summary can't drop it, you've doubled the side effect.

Cascade abandonment

The agent runs a 5step saga. Step 3 fails. The model's context summary discards the saga state. The agent forgets that compensations are owed, and the system silently stays inconsistent. (This is precisely why "compensation owed" must be persisted by the deterministic executor, not held in the model's context.)

Staleread divergence

The agent reads external state, reasons over it, and writes back, but the state changed between the read and the write. Without optimistic concurrency control, the agent overwrites a newer update with stale data.

Concurrentagent contention

Two agent instances (or two sessions) act on the same external resource simultaneously. Without locks, leases, or partition keys, the result is nondeterministic in the worst possible way.

These aren't theoretical. They surface predictably in any system that combines unreliable orchestration with external side effects, which is exactly what an agent is. Plan for them upfront; don't discover them under incidentresponse pressure.

Patterns That Actually Work

Pattern 1: Tool Risk Classification

Tag every tool in the catalog with a risk class:

This classification is the foundation that every safety pattern below builds on. Without it, every other guard is bolted on randomly.

Pattern 2: The Idempotency Layer

A thin middleware between the agent and any softmutate tool that:

Mints or accepts an idempotency key per logical operation, and owns that key itself, outside the model's context, so it survives summarization and replay
Stores the (key → outcome) mapping for a bounded window
Returns the cached outcome on repeat invocations rather than reexecuting

This is one of the highestleverage components you can build. The implementation is small; the failure class it eliminates is large.

Pattern 3: The Saga Executor

A small deterministic orchestrator that sits between the agent's intent and the tool layer:

Accepts a saga plan (sequence of steps with compensations) emitted by the LLM
Owns sequencing: the model proposes the plan, the executor decides the order of execution and compensation, not the model
Executes forward, persisting state after each step
On failure, runs compensations in reverse order, treating each compensation as an idempotent, retryable mutation
Escalates to a deadletter queue and a human when a compensation exhausts its retries
Logs every attempt and outcome to the agent's event log

The saga executor lets the agent reason about intent while the executor handles execution safety. Separating those two concerns, probabilistic intent from deterministic execution, is the design move.

Pattern 4: DryRun and Preview Modes

For highstakes tools, build a dryrun mode that returns "what would happen" without executing. The agent can chain dryruns to validate a plan before committing. As a bonus, this is what makes good explainability possible: show the user the dryrun trace before triggering the real one.

Pattern 5: Confirmation Buffers for Irreversible Actions

Many "irreversible" actions can be made softreversible by inserting a delay. Instead of sending an email or firing a webhook immediately, write it to a deferredaction queue with a short cancellation window. During that window the action can be cancelled, edited, or superseded, converting an irreversible operation into a recoverable one without changing the downstream system.

This is the single highestleverage technique for taming the irreversible class: a 30second deferred send turns "the email is already in their inbox" into "we caught it in the buffer." Pair it with humanintheloop confirmation for the actions where even a buffer isn't enough.

Pattern 6: The Outbox Pattern

If your agent's tool calls trigger downstream events (notifications, webhook fires, messagequeue publishes), use the outbox pattern: write the event to an outbox table in the same transaction as the state change, then have a separate publisher deliver it. This eliminates the dualwrite problem that plagues naively eventdriven agents.

Where MCP Fits, and Where It Doesn't

The Model Context Protocol standardizes how agents discover and invoke external tools. It's a protocol for tool discovery and invocation, and a useful one.

MCP does not standardize safety. It does not require tools to declare:

Their risk class
Their idempotency contract
Their compensating action
Their concurrency semantics

A tool exposed over MCP is exactly as safe as the tool itself: no more, no less. Adopting MCP and assuming safety comes with it is a real operational risk; a single network blip is enough to trigger a phantom retry against a nonidempotent endpoint.

The opportunity, however, is real: MCP server authors can voluntarily document safety contracts alongside their tool definitions. As an industry, we should push for that becoming convention. Until then, treat every MCP tool as riskclass unknown until proven otherwise. Trust is not a concurrency primitive.

The Safety Decision Matrix

Before adding any new tool to your agent's catalog, run it through this:

| Question | If YES → | | | | | Does this tool make a state change anywhere? | Treat as mutating | | Is the mutation reversible by another tool call? | Define the compensating action (and its idempotency contract) | | Can the tool be retried with no additional effect? | Softmutate (add idempotency key, minted outside the model) | | Could running it twice cause customer harm? | Hardmutate or irreversible: require saga or HITL | | Is the action irreversible but tolerant of a short delay? | Add a confirmation buffer to make it softreversible | | Does this tool depend on external state other agents can change? | Add optimistic concurrency control |

If you can't confidently answer all of these, the tool isn't safe to ship yet. That's not bureaucracy; it's the same bar we apply to any production code that touches external systems.

What to Build First

If your agent currently has tools but none of this infrastructure, build in this order:

A tool risk classifier. Even a YAML field per tool: risk: read | softmutate | hardmutate | irreversible. This is metadata. It's free. Add it.
An idempotency middleware. A key generator (owning keys outside the model) and a (key → outcome) store with TTL. The smallest version of this is a few dozen lines of code and prevents the entire class of duplicatesideeffect bugs.
A compensation catalog. For every hardmutate tool, write its compensating tool, and make that compensation idempotent and retryable. Even if you never need it, the act of writing it forces clarity about what the original tool actually does.
A saga executor. Start with synchronous, deterministic orchestration that owns sequencing and persists "compensation owed" state durably. Add a deadletter / escalation path for failed compensations. Add asynchronous execution later if you need it. Persist saga state to the same event log you built for episodic memory in the previous article.
Auditlog every tool invocation. Every attempt, every retry, every compensation. This is your debugging substrate and your compliance substrate at the same time.

Each of these is small in isolation. Together they form the safety envelope that makes agent actions trustable in production.

The Architect's Mental Model

The shift that separates teams shipping safe agents from teams that don't is this:

The model is a probabilistic component embedded in a deterministic system. The deterministic system is what makes the agent safe.

The LLM does the reasoning. The envelope around it (the idempotency layer, the saga executor, the compensation catalog, the risk classifier, the audit log) does the safety. The model proposes intent; the deterministic envelope owns execution, sequencing, and recovery.

That envelope is not optional, and it is not new. It's the same envelope distributed systems engineers built around databases in the ACID era, around microservices in the saga era, around eventdriven systems in the idempotentconsumer era.

The agentic AI era is the same problem with a new probabilistic component at the center. We don't need new fundamentals. We need to remember the ones we already have.

Reasoning is cheap. Side effects are forever. Design accordingly.

This is part 3 of Architecting Agents, a series on building agentic AI systems that survive production.

Reasoning Is Cheap, Side Effects Are Forever

The Two Categories of Tool Calls