Fault-Tolerant Execution Guarantees for Large Agent Fleets

Persistent and scheduled autonomous agents at scale need an execution layer that preserves state, repairs partial work, and limits irreversible effects across the fleet. For the infrastructure to be reliable, all three must be built into the runtime, beyond what retries, prompts, or controls around individual tool calls can do.

Why large agent fleets face a reliability crisis that incremental fixes cannot solve

Agents move money, deploy code, delete data, and share information. Their failures reach beyond the process running them. A refund agent may validate an order, reverse the charge, update the CRM, and send confirmation. A network timeout at the CRM step leaves the charge reversed while the customer remains uninformed. Better prompting cannot make those separate external systems agree about what happened.

Traditional orchestration frameworks often treat an agent pipeline as a single-process concern, with state in memory and visibility limited to framework logs. A worker failure can therefore interrupt the whole pipeline. Temporal’s Series D announcement identifies the surrounding execution systems, rather than model quality, as the obstacle to handling agentic AI in real-world conditions. The distinction becomes sharper as fleets grow, because a bug that causes one agent to retry can cause a thousand-agent fleet to produce a correlated burst.

The costs are not confined to interrupted tasks. A 2026 empirical study of 63 confirmed production incidents catalogs retry-loop cost overruns as a production failure class. Gartner predicts that 40% of agentic AI projects will be scrapped by 2027 because of escalating costs, unclear business value, and inadequate risk controls.

The three failure classes are structurally distinct and require separate solutions

A visible crash shows that execution has stopped. A worker dies, a timeout fires, or a process exits. Recovery is possible if the system persisted its state before the interruption. Without a committed checkpoint, though, a restart means either doing earlier work again or losing state.

An agent completes some steps, then reports success or simply stops while external systems disagree. The refund has gone through, but the CRM has not changed. In a trip-booking workflow, the flight and hotel may be booked while the car booking fails. Recovering consistency may require undoing successful steps.

What if the action cannot be fully undone? Money moved, code deployed, data deleted, and information disclosed can create exposure that recovery cannot erase. The runtime must constrain that exposure before commitment, including when individually permitted actions accumulate across agents. Durable journals restore lost execution state, compensation protocols undo completed work that is no longer wanted, and admission control limits irreversible exposure. None of those primitives substitutes for the other two.

Durable execution handles visible crashes by treating every committed step as a checkpoint

Under “Durability by Default,” each action that changes pipeline state is committed to a distributed log before execution. Agents act as deterministic replayers, so a replacement worker given the same log prefix can reproduce the recorded execution. A failed worker need not take pipeline state with it.

The Fault-Tolerant Distributed AI Pipeline Harness (DAPH) provides a concrete architecture for this approach. It separates Planner, Generator, and Evaluator roles, connects them through an append-only distributed event log, and uses a Raft-based coordinator for consensus. The DAPH evaluation reports a median fault recovery time of 1.2 seconds following a Generator node failure. Actor-model isolation adds another boundary by placing workers in lightweight, supervisor-monitored processes, allowing a crashed agent to restart without contaminating sibling state.

Durable execution provides “at least once” delivery and can fall short of “exactly once,” because a downstream service may already have processed a call when the worker loses contact. Idempotency keys let that service recognize a repeated request and return the original result. The journal establishes where execution should resume, while idempotency protects against duplication at the external boundary. On its own, neither can repair a workflow when its completed steps are no longer wanted.

Compensation protocols, not retries, are the right tool for silent partial completions

If a trip cannot proceed after the car booking fails, retrying that booking leaves the flight and hotel commitments in place. Compensation instead reverses completed steps in a defined order to restore consistency. Idempotency prevents a duplicate charge, but it does not decide which existing charges must be reversed.

Perera et al.’s Robust Agent Compensation (RAC), presented at ACM CAIS’26, makes that distinction part of execution. The log-based recovery approach wraps agent workflows with compensation logic through existing framework extension points. Users can opt in without changing their current agent code. In the booking example, recovery reverses the hotel and flight charges rather than merely stopping at the failed car reservation.

Could the model simply work out what to fix? The RAC paper reports substantially better latency and token economy on complex problems than LLM-based recovery approaches. Following a committed compensation log avoids asking the model to reconstruct the repair plan. That log must reflect the actual execution path, because ReAct agents and agents that plan before executing determine their order of work at runtime. Circuit breakers complement this recovery mechanism at the network boundary by diverting calls to a fallback when a tool or sub-agent sustains a high error rate.

Per-effect gates fail at fleet scale because they cannot see correlated, cross-agent exposure

Even when every step is allowed and finishes, the result can be unacceptable. Mohammadi & Bindschaedler’s “The Irreversibility Budget,” presented at AgenticOS @ SOSP 2026, describes procurement agents that each manage a product line within an individual purchasing cap. A shared supplier price spike causes them all to buy at once. Every order passes its local check, yet the fleet commits a position that no reviewer approved.

The study makes the size of this gap concrete. Its controlled experiments found that per-effect gates admitted fleet-level overdraws of up to 48× the tenant’s risk limit. The average overdraw was more than double the limit and grew as the fleet grew. No worker needed to crash, and no transaction needed to remain incomplete, for the shared risk constraint to fail.

This is not only a problem of correlated demand. A compromised or misdeclaring model can split an effect into smaller calls, passing each local cap while building exposure that the principal never authorized. A gate that checks one effect keeps no running balance across the fleet. Compensation cannot close this gap when the resulting effects are irreversible, so enforcement must happen before commitment and outside the agent’s control.

Treating irreversibility as a first-class runtime resource closes the fleet-level gap

An irreversibility budget gives the runtime a shared account for each principal. Each effect consumes part of that account, and the runtime denies or escalates the next effect if admitting it would overdraw the balance. The charge is based on the residual loss expected to remain after recovery, which can differ from the effect’s face value. That distinction allows different kinds of effects to draw on a common risk budget.

Memory accounting keeps one process from using shared RAM without regard for the others. In the same way, irreversibility accounting keeps a fleet run from using up a principal’s risk allowance even when each agent stays within its local cap. When Fifty procurement agents buy at the same time, the runtime makes one admission decision covering all fifty purchases.

Enforcement belongs below the agent, in a trusted runtime. Splitting calls still draws on a shared balance when every effect is correctly charged, and local controls become spending policies over that balance. The ledger is cheap to maintain, but pricing remains difficult. The same paper identifies misdeclared effect types and correlated losses as ways realized loss can exceed the amount accounted for. A runtime can enforce the budget it has been given without that budget accurately representing exposure, which leaves conservative, dependency-aware pricing as an open research problem.

How do you run hundreds of long-running AI agents reliably without losing state?

The baseline is a runtime that co-designs durable checkpointing, compensation-aware execution, and irreversibility accounting. Its append-only distributed log, deterministic replay, per-agent identity, and isolation provide the foundation for crash recovery. Raft or equivalent consensus keeps workers in agreement on pipeline state. These execution guarantees are useful from the first deployment.

The log must also record how completed work can be compensated. That metadata lets rollback follow the agent’s actual path. At the external boundary, a trusted ledger must keep a balance for each principal and use residual-loss pricing before it allows irreversible effects. These mechanisms need to agree about execution, recovery, and commitment, because a runtime with only two still leaves the third failure class open.

Operators also need visibility across those boundaries. End-to-end tracing through OpenTelemetry or an equivalent system lets a faulty LLM output be traced to the agent invocation that produced it. Without that connection, distinguishing a crashed worker from incomplete recovery or excessive aggregate exposure becomes harder. Locus DiSCO’s multi-agent logistics system illustrates the coordination demands, with hundreds of enterprise deployments in dozens of countries and hundreds of carriers under orchestration. Specialized agents collaborating on fleet deployment decisions need a shared execution foundation.

Why retrofitting general-purpose cloud tooling cannot satisfy these requirements

General-purpose cloud primitives can provide processes, storage, and communication, but changing their settings alone cannot give them these agent-specific guarantees. In a monolithic framework, adding an external log leaves a consistency boundary between the record and the state held in memory. Recording events is not equivalent to making the journal authoritative. Recovery depends on that distinction.

A record of what happened does not necessarily describe what undoing it means. RAC shows that framework extensions can add compensation without changing agent code, but this depends on an architectural recovery layer. Likewise, static workflow engines designed around deterministic DAGs do not naturally capture execution paths chosen dynamically through agent reasoning.

Irreversibility control imposes a separate trust requirement. A policy attached to an API gateway is insufficient if agents can route around it or configuration leaves effects outside enforcement. The control must mediate effects below the agent. The historical OS analogy helps explain the required change, as uniform resource management emerged through a new layer rather than through patches to applications managing their own memory. Cloud components may support that layer, but assembling them does not itself establish its guarantees.

What a purpose-built agent operating system looks like in practice

A purpose-built agent operating system is a policy-controlled execution environment for bounded, auditable, and recoverable actions. Individual agents run in isolated sandboxes with their own identity and access management. That identity supports both crash isolation and attribution to a principal’s risk ledger. A policy engine governs compliance rules, resource allocation, and escalation thresholds, while the runtime enforces them below the agent.

Persistent context is another part of recoverable execution. Intentional memory tiers include in-context working state, external key-value storage, episodic logs, and semantic vector memory. These let a restarted agent reconstruct its position. For proactive agents that wake on a schedule, infrastructure must therefore govern when a task starts and what context and execution guarantees remain available when it resumes.

The AgenticOS Workshop at SOSP 2026 in Prague on September 29, 2026 places this approach within a wider research agenda. Its work includes the irreversibility paper, Patient Bytes, a reliability-budgeted scheduler, and Externalization Barriers, an OS abstraction for untrusted agent exploration. A ranked shortlist of production platforms would require evidence beyond these research directions. The practical adoption requirement is less abstract, however, as developers need an environment they can install and run against without a long integration project. Runtime guarantees must be active from the first deployment, before agents begin changing the world.