Non-Determinism in LLM-Driven Agent Pipelines

A benefits appeal gets approved on Monday. The same fact pattern, submitted Tuesday, gets denied. Both decisions cite the correct policy. Nobody touched the code in between. Or picture a code review agent that flags a bug in one file during a morning run, then misses it entirely by afternoon, with no change to the pull request. Neither of these is a case of an LLM phrasing something a little differently. Both are the system reaching a different decision on identical input, and that is the subject of this piece: non-determinism in LLM-driven agent pipelines is a systemic architectural problem, not a model quirk that sampling controls can tune away.

Developers usually meet this as an unreproducible bug. Site reliability engineers look at the same incident and see nothing wrong at all: normal HTTP status, normal latency, no exception thrown anywhere in the stack. The outcome says broken.

The instinct is to reach for temperature. Set it to zero, pin the seed, assume the problem is solved. It helps, but it does not close the gap, because determinism breaks down at a level most engineers never look. When two top candidate tokens are close, that tiny shift is enough to flip which one wins. Add in batch composition changing how kernels reduce values, routing across data centers, mixture-of-experts models hitting capacity limits and dropping tokens, and techniques like speculative decoding and continuous batching, and the sources of drift multiply well past anything a seed value controls.

None of this means sampling controls are worthless. Pinning temperature, top-p, and seed is necessary. It just addresses one layer of a pipeline that has several, and the layers below sampling (tool calls, retrieval, state handoffs, memory) stay just as variable as before. Treating non-determinism as something to dial down at the model level misses where the real exposure sits: in the architecture that strings model calls together into a decision.

Cascading failures from compounding pipeline variance

Variance at one step in an agent pipeline rarely stays put. It moves forward, and every step downstream that depends on the drifted output inherits the drift along with it. That is what turns a small wobble in one token into a benefits appeal approved one day and denied the next.

Four sources of variance drive most of this. In the benefits appeal case, a early planning token choice could decide whether the agent checks the exception-handling policy first or the standard denial criteria first, and that ordering alone can change the outcome.

Retrieval variance compounds the same way. The code review agent that flags a bug in the morning and misses it in the afternoon may simply have chosen a different file to inspect first, with no instruction telling it to do otherwise.

State handoff variance is the fourth source, and it occurs in hierarchical pipelines where a Supervisor agent delegates subtasks to specialized workers. Each handoff is a point where whatever drift has already built up gets passed along, and the receiving sub-agent has no way to correct for drift it never sees. It sees only the state it was handed.

These four sources combine into trajectory divergence: identical input produces a different reasoning path on every run, shaped by whatever state, memory, and tool outputs happened to surface that time. When a deterministic pipeline breaks, it tends to break loudly, with a parse failure, a rejected validation, an alert firing somewhere. A non-deterministic pipeline drifts quietly instead: it produces output that looks complete and passes a surface-level health check while hiding the error underneath. A weak plan or a wrong tool call early in the run may not surface until much later. That delay is what makes root-causing these incidents so hard: the multi-step structure of the pipeline is the amplifier, not any single bad call inside it.

Memory and context management as a distinct propagation vector

Everything in the previous section happens within a single run. Memory is different: it is the layer through which non-determinism either gets contained or carried forward into every future session, because a memory layer that drifts or gets corrupted poisons every run that reads from it afterward.

An LLM has no memory between calls on its own. A bigger context window does not fix this, either. If the fifteenth most historically relevant memory keeps outranking something that just happened five minutes ago, that is a retrieval and curation failure, not a capacity problem, and adding more tokens to the window does not touch it.

Production systems tend to handle this with a two-tier model. Tier 1 is the context window acting as working memory: the recent turns, the current scratchpad, and a small set of memories retrieved specifically for the prompt in front of the agent. The retrieval step that decides what moves from Tier 2 into Tier 1 is itself a source of variance: whatever gets promoted into the working context changes what the agent reasons about on that turn. Two runs with identical Tier 2 contents can still behave differently depending on what got pulled forward.

The agent's message log, the append-only record of everything it has done, functions as its state. Treating context as a fixed prompt misses what it actually is in a production system: an evolving artifact that gets generated, reflected on, curated, and reused over time. When that curation process fails, every step downstream inherits corrupted state, and because memory persists across sessions, the damage does not stay contained to one run the way within-run variance does.

The cost of compounding non-determinism in production: debugging, compliance, and security

Compounding non-determinism produces three kinds of harm, and they do not arrive on the same schedule. Debugging cost appears daily, in the time engineers spend tracing a run. Compliance exposure becomes visible at audit time. Security harm can sit undetected for months. Each one gets more expensive as the pipeline grows more complex.

It takes longer to feel, but it costs more when it arrives. Financial, healthcare, and HR workflows commonly require that any transformation of a record be documented and stable. Traditional software executes logic that was written down in advance, but agents make runtime decisions, touch sensitive data, and take actions with real consequences, so they create governance obligations a deterministic system satisfies automatically and a non-deterministic one cannot.

Security harm is the quietest of the three and often the most expensive by the time it is found. Product teams run into a related problem in A/B testing: a treatment can appear to win on quality when it has actually just produced higher answer variance. Non-determinism can contaminate the measurement used to judge the system.

How the agent runtime layer bounds variance

None of the previous sections point back to the model as the fix. They point to the runtime, which controls execution flow, state persistence, tool dispatch, and error recovery across the whole pipeline. That is the layer with the structural reach to bound variance, because it is the only one that touches every step identified so far.

A purpose-built agent runtime breaks into four coordinating layers. A runtime layer keeps agents alive and coordinates multiple agents when they work on one task. A memory layer stores context and past actions so the agent carries state across sessions, starting each one with what it already knows. A model-routing layer picks among providers for each step, so you are not locked to one model regardless of task. A governance layer handles evaluations, access rules, audit logs, and risk controls. One pattern that depends on this governance layer working correctly is a plan-and-execute pattern, where a high-capability reasoning model breaks a request into a sequence of concrete subtasks that another component then carries out; without governance watching that handoff, there is no check on whether the plan that component received was sound.

Research on process-oriented systems (arXiv:2610.06255) frames the underlying issue clearly: LLM-based systems introduce variation into processes that were otherwise built to produce controlled, accountable state transitions. The fix it proposes is a controlled variation space, an explicit boundary that confines non-deterministic processing to defined regions and governs where the output of that processing is allowed to become authoritative. The controlled variation space is the design principle behind the next section: it marks which regions of the pipeline are non-deterministic, and requires their output to pass through a control gate before it can take effect.

Engineering interventions that reduce variance at each layer of the stack

You can reduce variance layer by layer if you match each intervention to the specific source it targets, instead of applying one fix everywhere and hoping it covers the rest.

At the sampling layer, the fix is pinning. Evaluation at this layer should compare for semantic equivalence rather than exact byte matches: run several rollouts per prompt and report the mean, the standard deviation, and the worst case for each metric, because a wide worst-case tail means the system is unsafe even when the average looks fine.

At the tool and action layer, constraining the action space produces the strongest results. OpenCodeReview (arXiv:2608.09290) treats non-determinism in code review agents as a design failure, not an inherent property of the model, and removes it along three axes. Tested on AACR-Bench, a benchmark of 200 real-world pull requests across 10 languages with 1,505 expert-verified comments, OpenCodeReview outperformed mainstream coding agents across six different LLM backends, scoring higher on SEM-F1 while using far fewer tokens. Constraining what the agent is allowed to do produced more depth and more reliability at lower cost, not a tradeoff between them.

At the execution harness layer, more designers now treat the harness itself, not the model, as the main place to control determinism. Harness engineering is what actually removes it, by fixing how tools get called and how outputs get validated before they count as final.

In practice, you design handoff points so an agent's non-deterministic output has to pass a deterministic validation gate before it can write to shared state or trigger anything downstream.

At the governance layer, the intervention is bounded autonomy. Clear operational limits, escalation paths to a human for high-stakes decisions, and full audit trails of agent actions reduce how far a drifted decision can travel before someone catches it. That is not only a compliance box to check. Bounding autonomy limits the blast radius of the variance that does slip through, which is the same goal the sampling, tool, harness, and process-boundary interventions are all working toward from different angles.

How observability makes non-determinism visible and measurable in production

Every intervention in the previous section can fail quietly if nothing is watching for it. Observability is what turns non-determinism from a risk a team assumes is handled into a property they can actually measure.

Standard application logs tell a team almost nothing when an agent hallucinates an answer, loops without end, or burns through its token budget for no visible reason. The structural unit for this is the parent-child span: each LLM call, retrieval step, or tool invocation becomes a child span nested under a parent span representing the user-facing operation. That hierarchy is what lets a team trace a spike in latency, cost, or variance back to the specific step that caused it.

Measuring non-determinism properly means running the same prompt multiple times and comparing the results, not relying on a single observation. For each prompt or task, running several rollouts and checking three things: answer agreement, whether repeated runs land on the same final answer, label, or JSON value; tool agreement, whether the same tool gets called with the same arguments and the same retry count; and trajectory agreement, whether the intermediate reasoning steps converge toward the same path or diverge early. And blaming temperature by default ignores that tool timeouts, cache misses, stale context, and provider-side routing can all produce the same divergence.

Evaluation needs to run at three levels at the same time: end-to-end, asking whether the task succeeded; trajectory-level, asking whether the path taken was efficient and sound; and component-level, asking which retriever, tool, or sub-agent introduced the variance. Gating releases on agreement thresholds across repeated runs is the more honest standard, and it is the only way to know whether the engineering interventions from the previous section are holding up over time.

A runtime designed around variance containment from the start

Bolting variance controls onto general-purpose cloud infrastructure runs into the same wall as bolting on memory or tool dispatch after the fact: the underlying substrate was never built around the constraint, so each control gets applied separately and nothing guarantees they work together.

Each layer covered here, sampling, retrieval, tool dispatch, state handoff, memory, and governance, needs its own containment mechanism, and those mechanisms have to coordinate. A governance gate does nothing if there is no audit log for it to write to. That coordination requirement is the real argument for a purpose-built agent operating system: the runtime, memory, model-routing, and governance layers need to be designed together, on the assumption that variance exists at every step and has to be actively contained.

If an agent cannot hold state across sessions, cannot run on a schedule, or cannot enforce a deterministic gate at each handoff point, you do not have a pipeline with a non-determinism problem waiting to be patched. It is a pipeline that was never built for production. Teams that choose the first path spend their engineering time on substrate problems. Teams that choose the second get to spend it on the agent logic that actually sets their product apart.