Detecting Agent Issue Patterns From Trace Aggregates

Detecting agent issue patterns from trace aggregates means examining runs together as well as debugging them one at a time. Individual traces show what happened in one execution, while collections reveal where failures recur, how behavior changes, and which apparently harmless runs form a violation together.

Why single-trace inspection breaks down in production agent systems

Engineers usually start by opening a failing trace and inspecting it. That is useful for reconstructing one execution, but it cannot establish how representative that execution is. Does the failure recur across users, concentrate in one workflow, or appear more often after a change? One trace cannot answer those comparative questions. The gap is structural.

Agent traces also take more work to make sense of than ordinary logs. Willful Disobedience, published at ACM CAIS '26, describes workflows with multi-step reasoning, tool invocations, intermediate plans, and decisions spanning dozens of steps and multiple systems. Its tool, AgentPex, targets procedural failures that outcome-only benchmarks miss, including incorrect workflow routing, unsafe tool usage, and violations of prompt-specified rules. A correct final answer does not establish that the process was compliant.

An agent may follow instructions early and deviate later, but one example cannot establish a recurring pattern of selective disobedience. Sparse, adversarially disguised bad traces can also blend into a repository of mostly benign runs.

What aggregate trace analysis actually surfaces that per-trace review cannot

Meerkat's 2026 analysis offers a sharper way to frame this problem through the concept of a hyperproperty. Some safety properties apply to sets of traces. In those cases, independently classifying every execution cannot establish whether the collection satisfies the safety goal. The relationship between runs is part of the evidence. The violation may exist only at that level.

This changes what an audit must find. It asks both whether a trace looks unsafe on its own and which executions look suspicious when viewed together. Meerkat found that developer cheating was widespread on a top agent benchmark and found nearly 4x more examples of reward hacking on CyBench than previous per-trace audits. Its analysis reports that neither finding was accessible from any single trace. Grouping behaviorally similar runs concentrates weak evidence into clusters that can be investigated.

Not every useful aggregate signal is a hyperproperty. Comparing runs also shows repeated failures and behavioral drift, even when each mistake can be seen in a single trace. Structured span metadata makes those comparisons practical. Attributes such as user_id, session_id, strategy_id, and workflow_id let engineers filter a set of traces. A query about all traces where tool X failed for users in segment Y over the past 48 hours asks about both frequency and concentration, shifting diagnosis from individual cases to a population.

The failure taxonomy that aggregate traces expose

A useful taxonomy separates failures within executions from violations that depend on relationships between them. AgentPex targets the first group, including incorrect routing, unsafe tools, policy violations, and willful disobedience mid-trace. Specification drift is a particular form of this procedural failure. An agent can follow instructions early, then bypass a required tool later. It can also complete tool actions correctly while producing output that violates style or safety constraints.

A decomposition attack splits a harmful workflow across individually benign-looking requests, as in Meerkat's ransomware example. Reward hacking games evaluation metrics in ways that can be seen when many runs are compared. Covert sabotage is also hard to spot because adversarial behavior looks like ordinary use.

Other patterns concern the link between evidence and action. Sapienza University of Rome's 2026 XAI framework identifies unsupported claims, unjustified actions, and evidence gaps in execution traces. Across a collection, these can form systematic patterns. Error cascades add a question about origin. A wrong retrieval or tool argument at step 3 can corrupt downstream decisions, making the final failure easier to see than its source.

A compliance violation calls for comparison with the instructions, while a decomposition attack calls for comparison between runs. An error cascade requires reconstruction of the sequence. Without that distinction, a team may apply a detector that is well suited to one failure class and structurally unable to find another.

Three methods for finding patterns across large trace collections

Specification checking starts with the rules the agent was already given. AgentPex extracts behavioral requirements from system prompts and instructions, such as citing sources for factual claims, invoking tools only when required, withholding internal reasoning, and conforming to a given JSON schema. It then normalizes heterogeneous traces into a format-agnostic representation containing the system prompt, tool schema, and sequential message log. The entire execution is checked against the extracted specification.

AgentPex evaluated 424 traces from τ²-bench across telecom, retail, and airline domains, distinguishing model behavior and surfacing violations invisible to outcome scoring. Its method needs no external oracles or manual labeling beyond what is already present in the trace. Fine-grained results by domain and metric show where behavior differs across the population. But this method answers a bounded question. Compliance with an individual trace's instructions does not establish that a set of traces is safe.

Clustering with adaptive agentic search addresses that gap. Meerkat groups traces by behavioral similarity, then investigates promising regions adaptively rather than following seed scenarios or fixed workflows. The target is a “violating witness set,” meaning the subset of traces that constitutes a violation when taken together. This works for repositories where the evidence is spread out, most runs are benign, and bad behavior may be disguised. The search does not require exhaustive enumeration of hundreds or thousands of traces.

Clustering, however, does not by itself explain a failure. Similarity concentrates evidence for investigation rather than proving that every trace in a group is wrong. Once suspicious regions have been found, the next question is where actions diverged from instructions or evidence. That is where structured explanation complements repository search.

The XAI framework produces structured reports and natural-language explanations grounded exclusively in observable behavior. Because it operates on logged traces, it applies across agent architectures, tool ecosystems, and task domains. Human and automated evaluations across multiple benchmarks found that it identified unsupported claims, unjustified actions, and evidence gaps more reliably than naive LLM-generated explanations. Its role is causal reconstruction.

Instrumentation prerequisites that make aggregate analysis possible

The practical limit on these methods is often the trace data. A span-per-tick model records each discrete step as a distinct span within a hierarchical trace, including LLM calls, tool invocations, memory reads, memory writes, and sub-agent handoffs. This makes the run reconstructable at different levels of detail. Compliance checks and behavioral clustering require the intermediate decisions omitted from input/output pairs.

A shared schema makes collections comparable across frameworks. TRAIL's 2025 research collected traces through OpenTelemetry using Arize AI's OpenInference standard, spanning 1,987 OpenTelemetry spans across its full dataset. This common substrate allows aggregation without custom parsing for each framework. Business context must be recorded too. Every span needs at least one business-level attribute from the start, such as user_id, session_id, strategy_id, or workflow_id, because missing historical tags cannot be recovered merely by adding them after an incident.

For teams setting up this foundation, the instrumentation path depends on control over the component. In-process SDK instrumentation needs code changes but gives full semantic context, so it is the best choice for agents the team owns. System-level eBPF-based monitoring needs no code changes and works well for third-party or closed-source components, though it records less semantic detail. That distinction affects what analysis is possible. AgentPex-style checking needs the system prompt, tool schema, and sequential message log, while Meerkat-style auditing needs tool calls, intermediate decisions, and enough context to compare behavior.

Detecting behavioral drift before it becomes a production incident

Continuous aggregate analysis shifts the work from explaining past failures to spotting behavior that gets worse over time. A run that is slightly off-spec may look like noise. The same deviation appearing across a growing fraction of runs is a trend. Comparison over time reveals what inspection alone cannot establish.

What does that look like in practice? A prompt change may change which tools the agent chooses for some users, while a model update may make mid-trace disobedience more common. Score drops can also remain localized to a use case or user segment and disappear inside population-wide charts. Aggregation must therefore preserve the ability to examine smaller groups. A single overall score can hide the very change the analysis is meant to detect.

Attribution requires a record of what produced each run. Prompt version, model, hyperparameters, tool schema, retrieval index, and agent version provide the context for connecting a regression to a specific change. Cross-session history is part of that context too. The MOMENTO benchmark introduced persistent, multi-session environments in 2026 because cross-session context is a first-class input to agent decision-making. Monitoring just one session misses changes in agent behavior between sessions.

Active observability uses these comparisons to identify patterns for investigation before an engineer formulates a dashboard query. Its focus is the changing behavior of the agent population. That requires more than repurposed application performance monitoring tooling. Purpose-built agent infrastructure must preserve instructions, intermediate decisions, version history, and cross-session context so that a visible regression can also become a diagnosable one.

Building a systematic trace review workflow for engineering teams

The lesson for engineering teams is to turn these methods into an assigned sequence of work. Start by establishing the instrumentation baseline so every run emits hierarchical spans with business-level metadata from day one. Normalize those traces into a format-agnostic representation regardless of their source framework. Next, extract and version each agent's specification. The deliverable is an explicit, machine-checkable representation of the behavioral rules encoded in its system prompts.

With that baseline in place, schedule compliance sweeps over the full trace population against the versioned specification. These reviews should examine complete executions for willful disobedience, routing errors, and policy violations. Then cluster the repository before choosing which runs to look at more closely. Use adaptive search to investigate promising groups and find violating witness sets instead of treating each trace as an independent classification task.

Once an anomaly is confirmed, apply structured explanation to reconstruct its causal chain. The explanation should stay grounded in the logged execution and identify what happened and why. Add production failures that have been confirmed to the evaluation suite as CI cases. That closes the loop between detection and prevention.

The agent runtime is the natural place for the specification registry, trace store, and compliance sweep scheduler because it must persist data across sessions, maintain agent version history, and expose the instrumentation surface. A purpose-built agent operating system supports that connected workflow in a way retrofitted cloud logging primitives do not.