Agent Evaluations: Why Output-Only Checks Fail

The test passed. The customer support agent returned the correct refund status, and the automated evaluation script marked it green. But when we looked at the backend logs, the agent had spent forty seconds calling a deprecated database API six times while exposing an internal customer ID in a hidden scratchpad.

The system got the right answer, but the path it took was a disaster. If that agent had been live, it would have run up a massive API bill while exposing sensitive database fields. Yet, by every standard output-based evaluation metric, the test was a success.

This is the "right answer, wrong path" trap. As AI engineering teams move from simple chatbots to complex agentic workflows, traditional evaluation methods are breaking down. If you only evaluate an AI agent's final output, you are flying blind. You must evaluate the trajectory, not just the destination.

Agent Evaluations vs. LLM Evaluations

To build reliable systems, we must first understand how agent evaluations differ from traditional llm evaluation. Standard llm evaluation focuses primarily on static text tasks like summarization, translation, or RAG (Retrieval-Augmented Generation). In those setups, the input goes in, the model processes it, and the output comes out. The intermediate steps are entirely internal to the model's weights.

Agentic workflows introduce an entirely different operational paradigm. Agents are active decision-makers. They interact with the physical and digital world by calling APIs, querying databases, searching the web, and running code. They operate in loops, deciding what to do next based on the results of their previous actions.

Here is how the two approaches compare:

• Scope: LLM evaluation grades a single, static response. Agent evaluation inspects a multi-step sequence of decisions, tool calls, and state changes.
• Failure Modes: LLM failures are mostly limited to style, tone, or factual hallucination. Agent failures include infinite loops, broken tool arguments, security leaks, and redundant API calls.
• Primary Metric: LLM evaluation relies on semantic similarity or answer correctness. Agent evaluation requires measuring operational efficiency and path accuracy.

Why Output-Based Agent Evaluations Create False Positives

Because agents have operational freedom, they can fail in ways that simple LLMs cannot. They can hallucinate tool parameters, run into infinite loops, leak data, or call deprecated endpoints. Most critically, they can do all of this and still manage to patch together a correct-looking final answer.

An agent can reach the right final answer through entirely the wrong sequence of tool calls. This is a failure invisible to any check that only grades the final output.

Relying solely on final output checks for agent evaluations creates a false positive machine. Your test suite tells you everything is green, while your production environment is quietly accumulating technical debt and burning through API budgets.

The Problem with Static Benchmarks in Agent Evaluations

To combat this, many teams turn to static benchmarks or pre-built golden datasets. They run their agents through a battery of offline tests before deployment, hoping to catch regressions.

But static benchmarks decay rapidly when applied to dynamic agents. The Foundational Layer article by Supriya Rao cites an audit of 26 widely used AI benchmarks that found a median longevity score of just 5 out of 100. This is a staggering rate of decay, proving that PR-stage evaluations alone cannot catch agents that are quietly getting worse.

Static benchmarks fail because:

• They do not reflect live environments. An offline benchmark cannot simulate a slow database or an unexpected API rate limit.
• They do not account for tool drift. When a tool's behavior changes even slightly, an agent's trajectory will change.
• They encourage overfitting. Engineering teams naturally optimize prompts to pass the specific queries in the benchmark, creating a system that fails on the first novel user request in production.

Categorizing Agent Evaluations: Dev-Time vs. Production

To build a resilient agentic system, we need to divide our evaluation strategy into two distinct phases.

• Development-Time Evaluations: These are run offline during your CI/CD pipeline to catch functional regressions, prompt drift, and tool-definition errors before code is merged. Here, we rely on curated datasets of "golden trajectories" to ensure the agent's logic remains sound.
• Production Monitoring and Observability: This is continuous, real-time evaluation of live user interactions. Since agents operate in dynamic environments, offline testing is never enough. This is where ai agent observability becomes the critical enabler, providing the raw data needed to understand live agent trajectories.

The Trajectory Trap: How Agents Fail Silently

When an agent falls into the "right answer, wrong path" trap, it usually manifests in one of three silent failures.

First, there are inefficient loops. An agent might query a database, get a slightly malformed response, and instead of gracefully handling the error, query the database again with a tiny variation. It might repeat this cycle ten times before finally getting the data it needs. The final output is correct, but the latency is unacceptable, and your token bill is ten times higher than it should be.

Second, agents frequently call deprecated or incorrect APIs. If an agent has access to multiple search or database tools, it might choose an older, slower, or less secure tool because its system prompt is slightly ambiguous.

Third, agents often hallucinate tool parameters or leak data. An agent might retrieve an entire database row containing sensitive customer data, process it in its internal scratchpad, and then output only the specific, non-sensitive answer the user asked for. The final output looks perfectly safe, but sensitive data has now been processed by an external LLM provider and stored in your application's trace logs.

Transitioning to Trajectory-Based Agent Evaluations

If evaluating the destination is not enough, we must evaluate the path. This is the core of trajectory evaluation.

The contract tells us what the agent should be doing. The trajectory tells us what the agent actually did. Evaluation compares the two.

In this framework, the contract is the set of rules and expected behaviors we define for our agentic workflow. The trajectory is the actual sequence of states, tool calls, arguments, and intermediate thoughts the agent generated during execution. We are no longer just asking whether the agent solved the problem. We are asking whether the agent solved the problem efficiently and according to the rules we set.

Core Metrics for Agent Evaluations

To move from high-level philosophy to practical engineering, we must track specific, quantifiable metrics across every agent trajectory. We focus on three key metrics:

• Tool Call Accuracy: The percentage of tool calls made with valid, schema-compliant arguments. This helps catch parameter hallucinations before they cause silent database failures.
• Step Efficiency: The ratio of necessary steps to total steps taken. A low efficiency score indicates the agent is repeating actions or getting stuck in soft loops.
• Token-to-Task Ratio: The total number of tokens consumed divided by the number of successfully completed tasks. This is your primary economic metric, warning you when agents are becoming too expensive to run at scale.

A Framework for Modern Agent Evaluation Strategy

Transitioning to trajectory evaluation requires a shift in both tooling and mindset. Here is how we can build a robust evaluation strategy for agentic workflows.

1. Implement Comprehensive Trace Logging

The foundation of trajectory evaluation is deep, structured tracing. Every single step the agent takes must be captured. This is where AI agent observability becomes essential. We need to log the exact prompt sent to the LLM at each step, the raw text returned, the specific tool selected, the arguments passed to that tool, the tool's response, the latency of each call, and the token consumption.

2. Define Step-Level Assertions

Just as we write unit tests for software, we must write assertions for our agent's trajectories. For example, we can assert that the agent never calls a specific database tool more than twice per run, or that it never passes null values to a critical payment API. If an assertion is violated, the evaluation fails, even if the final output is correct.

3. Use LLM-as-a-Judge for Trajectory Analysis

Some aspects of a trajectory are too complex for simple code assertions. For instance, did the agent use a logical reasoning path, or did it make wild, erratic guesses before finding the answer? We can use a separate, highly capable LLM as an evaluator. Instead of showing the evaluator just the final output, we feed it the entire trace of tool calls and intermediate thoughts. We ask the evaluator to grade the efficiency and safety of the trajectory.

4. Build Human-Verified Golden Trajectories

While AI-as-a-judge is highly scalable, human-in-the-loop validation remains the gold standard. We must establish a process where domain experts review complex traces and flag them as "golden trajectories." These human-verified paths serve as the baseline for our development-time evals, ensuring our automated judges are aligned with actual human expectations.

5. Establish Resource and Latency Budgets

A successful agentic workflow must be economically viable and performant. We should establish strict budgets for token spend and execution time per run. If an agent takes fifty tool calls and forty seconds to solve a simple query, that run is a failure. Tracking these metrics as part of our core agent evaluations helps us catch performance degradation before it impacts production budgets.

6. Monitor Production Trajectories Continuously

Since static benchmarks decay so quickly, we need a continuous feedback loop. We should monitor production trajectories and flag unusual patterns, such as runs with high step counts or frequent tool errors. These problematic trajectories should be sanitized, annotated, and added directly to our evaluation suite. This ensures our test coverage evolves alongside actual user behavior and real-world edge cases.

Frequently Asked Questions about Agent Evaluations

What is the difference between LLM evaluation and agent evaluation?

Standard LLM evaluation measures the quality and accuracy of a single static text output. Agent evaluation inspects the entire sequence of steps, tool calls, and decisions an autonomous system takes to reach a goal, focusing on operational efficiency and execution safety.

What is trajectory evaluation?

Trajectory evaluation is an evaluation methodology that analyzes the complete execution graph of an agent (including every intermediate thought, tool call, argument, and state transition) rather than just grading the final text output.

Why do static benchmarks fail for AI agents?

Static benchmarks decay rapidly because they cannot simulate dynamic live environments, unexpected API rate limits, or tool drift. They also encourage overfitting, where developers optimize prompts to pass specific static tests while failing on novel real-world user queries.

The Bottom Line

Evaluating AI agents on final output alone creates a dangerous illusion of stability while hiding chaotic and expensive behavior under the hood. By shifting to trajectory evaluation, we can inspect the actual path our agents take, ensuring they are not just stumbling into the right answers, but executing them reliably and efficiently.