Evaluating and Debugging LLM Agents: Strategies for Reliability in Non-Deterministic Systems

The hardest part of agent development is testing. Given the same input, an agent may call different tools or reason in a different order. Traditional software testing methods do not work.

Classifying Agent Failure Patterns

Failure type	Symptom	Cause
Wrong tool selection	Calls the wrong tool	Unclear description
Argument error	Wrong parameters	Unclear schema
Infinite loop	Repeatedly calls the same tool	No stop condition
Early termination	Answers before the task is complete	Missing validation
Hallucination	Ignores tool results and fabricates	Prompt design issue

Trace-Based Debugging

Python

import time, json
from dataclasses import dataclass, field, asdict

@dataclass
class ToolCall:
    tool_name: str
    arguments: dict
    result: object
    duration_ms: float
    error: str = None

@dataclass
class AgentTrace:
    session_id: str
    user_input: str
    final_output: str
    tool_calls: list = field(default_factory=list)
    total_duration_ms: float = 0
    success: bool = True

class TracedAgent:
    def __init__(self, agent):
        self.agent = agent

    def run(self, user_input):
        session_id = f"trace_{int(time.time())}"
        start = time.time()
        trace = AgentTrace(session_id=session_id, user_input=user_input, final_output="")

        original_dispatch = self.agent.dispatch_tool

        def traced_dispatch(name, args):
            t0 = time.time()
            try:
                result = original_dispatch(name, args)
                trace.tool_calls.append(ToolCall(
                    tool_name=name, arguments=args, result=result,
                    duration_ms=(time.time() - t0) * 1000
                ))
                return result
            except Exception as e:
                trace.tool_calls.append(ToolCall(
                    tool_name=name, arguments=args, result=None,
                    duration_ms=(time.time() - t0) * 1000, error=str(e)
                ))
                raise

        self.agent.dispatch_tool = traced_dispatch

        try:
            trace.final_output = self.agent.run(user_input)
        except Exception as e:
            trace.success = False
            trace.final_output = str(e)

        trace.total_duration_ms = (time.time() - start) * 1000
        return trace

Unit Testing Tool Calls

Python

import pytest
from unittest.mock import patch

class TestToolSelection:
    def setup_method(self):
        self.agent = MyAgent()

    def test_stock_query_calls_stock_tool(self):
 """ when get_stock_price call """
        with patch.object(self.agent, 'get_stock_price', return_value={"price": 100}) as mock:
 self.agent.run(" ")
            mock.assert_called_once()

    def test_no_tool_for_greeting(self):
 """ call """
        with patch.object(self.agent, 'dispatch_tool') as mock:
 self.agent.run("")
            mock.assert_not_called()

    def test_parallel_calls_for_multi_query(self):
 """ when call """
        with patch.object(self.agent, 'get_stock_price', return_value={"price": 100}) as mock:
 self.agent.run(" SK compare")
            assert mock.call_count == 2

LLM-as-Judge Evaluation

Python

import anthropic, json

eval_client = anthropic.Anthropic()

EVAL_PROMPT = """ reply 1-5 .

[] {question}
[ reply] {response}
[ ] {ground_truth}

 :
- (40%), (30%), (30%)

JSON : {{"score": 1-5, "reason": " "}}"""

def evaluate_response(question, response, ground_truth):
    result = eval_client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": EVAL_PROMPT.format(
                question=question, response=response, ground_truth=ground_truth
            )
        }]
    )
    return json.loads(result.content[0].text)

def run_evaluation(test_cases):
    scores = []
    for case in test_cases:
        response = agent.run(case["question"])
        eval_result = evaluate_response(case["question"], response, case["expected"])
        scores.append(eval_result["score"])

    return {
        "avg_score": sum(scores) / len(scores),
        "pass_rate": sum(1 for s in scores if s >= 4) / len(scores)
    }

Production Monitoring

Key metrics:

Python

metrics = {
 "tool_call_rate": "call — when ",
 "tool_error_rate": " call failed — external ",
 "avg_latency_ms": " reply when — ",
 "fallback_rate": " without quit — ",
 "user_retry_rate": " when — when",
}

#
alerts = {
 "tool_call_rate > 10": " , when ",
 "tool_error_rate > 5%": "external ",
 "avg_latency_ms > 10000": "SLA ",
}

Agent testing can never be perfect, but with trace analysis, tool-level unit tests, and LLM-as-Judge in place, you can catch most production issues before they ship.

Debugging Decision Table by Symptom

Symptom	Look here first	Action
Same input, intermittent failures	Tool-call arguments in the trace	Lower temperature; add examples to the tool schema
Always fails on a specific tool	Unit tests for that tool	Separate a bug in the tool itself from errors in the model's argument generation
Unusually long step count	Planning-stage output	Cap max steps + add a prompt that reconfirms intermediate goals
Quality degrades gradually	Evaluation-set score trends	Compare by rolling back recent changes to the model, prompt, or docs
Sudden cost spike	Per-session token logs	Detect loops; review the policy for resetting accumulated context

Minimum Evaluation Metrics (Start Here)

Task success rate — pass rate on 20–50 representative scenarios (deployment gate)
Step efficiency — average tool-call count on successful cases (regression detection)
Judge agreement rate — when using LLM-as-Judge, re-validate sample agreement with human raters every quarter
Without metrics, there is no improvement — document a manual evaluation rubric before you automate.

#LLM #Agents #Observability

✦ ✦ ✦

Sources and verification

Sources and checks vary by article. Only articles with a recorded check date, scope and sources below were checked, and only within that scope. For other articles, verify against the current docs for your version before applying anything, and email us about errors. Editorial policy · email us

This article has no recorded human check (date and scope) yet.

Published · May 20, 2026

Topic-related official docs (auto-linked · not per-claim sources)NIST CSRC ↗

How we write and review →Editorial policy →

Was this helpful?

Comments

Be the first to comment.