AI Agent Reliability Verification Guide, Part 2: Failure Detection and Auto-Recovery Strategies

Why Conventional Tests Don't Work for Agents

The premise of unit testing is determinism: same input → always the same output. LLM agents, however, produce different results depending on temperature, context noise, and model updates. This installment covers a testing strategy for systematically verifying non-deterministic systems.

The Four Layers of Agent Testing

Layer 4: End-to-End scenario
Layer 3: call
Layer 2:
Layer 1: /

You have to stabilize the lower layers first; otherwise, tests at the higher layers are meaningless.

Layer 1: Output Schema Validation

from pydantic import BaseModel, ValidationError

class AgentResponse(BaseModel):
    action: str
    parameters: dict
    confidence: float  # 0.0 ~ 1.0

def test_schema(raw_output: str):
    try:
        r = AgentResponse.model_validate_json(raw_output)
        assert 0.0 <= r.confidence <= 1.0
        return True
    except ValidationError as e:
        print(f"Schema error: {e}")
        return False

Enforcing Structured Output (JSON mode) can drive this layer's failure rate close to zero.

Layer 2: Probabilistic Assertions

Run the same input N times, then verify statistically.

async def probabilistic_assert(agent, prompt, expected, n=20, threshold=0.8):
    results = await asyncio.gather(*[agent.run(prompt) for _ in range(n)])
    rate = Counter(r.action for r in results)[expected] / n
 assert rate >= threshold, f" {rate:.0%} < {threshold:.0%}"

Threshold guidelines:

• Critical path: 0.95 or higher
• General functionality: 0.80 or higher
• Creative generation: 0.60 or higher (use alongside qualitative evaluation)

Layer 3: Tool-Call Accuracy Tests

class ToolCallRecorder:
    def __init__(self): self.calls = []
    def __call__(self, tool_name, **kwargs):
        self.calls.append({"tool": tool_name, "params": kwargs})
        return mock_response(tool_name, **kwargs)

def test_tool_selection():
    recorder = ToolCallRecorder()
    agent = build_agent(tools=recorder)
 agent.run(" ID 12345 ")
    assert recorder.calls[0]["tool"] == "get_orders"
    assert recorder.calls[0]["params"]["customer_id"] == "12345"

Layer 4: Replay Tests (LangSmith)

from langsmith import Client

client = Client()
def export_golden_tests(project: str, limit=100):
 #
    runs = client.list_runs(
        project_name=project,
        filter="eq(feedback_score, 1)",
        limit=limit
    )
    return [{"input": r.inputs, "expected": r.outputs} for r in runs]

Accumulating positive user feedback as a golden dataset enables automatic regression testing whenever the model is updated.

CI Pipeline Integration

# .github/workflows/agent-test.yml
on: [pull_request]
jobs:
  test:
    steps:
 - run: pytest tests/test_schema.py # ,
 - run: pytest tests/test_tool_calls.py # PR
 - run: pytest tests/test_replay.py #

Part 3 covers Circuit Breaker and Fallback patterns that automatically recover when an agent that passed tests still fails in production.

When Replay and Probabilistic Tests Fail Intermittently

Once agent tests are in CI, the first problem is usually "flaky" tests that pass and fail with no code change.

Causes

• A model alias (a name that points to the latest version) is moved to a new snapshot by the provider
• Tool responses depend on time or external data and change every run
• temperature isn't 0, or even at 0 the provider doesn't guarantee fully fixed output

Reproduce: run the same test N times and measure pass rate

for i in $(seq 1 20); do pytest -q tests/agent/test_refund_flow.py >/dev/null && echo pass || echo fail; done | sort | uniq -c

If the pass rate isn't 100%, treat that test as probabilistic, not deterministic.

Fix

• Pin models to dated snapshot names and change versions in a dedicated PR.
• Freeze tool responses with record/replay: save external responses to files on the first run and return the files afterwards, removing every source of variation except the model.
• Judge probabilistic tests by threshold, such as "at least 18 of 20 passes," and state the threshold in the test code.

Prevention

• Move tests that fall below threshold to a quarantine list with an owner and deadline. A growing quarantine list is a quality signal for model or prompt changes.