AI Agent Reliability Verification Guide, Part 3: Production Monitoring and SLA Management

Production Agents Will Inevitably Fail

LLM API timeouts, tool errors, infinite loops — even well-designed agents fail in unexpected ways in production. You need a recovery architecture so the entire system does not go down.

LLM Agent Failure Types

Type	Example	Frequency	Impact
API timeout	OpenAI exceeds 30s	Medium	High
Model refusal	Content policy violation	Low	Medium
Tool error	DB connection failure	High	High
Infinite loop	Repeatedly calling the same tool	Low	Critical
Context overflow	Token limit exceeded	Medium	Medium

Pattern 1: Circuit Breaker

When repeated failures are detected, automatically cut off that path to protect the entire system.

Python

from enum import Enum
from datetime import datetime, timedelta

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class AgentCircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.last_failure_time = None

    async def call(self, agent_fn, *args, **kwargs):
        if self.state == CircuitState.OPEN:
            if self._should_recover():
                self.state = CircuitState.HALF_OPEN
            else:
                raise Exception("Circuit breaker OPEN")
        try:
            result = await agent_fn(*args, **kwargs)
            self.failure_count = 0
            self.state = CircuitState.CLOSED
            return result
        except Exception:
            self.failure_count += 1
            self.last_failure_time = datetime.now()
            if self.failure_count >= self.failure_threshold:
                self.state = CircuitState.OPEN
 print(f" OPEN: {self.failure_count} failed")
            raise

    def _should_recover(self):
        elapsed = datetime.now() - self.last_failure_time
        return elapsed > timedelta(seconds=self.recovery_timeout)

Pattern 2: Fallback Chain

When the primary path fails, try alternatives in sequence.

Python

async def query_with_fallback(query: str) -> str:
    chain = [
 lambda q: claude_opus(q), # 1:
 lambda q: claude_sonnet(q), # 2:
 lambda q: cached_response(q), # 3: when
 lambda q: rule_based(q), # 4:
    ]
    for i, fn in enumerate(chain):
        try:
            return await asyncio.wait_for(fn(query), timeout=10.0)
        except Exception as e:
            if i > 0:
 print(f"Fallback level {i} ")
 return "when after when when."

Pattern 3: Graceful Degradation

Instead of shutting everything off, keep only core functionality.

Python

class DegradedService:
    def __init__(self):
        self.features = {
            "tool_use": True,
            "memory": True,
            "personalization": True
        }

    async def handle(self, request):
        if not self.features["tool_use"]:
            return await self.knowledge_only(request)
        return await self.full_agent(request)

    def disable(self, feature: str):
        self.features[feature] = False
 print(f" : {feature}")

Preventing Infinite Loops

Python

MAX_STEPS = 15

async def safe_run(agent, query):
    steps = 0
    async for step in agent.stream(query):
        steps += 1
        if steps >= MAX_STEPS:
            return {
                "result": step.partial_result,
 "warning": f" ({MAX_STEPS}) — Result "
            }
    return step.final_result

Key Operational Metrics

MTTR: Mean time to recover after a failure → target < 30 seconds
Circuit Open Rate: Share of requests that are blocked → alert if > 1%
Fallback Rate: Rate at which fallbacks run → track per model

This three-part series completes a three-layer defense for agent reliability (controlling non-determinism → testing → failure recovery).

Isolating the Cause When an SLA Is Breached

When an agent SLA is breached, you need to quickly split the cause between your code, the LLM provider, and external tool APIs.

Isolation order

Error type distribution: separate provider response codes (429 rate limit, 5xx server errors) from your own exceptions (timeouts, parse failures).
Latency by segment: in traces, check whether LLM calls, tool calls, or your own processing got slower.
External status: compare the provider status page's incident time with when your metrics degraded.

SQL

-- errors by type in the last hour
SELECT coalesce(provider_status::text, error_type) AS kind, count(*)
FROM agent_steps
WHERE created_at > now() - interval '1 hour' AND NOT success
GROUP BY 1 ORDER BY 2 DESC;

If each step emits an OpenTelemetry span (e.g., llm.call, tool.search, agent.plan), the slow segment is visible directly in the trace view.

Response

Mostly 429s: tune concurrency limits and backoff, and consider a higher provider quota or fallback to a secondary provider.
5xx or latency spikes coinciding with a provider incident: open the circuit breaker and switch to degraded mode (cached or simpler answers).
More parse failures on your side: suspect recent prompt or model changes first.

Prevention

Record the cause category (internal/provider/tool) in SLA reports and review quarterly which dependency consumes the most error budget.

#AI agents #SLA

✦ ✦ ✦

Sources and verification

Sources and checks vary by article. Only articles with a recorded check date, scope and sources below were checked, and only within that scope. For other articles, verify against the current docs for your version before applying anything, and email us about errors. Editorial policy · email us

This article has no recorded human check (date and scope) yet.

Published · May 20, 2026

How we write and review →Editorial policy →

Was this helpful?

Comments

Be the first to comment.