AI Agent Health Checks: Liveness, Readiness and Synthetic Runs

Idea In 30 Seconds

Health checks for AI agents show whether the system is truly ready for traffic right now.

They help detect degradation before incident: slow tools, timeouts, queue issues, or LLM-layer failures.

Without health checks, teams often see only the failure itself, not the early signals before it.

Core Problem

A system can be formally "up" while already unstable in practice.

API responds, runs start, but some steps already degrade: timeout rate grows, success rate drops, queue time increases. Without health checks, this is usually visible only after cascading failures.

Next, we break down how to read these signals and find which component enters degradation first.

In production this often looks like:

• everything looks "working" externally, but p95 latency is already above SLO;
• one tool starts returning timeout more often, but the issue is still unclear;
• synthetic run starts failing before main traffic;
• team reacts only after partial outage.

That is why the health layer should be monitored separately, not only through general run metrics.

How It Works

Health checks are usually built on two levels:

• component checks (tool_available, llm_reachable, queue_ok, db_ok);
• end-to-end checks (synthetic_run_ok, critical_workflow_ok).

These signals answer "is the system healthy right now". Logs and tracing are needed to explain why a concrete check failed.

A reachable process does not establish workflow readiness. Health checks can detect a failure before a user encounters it, depending on check frequency and scenario coverage. They do not guarantee earlier detection than error or latency metrics.

Liveness, readiness and synthetic runs make different decisions

• Liveness: should the process restart?
• Readiness: can it accept new tasks?
• Synthetic run: does a selected end-to-end scenario work?

Kubernetes documents liveness, readiness and startup probes.

For a support agent, an unavailable optional search service may allow a limited mode. An unavailable policy store required to authorize actions must stop admission of affected tasks. One failed critical check outweighs nine successful optional checks: an average health_score must not authorize traffic by itself.

Do not call an external LLM in every liveness probe. It adds cost, and a provider outage can trigger pointless restarts across replicas. Collect dependency results separately with timeouts; readiness reads a fresh snapshot. Give synthetic runs separate test data, a budget, and tools that cannot cause side effects (state changes) in production.

A readiness check you can run

Save this as readiness_demo.py and run python3 readiness_demo.py with Python 3.10+. This tests admission logic, not network probes. All timestamps use the same process-local monotonic clock; 30 seconds is an example to align with your check interval.

from dataclasses import dataclass


@dataclass(frozen=True)
class Check:
    ok: bool
    observed_at: float


def ready(checks, required, now, max_age=30.0):
    if not required or max_age <= 0:
        return False
    for name in required:
        check = checks.get(name)
        if check is None or not check.ok:
            return False
        if not 0 <= now - check.observed_at <= max_age:
            return False
    return True


required = ("queue", "policy_store")
checks = {
    "queue": Check(True, 100.0),
    "policy_store": Check(True, 100.0),
    "optional_search": Check(False, 100.0),
}
assert ready(checks, required, now=110.0)
assert not ready(checks, required, now=131.0)
assert not ready({}, required, now=110.0)
checks["policy_store"] = Check(False, 100.0)
assert not ready(checks, required, now=110.0)
print("readiness cases: 4 passed")

Expected output: readiness cases: 4 passed. The cases cover an optional dependency, stale results, missing results, and a failed critical dependency. A real collector must record failed checks and must not refresh the success timestamp after a timeout. Handling tasks already in progress requires a separate completion or recovery policy.

After a failure signal, continue with debugging the affected run.

Typical Production Health Metrics

health_score is usually computed on dashboard/health-service level as an aggregate of multiple checks, not as one "magic" metric.

To keep health metrics useful, they are usually segmented by release, region, workflow, and component type.

Important: do not add high-cardinality fields (run_id, request_id, user_id) to labels, or metric storage will overload quickly.

How To Read The Health Layer

What is checked -> what degrades -> which component breaks the workflow. These are three levels you should always read together.

Focus on time trends and release-to-release differences, not a one-off failure of one check.

Now look at signal combinations:

• health_check_pass_rate down + timeout_rate up -> system enters unstable mode;
• synthetic_run_success_rate down + degraded_component_count up -> issue already impacts critical workflow;
• queue_time_p95 up + run_count up -> capacity shortage, scaling needed;
• health_score down + error_rate up -> high incident risk in near term;
• tool_check_latency_p95 up + synthetic_run_success_rate down -> bottleneck in external tool layer.

When To Use

A full set of health checks is not always required.

For an early prototype, basic API availability check can be enough.

But detailed health checks become critical when:

• the system is already in production;
• there are stability SLO/SLA targets;
• the agent depends on multiple external tools or queues;
• early alerts are needed before mass failure starts.

Implementation Example

Below is a simplified Prometheus-style health-check loop. It shows baseline approach: component checks, synthetic run, and aggregated health score.

This is instrumentation scaffolding: install prometheus-client and supply your own checks with timeouts. The aggregate score supports an overview, not an automatic readiness decision.

import time
from dataclasses import dataclass
from prometheus_client import Counter, Gauge, Histogram

# perf_counter() is used for precise monotonic latency measurements
HEALTH_CHECK_TOTAL = Counter(
    "agent_health_check_total",
    "Total health checks by check name and status",
    ["check", "status", "release"],
)

HEALTH_CHECK_LATENCY_MS = Histogram(
    "agent_health_check_latency_ms",
    "Health check latency in milliseconds",
    ["check", "release"],
    buckets=(10, 20, 50, 100, 250, 500, 1000, 2000),
)

SYNTHETIC_RUN_TOTAL = Counter(
    "agent_synthetic_run_total",
    "Total synthetic runs by status",
    ["status", "release"],
)

DEGRADED_COMPONENT_TOTAL = Counter(
    "agent_degraded_component_total",
    "Total degraded component detections",
    ["component", "reason", "release"],
)

HEALTH_SCORE = Gauge(
    "agent_health_score",
    "Aggregated health score in range 0..1",
    ["release"],
)


@dataclass
class CheckResult:
    ok: bool
    reason: str = "ok"


def run_health_checks(checks, run_synthetic, release="2026-03-22"):
    total = len(checks)
    failed = 0

    for check_name, check_fn in checks.items():
        started_at = time.perf_counter()
        status = "error"  # default for any failure path
        try:
            result = check_fn()  # -> CheckResult
            status = "ok" if result.ok else "fail"
            if not result.ok:
                failed += 1
                DEGRADED_COMPONENT_TOTAL.labels(
                    component=check_name,
                    reason=result.reason,
                    release=release,
                ).inc()
        except Exception as error:
            failed += 1
            DEGRADED_COMPONENT_TOTAL.labels(
                component=check_name,
                reason=type(error).__name__,
                release=release,
            ).inc()
        finally:
            HEALTH_CHECK_TOTAL.labels(check=check_name, status=status, release=release).inc()
            HEALTH_CHECK_LATENCY_MS.labels(check=check_name, release=release).observe(
                (time.perf_counter() - started_at) * 1000
            )

    try:
        synthetic_ok = run_synthetic()
        SYNTHETIC_RUN_TOTAL.labels(
            status="ok" if synthetic_ok else "fail",
            release=release,
        ).inc()
        if not synthetic_ok:
            failed += 1
    except Exception:
        SYNTHETIC_RUN_TOTAL.labels(status="error", release=release).inc()
        failed += 1

    # 1.0 = all checks passed, 0.0 = all failed
    # +1 for synthetic run
    denominator = total + 1
    health_score = max(0.0, 1.0 - (failed / max(1, denominator)))
    HEALTH_SCORE.labels(release=release).set(health_score)
    return health_score

The following health metrics are illustrative values, not measurements from a production system:

Investigation

When a health alert fires:

1. find degraded check or workflow;
2. inspect problematic runs in tracing;
3. check timeout, stop_reason, and tool failures in logs;
4. find root cause (tool, LLM, queue, routing, release regression).

Common Mistakes

Even when health checks are added, they often fail to help because of common mistakes below.

Only ping/availability check exists

API availability does not guarantee agent workflow actually works. Without synthetic run, hidden degradation is easy to miss.

No checks for external dependencies

If tools, queues, or DB are not included in health checks, system can look "healthy" until first incident. In this case it is hard to quickly localize tool failure.

No breakdown by workflow and release

Without this, it is hard to understand which release or scenario degraded system state.

No alerts on health-score degradation

Without alerts, health checks become passive telemetry. This makes it easy to miss early multi-agent chaos signals.

High-cardinality labels

Adding run_id, request_id, or session_id to labels quickly overloads metric backend. Keep this data in logs and tracing.

Self-Check

Below is a short checklist of baseline health checks before release.

FAQ

Q: How are health checks different from regular metrics?
A: Metrics show trend, while health checks give a fast "ready or not right now" answer for concrete scenarios.

Q: What is the minimum health-check set to start with?
A: Start with llm_reachable, tool_available, queue_ok, and one synthetic run for critical workflow.

Q: Why is synthetic run so important?
A: It validates full execution path, not a single component. This best reflects real production state.

Q: If health score drops but API still responds, is it already incident?
A: It is early degradation. Investigation should start immediately before issue becomes mass failure.

Related Pages

Next on this topic:

• Observability for AI Agents — base model of tracing, logs, and metrics.
• Agent Metrics — general production system signals.
• Alerting In AI Agents — how to build early notifications.
• Agent Tracing — how to localize the step that breaks health check.
• Agent Latency Monitoring — how to detect speed degradation before incident.