AI agent reliability is the ability to complete bounded work correctly, surface failures promptly, recover without making the situation worse, and prove the final state. Accuracy from the underlying model matters, but production failures often come from the surrounding system: stale state, missing verification, uncontrolled processes, wrong credentials, or a recovery loop acting on a false diagnosis.

Hermanity’s incident record provides concrete examples because failures are documented alongside successful projects. The recurring pattern is simple: confidence is cheap; trustworthy closure requires independent evidence.

The main failure classes

Claimed completion without sink verification

The most common agent failure is a plausible completion report that was never checked against the final destination. A file may exist locally but not on the remote. A pipeline may be green while the live site still serves stale bytes. A browser click may have been delivered without changing the page.

Control: define the authoritative sink before execution, then read it back afterward. For source changes, inspect the remote commit and every CI job. For deployments, fetch the public URL with a cache-busting query. For account configuration, revisit the service’s own settings page.

Stale or partial state

Agents often reason from snapshots: a repository checkout, cached search result, monitoring payload, or prior-session memory. The snapshot may be internally consistent and still be wrong now.

Control: add freshness metadata and make stale state visible. Monitoring should show “stale” rather than the last known green state. Search snippets should be checked against the live page before diagnosing metadata. Repository work should begin with a fetch and exact branch comparison.

Recovery that amplifies the incident

Automatic recovery can create a second failure when it retries the same broken assumption, competes with a healthy process, or modifies shared state while investigating it. When Hermes Tried to Fix Itself and Took Down the Host documents this class directly.

Control: recovery should be bounded, observable, and fail closed. Retry only transient errors; when the same method fails repeatedly, change strategy instead of thrashing. Separate diagnosis from repair, and ensure only one recovery owner can mutate a shared resource.

Resource exhaustion

Browser sessions, model servers, test runners, and subagents can consume memory or GPU resources long after their useful work ends. The OOM incident showed how an apparently ordinary browser playtest could reboot the host.

Control: place heavy subprocesses outside the control-plane resource group, set memory and runtime ceilings, monitor progress rather than process existence, and terminate the full process group on timeout. Test the limit under a representative workload before calling it protective.

Model and route ambiguity

A client can request one model while a proxy or provider serves another. Evaluations then attribute behavior to the wrong system, and routing decisions learn from mislabeled evidence.

Control: record requested model, served model, provider path, runtime, and important sampling settings. The Model Garden field note and Model Bench show why model identity is an evidence field rather than UI decoration.

Vision and interface ambiguity

Screenshots can be dark, scaled, occluded, or semantically incomplete. A model can confidently invent state that is not present. Vision Models Hallucinate Game State documents one concrete case.

Control: use vision to identify visual defects, then verify text and state through the DOM, accessibility tree, API, or deterministic pixel checks. For state-changing input, capture fresh state before retrying an unverifiable action.

Reliability controls by layer

Before execution

Resolve the exact target, repository, account, or device.
Fetch current state rather than relying on memory.
Identify irreversible or identity-bound steps.
Define the expected artifact and verification command.
Isolate concurrent workers that could touch the same files.

During execution

Use explicit timeouts and resource ceilings.
Preserve logs and partial artifacts on failure.
Track progress with completed units, not merely a live PID.
Stop repeated identical failures and revise the hypothesis.
Keep credentials out of commands, repositories, and public output.

Before release

Run the full local suite, not only focused tests.
Inspect generated output and referenced assets.
Bind independent review to one exact commit.
Enumerate each CI job, including allowed-to-fail jobs.
Check that the diff contains only the intended scope.

After release

Fetch the deployed route from the public edge.
Verify canonical metadata and important rendered claims.
Exercise the changed interaction at desktop and mobile widths.
Confirm telemetry sees the new version without leaking private data.
Keep a rollback path until the live verification closes.

Reliability metrics that reveal real problems

A single success rate hides too much. Useful operational measures include:

valid completion rate by task family;
empty-response and hard-error rates;
p50 and tail latency;
recovery attempts per completed task;
stale-state detections;
resource-ceiling breaches;
verification failures after an apparently successful action;
requested-versus-served model mismatches; and
time from failure detection to a verified safe state.

The Skynet Breakage Atlas project treats empty replies and hard errors as first-class results. The Subagent Swarm field note illustrates another important rule: a reliability hypothesis can be useful even when measured evidence refutes it.

An evidence hierarchy for agent claims

From weakest to strongest:

The agent says it acted.
A tool reports that it accepted the action.
Local state appears changed.
Tests pass against the candidate.
The remote system contains the exact candidate.
Independent review approves that exact candidate.
The production destination serves or executes it correctly.
Monitoring shows the result remains healthy over the required window.

Not every task needs all eight levels. Production releases and identity-sensitive account changes often do. A local draft usually does not. Reliability means choosing the appropriate evidence floor before the agent starts congratulating itself.