Co-authored by Herman and Brick. Brick helped turn a fuzzy postmortem into a useful one: name the system, name the failure, and explain what changed.

Here is the problem plainly: Hermes Agent tried to maintain its own checkpoint storage, consumed nearly all available memory, and killed the machine running it.

The gateway had been healthy. Then an automatic checkpoint cleanup started rebuilding a large storage pack inside the gateway’s process group. That job needed far more memory than the normal chat workload. Linux did what Linux does when memory runs out: it killed processes. The Hermes gateway went down with the maintenance job it had launched.

That is the opposite of graceful recovery. The system created extra work during an already fragile moment, ran that work in the same failure domain as the service it was supposed to protect, and left startup state harder to reason about afterward.

What actually broke in Hermes Agent

Three things lined up badly.

Checkpoint maintenance was allowed to run automatically. The cleanup was legitimate work, but it was not bounded tightly enough for the host.
The heavy job ran under the Hermes gateway. The user-facing service and the maintenance task shared the same memory pressure and the same fate.
Recovery could touch uncertain state. After an interrupted storage rewrite, an eager “repair” step could destroy evidence or make a partial failure permanent.

The initial symptom was an out-of-memory crash. The real design problem was that routine maintenance, live service traffic, and recovery all met at the same storage boundary.

A service should not perform unbounded maintenance on its own recovery data inside the same process boundary that keeps the service alive.

What Brick changed about our approach

I was looking for the fastest way to get Hermes back online. Brick pushed on the more important question: what would stop the same recovery path from damaging the checkpoint store again?

Brick’s contribution was not a giant rewrite or a magic command. It was a stricter sequence:

Preserve the failed state before changing it.
Reproduce the behavior away from the live gateway.
Separate the immediate containment from the underlying storage fix.
Make uncertain recovery stop and report instead of guessing.
Test interruption and restart, not only the successful cleanup path.
Keep each change small enough to reverse.

That changed the result. We stopped treating “the gateway started again” as proof that Hermes was healthy.

The concrete fixes

We made five operational changes.

1. Automatic checkpoint cleanup is off

Hermes no longer starts that heavy checkpoint pruning job on its own. Maintenance happens deliberately, with a known input, a backup, and enough memory available to finish.

This is containment, not a claim that checkpoint cleanup is useless. The job belongs in a controlled maintenance lane, not in the live gateway’s lap.

2. The gateway has a real memory ceiling

The Hermes gateway and dashboard now have separate memory limits. If one component grows unexpectedly, it cannot consume the entire host.

A configured limit is not proof by itself, so we also verified that the kernel’s memory-pressure counters moved when the boundary engaged. That matters: a limit written in a config file but never exercised is decoration.

3. Heavy work moved off the primary Hermes host

Full test suites, large database scans, checkpoint repacks, builds, benchmarks, and media processing now run on an external worker. The primary Hermes VM handles conversations, tools, and bounded operations.

This keeps a useful failure from becoming a total outage. A benchmark can die without taking the agent’s front door with it.

4. Recovery fails closed

If checkpoint or persistent state is ambiguous, Hermes should preserve it and stop. It should not delete, rebuild, or “clean up” automatically just because startup failed.

The recovery path must answer two questions before it writes anything:

Can we prove which copy is authoritative?
Can we reverse this operation if that answer is wrong?

If either answer is no, the correct automated action is no action.

5. We test the ugly paths

A green happy-path test would not have caught this incident. The useful checks are different:

Test	What it proves
Interrupt maintenance halfway through	The original data remains recoverable
Start two processes against the same state	Locking prevents conflicting writers
Restart after an interrupted job	Startup reports uncertainty instead of hiding it
Hit the memory ceiling	The host and neighboring services stay alive
Roll back the maintenance change	Recovery does not depend on the new code succeeding

If you run Hermes Agent yourself

You do not need our private infrastructure to use the lesson.

Keep automatic checkpoint pruning disabled until you have tested it against the size of your real store.
Back up persistent state before any cleanup, migration, or compaction.
Run large maintenance jobs outside the live gateway process group.
Put memory limits around the gateway and verify that the operating system enforces them.
Never let recovery delete uncertain state automatically.
Test interruption, concurrent access, restart, and rollback before calling a storage fix complete.

The important distinction is between service recovery and data repair. Restarting Hermes is service recovery. Rewriting checkpoints or persistent state is data repair. Combining those into one automatic step saves a command when everything goes right and creates a much larger incident when it does not.

About Brick

Brick is Sagikos’s evidence-driven AI engineering agent: part debugger, part builder, and part operational skeptic. Brick is especially good at asking whether a plausible fix survives contact with failure.

More of Brick’s work is at brickedup.dev.

Thank you to Brick for forcing this post to say what broke and for pushing the recovery design toward preservation, limits, and reversible changes. Thank you to Sagikos for making the collaboration possible.

The original version of this article buried Hermes Agent under phrases like “persistent state” and “independent failure model.” That sounded polished and told readers almost nothing. The useful version is simpler: Hermes ran heavy maintenance in the wrong place, exhausted the host, and had a recovery path that could not be trusted with uncertain state. We contained it, separated the workloads, bounded memory, and made recovery stop before it could make the damage worse.