A Green Migration That Wasn't Complete
Our agent-host migration passed its scoped checks and still missed the tools that made the old machine usable. The lesson was not better copying; it was better definitions of complete.
The migration looked green twice.
The new host had the main agent workspace. It had the service state. It had the large data tree. The network paths worked, the agent account could read and write where expected, and the formal preflight returned a clean result. An independent review reached the same conclusion.
Then the operator logged in and typed the name of a familiar local command.
The shell answered: command not found.
That tiny failure invalidated the larger claim. We had proven that the parts named in the migration checklist were present. We had not proven that the new machine could replace the old one in daily use.
The word that caused the problem
The dangerous word was complete.
Our first checklist defined the system as the agent’s managed directories: its workspace, configuration, state database, repositories, and service files. That was reasonable from the application’s point of view. It was wrong from the operator’s point of view.
The old host also contained the accumulated tools around the application:
- small commands installed under the user’s local binary directory;
- terminal themes and shell customizations;
- bridge credentials stored outside the main configuration tree;
- provider CLI histories and project state;
- review receipts and dispatch evidence;
- SSH material used by narrowly scoped operational roles;
- scheduled jobs, helper scripts, and XDG configuration;
- mounts, container storage, and machine-specific service overrides.
None of those items was imaginary. The operator used them. The migration simply had no row for them.
That is a more serious defect than a failed copy. A failed copy produces an error. A missing category produces a confident success.
Two reviewers, one shared blind spot
The first review checked the paths in the plan and returned READY. A second, independent review inspected the same migration and also returned READY.
That sounds like corroboration. It was actually correlated scope.
Both reviewers asked, “Did every listed item arrive?” Neither asked, “What existed on the source machine that is not represented by this list?”
Independence of reviewers does not help when they inherit the same incomplete inventory. Two thermometers will agree that a room is comfortable while both are sitting outside.
The useful challenge is not another pass over the destination. It is a source-first difference audit:
- Inventory the source machine by artifact class.
- Inventory the destination using the same classes.
- Classify every difference as migrated, intentionally excluded, recreated, or missed.
- Block the word READY while any row is unclassified.
The nine classes we now check
We replaced the application-shaped checklist with a host-continuity checklist. It has nine classes.
| Class | What belongs in it | Typical failure |
|---|---|---|
| Secrets and bridge material | Tokens, scoped credentials, protected environment files | Main config arrives; top-level or profile-scoped secrets do not |
| Identity and access | SSH keys, authorized commands, role-specific access | Human login works; automation identity breaks |
| CLI accumulated state | Provider sessions, project histories, local indexes | CLI binary runs; its useful history is gone |
| Evidence and dispatch history | Audit receipts, review packets, prior run artifacts | Current service works; provenance disappears |
| Operator tools | Local commands, scripts, themes, helper binaries | Migration passes; familiar commands return not found |
| XDG and shell configuration | Config, data, and state outside the application tree | Tool exists but behaves differently |
| Schedulers and services | Timers, cron entries, user units, overrides | Files exist but recurring work silently stops |
| Storage and mounts | Network shares, container graph roots, permissions | Data is present through the wrong or fragile path |
| Runtime and hardware bindings | GPU, device, network, cgroup, and machine-specific settings | Software starts but cannot use the host it moved onto |
The important part is not the number nine. The important part is that every source artifact must land in one of three states:
- in scope and present;
- out of scope by design, with a reason;
- missed, which blocks completion.
An unclassified artifact is not harmless uncertainty. It is a failed gate.
Why backups did not solve it
We had backups. They were useful, but they did not answer the inventory question.
A backup proves that some bytes were preserved. It does not prove that the new host is using them, that permissions survived, that symlinks resolve to protected targets, that the right service account can read them, or that a command resolves through the new shell’s path.
Migration verification has to exercise behavior:
- invoke the operator’s local commands;
- authenticate each bridge without printing its secret;
- run read/write/delete canaries on shared storage;
- ask the container engine where its graph root actually is;
- enumerate scheduled jobs from the destination;
- compare source and destination inventories;
- verify secret file targets resolve to restrictive modes;
- confirm hardware-backed workloads can see the intended device.
Presence is one column. Usability is another.
The correction
We stopped defending the green checklist and widened the contract.
The missing operator command was reconstructed. External credential classes were audited separately. Profile-scoped environment files were added to the permissions sweep. Historical evidence was classified instead of assumed. The migration report was rewritten so its verdict named its scope explicitly.
Most importantly, READY stopped being a feeling attached to a large pile of passing checks. It became the output of a closed inventory:
Every source-machine artifact class is accounted for, every intentional omission is named, and every required behavior has been exercised on the destination.
That standard is slower than copying two directories and checking disk usage. It is much faster than discovering, one command at a time, that the old machine was the real documentation.
A practical migration gate
Before calling a replacement host complete, ask five questions:
- What did the operator use that the application did not own?
- What lives outside the obvious workspace and configuration roots?
- Which histories and receipts matter even though no service reads them at startup?
- Which destination checks prove behavior rather than file presence?
- Can every source-only artifact be explained without hand-waving?
If the answer to the fifth question is no, the migration may be operational for a narrow workload. It is not complete.
The original mistake was not that we forgot one command. The mistake was treating an application migration as a host migration, then using a host-sized word—READY—for an application-sized proof.