The agent update problem: how do you fix the thing you use to fix things?

There is one update in an MSP's environment that has no fallback path: the management agent updating itself. A failed application update means you push a fix. A failed agent update means the mechanism for pushing fixes is the casualty, and the recovery plan is a truck, a phone call, or a very awkward email to a client.

That single property should drive the design, and it usually does not — agent self-update tends to be treated as one more deployment rather than as the one deployment that removes your ability to deploy.

Transactional, not sequential

The design we landed treats an update as a transaction that either completes or leaves the previous working agent in place. Not a sequence of steps that stops wherever it failed — a sequence that stops wherever it failed is exactly how you get a half-updated service that starts, reports healthy, and cannot execute anything.

Concretely, that means: the update stages before it commits, identity and durable local state survive the swap, and a failure at any stage rolls back to the agent that was working ten minutes ago rather than forward into an unknown state. The agent that comes back up is either the new one, fully, or the old one, fully.

The success case of an agent update is boring. The design is entirely about the failure case.

Crash-safe means testing the crash

State that claims to be durable is a claim, not a property, until something kills the process at a bad moment. So we validate it by doing that on purpose: a forced child crash, a custom restart, and a full service-manager restart, each checked afterward for whether the agent came back with its identity and execution state intact rather than as a stranger asking to be re-enrolled.

Re-enrollment is worth calling out specifically. An agent that loses its identity during an update does not merely inconvenience you — it detaches from its device record, and the fleet view you use to notice the problem is the thing that stopped showing it.

What we have not finished

Self-update carries the highest risk tier in our capability registry and required its own recorded approval before it could be issued at all. It is implemented, it is validated on lab hardware, and the acceptance work that would justify running it across a real fleet — including code-signing the distributed binaries — is still ahead of us. Our installer currently ships unsigned under an explicit, recorded decision, and that gets fixed before wide distribution, not after.