Hallucination risk in IT automation, and the specific design choices that contain it

A language model can state something false with exactly the same confident tone it uses to state something true, and there is no reliable way to distinguish the two from the output alone — that's not a bug that gets patched in a future model version, it's a structural property of how these systems generate text. The useful question for IT automation isn't 'how do we stop hallucination,' because nobody can promise that honestly. It's 'what happens in our system when a hallucination occurs' — and the answer to that question is a design choice, not a model property, which means it's actually something a vendor can be held accountable for.

Where hallucination is dangerous versus where it's just annoying

A hallucinated fact in a drafted ticket summary is annoying — a technician reading it catches the wrong detail because they know the environment, fixes it, moves on. A hallucinated fact that directly triggers an executed action without a checkpoint is dangerous, because the system doesn't know the difference between a correct diagnosis and a confident-sounding wrong one, and by the time a human notices, something has already happened on a client's system.

The specific design choices that contain it

• A default-deny capability registry, so an agent can only initiate job classes that have been explicitly enabled — a hallucinated plan to run an unregistered action class simply has nowhere to execute, regardless of how convincing the reasoning looks
• Every job Ed25519-signed by the platform and verified on-device before it runs, so a fabricated instruction can't reach an endpoint through any path that skips the signing step
• A required human approval before anything destructive, irreversible, or outside a tenant's configured automation policy actually executes — the checkpoint exists specifically for the moment the model is confidently wrong
• Two separate, linked audit events for every AI draft and every human approval, so a hallucinated draft that somehow got approved anyway is traceable to a specific decision rather than buried in a single ambiguous log line

You cannot engineer a model into never being confidently wrong. You can engineer a system where being confidently wrong never translates into an unreviewed action — and that second thing is the actual containment strategy.

This is also why we're specific about which job classes are live end-to-end in our own practice — echo, ping, inventory collection, and patch deployment with ring rollout — rather than describing the whole capability surface as equally mature. Every job class rides the same signed-job pipeline and the same capability registry regardless of maturity, but the amount of real-world exposure behind each one differs, and pretending otherwise would be its own kind of hallucination risk — ours, not the model's. Ratifying a new capability class beyond that set is a deliberate, recorded decision on our side too, not a silent default flip, which is the same discipline we're asking the model architecture to respect.