Why 'fail closed' beats 'fail open' for a secrets system, even when it's less convenient

Every system that handles secrets has to make a decision about what happens when something's misconfigured — a missing key, an unreachable encryption service, a corrupted config value. There are exactly two options: fail open, where the system falls back to some degraded-but-functional state and keeps running, or fail closed, where it refuses to run at all until the misconfiguration is fixed. Fail open is more convenient in the moment. Fail closed is the only one of the two that doesn't turn a config mistake into a data-exposure incident.

The reason fail-open is so tempting is that it optimizes for uptime, and uptime is the metric everyone notices immediately when it's missing. A secrets system that fails open when its encryption key is unavailable might fall back to storing the secret in plaintext 'just this once, temporarily' — and it works, in the sense that the deployment doesn't crash and nobody gets paged. What actually happened is that a real secret got written to disk unencrypted, silently, and unless someone happens to audit storage at exactly the right moment, nobody finds out until it matters.

Why fail-closed is the only honest default

• A crash is loud. A silent plaintext fallback is not — and a security failure mode that doesn't announce itself is strictly worse than one that does, because it removes the operator's chance to notice and fix it before it's exploited.
• Fail-open failure modes compound — one plaintext secret written during a misconfiguration window can sit there indefinitely, surviving the fix to the original misconfiguration, because nothing ever flagged that it needed to be re-encrypted.
• The cost of fail-closed is borne by the deploying team, immediately, in a form they can't ignore — a service that won't start. The cost of fail-open is borne by whoever's data was in the secret, later, in a form nobody connects back to the original misconfiguration.

A system that refuses to start is a bad five minutes for whoever's deploying it. A system that quietly starts anyway with your secrets in plaintext is a bad five minutes you don't get to have, because you never find out it happened.

The honest counterargument for fail-open is real, not a strawman: for a lot of software, availability genuinely matters more than a narrow security property, and a fail-closed default in the wrong place can turn a minor config drift into an unnecessary outage. The right answer isn't 'always fail closed for everything' — it's being deliberate about which systems specifically warrant it, and secrets handling is one of the clearest cases where it does, because the alternative failure mode is silent and the blast radius is every secret the system touches until someone notices.

This is a real, shipped decision in Nexus, not an aspiration: production is configured fail-closed for secrets — if the encryption key required to protect an integration credential isn't properly available, the system refuses to store or use that secret rather than falling back to storing it in plaintext, full stop. It's a less forgiving default in the narrow window right after a misconfiguration, and that's the tradeoff on purpose. If you're evaluating a vendor's secrets handling, the specific question worth asking is what their system does — not what it's supposed to do — when the encryption dependency isn't available at startup. Most vendors haven't tested that path recently enough to answer with confidence.