Every system I build has two jobs, and they fight each other: it has to keep working when something breaks, and it has to tell someone something broke. Users see the first job. Nobody except the maintainers sees the second. And the better the first job goes, the worse the second gets. That tension is the whole subject. Resilience and visibility are not allies. Every fallback trades today’s calm for tomorrow’s ignorance.
The invisible failure is the fallback that works. Nothing errored, nothing paged. The monitor stayed green because the backup path did exactly what it was built to do. The user got an answer. And the system swallowed the break. Underneath, the real path is dead. Nobody knows, because nobody was ever told. So who was supposed to notice?
So who tells you? The rule I work by is simple: silent to users, loud to maintainers. A failure can be silent to the person using the system. It has to be louder for the people who maintain it. Graceful for the user and quiet for the maintainer is not graceful. It is hidden.
Loud has an address, and it is not the receiver. When a service fails and its caller falls back, the caller absorbed the failure. But alerting there chases the symptom. The failure has to be loud where it happened, because the receiver will never complain. It got an answer, or a spare path, and moved on. Only the origin knows something broke. That is where the noise has to live.
GitLab learned this in public. On January 31, 2017, an engineer deleting what he thought was a replica database deleted the primary instead, and about 300 gigabytes of GitLab.com’s production data went with it. The team reached for its backups and found what GitLab itself later described as five backup and replication techniques, none working reliably or set up in the first place. The nightly backup had been failing silently on a version mismatch, leaving zero-byte files. The failure emails never arrived, rejected by the mail server, so the absence of failure emails read as success. The snapshots that could have saved them were never enabled for the database servers, and the replica was already wiped. The restore came from a snapshot taken six hours earlier, by chance, for unrelated work. Eighteen hours down, six hours of data lost for good. The safety net was the invisible failure. Five mechanisms, believed alive, all dead. And the watcher failed silently too. GitLab is one instance. The take is wider than backups. Replicas, failovers, caches, standby systems, the backup path. All of them exist for the moment the main thing goes wrong. A resiliency mechanism that fails silently is worse than not having one. It hands you confidence you have not earned. You only discover it is dead the moment you need it. That is the worst moment to discover anything. The least-exercised path in any system is the most likely to be silently dead. Resiliency paths are exercised least of all, by design. Until you have actually fallen back onto it, it is a belief, not a fact.
That is how a fallback becomes the system. It does not happen in one decision. Nobody sits down and votes to make the degraded path permanent. It happens because it works. Users are fine, so nobody notices. Because nobody notices, nobody owns the fallback rate, so nobody reviews it. New engineers learn the degraded path as if it were the design. Runbooks get written around it. One day it breaks, and there is nothing underneath it, because the thing underneath it stopped existing a long time ago. Nobody marked the date. The sociologist Diane Vaughan gave this a name studying Challenger, where O-ring erosion kept recurring without disaster until it was treated as normal: normalization of deviance. A degraded condition, survived repeatedly, stops looking degraded.
When one surfaces, I don’t improvise. Fix what broke. Then the cause, or an alert in time if the cause can come back. Then the widening. I ask the same questions of everything like it: does it even have a watcher, does that watcher fire where the failure happens instead of where it lands, and does anyone actually own the alert when it fires. A monitor nobody owns is not a monitor. The fix is not the end of it. A broken thing is a chance to audit the whole class. And if you stop at the one that broke, the rest of the class stays exactly as blind.
It happens more often now. Especially when code is cheap. Every new path a system gains is another place for a failure to sit quietly, and cheap code adds paths fast. Defensive code is the worst of it. Every catch-all that returns a default is a new place something can break without anyone noticing.
Loud has a limit, though. Over-logging is not observability. Most of it is noise, and noise is its own kind of invisibility, because a team that hears everything stops hearing anything. Logs, metrics, and APM have to work together and tell one story of the system. Three separate piles of data are not a story. When they cohere you can see what the system did, and when they do not, you are back to guessing, with more data than before.
I run this on my own systems, in the same order. Fix, cause, widen. The audit starts at home, while fixing it is still cheap.