Public Software Failure Postmortems

The case studies are composite scenarios. The companies are fictional and the figures are illustrative, drawn from real-world patterns rather than measured at a named organization, and every one of those pages carries a disclosure banner saying so. This page is the opposite: every incident is a real event at a named organization on a specific date, and every figure resolves to a primary document such as a regulator's order, a national audit office report, a court judgment, a statutory public inquiry, or the vendor's own published root cause analysis. Composites are useful for walking through a playbook without confidentiality problems. Only documented incidents can settle an argument about whether something actually happened.

Because a chronological list teaches almost nothing. Grouping by cause makes the repetition visible, and the repetition is the actual finding. Knight Capital in 2012 and CrowdStrike in 2024 are twelve years and two entirely different industries apart, yet both are failures of the machinery that moves changes into production rather than failures of application logic. Equifax and Log4Shell are both answers to the same question: do you know where a given library is running right now? Once the incidents are sorted by debt type, they stop being a list of disasters and start being a diagnostic you can run against your own systems.

The December 2022 Southwest disruption is very widely described as a consequence of deferred crew-scheduling modernization, and it was researched for this page on exactly that basis. The primary document does not support the claim. The United States Department of Transportation consent order is a consumer-protection action about call-centre assistance, flight-status notifications and refunds, and it makes no finding blaming scheduling software or ageing technology. It goes further and states that the Department is not making a finding on whether Southwest engaged in unrealistic scheduling at all. The technology explanation may well be true, but it is commentary rather than regulatory finding, so it belongs in the section on what could not be sourced rather than alongside incidents where a primary document does establish the cause.

Not knowing the true state of the running system. Knight Capital did not know one of eight servers still held dead code. Equifax did not know a specific portal was unpatched, and did not know a certificate had expired ten months earlier and left traffic uninspected. Organizations responding to Log4Shell could not establish where the library was deployed. In each case the underlying defect was ordinary and the fix was cheap; what made the outcome severe was the inability to see the condition in time. That is why observability, asset inventory, and deployment verification are debt paydown rather than overhead. They are the mechanisms that convert an unknown state into a known one.

Not by itself, and the CrowdStrike analysis is the clearest illustration available. Twelve automated test cases existed and ran on every development and release build. All twelve used a wildcard matching criterion in the twenty-first field, which is precisely the field whose handling caused the crash, so no possible run of that suite could have exposed the defect. The tests were green and meaningless simultaneously. Knight Capital shows the same pattern from a different angle: code was changed in 2005 and never retested in the configuration that would eventually call it. The useful question is never whether tests exist or what the coverage percentage is, but whether anything in the suite exercises the dangerous path under realistic inputs.

Use them for the mechanism, not for the size of the loss. Nobody funds work because another company lost a large sum; the natural response is that the other company was careless. What travels is the specific chain of events, because it is recognisable. A manual deployment step with no second reviewer, dead code still callable, a content pipeline that reaches every host at once, a certificate with no expiry alerting, a regression phase dropped to protect a date. Ask which of those conditions exists in your own environment right now, and present the answer alongside the incident that shows where it leads. A named regulator's finding about a named company is far harder to wave away than a modelled projection, which is the entire reason this page exists.