What 35 Real Outages Teach That Tutorials Can't

Tutorials teach you what happens when everything goes right. Careers are made by people who know what happens when it doesn't.

That's why our curriculum is anchored to 35 real, publicly documented industry failures. Not hypotheticals, not sanitized case studies — actual postmortems, with names, timelines, and costs. Every War Room drill, and a large share of our daily material, traces back to one of them.

Three stories we drill

GitLab, January 2017. An engineer, tired and deep into a long fight with replication lag, ran a data-removal command against the primary database instead of the secondary. Roughly 300 GB of production data was gone in seconds. Then came the second, worse discovery: of the multiple backup mechanisms GitLab believed it had, none had been producing usable backups. Recovery came from a staging snapshot taken hours earlier; about six hours of issues, merge requests, and comments were lost for good. GitLab, to its lasting credit, ran the recovery in public — a live-streamed postmortem the whole industry learned from. The lesson we drill isn't 'don't typo'. It's: a backup you haven't restored is a rumor. Test restores, not backups.

AWS S3, February 2017. An engineer on the S3 team, debugging the billing system, executed an established playbook command — with one parameter mistyped. Instead of removing a small number of servers, it removed a large set, including ones supporting S3's index and placement subsystems in us-east-1. Those subsystems needed a full restart, something that hadn't been done in years, during which vast swaths of the internet — sites, apps, IoT devices — simply broke for roughly four hours. In a detail we make every learner sit with, the AWS status dashboard itself depended on S3 and couldn't display the outage. Lessons: put guardrails and rate limits on destructive tooling, know your blast radius, and never let your observability depend on the thing it observes.

Knight Capital, August 2012. A deploy reused an old feature flag that had once controlled different, retired behavior. Seven of eight servers got the new code; the eighth ran the old code and, when the flag flipped, began firing millions of unintended orders into the market. Forty-five minutes later Knight had lost about 440 million dollars and, within days, its independence. Lessons: delete dead code, never repurpose flags, and make deploys atomic with an instant rollback path.

Why failure is the better textbook

Knowledge acquired without stakes is stored without priority — and recalled without urgency.

Ericsson's work on deliberate practice found that experts are built by practice at the edge of ability with immediate, meaningful feedback. A tutorial has no edge and no stakes. A realistic incident has both: consequences make the material emotionally salient, and salience is what memory keys on. Ask any engineer which they remember better — the chapter on backups, or the night the restore failed.

There's a second reason. Outages are where the 28 competencies stop being separate subjects. The GitLab incident is simultaneously a databases lesson, an ops lesson, a fatigue-and-process lesson, and a communication lesson. Real failures force the cross-layer thinking that [T-shaped engineers](/program) are defined by.

And a third: postmortems teach judgment about people. Every one of these incidents involved a competent engineer doing something locally reasonable. Studying them builds the reflex of asking 'what system allowed this?' instead of 'who was stupid?' — the blameless instinct that separates mature engineering cultures from finger-pointing ones.

From reading to rehearsing

Reading a postmortem is better than nothing. Rehearsing one is better still — the testing effect applies to incident response as much as to flashcards; you retain what you retrieve under pressure.

So in our [War Rooms](/incident-response), learners don't read about the S3 outage. They're paged into a simulation of one: dashboards red, status page lying, a stakeholder demanding an ETA. They triage, form hypotheses, choose what to roll back, and draft the customer communication — and get graded the way a staff SRE would grade them: on process, prioritization, and communication, not on lucky guesses.

Borrow scars, don't earn them all

The tuition for these lessons has already been paid — in the hundreds of millions, by someone else. The postmortems are public. The only thing missing from most engineers' education is anyone requiring them to actually internalize them.

Start with the three above; read the original write-ups, they're all public. Then, before your next deploy, ask the Knight question: is there any flag, path, or host that could run old code? And this weekend, run the GitLab drill: restore a backup, end to end, and time it.

If you'd rather rehearse with the pressure on and a grader watching, [that's what we built](/program).