Blameless Post Mortems

How do you respond when things go wrong?

By Ken Norton

Photo of surgical tools

The NBA’s Steph Curry missed almost 55% of his three-point shots this season. Last year, baseball’s Dee Gordon didn’t get a hit two-thirds of the times he stepped up to the plate. SpaceX failed to land their Falcon 9 reusable booster rocket in 80% of their attempts.

That’s one way to look at it. Another way: Steph Curry just ended the greatest three-point shooting season in history, Dee Gordon was the National League’s best hitter, and SpaceX recently made history by landing a rocket on a freaking barge.

If you’ve read what I’ve written, you know I’m fascinated by failure. As Jeff Bezos says, “failure and invention are inseparable twins.” You can’t do something extraordinary without having been defeated a lot on the way.

That attitude starts small. When an engineer makes a mistake that leads to catastrophe, how does your organization respond? Do you point fingers and name and shame? Does your CEO look for a throat to choke? Or do you welcome open discussion, learn from it, get back up and try again?

I’ve always admired Google’s post mortem culture. The approach is simple: when something goes wrong, the team does an in-depth analysis with an emphasis on process over people. They write down everything they learned and share the document as widely as possible to make sure it never happens again. These post mortems are blameless because we assume everyone makes mistakes from time to time. Post mortems aren’t criminal investigations, they’re an affirmative process designed to make us all a little smarter:

A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or teams for doing the “wrong” thing prevails, people will not bring issues to light for fear of punishment.

That’s from the new O’Reilly book, Site Reliability Engineering: How Google Runs Production Systems, written by members of Google’s Site Reliability Engineering (SRE) team. The philosophy is shared by lots of other companies, most notably Etsy, whose CTO John Allspaw writes:

A funny thing happens when engineers make mistakes and feel safe when giving details about it: they are not only willing to be held accountable, they are also enthusiastic in helping the rest of the company avoid the same error in the future. They are, after all, the most expert in their own error. They ought to be heavily involved in coming up with remediation items.

Blameless post mortems are indispensable if you value a culture where people can openly discuss their mistakes and learn from them. Don’t create scapegoats, make “error experts.” Google even shared their post mortem template so you can get started the next time the $#*! hits the fan.

Good Reads

More from Etsy: Morgue is their post mortem tracker, and the code is available on GitHub. Etsy’s Daniel Schauenberg also wrote up a more detailed look into their process.

Blameless doesn’t just mean not blaming others, it also means not blaming yourself. Jessica Harllee, an Etsy employee, shares her experience with blameless post mortems. “The guilt prevented me from going back on my own and reviewing what actually happened because I didn’t really want to relive it.”

“If you want your team to be audacious, you have to make being audacious the path of least resistance.” Astro Teller shares some of the ways X—formerly Google[x]—encourages open conversation about failure. I especially like his idea for pre-mortems where teams “speak up about things that might someday kill a project or slow it down — and make it business-as-usual for managers to hear and respond to these things.”

Goodbye Coach. Silicon Valley lost a giant over the weekend when the legendary Bill Campbell died. I only met Bill once and exchanged email with him a couple times, but his impact can be felt everywhere. The most poignant remembrance comes from Ben Horowitz, who shows us that above all, it was Bill’s humanity that made him one of a kind. So head to The Old Pro, toss out a few f-bombs, and knock one back for the Coach.

Originally published: April 20, 2016

Ken Norton is an executive coach who spent more than fourteen years at Google where he led product initiatives for Docs, Calendar, Google Mobile Maps, and GV (formerly Google Ventures).

  • How to Hire a Product Manager: the Classic Essay

    The classic essay that defined the product manager role
    What is product management? What makes a great product manager, and how do you become one? This is Ken Norton's classic essay on the role of product management that launched thousands of PM careers.

  • 10x Not 10%: Bold Product Strategy and Vision

    Product management by orders of magnitude
    In this ambitious essay, Ken Norton looks at the history of innovation and challenges product managers and product leaders to think bigger, to aim for 10x, not 10%.

  • Please Make Yourself Uncomfortable: Jazz and PMs

    What product managers can learn from jazz musicians
    What can product managers and product leaders learn from jazz, an art form that is all about improvisation, collaboration, and being willing to take risks?

  • Best Books for Product Managers [Updated for 2023]

    Essential product management reading
    Ken Norton shares his recommended books for product managers. The best books on product leadership, innovation, management, shipping winning products, and design thinking.

  • Ants & Aliens: Long-Term Product Vision & Strategy

    Why you need a thirty-year product vision (yes, thirty)
    How do you plan for the future and deliver an innovative and compelling product vision that will inspire your team to deliver winning products?

  • Meetings That Don’t Suck

    Break free from the tyranny of the conference room
    Most meetings suck, but it doesn't have to be that way. Ken Norton shows us how to break free and unsuck our meetings.

  • Building Products at Stripe

    Go deep, move fast, and build multi-decade abstractions
    What is Stripe's product culture like? Interview with a Stripe product leader demonstrate an embrace of going deep, moving fast, and maintaining a multi-decade perspective.

  • What Makes A Strong Product Culture?

    How a company's view of technology, product leadership, and empowerment contribute to product success
    Strong product cultures can produce winning products. They're places where product management is practiced (as we define it), where it is valued by the business, and where PMs can thrive and grow.

  • Building Products at Airbnb

    Snow White, storytelling, and a relentless focus on experiences
    What is Airbnb's product culture like? Interviews with Airbnb PMs demonstrate an embrace of Snow White, storytelling, and a relentless focus on experiences.

Executive Coaching

If you are interested in growing as a leader, I offer executive coaching. Schedule a free exploratory session.

Learn more »