Incident response

What does the communications role do?

Gives status on a fixed timer, using the impact and the mitigation the commander has already confirmed. Guessed causes stay off that update until they are checked.

What does the operations lead do?

Directs the technical checks and the mitigation. One person driving keeps the others from trying conflicting fixes.

What sets severity?

Customer impact and how wide it is. An internal dashboard being slow is not the same incident as payments being down.

An internal tool is slow and no customer is affected. What do you do?

Treat it as work to schedule, unless it is blocking the people fixing a customer incident. It does not get a customer-facing severity.

What comes before a root cause?

A mitigation that restores the user: a rollback, a drain, a shed. Finding the bug can wait until the bleeding has stopped.

Why assign roles out loud?

So everyone knows who decides and who updates. A room of helpers with no roles debugs the same node and updates nobody.

What is a status cadence?

A promised interval for the next update, kept even when nothing changed. Silence reads as 'nobody is on it.'

Why one incident channel?

The facts live in one place. Side messages to individual engineers split the picture and interrupt the person fixing it.

What is a misrouted page?

The team paged cannot act. The page should name the service that can, or the paging rule is wrong and belongs in the review.

Three layers each retry three times. What happens?

One failure at the bottom becomes dozens of calls. The bottom stays down, and the top amplifies it.

A sale has overloaded the site. What do you shed first?

The traffic that is not the purchase, and the anonymous browsers, before you shed the people already paying.

What makes a dependency soft?

The feature still completes its main job when that call fails. If it cannot, the dependency is hard and needs a timeout and a capacity plan.

What is containment?

Stopping the failure from spreading: a breaker, a drain, a flag. It is a success even before you know the bug.

When is rollback the mitigation?

When the last change lines up with the start, and the previous version can still run on the current data.

When is rollback the wrong first move?

When the data has moved to a shape the old version cannot read, or the harm is a bad write that a rollback will not undo.

What is an incident freeze?

Stopping unrelated deploys so the picture stops changing. It has an owner and it ends when the incident ends.

Sales is messaging engineers for news. What should happen?

They get the cadence from the communications role. Engineers stay on the mitigation.

What goes on the timeline as it happens?

What changed, who did it, and the time. Reconstructing it the next day from memory drops the minute that mattered.

What does an incident handoff include?

The current mitigation, what was already tried, and the next check, said to a named person. They repeat it back before the previous lead leaves the channel.

When do you hand the commander role over?

Before judgment goes. A long incident with one tired commander starts repeating failed actions.

Why write severity definitions down?

So two incidents with the same impact get the same response. Otherwise the loudest customer sets the level.

What happens with a finished postmortem?

The actions get owners and dates, and the write-up is shared. A document nobody reads does not change the next incident.

What does blameless require in the write-up?

Names of systems and gaps, not a hunt for who typed the command. People speak up when that is true.

What is a useful action item?

A change in paging, a limit, or a dependency, with an owner and a date. The follow-up is done when that change is in production, which the next incident will test.

What is a tabletop exercise?

Walking through an incident on paper, in roles, before it happens. It finds the missing page and the missing access.

What access do you check before you need it?

Who can roll back, drain, and read the logs at 2 a.m. An incident is a bad time to discover the right is missing.

What belongs on a bridge call?

Decisions and blockers. Debugging out loud on the bridge drowns the commander. Take the deep dive in a side effort and report back.

What does a customer update say?

What is broken, what to do meanwhile, and when you will update. A suspected root cause can wait until you know it.

When do you raise severity?

When the impact grew or the mitigation failed. Leaving it low because you hope it is almost over hides the need for more help.

When do you call the incident over?

When the user-facing measure has recovered and stayed there, not when the dashboard first blips green.

What is the follow-up window?

The time after recovery when someone watches for the same symptom. Closing the channel at the first green misses the relapse.

Two people start two mitigations. What do you do?

The operations lead picks one. Parallel fixes that both change state make the next observation meaningless.

A deploy is in progress as the incident starts. What do you do?

Stop the rollout. Whether you roll back is a separate decision, made once, by the operations lead.

A flag is the suspected cause. What do you record?

Who turned it off, at what time, and what the SLI did after. The timeline is how you know it was the flag.

A bad write is the harm. What besides the code?

Stop the writer, then decide how to repair the rows. Restarting the process can write more bad rows.

What if you have no customer-comms person?

The commander names one. Leaving updates to whoever is free means they arrive late and disagree.

The failure is a vendor. What do you still own?

The mitigation users feel: a failover, a reduced feature, or a status page that says what still works and when the next update will come.

When does an incident become a security incident?

When data may have been exposed or access misused. The response then includes containment of access, not only restoring the error rate.

Ten pages fire for one outage. What do you want instead?

One page on the user symptom, with the rest as attached clues. Ten pages split the team across ten tickets.

When do you suppress a noisy page?

When it cannot be acted on, and you have a ticket to fix the rule. Suppressing a real symptom to get quiet is how the next outage hides.

The runbook is wrong in the incident. What do you do?

Deviate, say so, and fix the runbook in the actions. Following a wrong step because it is written extends the outage.

How do you state impact?

Which action failed, how many users or how much traffic, and the start time. The severity is chosen from those three, and the status update repeats them.

Why use one clock on the timeline?

Offsets between a log, a pager, and a laptop hide the order. Pick UTC and stick to it.

What do you not put in a customer update?

Guesses about blame, another customer's data, and a cause you have not confirmed.

What is the first hour after recovery for?

Confirming the measure stays healthy, capturing the timeline, and assigning the write-up while the memory is fresh.

Why practice the roles when nothing is wrong?

The first time someone commands should be a tabletop, with a clock and the real roles. You find a missing login or a missing page while users are still unaffected.

What is a deputy commander for?

To take over cleanly, and to watch the commander's blind spot. A single commander with no backup is a long incident away from a mistake.

How do you keep the incident from absorbing the company?

Severity decides who is required. Everyone else stays out and gets the status. A crowd in the channel slows the people typing the fix.

When is an action from last time actually done?

When the change is in production, not when the ticket moved. The next incident will test the ticket.