1. Manage the client's expectations. Make sure the client knows from the start that critical incidents are natural. Disclose your backup plan and recovery process. 2. Assess the severity of the situation: What happened, who was affected and what's the impact of the issue? 3. Declare an incident: Does the issue have impact and complexity that requires a team effort? 4. Assign clear responsibilities for the team: Who will communicate the client? Who will fix the issue? Who will work on the restoration? 5. Have a transparency policy: Notify the client and take responsibility as a team. 6. Define a recovery and data restoration plan: Identify the bug causing the incident and issue a hotfix. Identify the latest backup with valid data. Define the time-frame not covered by the backup. Retrace the state of the system during the time-frame not covered by the backup. Write data restoration scripts. Specify all commands and steps required for the restoration. 7. Execute the plan while providing rapid status updates: Test the restoration locally. Backup the data. Restore the lost data. Identify the data that could not be recovered. Disclose the lost data to the client. 8. Write an incident postmortem with the team: What happened? Why the incident occurred? What was the resolution? And how effective? What would the team do differently? What problems did the team encounter? What actions will be taken to make sure the incident doesn’t happen again? 9. Update existing practices: Take time after the incident to read the postmortem and update what was necessary.
