1. Manage the client's expectations.
- Make sure the client knows from the start that critical incidents are natural.
- Disclose your backup plan and recovery process.
2. Assess the severity of the situation:
- What happened, who was affected and what's the impact of the issue?
3. Declare an incident:
- Does the issue have impact and complexity that requires a team effort?
4. Assign clear responsibilities for the team:
- Who will communicate the client?
- Who will fix the issue?
- Who will work on the restoration?
5. Have a transparency policy:
- Notify the client and take responsibility as a team.
6. Define a recovery and data restoration plan:
- Identify the bug causing the incident and issue a hotfix.
- Identify the latest backup with valid data.
- Define the time-frame not covered by the backup.
- Retrace the state of the system during the time-frame not covered by the backup.
- Write data restoration scripts.
- Specify all commands and steps required for the restoration.
7. Execute the plan while providing rapid status updates:
- Test the restoration locally.
- Backup the data.
- Restore the lost data.
- Identify the data that could not be recovered.
- Disclose the lost data to the client.
8. Write an incident postmortem with the team:
- What happened?
- Why the incident occurred?
- What was the resolution? And how effective?
- What would the team do differently?
- What problems did the team encounter?
- What actions will be taken to make sure the incident doesn’t happen again?
9. Update existing practices:
- Take time after the incident to read the postmortem and update what was necessary.
