Postmortem for Aug 9th, 2023: Users unable to enter edits to workbooks

Incident Start Time: 2023-08-09 21:56 UTC
Incident End Time: 2023-08-09 22:22 UTC


On August 9th, 2023 for a period of about 30 minutes, users who had recently made edits to a workbook received an error while attempting to make further edits to that Workbook. This affected roughly 20 individual external users, who were unable to make edits during that time.

Root Cause

This incident was caused by a change to our code which handles Workbook edits made during live edit sessions. This change resulted in new Workbook edits for users with current live edit sessions being improperly formed such that they could not be retained.

Timeline (UTC)

2023-08-09 at 19:50: Changes to our live edit code were pushed to internal testing environment. Engineering monitors for errors.
2023-08-09 at 21:56: Changes to live edit code deployed to production.
2023-08-09 at 21:59: Reports of users unable to make edits in workbooks on internal testing environment. Engineering investigates causes.
2023-08-09 at 22:22: Engineering rolls back changes to live edit code in production. Users are able to make edits again.


Changes to live edit code were removed from production within 30 minutes of first reports of the issue in internal testing environment

Forward-Looking Preventative Measures

  • Improve live edit code to better handle similar changes which could result in improperly formed edits, including typescript checks as an additional layer of prevention
  • Add unit tests which recreate this scenario
  • Add checks for health of live edits in internal testing environment

Added Live-Edit, Workbooks