Incident Start Time: 2023-08-14 at 13:17 UTC
Incident End Time: 2023-08-14 16:19 UTC
Summary
On August 14th, 2023 for a period of 3 hours, scheduled exports that contained elements with conditional formatting rules with fill color set to “None” failed with an error message of “An error has occurred, please try again later”.
Root Cause
This incident was caused by a code change which treated a variable for color conditional formatting as optional when it was in fact functionally required at runtime.
Timeline (UTC)
2023-08-14 at 13:17: First customer report of scheduled exports failing
2023-08-14 at 14:35: Issue is escalated, engineering begins investigating
2023-08-14 at 15:12: Root cause is identified, fix is built
2023-08-14 at 16:19: Fix is implemented in production, issue is mitigated
Response
- Root cause was identified within two hours of first report
- Fixed in production at three hours from first report
Forward-Looking Preventative Measures
- Improvements to proactive alerts on scheduled export health
- Implementing active monitoring of deployments by our incident resolution team
- Improved testing for changes to conditional formatting logic