Postmortem For April 20, 2023 Materialization and Scheduled Export Incident - Sigma AWS & Sigma GCP


On April 20th 2023, several Sigma organizations experienced degraded performance and sporadic failures of materializations and scheduled exports. A code change to our connection indexing system, which catalogs database objects for all organizations’ connections, introduced a bug that overloaded our task scheduling system, preventing scheduled runs for materializations and scheduled exports from running. A fix for this issue was implemented at approximately 12:20am UTC on April 21st, 2023.

Incident Start Time: Approximately 11:50am UTC April 20th, 2023

Incident End Time: Approximately 12:20am UTC April 21st, 2023

Root Cause

A code change to our connection indexing logic contained a bug that led to Sigma running indexing tasks indefinitely. The indexer shares infrastructure with scheduled workloads like materializations and scheduled exports. Due to contention in the task runner, users experienced degraded performance and sporadic failures of materializations and scheduled exports.

Mitigations & Fixes

At 4:55am UTC on Thursday, April 20th, timeout errors as a result of connection indexing were identified. In order to mitigate these timeout errors, connection indexing tasks were halted, which may have resulted in users experiencing a slowdown in navigating and searching through newly created connections. Our team identified and fixed the bug in the connection indexing logic, after which we re-enabled new connection indexing tasks. As a result, materializations and scheduled exports were able to run without issue.

Future Corrective Actions

  • Actively prioritizing the improvement of observability across our system, including in our connection cataloging service, and conducting comprehensive reviews to identify any areas that require further enhancement.
  • Implementation of supplementary tests to ensure similar issues do not impact our production environment
  • Improved isolation of time-sensitive tasks like materialization and scheduled exports to prevent them from being affected by delays in other task processing
1 Like

Added AWS, Export, GCP, Materialization, Scheduled