Summary
On Tuesday, 8/29, around 15:16 UTC, a few of our customers started reporting that tables sourced from Snowflake appeared empty (displaying “No Data” message) but the table summary showed a non-zero row count.
Our engineering team collaborated with Snowflake’s engineering team to address potential issues with Snowflake’s driver that were leading to incomplete payloads in Snowflake responses. After confirming it was a bug on Snowflake’s end, they rolled back server changes, resolving the issue.
Incident Start Time: Approximately 15:16 UTC Aug 29, 2023
Incident End Time: Approximately 15:26 UTC Aug 31, 2023
Root cause
The problem arose due to a combination of Snowflake’s server version 7.30.0, the configuration of the AWS VPC Interface Endpoints for Internal Stages feature by the customer, and the asynchronous query execution using the Snowflake driver.
This resulted in a bug that consistently returned malformed responses for sufficiently large results. A small number of Sigma Snowflake orgs had adopted server version 7.30.0 with the use of AWS VPC Interface Endpoints for Internal Stages feature, thus causing them to be Impacted.
Timeline
2023-08-29 15:16 UTC: We received reports of “No Data” messages within Sigma dashboards from a few of our customers.
2023-08-30 02:48 UTC: Sigma filed the support case with Snowflake.
2023-08-30 07:28 UTC: Snowflake support confirmed that there was a “new server roll out that coincided with when the issue was first observed”
2023-08-31 06:12 UTC: We reported to Snowflake that we’re able to consistently reproduce the issue with the standard Snowflake driver in async mode.
2023-08-31 15:26 UTC: Snowflake rolled back to 7.29.1 for impacted customers, resolving the issue.
Mitigations and current fixes
Snowflake rolled back the problematic server upgrade for the impacted customers.
Future corrective actions
-
We are enhancing our ability to replicate customer behavior for improved observability. As part of this effort, we have improved alerts to promptly identify issues of this kind. This involves enabling the execution of queries through the standard Snowflake driver in both synchronous and asynchronous modes, allowing us to more rapidly isolate such problems.
-
We’re enhancing our processes for a smoother escalation path with Snowflake.
We deeply apologize for the disruption and inconvenience you experienced as a result of this incident. Your trust is of utmost importance to us, and we are committed to taking the necessary measures to prevent similar incidents in the future. If you have any questions or concerns, please reach out to our Support team.
Thank you for your understanding.