Time: Aug. 26, 2022, 6:46 p.m.

Status: Resolved

Update: This incident has been resolved and the Drag-and-Drop Editor is fully operational.

Time: Aug. 26, 2022, 6:07 p.m.

Status: Monitoring

Update: The issue has been identified and a fix has been implemented. We are working on a full resolution of any remaining impact.

Time: Aug. 26, 2022, 3:53 p.m.

Status: Identified

Update: We are currently experiencing an outage to our Drag-and-Drop Editor used by all Simon Mail customers. This is leading to errors creating, editing, and viewing templates in the Drag-and-Drop Editor. Any Simon Mail flows leveraging templates that had already been saved are not impacted and templates are being sent appropriately to customers.

Time: Aug. 5, 2022, 8:16 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 5, 2022, 2:50 p.m.

Status: Investigating

Update: We've discovered a transient error with the Attentive integration and are currently investigating. While we investigate the issue, we have paused the integration.

Time: Aug. 1, 2022, 7:51 p.m.

Status: Postmortem

Update: # **Overview** Last week on July 27th, Amazon Web Services \(AWS\) was performing routine maintenance on a piece of infrastructure that Simon uses to power segmentation. During the maintenance window, AWS' routine failed and this left parts of Simon's segmentation functionality temporarily unavailable. This had ramifications for use of several Simon Data platform interfaces as well as on-time delivery of downstream flows & journeys. The attached file documents the steps we took to react to and remediate the issue while in parallel escalating the emergency to AWS who ultimately was able to identify the source of the issue and fix. AWS acknowledged that this bug was introduced by AWS and that the remediation they implemented on our infrastructure is a permanent solution.Executive Summary # **Executive Summary** In the early morning EST of July 27th 2022, a maintenance routine to an Amazon-managed service that Simon uses to power segmentation left that service unavailable. Simon was immediately paged and our on-call support reacted. Unfortunately, the most direct and fastest-acting remediations were not possible due to persistent hardware failure from Amazon, requiring Simon to fall back on reconstructing data on new services. In the end, Amazon rectified the bug in their maintenance program and remediation took effect before Simon finished reconstructing data on new services - rendering that effort unnecessary. No data or messages were lost, but both data refreshes and active or planned campaigns were delayed while Simon performed an emergency migration to a functional database. This resulted in delayed data refreshes and campaign launches, in addition to leaving the core segmentation product, and those products that depend upon it \(like unified contact view and selecting sample contacts in content\), to be unavailable until the migration completed. # **Root Cause** Simon Data splits data, leveraged by our platform, by kind into different databases and business logic into other databases. Each database comes equipped with multiple, redundant nodes. We routinely exercise node failover logic during upgrades, maintenance, and failures. Maintenance on active Amazon Web Services \(AWS\) databases that Simon Data uses is frequent and rarely are there unplanned outages because maintenance processes with AWS are typically predictable. On July 27th, 2022, AWS’ maintenance routine introduced a subtle bug that prevented its maintenance from finishing. As a result, the segmentation database was being reported as still under maintenance. The Simon Data on-call team was alerted and immediately reacted, however contacting our technical support at AWS took longer than expected and the Simon on-call team only received 1 downstream page. While the Simon team performed an emergency data reconstruction / migration to healthy infrastructure, the Simon team escalated through multiple teams at AWS. Despite the parallel escalations, it took longer than normal to have AWS view this incident as an emergency situation instead of an unhealthy situation. Once recognized as an emergency situation, AWS resolved quickly and this happened before Simon’s internal reconstruction / migration completed. AWS has provided their post mortem to Simon Data: “_temporary tables were inadvertently created \[by AWS\] during a maintenance period that caused a corruption of metadata and prevented the cluster from powering on”_ # **Impact Analysis** All customers using one of our specific segmentation databases saw interruptions in service in Unified Contact View \(UCV\), Segmentation, and content loading tools in Simon # **Remediation Plan** ## **Quicker Time to Detection & Alerting** Simon Data has reviewed the process we use for publishing incidents to our status page. We have cut out the manual steps which delayed the alert to our customers longer than expected. ## **Quicker Time to Resolution & Recovery** Simon Data has revisited design and usage of our segmentation databases such that a new process was added to validate that maintenance has been conducted correctly. If it doesn’t finish as intended, immediate migration to another segmentation database will occur to stem interruption of service for Simon Data customers.

Time: July 27, 2022, 6:52 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: July 27, 2022, 2:21 p.m.

Status: Monitoring

Update: Approximately 20% of affected client organization's pipes have been fully restored. We will continue to update here as we restore the remainder of affected pipes.

Time: July 27, 2022, 2:17 p.m.

Status: Monitoring

Update: The fix has been implemented and we are starting to recover pipes of affected client orgs.

Time: July 27, 2022, 12:23 p.m.

Status: Monitoring

Update: A fix has been identified and is currently being implemented. We will provide further updates as the fix is rolled out.

Time: July 27, 2022, 11:39 a.m.

Status: Identified

Update: We have identified this issue due to unplanned maintenance by AWS. We have escalated to our technical support team at the provider for immediate assistance and will update this status soon.

Time: June 16, 2022, 8:02 p.m.

Status: Resolved

Update: This issue is now resolved. The root cause was a build up of queued tasks, resulting in a slowdown of site performance. We have addressed the performance impact and are also working on improvements to increase our processing capacity. Thank you for your patience as we worked to resolve this!

Time: June 16, 2022, 8:02 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: June 16, 2022, 7:25 p.m.

Status: Identified

Update: The Simon Data platform is currently experience an issue that is causing a slower than normal user experience. We are actively remediating the issue and will post an update as soon as the issue is resolved.

Time: June 11, 2022, 5:51 a.m.

Status: Resolved

Update: This incident has been resolved and all ETFs are now in full recovery.

Time: June 11, 2022, 5:18 a.m.

Status: Investigating

Update: We are currently seeing issues with the AWS Lambda service that writes events from Kinesis to our Postgres DB. This is leading to a failure of some Event Triggered Flows. Our engineering team is currently investigating this issue.