Time: Aug. 7, 2024, 7:45 p.m.

Status: Resolved

Update: A hotfix has deployed which has been verified to correct the underlying behavior of the author alias reverting on campaign drafts. Marking all components fully operational and resolving incident.

Time: Aug. 7, 2024, 6:01 p.m.

Status: Identified

Update: We have determined that this problem is related to a regression introduced by a bug fix where a restricted user could potentially publish as an unassigned email alias. We are going to roll back that change in a forthcoming hotfix which should mitigate the problem, and rework a more comprehensive fix to the original bug. We are going through the change management process now to get the hotfix reviewed, verified, and pushed to production.

Time: Aug. 7, 2024, 5:28 p.m.

Status: Investigating

Update: We are continuing to investigate this Studio service disruption. Another update will be provided within the next 1 hour.

Time: Aug. 7, 2024, 4:31 p.m.

Status: Investigating

Update: We are currently investigating reports of email alias in a campaign results reverting to the default in Studio. We will provide you with an update with 1 hour.

Time: Aug. 8, 2024, 12:58 a.m.

Status: Postmortem

Update: **Summary:** On July 23rd, 2024, beginning at approximately 6:04 AM ET \(10:04 UTC\), we started receiving reports of published email campaigns not being received by the intended audiences. The reports were from customers hosted on the EU Firstup platform. ‌ **Scope and Impact:** The scope of this incident was restricted to customers hosted on the EU Firstup platform and impacted email campaigns on the EU Firstup platform published between 4:00 AM ET \(08:00 UTC\) and 7:43 AM ET \(11:43 UTC\). ‌ **Root Cause:** An update to a user’s profile erroneously triggered an update of over 400 campaigns authored by that user simultaneously, which overloaded the backend user planning processes, and caused delivery delays of other campaigns. ‌ **Mitigation:** The system recovered on its own after all this user’s campaigns were updated, and processing for the delivery of other campaigns resumed. ‌ **Recurrence Prevention:** The below changes have been implemented to ensure campaign delays are prevented due to processing spikes from specific user planning processes: * Enhanced Monitoring: By the time investigation into the incident began, the processing queue self recovered and resumed delivery. To prevent this, we have added a Delivery specific time threshold to our processing monitors which will automatically bring additional cloud compute capacity online to help prevent a deadlock condition due to a single planning process usurping shared resources. * Updated our regression testing and runbooks to include steps to troubleshoot delayed campaigns linked from planner queues activities.

Time: Aug. 6, 2024, 6:32 p.m.

Status: Resolved

Update: This platform service degradation is now resolved, and an RCA will be provided once a full incident postmortem has been completed.

Time: July 23, 2024, 11:47 a.m.

Status: Monitoring

Update: Queued campaigns are now successfully being delivered and we are currently monitoring the issue.

Time: July 23, 2024, 11:42 a.m.

Status: Investigating

Update: We are investigating a potential service disruption affecting campaign sending from EU communities. Users in communities hosted on our EU infrastructure may not be receiving campaign emails, or facing email delays at this time. Our next update will be in 30 minutes.

Time: July 26, 2024, 5:42 p.m.

Status: Postmortem

Update: **Summary:** On July 10th, 2024, beginning at approximately 1:30 PM ET \(17:30 UTC\), we started receiving reports of published email campaigns that hadn’t been delivered to the intended audiences in over 1 hour. Due to the number of reports received, a platform incident was declared at 2:18 PM ET \(18:18 UTC\) and an incident response team began investigating these reports. Another platform incident was declared on July 11th, 2024, after reports of audiences being inaccessible or taking too long to load were received. ‌ **Impact:** The service degradation was intermittent in nature, and the impact was restricted to the US platform for access to some audiences and some campaigns published on July 10th, 2024, at or after 11:20 AM ET \(15:20 UTC\) through July 15th, 2024, at 6:23 PM ET \(22:23 UTC\). **Root Cause:** Both incidents stemmed from an overload of the ElasticSearch service, which resolves Audiences to User IDs and email addresses. A surge in error messages temporarily stored in a queue \(for messages ElasticSearch couldn't process\) overwhelmed the service, causing it to intermittently stop serving requests until it could catch up. ‌ **Mitigation:** The issue was immediately addressed by reducing the number of workers sending requests to ElasticSearch and increasing the number of nodes processing those requests. This reduced the strain on ElasticSearch, allowing the request queue to clear faster. Additionally, the error messages were manually reprocessed, making audiences accessible and campaigns publishable again. ‌ **Recurrence Prevention:** Errors in the queue are normal and typically resolve through automatic reprocessing. However, to prevent future occurrences: * We doubled ElasticSearch’s processing power on July 15th, 2024, at 6:23 PM ET to better handle any spikes. * We enabled additional monitoring and dashboards for early detection and mitigation of potential issues. * We will investigate and address the sources of the errors to ensure a healthier service.

Time: July 26, 2024, 5:42 p.m.

Status: Postmortem

Update: **Summary:** On July 10th, 2024, beginning at approximately 1:30 PM ET \(17:30 UTC\), we started receiving reports of published email campaigns that hadn’t been delivered to the intended audiences in over 1 hour. Due to the number of reports received, a platform incident was declared at 2:18 PM ET \(18:18 UTC\) and an incident response team began investigating these reports. Another platform incident was declared on July 11th, 2024, after reports of audiences being inaccessible or taking too long to load were received. ‌ **Impact:** The service degradation was intermittent in nature, and the impact was restricted to the US platform for access to some audiences and some campaigns published on July 10th, 2024, at or after 11:20 AM ET \(15:20 UTC\) through July 15th, 2024, at 6:23 PM ET \(22:23 UTC\). **Root Cause:** Both incidents stemmed from an overload of the ElasticSearch service, which resolves Audiences to User IDs and email addresses. A surge in error messages temporarily stored in a queue \(for messages ElasticSearch couldn't process\) overwhelmed the service, causing it to intermittently stop serving requests until it could catch up. ‌ **Mitigation:** The issue was immediately addressed by reducing the number of workers sending requests to ElasticSearch and increasing the number of nodes processing those requests. This reduced the strain on ElasticSearch, allowing the request queue to clear faster. Additionally, the error messages were manually reprocessed, making audiences accessible and campaigns publishable again. ‌ **Recurrence Prevention:** Errors in the queue are normal and typically resolve through automatic reprocessing. However, to prevent future occurrences: * We doubled ElasticSearch’s processing power on July 15th, 2024, at 6:23 PM ET to better handle any spikes. * We enabled additional monitoring and dashboards for early detection and mitigation of potential issues. * We will investigate and address the sources of the errors to ensure a healthier service.

Time: July 26, 2024, 5:42 p.m.

Status: Resolved

Update: This incident is now resolved, and all associated services are fully operational.

Time: July 26, 2024, 5:42 p.m.

Status: Resolved

Update: This incident is now resolved, and all associated services are fully operational.

Time: July 15, 2024, 8:59 p.m.

Status: Monitoring

Update: Studio performance has now been restored and all functionality should be available. We will be placing these services under monitoring for now.

Time: July 15, 2024, 8:59 p.m.

Status: Monitoring

Update: Studio performance has now been restored and all functionality should be available. We will be placing these services under monitoring for now.

Time: July 15, 2024, 8:15 p.m.

Status: Investigating

Update: We continue to investigate the recurrence of this issue and will provide another update within 1 hour.

Time: July 15, 2024, 8:15 p.m.

Status: Investigating

Update: We continue to investigate the recurrence of this issue and will provide another update within 1 hour.

Time: July 15, 2024, 7:13 p.m.

Status: Investigating

Update: We have received reports of a recurrence of this issue (including delayed campaign deliveries), and are actively investigating it again. Another update within 1 hour.

Time: July 15, 2024, 7:13 p.m.

Status: Investigating

Update: We have received reports of a recurrence of this issue (including delayed campaign deliveries), and are actively investigating it again. Another update within 1 hour.

Time: July 11, 2024, 9:32 p.m.

Status: Monitoring

Update: Studio performance has now been restored and all functionality should be available. We will be placing these services under monitoring for now.

Time: July 11, 2024, 9:32 p.m.

Status: Monitoring

Update: Studio performance has now been restored and all functionality should be available. We will be placing these services under monitoring for now.

Time: July 11, 2024, 9:21 p.m.

Status: Identified

Update: We continue to work on a fix for this issue and will provide another update within the next hour, or as soon as the fix is deployed.

Time: July 11, 2024, 9:21 p.m.

Status: Identified

Update: We continue to work on a fix for this issue and will provide another update within the next hour, or as soon as the fix is deployed.

Time: July 11, 2024, 8:16 p.m.

Status: Identified

Update: We continue to work on a fix for this issue and will provide another update within the next hour, or as soon as the fix is deployed.

Time: July 11, 2024, 7:15 p.m.

Status: Identified

Update: We have identified the cause of this service degradation, and are working to mitigate the issue. Another update within 1 hour.

Time: July 11, 2024, 7:15 p.m.

Status: Identified

Update: We have identified the cause of this service degradation, and are working to mitigate the issue. Another update within 1 hour.

Time: July 11, 2024, 7:07 p.m.

Status: Investigating

Update: We are currently investigating reports of slow Studio performance with errors loading some pages, including Audiences. We will provide you with an update with 1 hour.

Time: July 11, 2024, 7:07 p.m.

Status: Investigating

Update: We are currently investigating reports of slow Studio performance with errors loading some pages, including Audiences. We will provide you with an update with 1 hour.

Time: July 26, 2024, 5:40 p.m.

Status: Postmortem

Update: **Summary:** On July 10th, 2024, beginning at approximately 1:30 PM ET \(17:30 UTC\), we started receiving reports of published email campaigns that hadn’t been delivered to the intended audiences in over 1 hour. Due to the number of reports received, a platform incident was declared at 2:18 PM ET \(18:18 UTC\) and an incident response team began investigating these reports. Another platform incident was declared on July 11th, 2024, after reports of audiences being inaccessible or taking too long to load were received. ‌ **Impact:** The service degradation was intermittent in nature, and the impact was restricted to the US platform for access to some audiences and some campaigns published on July 10th, 2024, at or after 11:20 AM ET \(15:20 UTC\) through July 15th, 2024, at 6:23 PM ET \(22:23 UTC\). **Root Cause:** Both incidents stemmed from an overload of the ElasticSearch service, which resolves Audiences to User IDs and email addresses. A surge in error messages temporarily stored in a queue \(for messages ElasticSearch couldn't process\) overwhelmed the service, causing it to intermittently stop serving requests until it could catch up. ‌ **Mitigation:** The issue was immediately addressed by reducing the number of workers sending requests to ElasticSearch and increasing the number of nodes processing those requests. This reduced the strain on ElasticSearch, allowing the request queue to clear faster. Additionally, the error messages were manually reprocessed, making audiences accessible and campaigns publishable again. ‌ **Recurrence Prevention:** Errors in the queue are normal and typically resolve through automatic reprocessing. However, to prevent future occurrences: * We doubled ElasticSearch’s processing power on July 15th, 2024, at 6:23 PM ET to better handle any spikes. * We enabled additional monitoring and dashboards for early detection and mitigation of potential issues. * We will investigate and address the sources of the errors to ensure a healthier service.

Time: July 26, 2024, 5:39 p.m.

Status: Resolved

Update: This incident is now resolved, and all associated services are fully operational.

Time: July 10, 2024, 6:35 p.m.

Status: Monitoring

Update: This issue has now been mitigated, and customers whose campaign emails had not been delivered should start seeing them coming in now. We will provide you with another update as more information is made available.

Time: July 10, 2024, 6:24 p.m.

Status: Investigating

Update: We have received reports where campaign emails are delayed or have not been delivered at all after an extended period of time. We are investigating these reports and will provide you with an update within 1 hour.

Time: July 26, 2024, 5:40 p.m.

Status: Postmortem

Update: **Summary:** On July 10th, 2024, beginning at approximately 1:30 PM ET \(17:30 UTC\), we started receiving reports of published email campaigns that hadn’t been delivered to the intended audiences in over 1 hour. Due to the number of reports received, a platform incident was declared at 2:18 PM ET \(18:18 UTC\) and an incident response team began investigating these reports. Another platform incident was declared on July 11th, 2024, after reports of audiences being inaccessible or taking too long to load were received. ‌ **Impact:** The service degradation was intermittent in nature, and the impact was restricted to the US platform for access to some audiences and some campaigns published on July 10th, 2024, at or after 11:20 AM ET \(15:20 UTC\) through July 15th, 2024, at 6:23 PM ET \(22:23 UTC\). **Root Cause:** Both incidents stemmed from an overload of the ElasticSearch service, which resolves Audiences to User IDs and email addresses. A surge in error messages temporarily stored in a queue \(for messages ElasticSearch couldn't process\) overwhelmed the service, causing it to intermittently stop serving requests until it could catch up. ‌ **Mitigation:** The issue was immediately addressed by reducing the number of workers sending requests to ElasticSearch and increasing the number of nodes processing those requests. This reduced the strain on ElasticSearch, allowing the request queue to clear faster. Additionally, the error messages were manually reprocessed, making audiences accessible and campaigns publishable again. ‌ **Recurrence Prevention:** Errors in the queue are normal and typically resolve through automatic reprocessing. However, to prevent future occurrences: * We doubled ElasticSearch’s processing power on July 15th, 2024, at 6:23 PM ET to better handle any spikes. * We enabled additional monitoring and dashboards for early detection and mitigation of potential issues. * We will investigate and address the sources of the errors to ensure a healthier service.

Time: July 26, 2024, 5:39 p.m.

Status: Resolved

Update: This incident is now resolved, and all associated services are fully operational.

Time: July 10, 2024, 6:35 p.m.

Status: Monitoring

Update: This issue has now been mitigated, and customers whose campaign emails had not been delivered should start seeing them coming in now. We will provide you with another update as more information is made available.

Time: July 10, 2024, 6:24 p.m.

Status: Investigating

Update: We have received reports where campaign emails are delayed or have not been delivered at all after an extended period of time. We are investigating these reports and will provide you with an update within 1 hour.