Time: May 21, 2020, 5:24 p.m.

Status: Postmortem

Update: **19 May 2020** ## **Summary** On May 19, 2020 at approximately ~16:15 UTC our third party SMTP email provider enforced a password reset on a subset of SEDNA sub-user accounts. This reset resulted in outbound message deliveries to fail. Once our records were updated, all sent messages were successfully sent. ## **Why this happened** Our third party provider alerted us via email that they were enacting this forced password reset due to a suspicion that login information had been posted online. Our IP logs show no access from an unauthorized location and there is no evidence that data in our accounts were accessed by a bad actor. As soon as we were alerted by our monitoring systems, we began resetting the sub-user accounts and updating our records. This restored email deliverability and all queued messages were delivered. ### **Actions and opportunities for improvement** 1. We are working with our third party SMTP provider to ensure that should something like this happen again, we have improved notification such that we can reset the accounts ourselves - **IN PROGRESS** 2. We will be adjusting our outbound Non-Delivery Receipts to include better messaging, specifically informing you that messages will be re-tried and sent via SEDNA. - **NOT STARTED** For any follow up questions please reach out to [[email protected]](mailto:[email protected]).

Time: May 19, 2020, 6:35 p.m.

Status: Resolved

Update: We have now fully resolved this issue and any messages that were not delivered have now been re-sent. We will be conducting an internal review of the incident in the next few days and will be posting our findings on the status page for public review. We apologies for the inconvenience and appreciate your patience while we resolved this issue.

Time: May 19, 2020, 6:11 p.m.

Status: Monitoring

Update: We have implemented a fix and are currently monitoring the results, we will provide a final update once we can confirm the issue is fully resolved.

Time: May 19, 2020, 5:23 p.m.

Status: Identified

Update: The team has identified the cause of this issue where messages are unable to send from SEDNA. They are currently looking into ways to resolve this.

Time: May 19, 2020, 4:59 p.m.

Status: Investigating

Update: We are currently investigating the cause of this issue and will provide more updates as they come available.

Time: April 24, 2020, 6:23 p.m.

Status: Postmortem

Update: Incident Report **SEDNA Incident** **21 April 2020** ## **Summary** On April 21st, 2020 SEDNA experienced an issue where our ingest service was operating at a much slower rate than usual which caused a large number of messages to be queued waiting to be ingested into SEDNA. This resulted in customers seeing major delays in messages arriving. ## **Why this happened** This issue occurred due to a misconfiguration in the ingest service which caused it to operate at a much slower rate. We reverted to a previous instance which allowed the ingest service to operate at its usual speed. **Actions and opportunities for improvement** To resolve the issue we rolled back the update, rewrote the code that resulted in the misconfiguration, and re-deployed the code.

Time: April 24, 2020, 6:23 p.m.

Status: Postmortem

Update: Incident Report **SEDNA Incident** **21 April 2020** ## **Summary** On April 21st, 2020 SEDNA experienced an issue where our ingest service was operating at a much slower rate than usual which caused a large number of messages to be queued waiting to be ingested into SEDNA. This resulted in customers seeing major delays in messages arriving. ## **Why this happened** This issue occurred due to a misconfiguration in the ingest service which caused it to operate at a much slower rate. We reverted to a previous instance which allowed the ingest service to operate at its usual speed. **Actions and opportunities for improvement** To resolve the issue we rolled back the update, rewrote the code that resulted in the misconfiguration, and re-deployed the code.

Time: April 21, 2020, 9:14 a.m.

Status: Resolved

Update: This incident has now been resolved and we can see that all messages have now been received into SEDNA.

Time: April 21, 2020, 9:14 a.m.

Status: Resolved

Update: This incident has now been resolved and we can see that all messages have now been received into SEDNA.

Time: April 21, 2020, 8:57 a.m.

Status: Monitoring

Update: A fix has been implemented and we are seeing queued messages now coming into SEDNA again, we will continue to monitor the results.

Time: April 21, 2020, 8:57 a.m.

Status: Monitoring

Update: A fix has been implemented and we are seeing queued messages now coming into SEDNA again, we will continue to monitor the results.

Time: April 21, 2020, 8:41 a.m.

Status: Investigating

Update: We have raised this with our development team who are actively investigating the issue. Currently appears to be affecting all customers with messages not being received into SEDNA, will revert when we have more information to provide.

Time: April 21, 2020, 8:41 a.m.

Status: Investigating

Update: We have raised this with our development team who are actively investigating the issue. Currently appears to be affecting all customers with messages not being received into SEDNA, will revert when we have more information to provide.