Is there an Sedna Systems outage?
Sedna Systems status: Systems Active
Last checked: 9 seconds ago
Get notified about any outages, downtime or incidents for Sedna Systems and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 9 seconds ago
Get notified about any outages, downtime or incidents for Sedna Systems and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Sedna Systems.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| SEDNA - Private Hosting | Active |
| SEDNA - www.sednanetwork.com | Active |
View the latest incidents for Sedna Systems and check for official updates:
Description: **SEDNA Incident** **30 September 2020** ## **Summary** Between the hours of 12:01-13:53 UTC users experienced an issue where messages they were sending were not being sent to the end recipients. ## **Why this happened** This happened due to a billing bug with SendGrid, who are our third party email provider which sends messages out from SEDNA, and a prematurely closed SendGrid support ticket opened 13 days prior to the incident, when the issue was first noticed. During the incident, a representative of all of SEDNA’s teams across the globe, including VP of Engineering and CEO were opening up every line of communication with SendGrid available. Unfortunately, SendGrid Support was experiencing limited Support Capacity due to a company event. Our time zone spanning team's persistence led to a line of communication opening, and an indefinite extension added to SEDNA’s account, allowing the mail to once again send. **Actions and opportunities for improvement** 1. SEDNA is increasing inter-department communication to deliberately and proactively inform teams of any potential downtime that may not be technical in nature, such as third party vendors experiencing internal billing issues. - **In Progress** 2. SEDNA is in talks with SendGrid to upgrade our SendGrid account with more dedicated support access - **In Progress** 3. SEDNA is considering a proposal to add a feature to the 2021 Roadmap to explore alternate options to SendGrid to fall back on in the event of SendGrid not being functional with enough notice. **- In Progress** 4. SEDNA’s SendGrid account has been adjusted until SendGrid resolve the bug on their end to ensure the billing issue does not arise again. - **Completed**
Status: Postmortem
Impact: Critical | Started At: Sept. 30, 2020, 12:09 p.m.
Description: Our 3rd party mail delivery platform has confirmed full resolution of the issue. As of 18:34 UTC all outbound SEDNA messages were resurrected and all services were functioning as intended. We do apologize for any teams that were met with the delivery delays. If there are any questions please do reach out to us at [email protected]. Thank you.
Status: Resolved
Impact: None | Started At: July 28, 2020, 5:10 p.m.
Description: **16 June 2020** ## **Summary** On June 16, 2020 at approximately ~14:38 UTC, we deployed a new version of SEDNA which included changes to how we retrieve events that are related to a message. This change had negative side effects on some of our larger clusters due to the large amount of data they contained, which caused the CPU on our database instances to be overwhelmed. This caused SEDNA to become unresponsive and failed to load content. ## **Why this happened** While we had tested our changes thoroughly, we failed to anticipate the stress on our larger database instances. As soon as we were alerted by our monitoring systems, we were able to rollback our changes to all of our clusters. **Actions and opportunities for improvement** 1. We are investigating putting restrictions on our database usage such that a single query can not overwhelm the entire database. 2. We will be looking at ways we can detect these kinds of problems much earlier, by having our test environments contain much larger sets of data. 3. Improved error messaging to provide better and earlier customer facing information so that customers are alerted sooner of an ongoing incident. For any follow-up questions please reach out to [[email protected]](mailto:[email protected]).
Status: Postmortem
Impact: Critical | Started At: June 16, 2020, 5:07 p.m.
Description: **16 June 2020** ## **Summary** On June 16, 2020 at approximately ~14:38 UTC, we deployed a new version of SEDNA which included changes to how we retrieve events that are related to a message. This change had negative side effects on some of our larger clusters due to the large amount of data they contained, which caused the CPU on our database instances to be overwhelmed. This caused SEDNA to become unresponsive and failed to load content. ## **Why this happened** While we had tested our changes thoroughly, we failed to anticipate the stress on our larger database instances. As soon as we were alerted by our monitoring systems, we were able to rollback our changes to all of our clusters. **Actions and opportunities for improvement** 1. We are investigating putting restrictions on our database usage such that a single query can not overwhelm the entire database. 2. We will be looking at ways we can detect these kinds of problems much earlier, by having our test environments contain much larger sets of data. 3. Improved error messaging to provide better and earlier customer facing information so that customers are alerted sooner of an ongoing incident. For any follow-up questions please reach out to [[email protected]](mailto:[email protected]).
Status: Postmortem
Impact: Critical | Started At: June 16, 2020, 5:07 p.m.
Description: **19 May 2020** ## **Summary** On May 19, 2020 at approximately ~16:15 UTC our third party SMTP email provider enforced a password reset on a subset of SEDNA sub-user accounts. This reset resulted in outbound message deliveries to fail. Once our records were updated, all sent messages were successfully sent. ## **Why this happened** Our third party provider alerted us via email that they were enacting this forced password reset due to a suspicion that login information had been posted online. Our IP logs show no access from an unauthorized location and there is no evidence that data in our accounts were accessed by a bad actor. As soon as we were alerted by our monitoring systems, we began resetting the sub-user accounts and updating our records. This restored email deliverability and all queued messages were delivered. ### **Actions and opportunities for improvement** 1. We are working with our third party SMTP provider to ensure that should something like this happen again, we have improved notification such that we can reset the accounts ourselves - **IN PROGRESS** 2. We will be adjusting our outbound Non-Delivery Receipts to include better messaging, specifically informing you that messages will be re-tried and sent via SEDNA. - **NOT STARTED** For any follow up questions please reach out to [[email protected]](mailto:[email protected]).
Status: Postmortem
Impact: Major | Started At: May 19, 2020, 4:59 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
