Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Courier and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Courier.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
Courier | Active |
API | Active |
Automations | Active |
Courier Inbox | Active |
Observability | Active |
Web Application | Active |
Hosting Infrastructure | Active |
AWS dynamodb-us-east-1 | Active |
AWS elasticsearch-us-east-1 | Active |
AWS kinesis-us-east-1 | Active |
AWS lambda-us-east-1 | Active |
AWS sqs-us-east-1 | Active |
Integrations: Inbound to Courier | Active |
Segment Destinations | Active |
Integrations: Outbound from Courier | Maintenance |
Mailgun API | Active |
Mailgun Outbound Delivery | Active |
Mailjet REST API | Active |
Mailjet SEND API | Active |
Nexmo Outbound SMS | Active |
Plivo SMS API | Active |
Segment Cloud Sources | Active |
Segment Data Ingestion (Tracking) API | Active |
SendGrid API | Active |
SendGrid SMTP | Active |
Slack Apps/Integrations/APIs | Active |
SparkPost SMTP API - EUROPE | Active |
SparkPost SMTP API - USA | Active |
SparkPost SMTP Delivery - EUROPE | Active |
SparkPost SMTP Delivery - USA | Maintenance |
Twilio AUTOPILOT | Active |
Twilio SMS | Active |
View the latest incidents for Courier and check for official updates:
Description: We have finished measuring the impact and will be reaching out to affected customers.
Status: Resolved
Impact: Major | Started At: July 25, 2022, 8:45 p.m.
Description: ### Impact Courier experienced delayed message delivery in its send pipeline impacting 0.1% of messages from 12:50pm to 21:50pm PT on 7/14. No messages were dropped as a result of the incident. 99.9% of send calls experienced no delivery delay. The average message send delay was 3 hours and 20 minutes for impacted messages. #### Root Cause Courier uses feature flags to safely roll out new features. Due to a misconfiguration of a flag, a larger than expected volume of send requests were included in a validation experiment meant to verify a refactor of the send pipeline was safe to rollout. These requests added significant additional load on key stages of the send pipeline, and caused non-validation related requests to queue. #### Remediation Courier incrementally scaled up processing capacity in the send pipeline to work through the large accumulated backlog of messages. Additionally, a hotfix release was pushed to production in order to drop validation messages that had already entered the send pipeline. #### Follow up actions * Courier has established a process to better validate flag configuration in the future, as well as made changes to its feature flag helper library to make use less error-prone. * Courier has created an incident playbook to guide on-call engineers through options to quickly scale up message processing in the send pipeline.
Status: Postmortem
Impact: Minor | Started At: July 14, 2022, 9:34 p.m.
Description: ### Impact Courier experienced delayed message delivery in its send pipeline impacting 0.1% of messages from 12:50pm to 21:50pm PT on 7/14. No messages were dropped as a result of the incident. 99.9% of send calls experienced no delivery delay. The average message send delay was 3 hours and 20 minutes for impacted messages. #### Root Cause Courier uses feature flags to safely roll out new features. Due to a misconfiguration of a flag, a larger than expected volume of send requests were included in a validation experiment meant to verify a refactor of the send pipeline was safe to rollout. These requests added significant additional load on key stages of the send pipeline, and caused non-validation related requests to queue. #### Remediation Courier incrementally scaled up processing capacity in the send pipeline to work through the large accumulated backlog of messages. Additionally, a hotfix release was pushed to production in order to drop validation messages that had already entered the send pipeline. #### Follow up actions * Courier has established a process to better validate flag configuration in the future, as well as made changes to its feature flag helper library to make use less error-prone. * Courier has created an incident playbook to guide on-call engineers through options to quickly scale up message processing in the send pipeline.
Status: Postmortem
Impact: Minor | Started At: July 14, 2022, 9:34 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: June 21, 2022, 10:41 p.m.
Description: AWS issue has been resolved with services operating normally. Courier systems are operational and healthy.
Status: Resolved
Impact: Minor | Started At: June 10, 2022, 3:58 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.