Is there an Zonos outage?
Zonos status: Systems Active
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Zonos and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Zonos and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Zonos.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Classify | Active |
| Dashboard | Active |
| International Checkout | Active |
| Quoter | Active |
| Landed Cost | Active |
| Landed Cost API | Active |
| Landed Cost API (GraphQL) | Active |
| Landed Cost API (Legacy) | Active |
| Plugins | Active |
| BigCommerce Duty Tax | Active |
| Magento Duty Tax | Active |
| Salesforce Duty Tax | Active |
| Shopify Checkout | Active |
| Shopify Duty Tax | Active |
View the latest incidents for Zonos and check for official updates:
Description: **What products were affected and what was the impact?** Zonos Dashboard Impact: CRITICAL **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | Jun 13, 2023\] | 12:54 to 13:46 MDT | **How was the issue detected?** Internal reports of authorization failures and Dashboard becoming inaccessible. **What functionality was affected?** Zonos Dashboard was not accessible. **What problems did this cause?** Users were unable to access Dashboard to complete tasks. **What was the resolution of the problem and steps that are being taken for continued follow-up?** The issue was identified as an AWS Operational issue in the US-EAST-1 Region impacting an upstream service provider hosting our Front-End services for Dashboard. We were able to redeploy those services to an unaffected region to restore functionality. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** We are continually assessing and improving business continuity solutions throughout every layer of our tech stack to minimize downtime and automate recovery where possible.
Status: Postmortem
Impact: Critical | Started At: June 13, 2023, 7:25 p.m.
Description: **What products were affected and what was the impact?** Shopify Duty & Tax Impact: CRITICAL **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | May 20th, 2023 | 09:30 MST | | To: | May 20th, 2023 | 10:30 MST | **How was the issue detected?** At 9:31 AM MST, we began receiving alerts of increased latency with some Shopify Duty & Tax quote requests. **What functionality was affected?** Shopify Duty & Tax quotes for some customers. **What problems did this cause?** From approximately 9:30 am to 10:30 am MST, latency was sufficiently elevated to cause some quote requests from our Shopify Duty & Tax plugin to fail. **What was the resolution of the problem and steps that are being taken for continued follow-up?** We identified that the issue was caused by a scheduled maintenance job that obtained a lock on the database. The database lock significantly increased latency. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** The database lock was not necessary, and the locking code has been removed. This will prevent this issue from happening again in the future. Also, we are educating our engineers on the proper usage of database locking strategies, and implementing protection measures during the code review process. Additionally, we are improving our monitoring and on-call coverage to ensure faster response times to issues that impact shoppers.
Status: Postmortem
Impact: Critical | Started At: May 20, 2023, 3:30 p.m.
Description: **What products were affected and what was the impact?** Dashboard Quoter Impact: CRITICAL **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | May 2nd, 2023 | 15:30 MST | | To: | May 3rd, 2023 | 08:35 MST | **How was the issue detected?** A developer was using [dashboard.zonos.com](http://dashboard.zonos.com/) to make a quote and discovered it was broken. **What functionality was affected?** 100% of Quoter requests failed. **What problems did this cause?** Customers using Quoter were unable to get quotes. **What was the resolution of the problem and steps that are being taken for continued follow-up?** It was discovered that a deployment of Zonos Dashboard had a missing environment variable. The missing environment variable was added to the deployment, and Quoter functionality was restored. This was validated both via server logs and manual testing. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * We have improved our testing procedures to catch similar deployment issues in QA before the deployment reaches production. * We will no longer allow a build if an environment variable is missing. * We have modified our release schedule to allow for greater support coverage around releases. * We are working to improve alerting for issues related to Dashboard functionality and deployments.
Status: Postmortem
Impact: Critical | Started At: May 3, 2023, 2:09 p.m.
Description: **What products were affected and what was the impact?** Dashboard Quoter Impact: CRITICAL **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | May 2nd, 2023 | 15:30 MST | | To: | May 3rd, 2023 | 08:35 MST | **How was the issue detected?** A developer was using [dashboard.zonos.com](http://dashboard.zonos.com/) to make a quote and discovered it was broken. **What functionality was affected?** 100% of Quoter requests failed. **What problems did this cause?** Customers using Quoter were unable to get quotes. **What was the resolution of the problem and steps that are being taken for continued follow-up?** It was discovered that a deployment of Zonos Dashboard had a missing environment variable. The missing environment variable was added to the deployment, and Quoter functionality was restored. This was validated both via server logs and manual testing. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * We have improved our testing procedures to catch similar deployment issues in QA before the deployment reaches production. * We will no longer allow a build if an environment variable is missing. * We have modified our release schedule to allow for greater support coverage around releases. * We are working to improve alerting for issues related to Dashboard functionality and deployments.
Status: Postmortem
Impact: Critical | Started At: May 3, 2023, 2:09 p.m.
Description: **What products were affected and what was the impact?** All Zonos GraphQL services. Impact: CRITICAL **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | Mar 31, 2023 | Starting at 18:00 MDT | | Apr 1, 2023 | Ending at 12:45 MDT | **How was the issue detected?** On the morning of April 1, Shopify GraphQL customers began noticing issues with landed cost quotes and notified CS, who then escalated the issue to the Engineering team. **What functionality was affected?** All GraphQL services in the Zonos Cloud were impacted. **What problems did this cause?** Merchants on GraphQL were unable to receive shipment ratings and landed cost quotes. **What was the resolution of the problem and steps that are being taken for continued follow-up?** After being notified of the issue, we worked quickly to switch GraphQL merchants over to our REST endpoints, which were not experiencing any issues. We then identified the root cause of the issue with GraphQL: a code deployment that caused broke event serialization and caused synchronous events to fail. A weakness with synchronous event handling then caused the event failure to cascade to the cluster-level. We immediately released a fix to prevent future occurrences. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** Our monitoring and notification channels for production server clusters were focused on unhealthy target groups and container failures. Due to the nature of the failure, we didn't receive notifications for either. This is a clear gap in monitoring coverage at a cluster-wide level. To make sure this never happens again, we are configuring task-based monitoring outside of the clusters where we will: * query each service in the cluster directly for the minimum amount of tasks that should be running and the actual number of tasks that are running, * make mock requests to each service to make sure they are returning correct responses, and * direct these notifications to our alerting platform with "on-call" rotations to make sure there are no lapses in coverage. We have also improved the resiliency of our event system, such that even if there were a future issue with event serialization, it would have no effect upon our public GraphQL services.
Status: Postmortem
Impact: Critical | Started At: April 1, 2023, 6:27 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
