Last checked: 7 minutes ago
Get notified about any outages, downtime or incidents for Wasabi Technologies and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Wasabi Technologies.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| AP-Northeast-1 (Tokyo) | Active |
| AP-Northeast-2 (Osaka) | Active |
| AP-Southeast-1 (Singapore) | Active |
| AP-Southeast-2 (Sydney) | Active |
| CA-Central-1 (Toronto) | Active |
| EU-Central-1 (Amsterdam) | Active |
| EU-Central-2 (Frankfurt) | Active |
| EU-South-1 (Milan) | Active |
| EU-West-1 (London) | Active |
| EU-West-2 (Paris) | Active |
| US-Central-1 (Texas) | Active |
| US-East-1 (N. Virginia) | Active |
| US-East-2 (N. Virginia) | Active |
| US-West-1 (Oregon) | Active |
| Wasabi Account Control API | Active |
| Wasabi Account Control Manager Console | Active |
| Wasabi Management Console | Active |
View the latest incidents for Wasabi Technologies and check for official updates:
Description: Between 10:32 UTC 2024-08-06 and 20:40 UTC 2024-08-07, we experienced three outages affecting both S3 and user services in all regions. Starting at 10:32 UTC 2024-08-06, our queueing service reached a full capacity state which impacted our database cache causing it to become unresponsive. The Wasabi Operations team initiated a restart to the primary database in an attempt to clear out all stale connections to the database while simultaneously clearing the queuing service queue. When this action failed to bring the database into a fully operational state, the secondary database instance was promoted to primary. At 11:20 UTC the S3 service was fully operational again. Between 13:17 UTC and 13:23 UTC, the database was restarted once more by Operations in order to fully incorporate our queueing service library. Between 02:55 UTC to 03:35 UTC on 2024-08-07, a second event occurred when our Operations team identified a configuration issue within the queueing service and the previously promoted secondary database instance. This configuration issue was causing timeouts to occur on user services such as our Web Console, WAC API, and WACM interface. Our Operations team then promoted the primary database back to production, alleviating these issues. There was no impact to S3 services during this event. Between 20:30 UTC to 20:44 UTC on 2024-08-07, a third event occurred when an automation cluster was failing to be seen by our automation service, causing a small decrease in accepted traffic to our S3 vaults. Our Operations team then recreated and redeployed this cluster, fully restoring the S3 service.
Status: Postmortem
Impact: Minor | Started At: Aug. 6, 2024, 12:09 p.m.
Description: Between 10:32 UTC 2024-08-06 and 20:40 UTC 2024-08-07, we experienced three instances affecting both S3 and user services in all regions. Starting at 10:32 UTC 2024-08-06, our queueing service reached a full capacity state which impacted our database cache causing it to become unresponsive. The Wasabi Operations team initiated a restart to the primary database in an attempt to clear out all stale connections to the database while simultaneously clearing the queuing service queue. When this action failed to bring the database into a fully operational state, the secondary database instance was promoted to primary. At 11:20 UTC the S3 service was fully operational again. Between 13:17 UTC and 13:23 UTC, the database was restarted once more by Operations in order to fully incorporate our queueing service library. Between 02:55 UTC to 03:35 UTC on 2024-08-07, a second event occurred when our Operations team identified a configuration issue within the queueing service and the previously promoted secondary database instance. This configuration issue was causing timeouts to occur on user services such as our Web Console, WAC API, and WACM interface. Our Operations team then promoted the primary database back to production, alleviating these issues. There was no impact to S3 services during this event. Between 20:30 UTC to 20:44 UTC on 2024-08-07, a third event occurred when an automation cluster was failing to be seen by our automation service, causing a small decrease in accepted traffic to our S3 vaults. Our Operations team then recreated and redeployed this cluster, fully restoring the S3 service.
Status: Postmortem
Impact: None | Started At: Aug. 6, 2024, 10:56 a.m.
Description: From 2024-06-19 00:57 UTC to 2024-06-19 04:36 UTC, we experienced issues in all Wasabi regions affecting S3, Wasabi Account Control \(WAC\) API, and Console services. Our WAC API and Console services were affected between 00:57 UTC - 02:45 UTC and 03:55 UTC - 04:36 UTC, while our S3 service was affected between 01:55 UT - 02:45 UTC. At 00:57 UTC it was noticed by our Operations Team that there were connection issues between our Console and WAC API services and our global database with no impact to other \(S3\) services. While debugging these connection issues, it was noted at 1:55 UTC that S3 services were now also affected. After further investigation from the Operations Team, it was seen that our queuing service within the global database had failed, resulting in these services unable to communicate with the database. To correct this issue, Operations had to restart both the global database instance, as well as the queuing service to restore connections for the affected services. Once both the database and queuing service had re-initialized at 2:45 UTC, services were restored. At 03:55 UTC, it was noticed that the Console and WAC API were beginning to exhibit the same symptoms as before with a loss of connectivity to the database. Upon further investigation from the Operations Team, it was noted that the queuing service had run low on resources. Operations increased available resources to the service and at 04:10 UTC the services were brought back online. At 04:36 UTC it was confirmed that all services were operational.
Status: Postmortem
Impact: Minor | Started At: June 19, 2024, 1:04 a.m.
Description: From 2024-06-10 18:25 UTC to 2024-06-10 18:43 UTC, we experienced an issue in our US-EAST-1 \(Ashburn\) vault which resulted in degraded performance in the region. This impacted customer’s ability to send and retrieve data to/from their buckets in the region, and access services such as the Wasabi Web Console and Wasabi Account Control API. At around 17:48 UTC, the Wasabi Operations team began deploying updates in the US-EAST-1 region. After the update was completed and the process of restarting internal services was started, one service failed, causing communication between our client servers and database to fail. Our Operations Team took action by removing the applied configuration of the failed service and redeploying the service. After this action, the internal services came up successfully and communication between our client servers and database was restored, restoring service in the region. At 18:43 UTC all services in the US-EAST-1 region were operational.
Status: Postmortem
Impact: Minor | Started At: June 10, 2024, 6:33 p.m.
Description: From 2024-05-30 15:36 UTC to 2024-05-31 05:00 UTC, we experienced elevated temperatures in our us-east-1 and us-east-2 regions. This issue was due to a cooling system failure in one of the Iron Mountain Datacenter \(IMDC\) buildings which hosts our storage sub-system and database servers. This failure caused temperatures to pass a safe operating threshold, causing systems to involuntarily shutdown in order to prevent any damage. Between 2024-05-30 17:00:00 UTC and 2024-05-31 05:00:00 UTC, our Operations Team worked on each server rack to bring each individual component back up safely, ran integrity checks on the hardware and replaced faulty equipment. At 2024-05-31 05:00:00 UTC, our services were returned to a fully operational status.
Status: Postmortem
Impact: Minor | Started At: May 30, 2024, 3:58 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
