Is there an Sauce Labs outage?
Sauce Labs status: Systems Active
Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Sauce Labs and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Sauce Labs and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Sauce Labs.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| API Testing | Active |
| EU-Central | Active |
| US-West | Active |
| Automated Browser Testing | Active |
| EU-Central | Active |
| US-West | Active |
| Automated Real Device Testing | Active |
| EU-Central | Active |
| US-East | Active |
| US-West | Active |
| Automated Virtual Mobile Device Testing | Active |
| EU-Central | Active |
| US-West | Active |
| Billing | Active |
| EU-Central | Active |
| US-West | Active |
| Insights | Active |
| EU-Central | Active |
| US-West | Active |
| IPSec VPN | Active |
| EU-Central | Active |
| US-West | Active |
| Live Browser Testing | Active |
| EU-Central | Active |
| US-West | Active |
| Live Real Device Testing | Active |
| EU-Central | Active |
| US-East | Active |
| US-West | Active |
| Live Virtual Mobile Device Testing | Active |
| EU-Central | Active |
| US-West | Active |
| Mobile App Distribution (TestFairy) | Active |
| TestFairy Platform | Active |
| TestFairy UI | Active |
| Native Framework Mobile App Testing | Active |
| EU-Central | Active |
| US-East | Active |
| US-West | Active |
| Other | Active |
| saucelabs.com | Active |
| Sauce Labs Documentation | Active |
| support.saucelabs.com | Active |
| Sauce Connect | Active |
| EU-Central | Active |
| US-East | Active |
| US-West | Active |
| Sauce Labs Dashboard | Active |
| EU-Central | Active |
| US-West | Active |
| Sauce Labs REST API | Active |
| EU-Central | Active |
| US-West | Active |
| Sauce Orchestrate | Active |
| EU-Central | Active |
| US-West | Active |
| Visual Testing | Active |
| Legacy Visual Testing (Screener) UI | Active |
| Visual Testing Hub | Active |
| Visual Testing Infrastructure | Active |
| Visual Testing REST API | Active |
View the latest incidents for Sauce Labs and check for official updates:
Description: ### **Dates:** Thursday December 14th 2023, 18:36 - Friday December 15th 2023, 14:24 UTC ### **What happened:** During the incident, tests running on Windows virtual machines did not record a video of the test session. ### **Why it happened:** A new version of ffmpeg was added to the Windows side disk as part of an unrelated change for Windows. This ffmpeg package had a different directory structure than previous bundles, resulting in the video failing to be recorded. ### **How we fixed it:** We added temporary logic during the incident to ignore the invalid ffmpeg version on the Windows side disk. ### **What we are doing to prevent it from happening again:** The team is working on ways to automatically validate video recordings for various OS/Browser versions to detect these issues better.
Status: Postmortem
Impact: Minor | Started At: Dec. 15, 2023, 12:39 p.m.
Description: ### **Dates:** Monday November 27 2023, 20:00 - Tuesday November 28 2023, 15:36 UTC ### **What happened:** Pixel devices in Europe failed to recover from a specific test scenario, resulting in decreased availability of Pixel devices over time. ### **Why it happened:** A combination of defects resulted in devices failing to be properly cleaned following a specific customer test scenario involving audio capture. ### **How we fixed it:** Multiple software defects were fixed and deployed. ### **What we are doing to prevent it from happening again:** The defects were resolved. Additionally, new observability work is under investigation to improve detection time for similar availability situations in the future.
Status: Postmortem
Impact: Minor | Started At: Nov. 28, 2023, 10:22 a.m.
Description: ### **Dates:** Thursday November 16th 2023, 16:15 - Thursday December 7th 21:46 UTC ### **What happened:** Customers encountered sporadic VDC and RDC test failures throughout the incident. The symptoms visible to customers were alleviated within the first two weeks by implementing various workarounds. The incident was completely resolved on December 7th by applying a fix provided by a third party to all our regions. ### **Why it happened:** During a routine upgrade to our Google-managed Kubernetes clusters \(version 1.25 > 1.26\), an undocumented change was introduced to the Container Network Interface \(Cilium\) causing port conflicts with Kubernetes NodePorts during SNAT. This resulted in occasional dropped SYN-ACK packets and, ultimately, communication failures for services at random times. ### **How we fixed it:** Teams successfully restored functionality by implementing retry logic to services experiencing connection issues. This restored service for customers but there were still background issues with dropped connectivity to services we could not directly manage. To address these background issues, we worked with Google to identify a corrective action that eliminated the Google-managed component, ip-masq-agent. Once removed, we replaced it with a version we could manage and omit the configuration flag causing the issue. ### **What we are doing to prevent it from happening again:** Although taking over management of the Google-managed component resolved the incident, we anticipate an official fix from Google, slated for completion by the end of January. Concurrently, we are engaged in collaborative efforts with Google to enhance our joint ability to identify such issues in the future, especially as they roll out new versions of these managed components.
Status: Postmortem
Impact: Major | Started At: Nov. 22, 2023, 2:28 p.m.
Description: ### **Dates:** Thursday November 16th 2023, 16:11 - Sunday 19 November 06:20 UTC ### **What happened:** A subset of customers and prospects intermittently experienced 403 responses and/or partial page loads for the Sauce Labs marketing website. ### **Why it happened:** While rolling out new network security features, a specific rule was set to a sensitivity level that resulted in false positives triggering in some conditions. ### **How we fixed it:** The sensitivity level of the offending rule was decreased. ### **What we are doing to prevent it from happening again:** We have improved the dashboard for all rules, introduced alerts for rule thresholds being exceeded, and improved our ability to quickly filter through logs to identify false positives.
Status: Postmortem
Impact: Minor | Started At: Nov. 19, 2023, 5:34 a.m.
Description: ### **Dates:** Friday November 1st 2023, 10:15 - 18:00 UTC ### **What happened:** An individual customer was continuously running a large number of misconfigured tests, impacting the overall error rate in the US-West region and ultimately triggering our alerting system. Though the error rate alerting threshold was surpassed, no other customer tests were impacted. ### **Why it happened:** Due to the individual customer’s misconfiguration, all tests they ran failed to start. Our test validation service did not properly identify the misconfiguration, causing VMs to be allocated to them. The large number of failing tests tripped our alerting system, though only the individual customer was impacted. ### **How we fixed it:** We worked with the individual customer to stop and reconfigure their tests. ### **What we are doing to prevent it from happening again:** We are enhancing our test configuration validation to better inform customers of these issues before a VM is allocated to a test that will ultimately fail. We are also improving our visibility into test errors, to better identify and monitor individual customer issues versus overall system issues.
Status: Postmortem
Impact: Minor | Started At: Nov. 1, 2023, 12:33 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
