Last checked: 11 minutes ago
Get notified about any outages, downtime or incidents for Sauce Labs and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Sauce Labs.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
API Testing | Active |
EU-Central | Active |
US-West | Active |
Automated Browser Testing | Active |
EU-Central | Active |
US-West | Active |
Automated Real Device Testing | Active |
EU-Central | Active |
US-East | Active |
US-West | Active |
Automated Virtual Mobile Device Testing | Active |
EU-Central | Active |
US-West | Active |
Billing | Active |
EU-Central | Active |
US-West | Active |
Insights | Active |
EU-Central | Active |
US-West | Active |
IPSec VPN | Active |
EU-Central | Active |
US-West | Active |
Live Browser Testing | Active |
EU-Central | Active |
US-West | Active |
Live Real Device Testing | Active |
EU-Central | Active |
US-East | Active |
US-West | Active |
Live Virtual Mobile Device Testing | Active |
EU-Central | Active |
US-West | Active |
Mobile App Distribution (TestFairy) | Active |
TestFairy Platform | Active |
TestFairy UI | Active |
Native Framework Mobile App Testing | Active |
EU-Central | Active |
US-East | Active |
US-West | Active |
Other | Active |
saucelabs.com | Active |
Sauce Labs Documentation | Active |
support.saucelabs.com | Active |
Sauce Connect | Active |
EU-Central | Active |
US-East | Active |
US-West | Active |
Sauce Labs Dashboard | Active |
EU-Central | Active |
US-West | Active |
Sauce Labs REST API | Active |
EU-Central | Active |
US-West | Active |
Sauce Orchestrate | Active |
EU-Central | Active |
US-West | Active |
Visual Testing | Active |
Legacy Visual Testing (Screener) UI | Active |
Visual Testing Hub | Active |
Visual Testing Infrastructure | Active |
Visual Testing REST API | Active |
View the latest incidents for Sauce Labs and check for official updates:
Description: See [https://stspg.io/906109z92sfj](https://stspg.io/906109z92sfj)
Status: Postmortem
Impact: Minor | Started At: Oct. 4, 2023, 8:27 a.m.
Description: ### **Dates:** Monday October 2nd 2023, 11:51 UTC - Wednesday October 4th 15:20 UTC ### **What happened:** A small percentage of customer Appium tests failed when certain commands were executed, like capturing screenshots or executing custom scripts. ### **Why it happened:** A connection pool used by our Appium server was intermittently exhausted during spikes in the usage of mid-session install scripts. During these spikes, we were leaking HTTP client connections, which caused timeouts and eventual connection issues to the Appium server. ### **How we fixed it:** We restored the functionality of the service in two ways: * Firstly, we gradually recycled pods to see if cleanly starting these processes would clear the issue. * That bought us time to further debug the issue and we discovered the leak of HTTP client connections which led to this timeout issue. The team then put together a fix for the issue and deployed it to production. ### **What we are doing to prevent it from happening again:** During the incident, we addressed the HTTP client connection leak and enhanced monitoring to better identify this specific issue should it happen again in the future.
Status: Postmortem
Impact: Minor | Started At: Oct. 3, 2023, 3:03 p.m.
Description: ### **Dates:** Thursday September 28th 2023, 21:10 UTC - Friday September 29th 2023, 00:43 UTC ### **What happened:** Users attempting to use live test sessions in the Sauce Labs UI were either given an error or a blank screen for the duration of the incident. ### **Why it happened:** When attempting to insert a non-serializable value into our memory store, an error was thrown, preventing actions such as starting a live session. This error began occurring after a middleware was added to our memory store that attempted to serialize any payload, which, upon failing to do so, would throw an error preventing that action from being committed. ### **How we fixed it:** We rolled back to the previous version of live testing to restore service. ### **What we are doing to prevent it from happening again:** We are adding a blocking stage to our pipeline to check for these issues by running live sessions for both real devices and web sessions.
Status: Postmortem
Impact: Major | Started At: Sept. 28, 2023, 11 p.m.
Description: ### **Dates:** Thursday September 28th 2023, 17:05 - 17:36 UTC ### **What happened:** For the duration of the incident, customers could not start new tests in all regions. ### **Why it happened:** A change that removed public cloud provider storage interfaces from the core Kubernetes codebase was introduced during a Kubernetes upgrade. We needed to rework large sections of our infrastructure management code to handle this change. In making these changes, we introduced an issue with applying firewall tags that broke key traffic outside the Kubernetes cluster. This was not detected in lower environments. ### **How we fixed it:** We manually rolled back the configuration that caused the issue. ### **What we are doing to prevent it from happening again:** We are addressing the underlying issue that caused the broken cluster networking. Once we have that fix we will plan a safe rollout.
Status: Postmortem
Impact: Major | Started At: Sept. 28, 2023, 5:33 p.m.
Description: ### **Dates:** Wednesday September 20th 2023, 09:39 - 15:38 UTC ### **What happened:** Sauce Connect users could not load their websites behind Sauce Connect tunnels until they closed a pop-up informing them, "Your connection is not private”. ### **Why it happened:** Windows virtual machines did not import the certificates needed to run tunnels properly due to a change in how they were stored on the virtual machine. A startup script references several locations to load certificates from, and not all of them were updated to reflect the new location on the virtual machine. ### **How we fixed it:** The startup script was updated to reflect the new certificate locations. ### **What we are doing to prevent it from happening again:** We reviewed the tunnel startup code across all platforms to ensure no other areas referenced the old locations. We also added unit tests that check whether there is a mismatch between where the certificates are stored and their references in the startup script.
Status: Postmortem
Impact: Minor | Started At: Sept. 20, 2023, 5:59 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.