Is there an SecureAuth outage?
SecureAuth status: Systems Active
Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for SecureAuth and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for SecureAuth and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for SecureAuth.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Operator Alerts | Active |
| CIAM | Active |
| Application | Active |
| AU Region | Active |
| EU Region | Active |
| Push | Active |
| SMS | Active |
| US Region | Active |
| Voice | Active |
| Passwordless | Active |
| Application | Active |
| Device Trust | Active |
| Mobile App | Active |
| Push | Active |
| SMS | Active |
| Voice | Active |
| SaaS/Full Cloud Components | Active |
| SaaS/Full Cloud Identity Platform | Active |
| SecureAuth Connector | Active |
| SecureAuth Cloud Services | Active |
| 3rd Party Mobile Carrier | Active |
| Enhanced Geolocation Resolution Service - US1 | Active |
| Enhanced Geolocation Resolution Service - US2 | Active |
| Fraud Service - US1 | Active |
| Fraud Service - US2 | Active |
| Geolocation Resolution Service - US1 | Active |
| Geolocation Resolution Service - US2 | Active |
| Nexmo Voice API | Active |
| Push-to-Accept Service - US1 | Active |
| Push-to-Accept Service - US2 | Active |
| SMS Service - US1 | Active |
| SMS Service - US2 | Active |
| Telephony Extension/DTMF Service - US1 | Active |
| Telephony Extension/DTMF Service - US2 | Active |
| Telephony Provider SMS API | Active |
| Telephony Service - US1 | Active |
| Telephony Service - US2 | Active |
| Threat Service - US1 | Active |
| Threat Service - US2 | Active |
| X.509 Certificate Service (SHA2) - US1 | Active |
| X.509 Certificate Service (SHA2) - US2 | Active |
| SecureAuth IdP Frontend Services | Active |
| SecureAuth Application Templates | Active |
| SecureAuth Web Admin | Active |
| SecureAuth Polaris Services | Active |
| FIDO Service | Active |
| Mobile Services | Active |
| Polaris Base Infrastructure | Active |
| SaaS IdP Broker | Active |
| SecureAuth Titan Services | Active |
| Dashboard Service | Active |
| Device Enrollment Service | Active |
| Fraud Service | Active |
| IP Blocking Service | Active |
| IP Intel Service | Active |
| IP Reporting Service | Active |
| Licensing Service | Active |
| Link-to-Accept Service | Active |
| OAuth Service | Active |
| Push-to-Accept Service | Active |
| SMS Service | Active |
| Symbol-to-Accept Service | Active |
| Telephony Service | Active |
| Titan Proxy Services (SA IdP 9.3 and older only) | Active |
| Transaction Service | Active |
| User Risk Scoring Service | Active |
| User Risk Service | Active |
| User Stats Service | Active |
| Workforce | Active |
| Certificate Enrollment | Active |
| Cloud IdP | Active |
| Dashboard | Active |
| FIDO WebAuthn | Active |
| Kerberos Authentication | Active |
| Link-to-Accept Service | Active |
| Mobile App | Active |
| Push | Active |
| SMS | Active |
| Voice | Active |
View the latest incidents for SecureAuth and check for official updates:
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: Oct. 31, 2024, 7:50 p.m.
Description: Verizon has reported that they have fully restored their services.
Status: Resolved
Impact: None | Started At: Sept. 30, 2024, 4:21 p.m.
Description: **RCA – SecureAuth Authenticate iOS Release** **Problem Description:** On September 19, 2024 at 10:00AM PDT, SecureAuth released SecureAuth Authenticate version 25.0.18 to the IOS App Store. The application was targeted to 1% of the userbase with the previous version installed; however, users could still download the new version manually if desired. At approximately 11:00AM PDT, we receive internal and external reports that version 25.0.18 of SecureAuth Authenticate has broken preexisting enrollments, and users were unable to re-enroll to resolve the issue. **Cause:** Within the update process for SecureAuth Authenticate version 25.0.18, a format conversion of account data to a new arc-6 architecture is performed. The format conversion process failed due to missing group entitlements within the project configuration. This resulted in the SecureAuth Authenticate App’s inability to properly load account information, thus breaking functionality for end users. While no account data was lost, previous enrollments were inaccessible, and users were prompted to enroll, but could not. These entitlements were not discovered during QA due to a caching of configurations inside of Apple’s TestFlight system where the application was being tested. The only way to clear this cache is a factory reset of the device. Unknown to developers, these cached configurations held onto the missing group entitlements that were not present in the GA version of the application that was released. **Recovery:** The engineering team initiated resolution efforts on two fronts: Front 1: Revert to the previous version as an initial interim mitigation to restore functionality to impacted users as quickly as possible while a separate team focused on providing permanent fix for the following release. However, upon revert efforts, it was determined that this option was not viable due to technical complications brought on by the update and Apple compliance standards related to the Apple Watch application. Focus of efforts quickly shifted to Front 2. Front 2: Identify the cause of the format conversion failure and implement the fix. Upon successful completion of QA validation, submit to the iOS App Store for urgent review and publishment. **Resolution:** The format conversion process failed due to missing group entitlements within the project configuration. The fix reapplied the necessary group entitlements, restoring the app’s ability access the account data. **Timeline:** Sep 19, 2024 • 10:00 AM PDT - SecureAuth Authenticate version 25.0.18 released in iOS App Store • 11:00 AM PDT – Internal teams discover the issue with App release and Engineering Teams are notified • 11:09 AM PDT – Incident bridge started and Engineering teams begin investigating the issue • 11:15 AM PDT – Engineering Teams begin efforts to revert App to previous version in iOS App Store. • 11:20AM PDT – Status Page updated to inform customers to hold from application updates until further notice • 11:20 AM PDT – Engineering Teams continue to investigate root cause of the issue while also working on reverting to previous version on the iOS App Store • 12:30 PM PDT – Confirmed that the issue was not isolated to iOS 18 • 1:00 PM PDT – Discarded rollout option due to complications with Apple compliance standards for Wearable App • 1:00 PM PDT – Engineering teams refocus to provide patch for the Authenticate app. • 2:05 PM PDT – Cause of the issue identified as a failure of the format conversion process due to missing group entitlements within project configuration • 2:25 PM PDT – Fix is implemented and QA validation is initiated • 3:00 PM PDT – New build published for Urgent Review to App Store. • 4:12 PM - New version is deployed. Impacted users were notified to download the version 25.1.18 with preexisting enrollments intact. Support and Engineering teams continue to monitor the situation closely with customers. **Corrective Actions:** • Work with Apple to review their TestFlight requirements and determine why configurations were being cached, discover the standard duration of the cache period, and identify the steps needed to ensure the cache is cleared and updated configurations are being used during TestFlight QA processes. • Improve the current Pull Request and Code Review process in Mobile Development in order to mitigate the impacts of missing configurations and improve code release standards. • Add test cases to our QA suite to cover fresh devices, as it was determined that if wiped or “new” devices were being used for testing, the cached configurations would have been discovered.
Status: Postmortem
Impact: Major | Started At: Sept. 19, 2024, 6:21 p.m.
Description: **Polaris Twilight Outage RCA - September 12, 2024** **Problem Description** On September 11, 2024 at 7:16PM, the SecureAuth Cloud Infrastructure encountered widespread connection issues with databases systems which resulted in authentication failures for impacted customers. **Cause** The SecureAuth Cloud Operations team was alerted of connections issues with the Twilight service \(integral service which other microservices are reliant\). Upon investigation, we identified that the service was experiencing database latency due to CPU utilization spikes on the database. The CPU spikes triggered mass restarts of the Twilight Service which in turn caused extended CPU spikes on the database. The root cause was due to legacy dependencies on the database that were negatively affected during a redistribution exercise related to the Vault migration performed on August 29, 2024. Those legacy dependencies were originally determined to be benign, and therefore assumed to have no impact to the customer base after the Vault migration. It was determined that the CPU spikes were caused by the interface between the service and the database in form of health checks that created a snowball effect, resulting in the aforementioned issues with the Twilight service. Due to the nature of this issue, not all customers were immediately impacted; however, the recovery and resolution of this issue impacted all customer cloud services as a result of the scaling operations. **Recovery** To mitigate this issue, the cloud services were scaled down alleviate database pressure. Once the database stabilized, the services were scaled back up in a controlled manner until all services were fully restored. **Timeline:**` `Sep 11, 2024 * 7:16PM PST – Twilight connection issues begin and alerts were triggered * 7:17PM PST – Cloud Operations team join bridge to investigate alerts * 7:27PM PST – Issue is understood and mitigation efforts begin * 7:27PM PST – Scale down of cloud services to alleviate database pressure begins. * 7:40PM PST – Scale down complete and database CPU utilization stabilizes * 7:41PM PST – Controlled \(staggered\) scale up of cloud services begins * 8:30PM PST – Controlled scale up of cloud services is completed * 8:40PM PST – All services in running state * 9:00PM PST – Validation testing complete and incident resolved * Post-9:00PM PST – Continued to monitor closely while working with some customers as needed to resolve intermittent issues caused by the incident. Corrective Actions * Engineering to review and improve the Twilight to Cockroach Database interface and determine a more elegant solution to the health check actions that would diminish the result of mass-restarts of the service during periods of high-usage spikes. * Leadership review of database alternatives to the solution architecture * Improve decision-making accuracy by increasing team knowledge around legacy systems to ensure end to end awareness of potential impacts to assumed benign configuration changes. * Introduce additional gates into the existing CAB \(Change Advisory Board\) process, including additional Engineering leadership, including cross-functional Subject Matter Experts
Status: Postmortem
Impact: Minor | Started At: Sept. 12, 2024, 2:43 a.m.
Description: There was an interruption of service for the Passwordless platform, previously known as Arculix, between 12:01pm PDT until 12:13pm PDT. Service was restored and no further interruptions are expected. If you have any further questions or inquiries, please reach out to support at https://support.secureauth.com.
Status: Resolved
Impact: None | Started At: Sept. 6, 2024, 7:40 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
