Time: April 5, 2024, 3:38 p.m.

Status: Postmortem

Update: **Title**_:_ Secret Server Cloud \(SSC\) Issues in East US starting on April 3rd, 2024. **Impact:** Beginning on April 3rd, 2024, at approximately 7:52:40 PM ET, SSC customers would have started experiencing issues accessing their systems. **Issue:** On April 3rd, 2024, at approximately 7:52:40 PM ET, Azure SQL Databases in East US started experiencing issues. New connections to databases in this region may have resulted in an error or timeout. Existing connections remained available to accept new requests, however if those connections were terminated and re-established, they may have failed. Microsoft suspects that a potential deployment on their frontend gateways caused SQL database availability issues impacting customer connectivity. Microsoft worked to mitigate the issue by stopping the active rollout of their deployment. **Resolution:** At approximately 7:54 PM our monitoring systems started alerting regarding an issue. After initial triage, we estimated that roughly 25% of US based SSC customers were experiencing issues. At approximately 8:26 PM ET,  SC failover was initiated. However, due to an issue with the automated tool to perform failover, we began to manually perform failover, which increased our response and resolution time. At approximately 10:36 PM ET the  failover had completed, and all SSC systems were operational. Our engineers monitored the Azure incident and waited for it to be fully resolved before closing the incident on our side. At approximately 11:29 PM ET on April 3rd, 2024, the Azure team declared the incident mitigated. At approximately 3:56 AM ET on April 4th, 2024, a rollback to restore functionality in the primary East US region for SSC was completed. **Action Items:** 1\.    Investigate issues with tooling that facilitates automated failover. **To address this and prevent future occurrences:** IN PROGRESS – Improve failover automation tooling and capabilities. **Incident Start Time:** _April 3rd, 2024, 07:52:40 PM ET_ **Incident End Time:** _April 3rd, 2024, 11:29 PM ET_

Time: April 4, 2024, 4:51 a.m.

Status: Resolved

Update: We’re pleased to inform you that the incident affecting the service(s) listed below has been resolved. All systems are now operating normally. We apologize for any inconvenience this incident may have caused, and we appreciate your understanding and support. For any questions or concerns, please reach out to our support team at https://support.delinea.com.

Time: April 4, 2024, 1:43 a.m.

Status: Monitoring

Update: We wanted to provide you with an update on the incident affecting the service(s) listed below. A subset of customers are coming back online. We are continuing to monitor the incident and will provide updates as warranted. For any questions or concerns, please reach out to our support team at https://support.delinea.com.

Time: April 4, 2024, 1:36 a.m.

Status: Identified

Update: We wanted to provide you with an update on the incident affecting the service(s) listed below. Our team has been working diligently to address the issue, and we have made significant progress. We appreciate your continued patience and support as we work to fully restore normal service. For any questions or concerns, please reach out to our support team at https://support.delinea.com.

Time: April 4, 2024, 12:30 a.m.

Status: Investigating

Update: We are currently investigating issues in Secret Server Cloud US

Time: March 26, 2024, 10:06 p.m.

Status: Postmortem

Update: **Impact:** Beginning on 3/22/24 at approx. 11:30 PM UTC, installed Secret Server SDKs may have been unable to retrieve Secret credentials. **Issue:** A change to enhance security for IP Addresses introduced an issue where the IP Address was not correctly retrieved for calls from the Secret Server SDK. This blocked the SDK from authenticating in order to retrieve credentials due to internal ongoing security vetting of calls from the SDK. SDKs with a pre-existing valid token would continue to function correctly until their token expired and they needed to re-authenticate to get a new token. **Resolution:** The issue was resolved by rolling back the update. By 10:49 PM UTC on 3/23/24 the issue was resolved. **Action Items:** **To address this and prevent future occurrences:** Add automation tests to cover installation of the Secret Server SDK and retrieval of credentials using it. Add missing unit tests to cover the specific code area that was responsible for the issue. Add additional monitoring for anomalous changes in the authentication endpoint to provide back-end visibility of SDK calls.

Time: March 26, 2024, 10:06 p.m.

Status: Resolved

Update: Customers experienced issues with the Secret Server client sdk when authenticating after a patch release.

Time: March 26, 2024, 10:06 p.m.

Status: Postmortem

Update: **Impact:** Beginning on 3/22/24 at approx. 11:30 PM UTC, installed Secret Server SDKs may have been unable to retrieve Secret credentials. **Issue:** A change to enhance security for IP Addresses introduced an issue where the IP Address was not correctly retrieved for calls from the Secret Server SDK. This blocked the SDK from authenticating in order to retrieve credentials due to internal ongoing security vetting of calls from the SDK. SDKs with a pre-existing valid token would continue to function correctly until their token expired and they needed to re-authenticate to get a new token. **Resolution:** The issue was resolved by rolling back the update. By 10:49 PM UTC on 3/23/24 the issue was resolved. **Action Items:** **To address this and prevent future occurrences:** Add automation tests to cover installation of the Secret Server SDK and retrieval of credentials using it. Add missing unit tests to cover the specific code area that was responsible for the issue. Add additional monitoring for anomalous changes in the authentication endpoint to provide back-end visibility of SDK calls.

Time: March 26, 2024, 10:06 p.m.

Status: Resolved

Update: Customers experienced issues with the Secret Server client sdk when authenticating after a patch release.

Time: March 4, 2024, 1:45 p.m.

Status: Postmortem

Update: **Title**:  A subset of customers were briefly affected by a SQL health issue.   **Impact:**   Beginning at approximately 18:30 UTC on 2/28/2024 a single Azure SQL server had an intermittent outage that affected some databases on the server. The issue was fully resolved by 18:50 UTC.   **Issue:**  An Azure SQL Server had a brief outage which caused some connections to databases on that server to be rejected until it recovered. Due to the small number of databases affected, an alert was not initially triggered, by the time Azure raised any health alerts and those were acknowledged the database server had recovered.   **Resolution:**  The Azure SQL database recovered with no intervention.

Time: Feb. 28, 2024, 7:28 p.m.

Status: Resolved

Update: One of the Secret Server Cloud SQL Servers experienced a brief outage due to Azure issues that was resolved in a short amount of time. Engineers are investigating root cause. A small subset of tenants were affected.

Time: March 4, 2024, 1:45 p.m.

Status: Postmortem

Update: **Title**:  A subset of customers were briefly affected by a SQL health issue.   **Impact:**   Beginning at approximately 18:30 UTC on 2/28/2024 a single Azure SQL server had an intermittent outage that affected some databases on the server. The issue was fully resolved by 18:50 UTC.   **Issue:**  An Azure SQL Server had a brief outage which caused some connections to databases on that server to be rejected until it recovered. Due to the small number of databases affected, an alert was not initially triggered, by the time Azure raised any health alerts and those were acknowledged the database server had recovered.   **Resolution:**  The Azure SQL database recovered with no intervention.

Time: Feb. 28, 2024, 7:28 p.m.

Status: Resolved

Update: One of the Secret Server Cloud SQL Servers experienced a brief outage due to Azure issues that was resolved in a short amount of time. Engineers are investigating root cause. A small subset of tenants were affected.