Time: July 6, 2023, 5 p.m.

Status: Postmortem

Update: The incident was primarily caused by a critical microservice responsible for data processing experienced an unhandled exception that led to a crash. Unfortunately, this crash went undetected, causing a data processing stall. As a result, data processing tasks were not completed within the expected timeframe Upon discovering the data processing stall, our operations team promptly initiated an investigation to identify the cause. To resolve the issue, we first restarted the crashed microservice, which allowed the data processing to resume. With the investigation we did not detect any data loss. We are continuiously working on improving our monitoring infrastructure to include automated checks for microservice health and performance. This includes proactive monitoring of critical metrics, such as resource usage, response times, and error rates, to detect and alert us of any anomalies or crashes. We regret the inconvenience caused by the data processing stall resulting from the undetected microservice crash, and we are committed to implementing the necessary measures to prevent such incidents in the future.

Time: July 5, 2023, 1:46 p.m.

Status: Resolved

Update: We have been monitoring the service since the fix has been implemented and have not seen the issue arise. All Non-Streamed data going to CPM Monitoring has been processed and no service delays are observed.

Time: July 3, 2023, 7:40 p.m.

Status: Monitoring

Update: The service responsible for sending Non-Streamed data to CPM Monitoring had failed. The service was started and Non-Streamed data is now making it through. The service is processing all the data, therefore a delay in Microsoft Teams and Cisco Call Manager will be observed. We will monitor the progress and update once complete.

Time: July 3, 2023, 7 p.m.

Status: Investigating

Update: We are currently investigating reports of Non-Streamed data missing from CPM Monitoring. This currently impacts Microsoft Teams calls and Cisco Call Manager.

Time: July 6, 2023, 4:59 p.m.

Status: Postmortem

Update: The failure was a result of a code change that caused a disruption in the authentication process. We apologize for any inconvenience caused by the outage. Upon discovering the authentication system failure, our engineering team initiated an investigation to identify the cause and restore the system's functionality. To address the issue, we decided to back out the code change, effectively reverting the system to the previous stable version. This rollback allowed us to restore the authentication process and resolve the incident. We apologize for the inconvenience caused by the authentication system failure and assure you that we are taking this incident seriously. We have taken immediate steps to address the issue and have implemented measures to prevent similar incidents from occurring in the future.

Time: June 28, 2023, 2:37 p.m.

Status: Resolved

Update: We have been monitoring the platform since the fix has been implemented and have not seen the issue arise. A RCA will be provided within a week.

Time: June 27, 2023, 1:53 a.m.

Status: Monitoring

Update: A fix has been implemented and we are currently monitoring the platform. Users are now able to access CPM Monitoring and Analytics. The alert generator was shutdown to minimize false alerts from being sent out, and will be turned back on in an hour.

Time: June 27, 2023, 1:37 a.m.

Status: Identified

Update: The issue has been identified and our engineers are working on a fix.

Time: June 27, 2023, 1:20 a.m.

Status: Investigating

Update: We are currently investigating reports of users unable to log in. Users using my.vyopta.com are getting a "Failed to fetch" message while SSO users are getting a "502 Bad Gateway" We will provide an update once we have identified the issue.

Time: July 6, 2023, 5:02 p.m.

Status: Postmortem

Update: The outage was caused by a code issue that was inadvertently introduced into our system. Unfortunately, this change contained an issue that resulted in a service degradation. Upon discovering the code issue, our engineering team immediately initiated an investigation to identify and resolve the problem. In order to restore the services as quickly as possible, we decided to revert the code change to a previous stable version.

Time: June 26, 2023, 9:07 p.m.

Status: Resolved

Update: The issue reported with datasets not loading and the UI being slow has been rectified. We will provide a RCA by the end of the week.

Time: June 26, 2023, 4:43 p.m.

Status: Monitoring

Update: A fix has been implemented and we're currently monitoring the results.

Time: June 26, 2023, 3:56 p.m.

Status: Investigating

Update: We are currently investigating a report of datasets in CPM Analytics not loading and the user interface being slow.

Time: June 7, 2023, 10:05 p.m.

Status: Resolved

Update: Issue has been resolved and data is no longer backlogged. All data should be visible now via CPM Monitoring.

Time: June 7, 2023, 9:10 p.m.

Status: Monitoring

Update: A fix has been implemented and our services are currently processing data in our backlog. It may take a few hours to start seeing real time data. We will continue monitoring and will provide updates should the status change.

Time: June 7, 2023, 8:27 p.m.

Status: Identified

Update: Issue has been identified by our engineering team and are currently working on a fix.

Time: June 7, 2023, 7:34 p.m.

Status: Investigating

Update: We have received a report of no call/meeting data in CPM Monitoring for our Gov Cloud. We're currently investigating.

Time: May 10, 2023, 5:11 p.m.

Status: Resolved

Update: This issue has been resolved. It was related to a data replication issue with the reported customer.

Time: May 10, 2023, 2:13 p.m.

Status: Identified

Update: We have had a report of data not being accessible via CPM Analytics (CPM Monitoring is not affected). We have no reason to believe that this issue affects many customers. We have identified the issue and are working towards its resolution.