Time: Feb. 29, 2024, 8:49 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 29, 2024, 8:29 p.m.

Status: Investigating

Update: We are observing a slowdown in ABBYY Vantage EU service. Operations teams are actively working to resolve the issue. We expect to provide another update within 30 min or as soon as more information becomes available.

Time: Feb. 29, 2024, 8:47 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 29, 2024, 8:36 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Feb. 29, 2024, 8:09 p.m.

Status: Investigating

Update: We are observing a slowdown in ABBYY Vantage AU service. System scaling does not work caused by security improvements Operations teams are actively working to resolve the issue. We expect to provide another update within 30 min or as soon as more information becomes available.

Time: Feb. 28, 2024, 5:57 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 28, 2024, 5:51 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: Feb. 28, 2024, 4:51 p.m.

Status: Monitoring

Update: MS Azure Impact Statement: There is an ongoing outage in MS Azure Eastus2 region, please find below details. MS AZURE ISSUE: VM CRUD failure outage in eastus2 Starting at 11:05 UTC on 28 Feb 2024, you have been identified as a customer using Virtual Machines in East US 2 who may experience errors when performing Create, Read, Update or Delete (CRUD) operations. Retries may be successful. We have engaged additional teams and are currently investigating. The next update will be provided in 60 minutes, or as events warrant. ABBYY Impact Statement: Incident partly mitigated. Processing restored. We are still expecting slight delays in document processing and API calls. The duty team continues to monitor the situation diligently to ensure everything is functioning properly.

Time: Feb. 28, 2024, 4:44 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Feb. 28, 2024, 4:13 p.m.

Status: Investigating

Update: Due to technical issues in Azure Infrastructure, the processing time can be increased up to several times. The API methods can take more time than usual, and some tasks can fail with errors.

Time: Jan. 31, 2024, 3:51 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 31, 2024, 3:40 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Jan. 31, 2024, 3:37 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Jan. 31, 2024, 3:07 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Jan. 31, 2024, 2:28 p.m.

Status: Investigating

Update: Processing with previously created skills remains unaffected: folders created earlier for import and export functions are fully operational and have experienced no issues. However, folders recently created for new Process skills, or those which are newly created for the modified Process skills, might be inaccessible to users attempting to use them.

Time: Jan. 31, 2024, 2:27 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Jan. 31, 2024, noon

Status: Investigating

Update: We are currently encountering difficulties with the SFTP (Secure File Transfer Protocol) functionality on the Vantage AU service. Users may experience disruptions or when attempting to obtain the output files using SFTP folder. Our team is actively investigating the issue and working towards a resolution. We apologize for any inconvenience caused and appreciate your patience.

Time: Feb. 7, 2024, 9:43 a.m.

Status: Postmortem

Update: **Dear Customer,** On January 29 and January 30, 2024, ABBYY Vantage EU experienced interruption in operation of the Redis service, resulting in outages of the Vantage platform. We are glad to inform you that the issues have been mitigated and the service is now fully functional. Please review the following incident Root Cause Analysis \(RCA\) information: **Cloud instance** * Western Europe **Incident timeframe** * January 29, 2024 * 15:42 – 17:20 UTC * January 30, 2024 * 10:30 – 12:10 UTC **Incident status** * Fully mitigated **Customer impact** * The platform was unable to launch new or complete existing transactions. * Skills which were being edited during the outage could become unresponsive. * Edits being done to skills during the outage could be lost. * Subscription status information was not accessible. **Incident history** * January 29, 2024 * 15:42 UTC: The health monitoring system of the service reported a failure of the Redis master node. The on-duty operations team discovered an unusually high memory consumption of the Redis service and performed its manual failover. * 16:02 UTC: The Redis master node started consuming an unusually high amount of memory again. The team performed another manual failover and cleaned up the Redis stream to reduce the consumed memory. * 17:20 UTC: All internal services returned to normal operation. The incident was considered mitigated. * January 30, 2024 * 10:30 UTC: The health monitoring system of the service reported a new failure of the Redis master node due to high memory consumption. The on-duty operations team scaled down the Transaction service to zero to prevent new data flowing to Redis and deleted all Workflow service objects in all Redis master nodes. They then scaled up the Transaction service again. * 12:10 UTC: All internal services returned to normal operation. The incident was considered mitigated. **Root cause** * The failures of Redis were caused by too high an amount of memory set allowed to be occupied by the Redis service in its hosting VMs. * As a result, the Redis master node was unable to process data requests but was successfully responding about its availability. This caused an automatic failover not to be triggered. **Mitigation measures** * Manual failover of the Redis service, followed by cleaning up the Redis stream to reduce the consumed memory. **Prevention measures** * Optimize the implementation of communication with Redis for reading and writing data to reduce memory consumption. * Replace the VMs hosting the Redis service with a native fully managed Redis service on MS Azure. We apologize for any inconvenience and most importantly, for the potential impact on your business. We are committed to preventing such issues in the future and will continue working on improving our infrastructure and monitoring solutions. Thank you for using ABBYY Vantage! If you have any questions or feedback, please feel free to contact our support team via the [Help Center](https://support.abbyy.com/hc/en-us/requests/new) portal. Yours faithfully, [ABBYY Vantage](https://vantage-eu.abbyy.com) Team

Time: Jan. 30, 2024, 12:21 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 30, 2024, 12:20 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Jan. 30, 2024, 10:43 a.m.

Status: Investigating

Update: We are currently experiencing difficulties in completing transactions and encountering errors when attempting to create new skills within our platform. Our team is actively investigating the issue and working towards a resolution. We apologize for any inconvenience caused and appreciate your patience.