-
Time: May 18, 2023, 4:32 p.m.
Status: Postmortem
Update: ### Updated 5/22/23:
> These updates aim to provide clear and concise information regarding when the code defect was introduced and test coverage moving forward.
# Summary:
We recently encountered intermittent outages with the Workflow API, which impacted certain functionalities of PopFlow, including ScreenPops, Data Retrieval and Lookups, and General Use.
## What was impacted:
* PopFlow workflow functionality
* \(e.g., ScreenPops, Data Retrieval and Lookups, General Use\)
# Remediation:
The root cause of the issue was identified as a code defect that caused Workflow Requests to process repeatedly, resulting in the depletion of CPU and memory resources for the Workflow API. The code defect was introduced in a prior maintenance event, listed [here](https://status.openmethods.com/incidents/47572qj6my37).
To address and prevent similar outages in the future, we have resolved the code defect by removing the flaw from the production code. We have updated validation and testing efforts to cover additional areas to remedy feature occurrences and issues.
We sincerely appreciate your understanding and cooperation throughout this entire process. Should you have any further questions or concerns, please do not hesitate to reach out to our dedicated support team.
Thank you once again for your patience and support.
-
Time: May 18, 2023, 4:32 p.m.
Status: Resolved
Update: We are pleased to announce a significant update regarding the recent issues that affected our Workflow API service. After thorough monitoring and implementing the necessary fixes, we are delighted to inform you that the problem has been successfully resolved.
As a result, we will now proceed to close this incident on the status page.
We extend our sincere gratitude for your patience and cooperation throughout this process. Please see the postmortem for more information!
-
Time: May 18, 2023, 12:11 a.m.
Status: Monitoring
Update: We are still actively monitoring the changes made in our environment to help reduce and resolve issues with the Workflow API. Our team will continue to prioritize the stability and performance of the Workflow API service.
<b>Next Steps:</b>
We plan to proactively monitor for the next 12-24 hours and address any emerging problems to ensure a seamless user experience.
We appreciate your understanding and cooperation throughout this process. If you have any further questions or concerns, don't hesitate to contact our support team.
-
Time: May 17, 2023, 7:13 p.m.
Status: Monitoring
Update: <b>Current Status:</b>
A fix has been implemented and has been actively in monitoring. However, we have noticed some ongoing latency issues that are currently under review.
Our team is actively investigating these cases to identify the root causes and work towards resolving them. We understand the impact this may have on your experience and want to assure you that addressing these latency issues is our top priority.
We appreciate your patience and understanding as we continue to analyze and troubleshoot the situation.
-
Time: May 17, 2023, 6:06 p.m.
Status: Monitoring
Update: We are pleased to provide you with an update on the recent fix that has been implemented to address the issues affecting our services. The fix has been deployed, and we are currently monitoring its effectiveness and impact on the overall system stability. Our team is actively continuing the investigation to identify the root causes and implement effective solutions.
<b>Fix & Monitoring Implementation:</b>
Our engineering team has successfully implemented a fix based on our thorough investigation and analysis. The necessary updates and optimizations have been applied to our services to address the issues encountered.
<b>Current Status:</b>
The fix is now in place and actively being monitored. Our team is closely observing the system's performance, stability, and error rates to evaluate the impact of the implemented solution.
We will continue to provide updates through this status page to ensure transparency and keep you informed of our progress.
-
Time: May 17, 2023, 5:27 p.m.
Status: Monitoring
Update: <b>Current Status:</b>
The investigation into the issues impacting the Workflow API service is still underway. We are thoroughly monitoring system metrics to gain deeper insights into the patterns and triggers of the reported issues.
While we are actively researching and resolving the issues, it is possible that you may experience intermittent disruptions. We apologize for any inconvenience caused and appreciate your patience and understanding.
-
Time: May 17, 2023, 4:25 p.m.
Status: Monitoring
Update: <b>Current Status:</b>
The investigation into the issues impacting the Workflow API service is still in progress. We have implemented additional resources to help efforts in resolving the issues. These resources have helped reduce timeouts.
While we are actively researching and resolving the issues, it is possible that you may experience intermittent disruptions. We apologize for any inconvenience caused and appreciate your patience and understanding.
-
Time: May 17, 2023, 3:44 p.m.
Status: Monitoring
Update: <b>Current Status:</b>
The investigation into the issues impacting the Workflow API service is still underway. While we continue to research and resolve the issues, you may experience intermittent disruptions when using the Workflow API service.
We apologize for any inconvenience caused and appreciate your patience and understanding.
-
Time: May 17, 2023, 3:13 p.m.
Status: Monitoring
Update: We would like to provide you with a status update regarding the ongoing research into the issues affecting our Workflow API service. Our team is actively investigating the problem to identify the root causes and implement appropriate solutions.
<b>Current Status:</b>
The investigation into the issues impacting the Workflow API service is still underway.
<b>Mitigation Measures:</b>
While we continue to research and resolve the issues, you may experience intermittent disruptions when using the Workflow API service.
We apologize for any inconvenience caused and appreciate your patience and understanding.
-
Time: May 17, 2023, 2:37 p.m.
Status: Monitoring
Update: We want to provide you with an update regarding the ongoing investigation into the issues affecting our Workflow API service. Our team is actively researching the problem to identify the underlying causes and implement appropriate solutions.
<b>Current Status:</b>
Our engineers continue to investigate the issues impacting the Workflow API service.
<b>Symptoms:</b>
Symptoms of the ongoing issues affecting the Workflow API service include:
Intermittent Timeouts: Users may experience occasional timeouts when interacting with the Workflow API service. Requests may take longer than usual to complete or fail to execute within the expected timeframe.
Delays in Workflow Processing: Workflows that rely on the API service may encounter delays in their execution. This can result in a backlog of pending tasks or actions within the workflow, impacting overall efficiency and timely completion.
While we work diligently to resolve the issues, you may continue to experience disruptions with the Workflow API service. We understand the impact this may have on your workflows and apologize for any inconvenience caused.
-
Time: May 17, 2023, 1:50 p.m.
Status: Identified
Update: We are currently experiencing intermittent timeouts on our Workflow API service, and we apologize for any inconvenience caused. Our team is actively investigating the issue to identify the root cause and restore normal functionality as quickly as possible.
<b>Issue Summary:</b>
Intermittent timeouts are impacting the Workflow API service, resulting in delays and occasional failures when interacting with the service.
<b>Current Status:</b>
Our team is diligently investigating the issue to pinpoint the underlying cause and implement a resolution. We are actively monitoring the service and working towards a swift resolution.
We apologize for any inconvenience caused and appreciate your patience. We will continue to update this status page with new information as it becomes available. Thank you for your understanding and cooperation.