Time: Feb. 24, 2023, 2:21 a.m.

Status: Resolved

Update: The Mailgun service appears to have stabilized. During the past 12 hours, all emails sent from our system through Mailgun have been delivered as expected. We are marking our issue as resolved, and will provide additional updates from Mailgun if there are any of importance to our customers.

Time: Feb. 23, 2023, 3:56 p.m.

Status: Monitoring

Update: Mailgun continues to be under an active DDoS attack. We have observed improvements in email delivery rates over the past 24 hours which indicates that Mailgun's mitigation efforts are being effective. We will continue to monitor delivery rates and Mailgun reports for the immediate future and provide updates when there is a substantial change.

Time: Feb. 22, 2023, 9:34 p.m.

Status: Identified

Update: The email service used by AchieveIt to send emails from our production applications is experiencing intermittent delays delivery certain emails to recipients. The service, Mailgun, has reported that they have been under a distributed denial of service attack since February 21, 2023. During that time, Mailgun has been working with its hosting and infrastructure providers to reduce the impact of the attack and restore normal delivery. We have observed that a small percentage of emails from AchieveIt in the past 36 hours have been either delayed or not yet delivered. Most emails are, however, being delivered as expected. We will continue to monitor Mailgun's response and provide updates here. We are also evaluating options for delivering the remainder of the delayed emails.

Time: June 9, 2022, 4:58 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: June 9, 2022, 4:57 p.m.

Status: Monitoring

Update: We have verified that email flow has been completely restored and all delayed emails have either been delivered or are in the process of being delivered.

Time: June 9, 2022, 2:09 p.m.

Status: Monitoring

Update: Mailgun has confirmed that they have restored all services but are still seeing delays in delivering emails. We are seeing many emails being successfully delivered, but it is likely that some emails are also being delayed. All delayed emails will be delivered once Mailgun resolves their system problem.

Time: June 9, 2022, 1:46 p.m.

Status: Monitoring

Update: Mailgun has at least partially resolved their system outage and we are beginning to see emails flow again. We will continue to monitor to ensure that emails continue to be delivered as expected and post an update soon.

Time: June 9, 2022, 1:08 p.m.

Status: Identified

Update: The service that AchieveIt uses to send emails from our platform, Mailgun, is experiencing a service interruption that is causing emails sent from our system to be delayed. We are monitoring Mailgun's updates as they work toward a fix and will publish updates here as we know more. Any email that should be sent will still be delivered once the email service has been restored.

Time: June 9, 2022, 4:58 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: June 9, 2022, 4:57 p.m.

Status: Monitoring

Update: We have verified that email flow has been completely restored and all delayed emails have either been delivered or are in the process of being delivered.

Time: June 9, 2022, 2:09 p.m.

Status: Monitoring

Update: Mailgun has confirmed that they have restored all services but are still seeing delays in delivering emails. We are seeing many emails being successfully delivered, but it is likely that some emails are also being delayed. All delayed emails will be delivered once Mailgun resolves their system problem.

Time: June 9, 2022, 1:46 p.m.

Status: Monitoring

Update: Mailgun has at least partially resolved their system outage and we are beginning to see emails flow again. We will continue to monitor to ensure that emails continue to be delivered as expected and post an update soon.

Time: June 9, 2022, 1:08 p.m.

Status: Identified

Update: The service that AchieveIt uses to send emails from our platform, Mailgun, is experiencing a service interruption that is causing emails sent from our system to be delayed. We are monitoring Mailgun's updates as they work toward a fix and will publish updates here as we know more. Any email that should be sent will still be delivered once the email service has been restored.

Time: Jan. 14, 2022, 1:14 p.m.

Status: Postmortem

Update: ## **Summary** On Wednesday, January 12, 2022 at approximately 14:52 UTC \(9:52 AM EST\) we were notified that some customers were experiencing errors when trying to load plan item data in certain views, including List View and list widgets in Custom Dashboards. We were able to replicate the errors in certain scenarios and began investigating the cause within 10 minutes of the initial report. We also confirmed that most application features and operations on plans and plan items continued to function normally. By 17:00 UTC \(12:00 PM\), we confirmed that the issue was likely related to an increase in the amount of time it was taking to return some data from queries for plan items and that the issue only began on January 12. Over the succeeding 9 hours, we continued to research and make adjustments to the database. Some of these adjustments caused increase overall load, particularly between 21:00 UTC and 23:00 UTC \(4:00 PM - 6:00 PM EST\). During this time customers likely experienced significant slowness through the application. At approximately 2:00 UTC on January 13 \(9:00 PM EST on January 12\) we identified that there was a misconfiguration in the primary production database. We corrected that configuration and saw all system operations immediately return to normal. This incident did not affect the US Government production environment. No data was lost or corrupted in either environment. ## **Root Cause** The database misconfiguration was caused by maintenance activities we performed during a production release the evening of January 11, 2022. During the maintenance we inadvertently missed a specific configuration related to the performance of retrieving comments on plan item progress updates. This caused queries that included this comment data to, in certain instances, take much longer to complete. At times, the application would time out waiting on the query and display an error to the user. ## **Mitigation Actions** We manage all configurations for our environments in version control just as if it were code. We have already reviewed and corrected that the database configuration under version control to ensure that it is not missing any other configuration information. We are also in the process of changing how we store database configuration changes, which are done from time to time to manage system performance and security, to be in a more centralized location to improve review of these changes in our quality processes.

Time: Jan. 13, 2022, 2:38 a.m.

Status: Resolved

Update: We have confirmed that all plans and plan items are loading as expected. We will publish a post mortem with additional information about the incident in the next two days.

Time: Jan. 13, 2022, 1:15 a.m.

Status: Monitoring

Update: We have confirmed system performance has returned to normal. We will continue to monitor activity for the next hour to confirm that all performance issues have been resolved.

Time: Jan. 12, 2022, 11:33 p.m.

Status: Identified

Update: We have finished applying the changes to the database to improve the performance. We are seeing improvements but are continuing to evaluate to identify if there are any scenarios where displaying list views of plans are not functioning as expected.

Time: Jan. 12, 2022, 9:32 p.m.

Status: Identified

Update: We are in the process of applying changes to the database to alleviate the performance issues. In the process of making these changes, some areas of the application may have limited functionality. We will continue to update as we complete these changes.

Time: Jan. 12, 2022, 8:44 p.m.

Status: Identified

Update: We have identified that the performance degradation is related to a database configuration. We are working to confirm the root cause and also make configuration adjustments to alleviate the immediate performance issue.

Time: Jan. 12, 2022, 6:52 p.m.

Status: Investigating

Update: We are investigating an issue that appears to be causing information for some plans to load slowly, and in some instances fail to load.

Time: Jan. 14, 2022, 1:14 p.m.

Status: Postmortem

Update: ## **Summary** On Wednesday, January 12, 2022 at approximately 14:52 UTC \(9:52 AM EST\) we were notified that some customers were experiencing errors when trying to load plan item data in certain views, including List View and list widgets in Custom Dashboards. We were able to replicate the errors in certain scenarios and began investigating the cause within 10 minutes of the initial report. We also confirmed that most application features and operations on plans and plan items continued to function normally. By 17:00 UTC \(12:00 PM\), we confirmed that the issue was likely related to an increase in the amount of time it was taking to return some data from queries for plan items and that the issue only began on January 12. Over the succeeding 9 hours, we continued to research and make adjustments to the database. Some of these adjustments caused increase overall load, particularly between 21:00 UTC and 23:00 UTC \(4:00 PM - 6:00 PM EST\). During this time customers likely experienced significant slowness through the application. At approximately 2:00 UTC on January 13 \(9:00 PM EST on January 12\) we identified that there was a misconfiguration in the primary production database. We corrected that configuration and saw all system operations immediately return to normal. This incident did not affect the US Government production environment. No data was lost or corrupted in either environment. ## **Root Cause** The database misconfiguration was caused by maintenance activities we performed during a production release the evening of January 11, 2022. During the maintenance we inadvertently missed a specific configuration related to the performance of retrieving comments on plan item progress updates. This caused queries that included this comment data to, in certain instances, take much longer to complete. At times, the application would time out waiting on the query and display an error to the user. ## **Mitigation Actions** We manage all configurations for our environments in version control just as if it were code. We have already reviewed and corrected that the database configuration under version control to ensure that it is not missing any other configuration information. We are also in the process of changing how we store database configuration changes, which are done from time to time to manage system performance and security, to be in a more centralized location to improve review of these changes in our quality processes.

Time: Jan. 13, 2022, 2:38 a.m.

Status: Resolved

Update: We have confirmed that all plans and plan items are loading as expected. We will publish a post mortem with additional information about the incident in the next two days.

Time: Jan. 13, 2022, 1:15 a.m.

Status: Monitoring

Update: We have confirmed system performance has returned to normal. We will continue to monitor activity for the next hour to confirm that all performance issues have been resolved.

Time: Jan. 12, 2022, 11:33 p.m.

Status: Identified

Update: We have finished applying the changes to the database to improve the performance. We are seeing improvements but are continuing to evaluate to identify if there are any scenarios where displaying list views of plans are not functioning as expected.

Time: Jan. 12, 2022, 9:32 p.m.

Status: Identified

Update: We are in the process of applying changes to the database to alleviate the performance issues. In the process of making these changes, some areas of the application may have limited functionality. We will continue to update as we complete these changes.

Time: Jan. 12, 2022, 8:44 p.m.

Status: Identified

Update: We have identified that the performance degradation is related to a database configuration. We are working to confirm the root cause and also make configuration adjustments to alleviate the immediate performance issue.

Time: Jan. 12, 2022, 6:52 p.m.

Status: Investigating

Update: We are investigating an issue that appears to be causing information for some plans to load slowly, and in some instances fail to load.