Time: Sept. 15, 2023, 8:11 p.m.

Status: Postmortem

Update: **Issue Summary** Customers experienced delays with items such as lab orders and referrals showing up in the message center between 9 AM EST and 6:45 PM EST. Between 3:30 PM EST and 6:45 PM EST, customers also experienced intermittent 503 errors while attempting to log into the system as well as slow/sluggish behavior while using the web application. During the latter incident window, 5% of all requests resulted in 503 errors while a total of 22% of requests failed to complete successfully \(some due to user canceling/refreshing\).   **How were customers impacted?** Customers experience delays in receiving items in the message center as well as general application slowness and at times, trouble logging in.   **Root Cause**  It was determined that a production hotfix deployed on Friday evening at 9 PM EST resulted in a flood of messages that overwhelmed our background processing capabilities for some activities in the application. As a result, our Amazon AWS-hosted queuing infrastructure tipped over under resource pressure from the increased load and took longer than expected to failover and recover. However, this flood was not experienced until hitting Monday traffic levels.   **Resolution**  The hotfix code was reverted and the message backlog was drained.   **Mitigation steps planned/ taken** * Update background processes that share the same processing queue to have dedicated queues. * Evaluate and improve the scalability and resiliency properties of our queue infrastructure. * Identify and remove unnecessary coupling of the web app to the message broker. * Improve rollback/revert efficiency * Improve performance testing in lower environments to better simulate production workloads for message queueing

Time: Sept. 13, 2023, 1:25 p.m.

Status: Resolved

Update: This incident is resolved. Thank you for your patience as we continued to monitor system performance over the last day. A Root Cause Analysis (RCA) will be available via our status page soon.

Time: Sept. 11, 2023, 11:21 p.m.

Status: Monitoring

Update: DrChrono believes all issues have been resolved with intermittent service outages, slow loading, reports not arriving to the message center, referrals not loading, lab order failure, and other related issues. If you are still trying these actions and not seeing the expected results please reach out to support.

Time: Sept. 11, 2023, 11:11 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Sept. 11, 2023, 8:56 p.m.

Status: Investigating

Update: Our engineering team continues to investigate today's ongoing performance issues including reports from some users that they are not able to log in to DrChrono. Thank you for your continued patience. We apologize for this inconvenience.

Time: Sept. 11, 2023, 6:20 p.m.

Status: Investigating

Update: We are continuing to investigate this issue with high priority. Additional updates will continue to be posted on our status page.

Time: Sept. 11, 2023, 5:05 p.m.

Status: Investigating

Update: We have received customer feedback that these issues have not been completely mitigated. Our engineering team is continuing to investigate.

Time: Sept. 11, 2023, 4:05 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Sept. 11, 2023, 3:30 p.m.

Status: Identified

Update: Our engineering team is continuing to work to implement a fix to resolve the performance issues. Thank you for your patience during this time.

Time: Sept. 11, 2023, 2:48 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Sept. 11, 2023, 2:23 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Sept. 11, 2023, 2:20 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Sept. 11, 2023, 2:19 p.m.

Status: Investigating

Update: DrChrono is currently experiencing intermittent service outages, slow loading, reports not arriving to the message center, referrals not loading, lab order failure, and other related issues. Please be aware that we take this very seriously and our engineering team is actively investigating and working to resolve this situation as quickly as possible. We will provide another update via this status page as soon as possible.

Time: June 14, 2023, 1:05 p.m.

Status: Resolved

Update: We've received an update that as of June 13th 3:37PM PDT, AWS services have been restored. DrChrono impacted functionalities should now be back to normal. Our team will be preparing an RCA and will be available via this status page.

Time: June 13, 2023, 9:45 p.m.

Status: Monitoring

Update: We've received information that affected AWS services are seeing improvement in their error rates and are working towards full recovery. You may see improvements in DrChrono related to previously reported errors on accessing and saving clinical notes including faxing as a result. If you continue to experience issues, please reach out to our Support team by submitting a ticket. Thank you for your patience.

Time: June 13, 2023, 7:44 p.m.

Status: Investigating

Update: We have received information that Amazon Web Services (AWS) services in the US-EAST are encountering significant issues due to AWS multiple service failure. We have identified that this impacts document-generating functions such as viewing, locking, downloading or saving clinical notes as well as sending out faxes. Our team is working on ways to assess the impact of this and create mitigation steps. We will post additional updates as more information becomes available.

Time: March 6, 2023, 8:16 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: March 6, 2023, 6:32 p.m.

Status: Monitoring

Update: Quest lab orders appear to be working again, though no engineering changes were implemented by DrChrono. There appears to have been some type of connectivity disruption between DrChrono and Quest that has resolved itself.

Time: March 6, 2023, 5:21 p.m.

Status: Investigating

Update: DrChrono is currently investigating a service disruption with sending lab orders to Quest Diagnostics. Please be aware that our engineering team is prioritizing the investigation of this issue and working to resolve this situation as quickly as possible. We will provide another update via this status page as soon as possible.

Time: Feb. 18, 2023, 3:37 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 17, 2023, 8:17 p.m.

Status: Identified

Update: The issue has been identified as stemming from a third party service our platform utilizes. We are reaching out to them in order to continue efforts to mitigate the problem. We will continue to post updates here as we have additional information.

Time: Feb. 17, 2023, 5:27 p.m.

Status: Investigating

Update: We are currently investigating reports from our users that they are unable to join telehealth video visit calls from the DrChrono web application. As a workaround, video visits can be joined from the EHR iOS application. We will provide an update as soon as we have additional information.

Time: Feb. 7, 2023, 2:19 p.m.

Status: Resolved

Update: Message from Square: We’re no longer seeing elevated levels of payment declines and card processing activity has returned to normal. Our team will continue to work with our partners on uncovering the underlying cause of the earlier disruption and putting in place safeguards to help prevent issues of this nature from recurring in the future. We understand how important it is for all of our services to be running for your business, thank you for your patience with us as we monitored the situation.

Time: Feb. 6, 2023, 11:11 p.m.

Status: Monitoring

Update: Update from Square: While we are seeing less elevated level of declines with most card carriers, we are continuing to see reports of declines when using Discover. If you continue to experience declines, please advise your customers to use an alternative card or form of payment. We appreciate your patience as we continue to work on resolving this disruption. We will continue monitoring the situation and provide updates as we have them.

Time: Feb. 6, 2023, 10:02 p.m.

Status: Investigating

Update: We received information from Square that they are currently encountering a system issue related to accepting payments. To get direct updates about this incident, please visit this link: https://www.issquareup.com/incidents/v952qqw12x1c We will continue to cross-post updates as we receive them.