Is there an DrChrono outage?
DrChrono status: Systems Active
Last checked: a minute ago
Get notified about any outages, downtime or incidents for DrChrono and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: a minute ago
Get notified about any outages, downtime or incidents for DrChrono and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for DrChrono.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| drchrono.com | Active |
| drchrono iPad Check-In Kiosk Application | Active |
| drchrono iPad EHR | Active |
| DrChrono Telehealth Platform | Active |
| onpatient.com | Active |
| onpatient iPhone PHR | Active |
| www.drchrono.com Public Website | Active |
| 3rd party services | Active |
| Box Uploads/Downloads | Active |
| Emdeon/Change Healthcare | Active |
| Health Gorilla | Active |
| HelloFax | Active |
| ID.ME | Active |
| Labcorp | Active |
| Quest Diagnostics | Active |
| Square | Active |
| Stripe API | Active |
| support.drchrono.com | Active |
| Trizetto | Active |
| Twilio Authy | Active |
| Twilio REST API | Active |
| Twilio SMS | Active |
| Twilio TwiML | Active |
| Updox | Active |
| Waystar | Active |
View the latest incidents for DrChrono and check for official updates:
Description: Our team has confirmed the the issue has been resolved. If you continue to experience issues, please reach out directly to the respective partner for further guidance on re-authenticating based on their individual workflow.
Status: Resolved
Impact: Minor | Started At: Dec. 19, 2022, 4:17 p.m.
Description: All affected Labcorp result files received between October 28th and November 8th have been reprocessed and are now available in the DrChrono platform. As mentioned yesterday, new results received from late November 8th onward are processing as normal. Thank you for your patience while we worked to resolve this issue.
Status: Resolved
Impact: Major | Started At: Nov. 7, 2022, 10:08 p.m.
Description: On 11/4, the task and message count numbers were temporarily deactivated as a means to mitigate the overall system performance issues by re-allocating resources to increase capacity resources and help site speeds. The full RCA is accessible in the postmortem for the overall slowness issue. [Click here to view the page](https://status.drchrono.com/incidents/gw0f7l9xx3gn).
Status: Postmortem
Impact: Minor | Started At: Nov. 4, 2022, 3:42 p.m.
Description: On 11/3/ users encountered difficulties accessing the website and other system functionalities. ## Timeline All times are PST Timezone. | **Date/Time** | **Activity** | | --- | --- | | 2022-11-02 22:00:00 | An Amazon Web Services system failure causes the primary database to be restarted. The DrChrono EHR platform becomes inaccessible. | | 2022-11-02 22:15:00 | Amazon Web Services automatically restarts the affected server, but a metadata loss causes other servers to stop being able to serve production database traffic as well. | | 2022-11-02 22:15:00 | An engineer on the DevOps team comes online after being notified of the issue from automated alarming systems and starts investigating. | | 2022-11-02 22:25:00 | The root cause of the issue is identified and the engineer reroutes traffic to preserve system availability. The DrChrono EHR platform is now accessible, but running at reduced capacity due to the loss of extra servers. | | 2022-11-02 23:30:00 | After the first attempts to recover the affected servers directly fail, the DevOps engineer initiates a backup and restore process to prepare capacity for the upcoming day traffic for one of the affected servers. | | 2022-11-03 01:30:00 | The rescue process does not succeed for the additional servers and it's determined that they will also need to be restored from backups. The process is initiated. | | 2022-11-03 05:44:00 | Status page was created to notify customers about “General system slowness on the DrChrono platform”. | | 2022-11-03 8:30:00 | One of the replica servers finalizes phase 1/3 of backup restore; the next steps are engaged immediately. | | 2022-11-03 11:45:00 | One of the replica servers completes phase 3/3. Additional replica server restoration processes are initiated. | | 2022-11-03 14:15:00 | Additional replica servers' process restoration fails. | | 2022-11-03 16:27:00 | Additional replica servers' process restoration was attempted but also fails. | | 2022-11-03 17:45:00 | A backup process from the primary DB started. | | 2022-11-03 20:55:00 | One additional replica server finishes process restoration. | | 2022-11-03 21:15:00 | Spun up 3 more database replicas as a backup solution. | | 2022-11-03 22:30:00 | Added an additional 3 database replicas. | | 2022-11-04 00:05:00 | 6 new database replicas are installed and configured. | | 2022-11-04 01:40:00 | The backup from the primary database is finalized. | | 2022-11-04 01:45:00 | The primary backup snapshot with fast restore option is started. | | 2022-11-04 02:10:00 | The standby server started phase 1/3 of backup restore from the primary database. | | 2022-11-04 04:00:00 | Restore process is completed. | | 2022-11-04 07:00:00 | Status page was updated to resolved. | | 2022-11-04 07:07:00 | One of the replica servers is removed. After reviewing the server response, the team found it was not at full capacity and will need a complete restoration. | | 2022-11-04 08:10:00 | Tasks and message count display were blocked increasing the capacity of responses from the third server and primary. | | 2022-11-04 08:42:00 | A status page was created to notify customers about the “Task and message counts are temporarily unavailable”. | | 2022-11-04 08:42:00 | A replica server finalizes phase 2/3 of backup restore; the final phase is engaged. | | 2022-11-05 17:00:00 | Replica restoration process and shutdown of other servers are completed. All DB servers were monitored for the next hours. | | 2022-11-06 09:50:00 | Announced the system returned to full capacity and operational. | | 2022-11-07 06:33:00 | Status page “Task and message counts are temporarily unavailable“ was resolved. | ## Contributing Factor\(s\) Sentry alerts contain information to identify the current issue. ## Stabilization Steps A configuration change was made in the database to match the previous behavior. ## Impact Users experienced overall system slowness affecting major functionalities such as errors copying notes, locking notes, saving information, and the likes. ## Corrective Actions The Ops team has fixed the issue in the database table and will roll out updates and connections to the database gradually. Outlining the exact process for any database configuration changes and mitigation/fallback procedures will prevent a similar situation from happening again.
Status: Postmortem
Impact: Minor | Started At: Nov. 3, 2022, 12:44 p.m.
Description: This issue has been resolved. Please note that our system is currently working through the backlog of orders and results that were queued for processing. It may take several hours for this backlog to be cleared this evening. In the meantime, new orders that are placed will expectedly populate a "Timeout" notice on the screen but will be added to the queue of orders that is currently processing. There is no need to try sending the order again. Thank you for your patience during this investigation and resolution.
Status: Resolved
Impact: Critical | Started At: Oct. 24, 2022, 9:30 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
