Time: Feb. 9, 2021, 12:23 p.m.

Status: Postmortem

Update: # What happened On Friday 5th February 2021, our US data center was not responsive for nearly three minutes, between the times of 16:53:45 UTC and 16:56:40 UTC. During this time, our API at [api.cronofy.com](http://api.cronofy.com) as well as our web application at [app.cronofy.com](http://app.cronofy.com) were not reachable. Our web application hosts the developer dashboard, Scheduler application, Real-Time Scheduling pages, and end-user authorization flows. Our background processing of jobs such as calendar syncronization was not affected. We first became aware of the issue at 16:55:07 UTC and began investigating, however the data center recovered automatically without direct action from ourselves. # Our investigation We identified that the symptoms of the issue matched an [incident from November 23 2020](https://status.cronofy.com/incidents/01syy96xwvpy). This time, we were able to identify the underlying cause with much more certainty. The root cause was determined to be a slowdown in the database table responsible for logging developers' API calls to us. As calls backed up, the application eventually stopped serving requests entirely. Our API and web applications are hosted together, and as a result both were affected by the downtime despite the underlying issue being specific to our API. AWS RDS Performance Insights identified an automatically scheduled housekeeping operation as being responsible for a disproportionate load to the database table, at an already busy time. # What we're doing Today, we are making changes to our database configuration for the relevant table. The effect of this will be that individual housekeeping operations will happen more often, but are smaller in scope, faster, and more predictable in their timing. We've previously made the same change to other tables that have similar access patterns and higher size and throughput, so are confident in its effect. In the near term, we will change the deployment of our applications, so that the API and web applications run more independently. This will help lessen the impact of any future issues of this nature.

Time: Feb. 5, 2021, 5:32 p.m.

Status: Resolved

Update: This outage appears very similar in nature to the previous US outage on Nov 23, 2020. Details of that and its post-mortem can be found here: https://status.cronofy.com/incidents/01syy96xwvpy We will conduct a full root cause analysis and publish the outcomes in the coming days. In the meantime, if you have further questions, please email [email protected].

Time: Feb. 5, 2021, 5:19 p.m.

Status: Monitoring

Update: The US data center started showing signs of the API being inaccessible from 16:53:45 (times UTC), it was fully accessible again from 16:56:40. Signs are that the primary database in this data center experienced an event starting at this time which ended around 16:56:30, with external access recovering a few seconds later. All metrics have returned to normal levels since this time.

Time: Feb. 5, 2021, 4:58 p.m.

Status: Investigating

Update: The US data center was temporarily unreachable, it has recovered automatically and we are investigating the cause.

Time: Jan. 15, 2021, 2:58 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 15, 2021, 2:42 a.m.

Status: Monitoring

Update: Errors for 365 calendars have been around normal levels for approximately 20 minutes. This Twitter thread contains some details of MO234433 and related EX234428 which appear to be the root cause: https://twitter.com/MSFT365Status/status/1349884557747695618 We will look to reinstate credentials where possible shortly.

Time: Jan. 15, 2021, 2:22 a.m.

Status: Monitoring

Update: Errors for 365 calendars remain higher, but otherwise systems have returned to normal. A small number of accounts (less that 1%) may have been incorrectly flagged as disconnected during this period. Once 365 errors have returned to normal levels we will attempt to reinstate credentials that have been flagged during this time. Microsoft have opened MO234433 "Unable to access multiple Microsoft 365 apps or services" which is potentially related to the errors we have observed.

Time: Jan. 15, 2021, 1:27 a.m.

Status: Identified

Update: The error rate when communicating with 365 calendars from our US data center has reduced significantly and overall background processing has returned to normal, though errors for 365 calendars remain higher than normal. Microsoft have opened MO234433 "Unable to access multiple Microsoft 365 apps or services" which is potentially related to the errors we have observed.

Time: Jan. 15, 2021, 1:18 a.m.

Status: Identified

Update: We are still seeing a high rate of errors when communicating with Microsoft 365 calendars. This is affecting all calls, not just push notification registration as previously reported. Service may be returning to normal. We are continuing to monitor and take mitigating actions where possible.

Time: Jan. 15, 2021, 1:01 a.m.

Status: Investigating

Update: We are seeing a high rate of errors when registering push notifications for 365 hosted calendars from our US data center. These requests are timing out or being refused with "service unavailable" responses. We have taken steps to mitigate the effects of this on background processing.

Time: Jan. 15, 2021, 12:40 a.m.

Status: Investigating

Update: We are currently investigating this issue.

Time: Jan. 15, 2021, 2:58 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 15, 2021, 2:42 a.m.

Status: Monitoring

Update: Errors for 365 calendars have been around normal levels for approximately 20 minutes. This Twitter thread contains some details of MO234433 and related EX234428 which appear to be the root cause: https://twitter.com/MSFT365Status/status/1349884557747695618 We will look to reinstate credentials where possible shortly.

Time: Jan. 15, 2021, 2:22 a.m.

Status: Monitoring

Update: Errors for 365 calendars remain higher, but otherwise systems have returned to normal. A small number of accounts (less that 1%) may have been incorrectly flagged as disconnected during this period. Once 365 errors have returned to normal levels we will attempt to reinstate credentials that have been flagged during this time. Microsoft have opened MO234433 "Unable to access multiple Microsoft 365 apps or services" which is potentially related to the errors we have observed.

Time: Jan. 15, 2021, 1:27 a.m.

Status: Identified

Update: The error rate when communicating with 365 calendars from our US data center has reduced significantly and overall background processing has returned to normal, though errors for 365 calendars remain higher than normal. Microsoft have opened MO234433 "Unable to access multiple Microsoft 365 apps or services" which is potentially related to the errors we have observed.

Time: Jan. 15, 2021, 1:18 a.m.

Status: Identified

Update: We are still seeing a high rate of errors when communicating with Microsoft 365 calendars. This is affecting all calls, not just push notification registration as previously reported. Service may be returning to normal. We are continuing to monitor and take mitigating actions where possible.

Time: Jan. 15, 2021, 1:01 a.m.

Status: Investigating

Update: We are seeing a high rate of errors when registering push notifications for 365 hosted calendars from our US data center. These requests are timing out or being refused with "service unavailable" responses. We have taken steps to mitigate the effects of this on background processing.

Time: Jan. 15, 2021, 12:40 a.m.

Status: Investigating

Update: We are currently investigating this issue.