Time: March 31, 2023, 4:54 p.m.

Status: Resolved

Update: From 16:30 UTC to 16:36 UTC, calls to the Read Events API encountered a higher level of 500 Server Error responses than usual. This was due to a bug released at this time which allowed a small number of events to cause an error while being rendered. This was immediately spotted and rolled back, which was completed at 16:36 UTC.

Time: Feb. 14, 2023, 5:40 p.m.

Status: Resolved

Update: At 15:58 UTC, communication between Cronofy and Apple began to timeout for a proportion of our requests. This quickly worsened, causing Apple calendar syncs to take several attempts. A change was deployed to slow down the rate of communication with Apple to reduce the pressure on their API and to avoid service being impacted for other calendar providers. At 16:50 UTC, we started to see most communication succeed again and monitored the situation until it had remained stable for 30 minutes. At 17:20 UTC, we removed the rate-limiting and continued to monitor the situation. Normal service has continued, so we are marking this incident as resolved. If you have any queries, please contact us at [email protected]

Time: Feb. 14, 2023, 4:52 p.m.

Status: Monitoring

Update: Apple API traffic has recovered, and normal operation has resumed. We are continuing to monitor the service.

Time: Feb. 14, 2023, 4:37 p.m.

Status: Identified

Update: We are seeing an increase in connection errors to Apple's API globally. Apple sync is still operating in a degraded state. We will update at 17:00 UTC.

Time: Feb. 14, 2023, 4:14 p.m.

Status: Investigating

Update: We are seeing increased errors from Apple's calendar API since 16:00 UTC; we are investigating and taking mitigating action.

Time: Feb. 13, 2023, 10:59 a.m.

Status: Resolved

Update: From 09:59 UTC to 10:07 UTC, calls to our API encountered a higher level of 500 Server Error responses than usual. This was due to a bug released at this time which was not caught by our automated tests or during code review. Once we were notified of the issue with the release, we initiated a rollback, which took about 7 minutes to roll out. We will be improving our test suite and adding better error handling in this area.

Time: Jan. 25, 2023, 10:18 a.m.

Status: Resolved

Update: The performance of syncing Microsoft 365-backed calendars was degraded for about 90 minutes. Microsoft 365 began returning an elevated number of errors and timing out just after 07:05 UTC until 08:30 UTC. This was part of a wider issue within Azure, where Microsoft 365 is hosted. Steps were taken to increase capacity and be more aggressive in timing out connections to Microsoft 365 to mitigate the impact on background processing. The sudden change to the volume of background jobs did lead to a slight delay in processing being present for much of the issue. Normal service resumed a few minutes after the Microsoft 365 APIs began to operate normally, once a surge of calendar update notifications was processed. We will be reviewing our mechanisms for reducing the side effects of such incidents in the future.

Time: Jan. 25, 2023, 8:44 a.m.

Status: Monitoring

Update: Error rates for Microsoft 365-backed calendars have returned to normal levels, and we have processed a surge in calendar updates. We will continue to monitor the service but believe this issue to now be resolved.

Time: Jan. 25, 2023, 8 a.m.

Status: Monitoring

Update: Microsoft have sent a 365 health notification (MO502273) relating to some users being unable to access multiple Microsoft 365 services. This is likely related to the issue we are seeing. Errors for 365-backed calendar continue to be elevated.

Time: Jan. 25, 2023, 7:35 a.m.

Status: Monitoring

Update: We are seeing a higher than usual number of errors affecting calendars hosted on Microsoft 365 across all our data centers for connections using Exchange Web Services (EWS) and Microsoft's Graph API. Background processing has scaled up to compensate but may be degraded as a result.

Time: Dec. 15, 2022, 6:36 p.m.

Status: Resolved

Update: Performance was degraded in our US data center for around 2 hours between 16:00 and 18:00 UTC. This was down to our primary database struggling under load. Steps were taken to remove any background processes to reduce the load as much as possible to aid the system to return to regular operation. API responses may have been slower than usual during this period, and background processing such as synchronizing calendar data will also have been slower than usual with messages taking up to 3 minutes to be picked up at the peak of the incident. We will be bringing forward the maintenance to upgrade this database cluster from this coming Sunday 18th December to tomorrow Friday 16th December. A notice for this maintenance change will be posted shortly.

Time: Dec. 15, 2022, 5:58 p.m.

Status: Monitoring

Update: The queued work has now been processed and we can see that performance is no longer degraded. We are continuing to monitor the situation.

Time: Dec. 15, 2022, 5:28 p.m.

Status: Identified

Update: Our US database is experiencing slower than usual disk performance. We have taken steps to ease the pressure, such as temporarily disabling maintenance tasks. The amount of queued work is reducing. We're continuing to work to bring performance levels back to the usual level. We have also added bigger database nodes to the cluster in case we need to failover to those. However, this would require a short outage and so we are holding off on failing over to those just yet.

Time: Dec. 15, 2022, 4:34 p.m.

Status: Investigating

Update: We have taken steps to ease the pressure on our US database. This has resulted in better, but still degraded, performance. We're continuing to investigate the root cause

Time: Dec. 15, 2022, 4:11 p.m.

Status: Investigating

Update: We are investigating degraded performance in our US data center