Time: June 27, 2023, 3:08 a.m.

Status: Postmortem

Update: ### **SUMMARY** On June 13 2023, from 6:49 PM UTC to June 14, 2023, 02:20 AM UTC, Atlassian customers using Jira Software, Jira Service Management, Jira Work Management, Confluence and Trello with services hosted in AWS us-east-1 region were impacted by Automation rule degradation. This event was triggered by an increased error rates and latencies for AWS Lambda function invocations in the us-east-1 region. Some other AWS services also experienced increased error rates and latencies as a result of degraded Lambda functions invocations.  This incident was automatically detected by multiple monitoring systems within 6 minutes, paging on-call teams. Recovery of the affected AWS Lambda service began after 116 minutes at June 13th 8:45 PM UTC.  Full recovery of all AWS services occurred at 10:37 PM UTC June 13th after the backlog of asynchronous Lambda events had been processed. Some Jira tenants with large event backlogs experienced delays in running schedule-based rule reruns. Full recovery of all Atlassian Cloud services was notified at June 14, 2023, 02:20 AM UTC. ### **IMPACT** The overall impact was between June 13, 2023, 06:49 PM UTC and June 14, 2023, 02:20 AM UTC.  Product-specific impacts are listed below. * Jira Software, Jira Service Management,  Jira Work Management - Automation rules were not executed for 2 hours between Jun 13, 06:49 PM UTC and Jun 13, 08:45 PM UTC.  Jira automation events generated during this period were unable to be rerun.  When AWS Lambda recovered delays were still experienced in our schedule-based and event-based rules for some larger tenants due to a large backlog of events. Full recovery was at June 14, 2023, 02:20 AM UTC. * Confluence - Automation rules were not executed for 2 hours between Jun 13, 06:49 PM UTC and Jun 13, 08:45 PM UTC.  On AWS service restoration Confluence automation recovered and Confluence automation events generated during this period were rerun and processed.  Full recovery was at June 14, 2023, 12:41 AM UTC. * Jira Product Discovery - Automation rules were not executed for 2 hours between Jun 13, 06:49 PM UTC and Jun 13, 08:45 PM UTC. Jira automation events generated during this period were unable to be rerun.  Sending feedback/filing a support ticket from the application did not work.  * Trello -  Email to board delays, card covers image upload failures, attachment preview generation failures, board background upload failures, custom sticker images upload failures, custom emoji upload failures. Trello automation was unaffected. Full recovery was at June 13, 2023, 10:08 PM UTC. The service disruption lasted for 7 hours and 1 minutes between June 13, 2023, 06:49 PM UTC and June 14, 2023, 02:20 AM UTC and caused service disruption to customers with services hosted in the US-EAST-1 region. ### **ROOT CAUSE** Atlassian uses Amazon Web Services \(AWS\) as a cloud service provider. The root cause was an issue with a subsystem responsible for capacity management for AWS Lambda in US-EAST-1 Region, which also impacted 104 AWS services.  This impacted Automation rules as the service is hosted exclusively in this region. There were no relevant Atlassian-driven events in the lead-up that have been identified to cause or contribute to this incident. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are prioritizing the following improvement actions to avoid repeating this type of incident: * Increase reliability of message delivery and recoverability from Jira to Automation platform to improve recovery times.  * Create a plan for multi-region impact mitigation for Automation. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Time: June 14, 2023, 12:57 a.m.

Status: Resolved

Update: From 6:49 PM UTC to 8:45 PM UTC, Trello experienced issues with image uploads that affected card covers, attachment previews, board backgrounds, and custom sticker images. Additionally, we observed degradation in our email-to-board and search features. The services are now operating as expected, and we consider this incident resolved.

Time: June 14, 2023, 12:57 a.m.

Status: Resolved

Update: From 6:49 PM UTC to 8:45 PM UTC, Trello experienced issues with image uploads that affected card covers, attachment previews, board backgrounds, and custom sticker images. Additionally, we observed degradation in our email-to-board and search features. The services are now operating as expected, and we consider this incident resolved.

Time: June 13, 2023, 10:33 p.m.

Status: Monitoring

Update: AWS Services have recovered from an outage. This issue affected the Attachment previews, email-to-board features, search, and card covers, and now all Trello services are back online. For now, we'll keep monitoring from our end.

Time: June 13, 2023, 10:33 p.m.

Status: Monitoring

Update: AWS Services have recovered from an outage. This issue affected the Attachment previews, email-to-board features, search, and card covers, and now all Trello services are back online. For now, we'll keep monitoring from our end.

Time: June 13, 2023, 9:07 p.m.

Status: Identified

Update: We are continuing to work with our third-party partner on a fix. Attachment previews, email-to-board features, search, and card covers are intermittently affected.

Time: June 13, 2023, 9:07 p.m.

Status: Identified

Update: We are continuing to work with our third-party partner on a fix. Attachment previews, email-to-board features, search, and card covers are intermittently affected.

Time: June 13, 2023, 9:05 p.m.

Status: Identified

Update: We have confirmed an issue with AWS services is causing the current issues. We are working with AWS to get the service restored back to normal as soon as possible.

Time: June 13, 2023, 9:05 p.m.

Status: Identified

Update: We have confirmed an issue with AWS services is causing the current issues. We are working with AWS to get the service restored back to normal as soon as possible.

Time: June 13, 2023, 8:19 p.m.

Status: Identified

Update: It has come to our attention that the Email-to-Board feature, along with other previously mentioned features, has been affected by a technical problem. Our team is currently collaborating with our third-party partner to identify and implement an effective solution for this matter.

Time: June 13, 2023, 8:19 p.m.

Status: Identified

Update: It has come to our attention that the Email-to-Board feature, along with other previously mentioned features, has been affected by a technical problem. Our team is currently collaborating with our third-party partner to identify and implement an effective solution for this matter.

Time: June 13, 2023, 7:45 p.m.

Status: Identified

Update: We have identified an issue that may cause problems for users while adding new card covers, viewing attachment previews, or using the search function. The issue has been identified with one of our third-party partners, and we are currently working on resolving it.

Time: June 13, 2023, 7:45 p.m.

Status: Identified

Update: We have identified an issue that may cause problems for users while adding new card covers, viewing attachment previews, or using the search function. The issue has been identified with one of our third-party partners, and we are currently working on resolving it.

Time: May 19, 2023, 8:04 p.m.

Status: Resolved

Update: After our third-party email service released the fix, all the problems with receiving emails from Trello were fixed.

Time: May 19, 2023, 7:01 p.m.

Status: Monitoring

Update: We got an update from our third-party email service provider, who has already implemented a fix. We're monitoring our side to ensure all emails are successfully delivered to our users. We will quickly provide more updates as more information becomes available.

Time: May 19, 2023, 5:18 p.m.

Status: Identified

Update: Our third-party email service provider has identified the problem on their side, and they're now working on a fix to be implemented. We will quickly provide more updates as more information becomes available.

Time: May 19, 2023, 4:41 p.m.

Status: Identified

Update: We have discovered an issue with our third-party email service provider, and we are currently awaiting their analysis to determine the root cause of the problem, which is hindering our users from receiving emails. We will quickly provide more updates as more information becomes available.

Time: May 19, 2023, 4:37 p.m.

Status: Investigating

Update: We're investigating an issue affecting the ability to receive any emails from Trello, except for marketing emails. Our team is still investigating, and we'll share a new update soon.

Time: May 19, 2023, 8:04 p.m.

Status: Resolved

Update: After our third-party email service released the fix, all the problems with receiving emails from Trello were fixed.

Time: May 19, 2023, 7:01 p.m.

Status: Monitoring

Update: We got an update from our third-party email service provider, who has already implemented a fix. We're monitoring our side to ensure all emails are successfully delivered to our users. We will quickly provide more updates as more information becomes available.

Time: May 19, 2023, 5:18 p.m.

Status: Identified

Update: Our third-party email service provider has identified the problem on their side, and they're now working on a fix to be implemented. We will quickly provide more updates as more information becomes available.

Time: May 19, 2023, 4:41 p.m.

Status: Identified

Update: We have discovered an issue with our third-party email service provider, and we are currently awaiting their analysis to determine the root cause of the problem, which is hindering our users from receiving emails. We will quickly provide more updates as more information becomes available.

Time: May 19, 2023, 4:37 p.m.

Status: Investigating

Update: We're investigating an issue affecting the ability to receive any emails from Trello, except for marketing emails. Our team is still investigating, and we'll share a new update soon.

Time: May 3, 2023, 6:51 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: May 3, 2023, 6:12 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: May 3, 2023, 4:30 p.m.

Status: Investigating

Update: Users may see issues logging in or signing up for Trello. Our engineering team is actively investigating this incident and working to resolve as soon as possible. We will update this page as we have additional information.

Time: May 3, 2023, 6:51 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: May 3, 2023, 6:12 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: May 3, 2023, 4:30 p.m.

Status: Investigating

Update: Users may see issues logging in or signing up for Trello. Our engineering team is actively investigating this incident and working to resolve as soon as possible. We will update this page as we have additional information.