Time: March 20, 2024, 10:41 p.m.

Status: Resolved

Update: Between 18:16 UTC and 22:XX UTC, March 20, 2024, we experienced creating issues and transitions for Jira Service Management, Jira Work Management, and Jira Software. The issue has been resolved and the service is operating normally.

Time: March 20, 2024, 10:05 p.m.

Status: Identified

Update: We are have identified an issue with creating issues and transitions that is impacting some Cloud customers. A fix is being rolled out now. Another update will be published in the next hour.

Time: March 15, 2024, 9 p.m.

Status: Postmortem

Update: ### SUMMARY On March 7, 2024, between 12:02 pm UTC and 12:21 pm UTC, some Atlassian customers using Jira in the EU-Central region were unable to interact with core experiences of the product, and some customers using Confluence in the EU-Central region had a degraded experience. This was caused by unexpected traffic and recent changes to automated scaling policies which resulted in nodes not being able to scale up quickly enough. The incident was detected within two minutes by automated monitoring and was mitigated by manually scaling up nodes which put Atlassian systems into a known good state. The total time to resolution was 19 minutes. ### **IMPACT** The overall impact was on March 7, 2024, between 12:02 pm UTC and 12:21 pm UTC, on Jira and Confluence products. The incident caused service disruption to some customers accessing the EU-central region. For Jira, the disruption caused core experiences of the product to be unavailable during this time. For Confluence, some non-core experiences were affected during this time. ### **ROOT CAUSE** This incident was caused by unexpected traffic and recent changes to automated scaling policies which resulted in nodes not being able to scale up quickly enough in the EU-Central region. As a result, the Atlassian GraphQL Gateway could not service all requests in EU-Central, leading to parts of Jira being unavailable for some customers and degradation of non-core features in Confluence in the incident time period. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We understand that outages impact your productivity. We have already implemented improvement actions to avoid repeating this type of incident and reduce the time to recovery. We have improved scaling out policies to be substantially more aggressive. This will help mitigate many different types of incidents. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Time: March 15, 2024, 9 p.m.

Status: Postmortem

Update: ### SUMMARY On March 7, 2024, between 12:02 pm UTC and 12:21 pm UTC, some Atlassian customers using Jira in the EU-Central region were unable to interact with core experiences of the product, and some customers using Confluence in the EU-Central region had a degraded experience. This was caused by unexpected traffic and recent changes to automated scaling policies which resulted in nodes not being able to scale up quickly enough. The incident was detected within two minutes by automated monitoring and was mitigated by manually scaling up nodes which put Atlassian systems into a known good state. The total time to resolution was 19 minutes. ### **IMPACT** The overall impact was on March 7, 2024, between 12:02 pm UTC and 12:21 pm UTC, on Jira and Confluence products. The incident caused service disruption to some customers accessing the EU-central region. For Jira, the disruption caused core experiences of the product to be unavailable during this time. For Confluence, some non-core experiences were affected during this time. ### **ROOT CAUSE** This incident was caused by unexpected traffic and recent changes to automated scaling policies which resulted in nodes not being able to scale up quickly enough in the EU-Central region. As a result, the Atlassian GraphQL Gateway could not service all requests in EU-Central, leading to parts of Jira being unavailable for some customers and degradation of non-core features in Confluence in the incident time period. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We understand that outages impact your productivity. We have already implemented improvement actions to avoid repeating this type of incident and reduce the time to recovery. We have improved scaling out policies to be substantially more aggressive. This will help mitigate many different types of incidents. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Time: March 7, 2024, 2:24 p.m.

Status: Resolved

Update: Between 12:00 March 7th and 12-25 March 7th UTC, we faced performance problems with the Jira Issue View for some EU customers. The issue has been resolved and the service is operating normally. We are closing this incident for now.

Time: March 7, 2024, 2:24 p.m.

Status: Resolved

Update: Between 12:00 March 7th and 12-25 March 7th UTC, we faced performance problems with the Jira Issue View for some EU customers. The issue has been resolved and the service is operating normally. We are closing this incident for now.

Time: March 7, 2024, 1:14 p.m.

Status: Investigating

Update: We have identified ongoing performance problems with the Jira Issue View for some EU customers. We are actively working to resolve this issue as quickly as possible.

Time: March 7, 2024, 1:14 p.m.

Status: Investigating

Update: We have identified ongoing performance problems with the Jira Issue View for some EU customers. We are actively working to resolve this issue as quickly as possible.

Time: March 6, 2024, 5:50 a.m.

Status: Resolved

Update: Between 20:25 March 5th and 04:49 March 6th UTC, we faced an issue with our incoming mail handlers, preventing the processing of new emails for some customers. The issue has been resolved and the service is operating normally.

Time: March 6, 2024, 5:01 a.m.

Status: Monitoring

Update: We have deployed the fix and we are seeing recovery on affected cloud sites. We are now monitoring closely.

Time: March 6, 2024, 4:07 a.m.

Status: Identified

Update: We have identified the root cause and we are currently working on deploying the fix. We'll update the status within the next hour.

Time: March 6, 2024, 12:12 a.m.

Status: Investigating

Update: We have identified an incident that has caused the failure of the incoming email handler to a small number of customers, affecting issue creation and comments via email. We are investigating and will post updates within the next hour.

Time: March 6, 2024, 5:50 a.m.

Status: Resolved

Update: Between 20:25 March 5th and 04:49 March 6th UTC, we faced an issue with our incoming mail handlers, preventing the processing of new emails for some customers. The issue has been resolved and the service is operating normally.

Time: March 6, 2024, 5:01 a.m.

Status: Monitoring

Update: We have deployed the fix and we are seeing recovery on affected cloud sites. We are now monitoring closely.

Time: March 6, 2024, 4:07 a.m.

Status: Identified

Update: We have identified the root cause and we are currently working on deploying the fix. We'll update the status within the next hour.

Time: March 6, 2024, 12:12 a.m.

Status: Investigating

Update: We have identified an incident that has caused the failure of the incoming email handler to a small number of customers, affecting issue creation and comments via email. We are investigating and will post updates within the next hour.

Time: April 5, 2024, 1:04 a.m.

Status: Postmortem

Update: ### Summary On February 29, 2024, between 05:11 and 08:46 UTC, customers of Jira Software, Jira Service Management, Jira Work Management, Jira Product Discovery, and Confluence experienced an incorrect data update of Automation Rules. This was due to an incomplete feature flag rollout and bugs in the data upgrade code. 99.99% of affected Automation Rules were remediated by 17:00 UTC on February 29, 2024. The remaining 0.01% of Rule Components were edited by users during the impacted time window and required a confirmation to ensure changes weren’t overridden. These customers were contacted proactively. ### **IMPACT** Some Rules containing Issue Edit, Issue Create, or Issue Clone actions in Jira products had the Advanced Fields section removed. These Rules continued to run as if the Advanced Fields section were empty. In addition, for affected Issue Edit actions, Jira notifications were not sent. Some Rules using the Send Email action in Jira and Confluence had the “From name” field removed. Emails continued to be sent but fell back to the Automation default for the “From name” field. ### **ROOT CAUSE** We were in the process of rolling out a new feature for Automation, which required us to perform data upgrades, which could be risky as the current system would run the upgrades automatically upon Rule activity. In an attempt to minimize such risk, we used feature flags to better control when the upgrade occurred. However, there was an error in the feature flag configuration which resulted in the upgrades immediately kicking in. Additionally, there were bugs in these upgrades that resulted in overriding customers' saved values with default values. This incorrect configuration was used for Rule runs until recovery. ### **REMEDIAL ACTION PLAN & NEXT STEPS** We are prioritizing the following improvement actions to avoid repeating this type of incident: * We are changing the approach for upgrading Rule configurations to allow better testing and prevent accidental upgrades. * Improving feature flag rollout and verification processes to avoid such incorrect configurations. * Increasing the frequency of backups of the Automation Rule data. We apologize to customers who were impacted during this incident; we are taking immediate steps to improve the reliability of Automation. Thanks, Atlassian Customer Support

Time: March 6, 2024, 4:53 a.m.

Status: Resolved

Update: All the impacted customers have been contacted and all the impacted rules have been fixed. We are now marking this issue as resolved.

Time: March 1, 2024, 2:52 p.m.

Status: Identified

Update: We have successfully restored the majority of automation rules that were affected, but we are currently in the process of manually fixing the remaining ones. Our support team will be contacting the customers impacted by this issue to proceed with resolving their cases.

Time: March 1, 2024, 3:18 a.m.

Status: Identified

Update: This issue has been mitigated, and we have restored the majority of automation rules that were affected. Those rules are working as normal, including the rules that had not been modified in the last 48 hours and the rules created after the incident. The remaining rules that haven't yet been restored are cases where customers have manually updated the rules after the incident happened. We are proceeding with caution to avoid overwriting these changes.

Time: Feb. 29, 2024, 5:30 p.m.

Status: Identified

Update: We are continuously working on restoring the affected automation rules configuration data. At the moment the majority of the impact has already been mitigated and the team is currently focusing on a few specific outstanding scenarios. We will continue to update you as we progress.

Time: Feb. 29, 2024, 3:20 p.m.

Status: Identified

Update: We are currently working on restoring the lost automation rules configuration data. The email notifications related to the automation will be addressed upon restoring the lost data. Customers can update the configurations with the Automation rules themselves, and our fix will not override those changes. We will share further updates as we progress.

Time: Feb. 29, 2024, 10:54 a.m.

Status: Identified

Update: We have identified the root cause and we are currently working on deploying fix to restore any lost configuration/rules. Next update will be provided in an hour.

Time: Feb. 29, 2024, 9:23 a.m.

Status: Investigating

Update: We continue to investigate the issue with the Automation rules losing configuration data that is impacting certain cloud products. We are actively working to resolve this issue as quickly as possible.

Time: Feb. 29, 2024, 8:22 a.m.

Status: Investigating

Update: We are investigating an issue with the Automation rules losing configuration data that is impacting some of the Cloud products. We are actively working to resolve this issue as quickly as possible.