Time: Dec. 28, 2023, 3:26 a.m.

Status: Postmortem

Update: ### **SUMMARY** On Dec 18, 2023, between 12:29 p.m. and 3:35 p.m. UTC, Dec 18, 2023, Atlassian's cloud customers using Atlas, Bitbucket Cloud, Compass, Confluence Cloud, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery products were unable to search for users or use the "@mention" functionality. Customers' user search results failed or were delayed as Atlassian's service returning user search results was degraded in several regions. The incident originated from a computationally intensive operation that was triggered multiple times in rapid succession, resulting in degraded performance of Atlassian's user search service across several regions. Notably, customers in the EU west region were most affected. The incident was detected within 2 minutes by automated monitoring, and our team promptly took action by recovering unhealthy systems and scaling up the service's infrastructure temporarily. The resolution process concluded in 3 hours and 06 minutes. ### **IMPACT** The overall impact was between Dec 18, 2023, between 12:29 p.m. UTC and Dec 18, 2023, 3:35 p.m. UTC. The Incident caused service disruption to cloud customers worldwide. Customers experienced delayed or failed user searches when using the following Atlassian cloud products: * Atlas * Bitbucket Cloud * Compass * Confluence Cloud * Jira Service Management * Jira Software * Jira Work Management * Jira Product Discovery ### **ROOT CAUSE** The incident stemmed from Atlassian's user search service receiving commands to process multiple computationally intensive operations in rapid succession. These operations were directed at the same customer data set, and therefore overloaded resources within a clustered database system, leading to memory exhaustion and subsequent unresponsiveness to user search requests.  ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** To prevent a recurrence of such incidents, we are implementing the following measures: * Implement a mechanism to queue computationally intensive operations in order to avoid overloading the resources within the systems and process them without impact on customer experience. * Fine-tune our clustered database settings to mitigate the impact of resource exhaustion on the overall system.  We apologize to customers whose services were affected during this incident; we are taking immediate steps to improve the service’s resiliency. Thanks, Atlassian Customer Support

Time: Dec. 28, 2023, 3:26 a.m.

Status: Postmortem

Update: ### **SUMMARY** On Dec 18, 2023, between 12:29 p.m. and 3:35 p.m. UTC, Dec 18, 2023, Atlassian's cloud customers using Atlas, Bitbucket Cloud, Compass, Confluence Cloud, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery products were unable to search for users or use the "@mention" functionality. Customers' user search results failed or were delayed as Atlassian's service returning user search results was degraded in several regions. The incident originated from a computationally intensive operation that was triggered multiple times in rapid succession, resulting in degraded performance of Atlassian's user search service across several regions. Notably, customers in the EU west region were most affected. The incident was detected within 2 minutes by automated monitoring, and our team promptly took action by recovering unhealthy systems and scaling up the service's infrastructure temporarily. The resolution process concluded in 3 hours and 06 minutes. ### **IMPACT** The overall impact was between Dec 18, 2023, between 12:29 p.m. UTC and Dec 18, 2023, 3:35 p.m. UTC. The Incident caused service disruption to cloud customers worldwide. Customers experienced delayed or failed user searches when using the following Atlassian cloud products: * Atlas * Bitbucket Cloud * Compass * Confluence Cloud * Jira Service Management * Jira Software * Jira Work Management * Jira Product Discovery ### **ROOT CAUSE** The incident stemmed from Atlassian's user search service receiving commands to process multiple computationally intensive operations in rapid succession. These operations were directed at the same customer data set, and therefore overloaded resources within a clustered database system, leading to memory exhaustion and subsequent unresponsiveness to user search requests.  ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** To prevent a recurrence of such incidents, we are implementing the following measures: * Implement a mechanism to queue computationally intensive operations in order to avoid overloading the resources within the systems and process them without impact on customer experience. * Fine-tune our clustered database settings to mitigate the impact of resource exhaustion on the overall system.  We apologize to customers whose services were affected during this incident; we are taking immediate steps to improve the service’s resiliency. Thanks, Atlassian Customer Support

Time: Dec. 19, 2023, 3:28 a.m.

Status: Resolved

Update: It has been resolved. Atlassian's cross product user search is working.

Time: Dec. 19, 2023, 3:28 a.m.

Status: Resolved

Update: It has been resolved. Atlassian's cross product user search is working.

Time: Dec. 18, 2023, 4:23 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is currently healthy. Searches for users within Atlassian products are working as expected. We are in the process of investigating the root cause of this incident.

Time: Dec. 18, 2023, 4:23 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is currently healthy. Searches for users within Atlassian products are working as expected. We are in the process of investigating the root cause of this incident.

Time: Dec. 18, 2023, 4:06 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Time: Dec. 18, 2023, 4:06 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Time: Dec. 18, 2023, 4:02 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Time: Dec. 18, 2023, 4:02 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Time: Dec. 18, 2023, 4:01 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Time: Dec. 18, 2023, 4:01 p.m.

Status: Investigating

Update: Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Time: Dec. 18, 2023, 3:04 p.m.

Status: Investigating

Update: We are investigating reports of intermittent errors for <SOME/ALL> Atlassian, Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details once we identify the root cause.

Time: Dec. 18, 2023, 3:04 p.m.

Status: Investigating

Update: We are investigating reports of intermittent errors for <SOME/ALL> Atlassian, Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details once we identify the root cause.

Time: Dec. 4, 2023, 5:15 p.m.

Status: Resolved

Update: The systems are stable after the fix and monitoring for a specified duration

Time: Dec. 4, 2023, 5:15 p.m.

Status: Resolved

Update: The systems are stable after the fix and monitoring for a specified duration

Time: Dec. 4, 2023, 4:59 p.m.

Status: Monitoring

Update: The issue was identified and a fix implemented. We are monitoring currently.

Time: Dec. 4, 2023, 4:59 p.m.

Status: Monitoring

Update: The issue was identified and a fix implemented. We are monitoring currently.

Time: Dec. 4, 2023, 4:54 p.m.

Status: Identified

Update: We are currently investigating an incident that result in outbound connections from Atlassian cloud in us-east-1 intermittently timing out. This affects Jira, Trello, Confluence, Ecosystem products. The features affected for these products are those that require opening a connection from Atlassian Cloud to public endpoints on the Internet

Time: Dec. 4, 2023, 4:54 p.m.

Status: Identified

Update: We are currently investigating an incident that result in outbound connections from Atlassian cloud in us-east-1 intermittently timing out. This affects Jira, Trello, Confluence, Ecosystem products. The features affected for these products are those that require opening a connection from Atlassian Cloud to public endpoints on the Internet

Time: Dec. 4, 2023, 4:43 p.m.

Status: Identified

Update: Including Atlassian Developer

Time: Dec. 4, 2023, 4:43 p.m.

Status: Identified

Update: Including Atlassian Developer

Time: Dec. 4, 2023, 4:33 p.m.

Status: Identified

Update: We are currently investigating an incident that result in connection time outs on service egress proxy. This affects Jira, JSM, Confluence, BitBucket, Trello, Ecosystem products. The features affected for these products are those that require a connection to service egress.

Time: Dec. 4, 2023, 4:33 p.m.

Status: Identified

Update: We are currently investigating an incident that result in connection time outs on service egress proxy. This affects Jira, JSM, Confluence, BitBucket, Trello, Ecosystem products. The features affected for these products are those that require a connection to service egress.

Time: Dec. 5, 2023, 6:33 p.m.

Status: Resolved

Update: We have not observed any further complaints of this issue and believe the matter is resolved.

Time: Dec. 4, 2023, 6:07 a.m.

Status: Monitoring

Update: We have identified some delays in the delivery of emails which should be resolved now. Customers who retried are expected to receive the exports. We are currently monitoring the situation.

Time: Dec. 1, 2023, 9:11 p.m.

Status: Investigating

Update: Some end users have confirmed that they are now able to receive the email to download the export file. The issue might still persist for a small number of customers and we will keep monitoring and investigating further. However, there are indications that the frequency for the issue has been reduced. We will provide further updates early next week.

Time: Dec. 1, 2023, 2:47 a.m.

Status: Investigating

Update: Some end users confirmed that they are now able to receive the email to download the export file. The issue still persists for some customers and we are working with them to identify and investigate this issue further.

Time: Dec. 1, 2023, 1:24 a.m.

Status: Investigating

Update: We are investigating cases where some customers aren't receiving email notification to download the file when exporting users from User Management. We will provide more details within the next hour.

Time: Dec. 5, 2023, 6:33 p.m.

Status: Resolved

Update: We have not observed any further complaints of this issue and believe the matter is resolved.

Time: Dec. 4, 2023, 6:07 a.m.

Status: Monitoring

Update: We have identified some delays in the delivery of emails which should be resolved now. Customers who retried are expected to receive the exports. We are currently monitoring the situation.

Time: Dec. 1, 2023, 9:11 p.m.

Status: Investigating

Update: Some end users have confirmed that they are now able to receive the email to download the export file. The issue might still persist for a small number of customers and we will keep monitoring and investigating further. However, there are indications that the frequency for the issue has been reduced. We will provide further updates early next week.

Time: Dec. 1, 2023, 2:47 a.m.

Status: Investigating

Update: Some end users confirmed that they are now able to receive the email to download the export file. The issue still persists for some customers and we are working with them to identify and investigate this issue further.

Time: Dec. 1, 2023, 1:24 a.m.

Status: Investigating

Update: We are investigating cases where some customers aren't receiving email notification to download the file when exporting users from User Management. We will provide more details within the next hour.

Time: Nov. 29, 2023, 5:59 a.m.

Status: Resolved

Update: Forge Invocations had an 8 minute outage between 2023-11-29 03:05:13 UTC to 2023-11-29 03:13:27 UTC resulting in Smart Links failing. This service has recovered post this time period.