Time: Nov. 29, 2023, 5:59 a.m.

Status: Resolved

Update: Forge Invocations had an 8 minute outage between 2023-11-29 03:05:13 UTC to 2023-11-29 03:13:27 UTC resulting in Smart Links failing. This service has recovered post this time period.

Time: Nov. 3, 2023, 5:06 a.m.

Status: Resolved

Update: Between 23:25 UTC and 23:46 UTC on 2nd November, we encountered a problem where automation rules in Jira that send out emails failed for certain customers. This was caused by spam-protection limits being incorrectly applied to the legitimate sending of emails. The issue has been resolved and the service is operating normally.

Time: Nov. 3, 2023, 5:06 a.m.

Status: Resolved

Update: Between 23:25 UTC and 23:46 UTC on 2nd November, we encountered a problem where automation rules in Jira that send out emails failed for certain customers. This was caused by spam-protection limits being incorrectly applied to the legitimate sending of emails. The issue has been resolved and the service is operating normally.

Time: Nov. 2, 2023, 6:05 a.m.

Status: Postmortem

Update: ### **SUMMARY** From 9pm UTC on October 23, 2023, until 8pm UTC on October 24, 2023, Atlassian customers using the Jira family products that have configured notifications from custom email domains experienced a degradation as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of a custom domain. The incident was detected by Atlassian customers as they started to receive notifications from the [atlassian.net](http://atlassian.net) domain instead of previously configured custom domains. It was mitigated by excluding redundant validations from the custom domain verification job which put Atlassian systems into a known good state. The total time to resolution was about 23 hours. ### **IMPACT** The overall impact was between October 23, 2023 9pm UTC and October 24 8pm UTC on Jira family products, including Jira Service Management, Jira Software and Jira Work Management. The incident caused service disruption to customers who have notifications from custom email domains enabled for their Jira products as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of the custom domain configured by the customer. ### **ROOT CAUSE** The issue was caused by a change to the service responsible for custom domain email notifications lifecycle that inadvertently enabled SPF record validation as part of a daily validation job. As a result, previously working custom domain configuration transitioned to an unverified state for some Atlassian customers, resulting in notifications from Jira products being sent from [atlassian.net](http://atlassian.net) domain instead of a previously configured custom domain. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Improve monitoring and alerting for custom domains lifecycle to detect such regressions before they reach production. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Time: Oct. 26, 2023, 4:37 a.m.

Status: Resolved

Update: Between 10-23-2023 21:00 UTC and 10-26-2023 03:00 UTC, some customers received emails stating that the domain verification was failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. The issue has been resolved and the service is operating normally. Additional details about the issue will be shared in our Post Incident Review

Time: Oct. 25, 2023, 6:02 a.m.

Status: Monitoring

Update: The fix for the issue has been rolled out and is expected to be complete by 3 am UTC, 26 October. In the meantime, we have refreshed all affected domains and the service is expected to be working normally. We will continue to monitor this issue for 24 hours to ensure it has been resolved and will share an update once confirmed.

Time: Oct. 25, 2023, 1:46 a.m.

Status: Monitoring

Update: We have identified the root cause of the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. A fix is currently being implemented, and we expect this issue to be resolved shortly.

Time: Oct. 25, 2023, 12:37 a.m.

Status: Identified

Update: We continue to work on resolving the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. We have identified the root cause and expect recovery shortly.

Time: Oct. 24, 2023, 11:27 p.m.

Status: Investigating

Update: We are investigating an issue with emails that have been sent out to admins stating domain verification is failing. Affected users may see emails sent from Atlassian's default domain instead of the custom domain they have configured We will provide more details within the next hour.

Time: Nov. 2, 2023, 6:05 a.m.

Status: Postmortem

Update: ### **SUMMARY** From 9pm UTC on October 23, 2023, until 8pm UTC on October 24, 2023, Atlassian customers using the Jira family products that have configured notifications from custom email domains experienced a degradation as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of a custom domain. The incident was detected by Atlassian customers as they started to receive notifications from the [atlassian.net](http://atlassian.net) domain instead of previously configured custom domains. It was mitigated by excluding redundant validations from the custom domain verification job which put Atlassian systems into a known good state. The total time to resolution was about 23 hours. ### **IMPACT** The overall impact was between October 23, 2023 9pm UTC and October 24 8pm UTC on Jira family products, including Jira Service Management, Jira Software and Jira Work Management. The incident caused service disruption to customers who have notifications from custom email domains enabled for their Jira products as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of the custom domain configured by the customer. ### **ROOT CAUSE** The issue was caused by a change to the service responsible for custom domain email notifications lifecycle that inadvertently enabled SPF record validation as part of a daily validation job. As a result, previously working custom domain configuration transitioned to an unverified state for some Atlassian customers, resulting in notifications from Jira products being sent from [atlassian.net](http://atlassian.net) domain instead of a previously configured custom domain. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Improve monitoring and alerting for custom domains lifecycle to detect such regressions before they reach production. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support

Time: Oct. 26, 2023, 4:37 a.m.

Status: Resolved

Update: Between 10-23-2023 21:00 UTC and 10-26-2023 03:00 UTC, some customers received emails stating that the domain verification was failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. The issue has been resolved and the service is operating normally. Additional details about the issue will be shared in our Post Incident Review

Time: Oct. 25, 2023, 6:02 a.m.

Status: Monitoring

Update: The fix for the issue has been rolled out and is expected to be complete by 3 am UTC, 26 October. In the meantime, we have refreshed all affected domains and the service is expected to be working normally. We will continue to monitor this issue for 24 hours to ensure it has been resolved and will share an update once confirmed.

Time: Oct. 25, 2023, 1:46 a.m.

Status: Monitoring

Update: We have identified the root cause of the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. A fix is currently being implemented, and we expect this issue to be resolved shortly.

Time: Oct. 25, 2023, 12:37 a.m.

Status: Identified

Update: We continue to work on resolving the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. We have identified the root cause and expect recovery shortly.

Time: Oct. 24, 2023, 11:27 p.m.

Status: Investigating

Update: We are investigating an issue with emails that have been sent out to admins stating domain verification is failing. Affected users may see emails sent from Atlassian's default domain instead of the custom domain they have configured We will provide more details within the next hour.