Last checked: 2 minutes ago
Get notified about any outages, downtime or incidents for Atlassian Bitbucket and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Atlassian Bitbucket.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| API | Active |
| Authentication and user management | Active |
| Email delivery | Active |
| Git LFS | Active |
| Git via HTTPS | Active |
| Git via SSH | Active |
| Pipelines | Active |
| Purchasing & Licensing | Active |
| Signup | Active |
| Source downloads | Active |
| Webhooks | Active |
| Website | Active |
View the latest incidents for Atlassian Bitbucket and check for official updates:
Description: ### **SUMMARY** On July 25, 2023, between 16:29 UTC and 16:44 UTC, Atlassian customers were unable to access the Bitbucket Cloud website. This was caused by a code bug that made it to our production environment. The incident was detected immediately by our internal monitoring and fully resolved within 15 minutes by reverting the change. ### **IMPACT** Customers who were impacted were unable to access the [Bitbucket | Git solution for teams using Jira](http://bitbucket.org/) website during the duration of the incident. APIs, Bitbucket Pipelines, and Git over HTTPS and SSH were fully functional. ### **ROOT CAUSE** The incident was caused by a bug that made it into our production environment. This bug affected our ability to communicate with our caches and prevented some of our internal services from starting up to serve traffic to our website. This bug has been fixed and deployed to production. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. We are prioritizing the following improvement actions to avoid repeating these types of incidents in the future and reduce recovery time: * Improve testing in non-production environments * Enhance monitoring around service deployments We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s availability. Thanks, Atlassian Customer Support
Status: Postmortem
Impact: Major | Started At: July 25, 2023, 4:47 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Major | Started At: July 18, 2023, 5:37 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Major | Started At: July 18, 2023, 5:37 p.m.
Description: ### **SUMMARY** On JUL 13, 2023, between 13:48 and 16:05 UTC, Atlassian customers using Bitbucket Cloud were experiencing degraded performance for git operations. The event was triggered by a bug that was deployed to production. The changes included the introduction of a bug that bypassed a critical cache for git operations which impacted all Bitbucket Cloud customers. The incident was detected within 10 minutes by automated monitoring and mitigated by rolling back the code changes and the redeployment of some services which put Atlassian systems into a known good state. The total time to resolution was about 2 hours & 17 minutes. ### **IMPACT** The overall impact was between JUL 13, 2023, 13:48 UTC and JUL 13, 2023, 16:05 UTC on Bitbucket Cloud products. The Incident caused service disruption to all customers where they experienced slow response times or failures when interacting with repository data. As a result of network saturation causing connections to queue, customers experienced increased latency and error rates across Bitbucket Cloud services. ### **ROOT CAUSE** The issue was caused by a change to a feature flag that contained a bug which bypassed a critical cache. As a result, more requests were directly accessing the disks which led to increased latency and eventually, degraded the performance of some operations. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because the impact was subtle and took approximately 48 hours to surface after initial deployment. This slow ramping was not picked up by our automated continuous test scripts because it requires significant load to reach the tipping point where systems start to become degraded. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Alerting on cache hit/miss rates * Improving the monitoring of network saturation for connections to our disks We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Status: Postmortem
Impact: Minor | Started At: July 13, 2023, 2:33 p.m.
Description: ### **SUMMARY** On July 13, 2023 between 4:04 AM UTC and 6:42 AM UTC, Atlassian customers using Bitbucket Cloud were unable to retrieve a list of branches. The event was triggered by a change in the rate limiting for that endpoint, causing them to be applied globally. The incident was detected within 5 minutes by automated monitoring and mitigated by reverting the change responsible which put Atlassian systems into a known good state. The total time to resolution was about 2 hours & 38 minutes. ### **IMPACT** Everywhere in Bitbucket Cloud that retrieved a list of repository branches was affected which includes the pull request creation pages and API as well as pipelines builds. The total time that customers experienced this impact was approximately 2.5 hours. ### **ROOT CAUSE** The issue was caused by a change to the repository branches list endpoint. As a result, users making calls to retrieve their list of branches received HTTP 429 errors. More specifically a change to the branches list endpoint caused the endpoint to incorrectly apply the repository level rate limit globally. The root cause of the incident was the failure in the detection of the bug by the deployment validations. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because the change was related to a very specific kind of legacy case that was not picked up by our automated continuous deployment suites and manual test scripts. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Prevent rate limiting from being able to be applied globally * Remove these rate limits from generic rules To minimize the impact of breaking changes to our environments, we will implement additional preventative measures such as: * Standardized way of disabling rate limiting to decrease time to resolution * Improved testing We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Status: Postmortem
Impact: Minor | Started At: July 13, 2023, 4:22 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
