Aptible outages and incidents
Outage and incident data over the last 30 days for Aptible.
There have been 0 outages or incidents for Aptible in the last 30 days.
Severity Breakdown:
Tired of searching for status updates?
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowComponents and Services Monitored for Aptible
Outlogger tracks the status of these components for Xero:
AWS EC2
| Component | Status |
|---|---|
| api.aptible.com | Active |
| Aptible Deploy | Active |
| auth.aptible.com | Active |
| dashboard.aptible.com | Active |
| Let's Encrypt | Active |
| AWS EC2 | Active |
| AWS EC2 (ap-northeast-1 — Tokyo) | Active |
| AWS EC2 (ap-northeast-2 — Seoul) | Active |
| AWS EC2 (ap-south-1 — Mumbai) | Active |
| AWS EC2 (ap-southeast-1 — Singapore) | Active |
| AWS EC2 (ap-southeast-2 — Sydney) | Active |
| AWS EC2 (ca-central-1 — Canada) | Active |
| AWS EC2 (eu-central-1 — Frankfurt) | Active |
| AWS EC2 (eu-west-1 — Ireland) | Active |
| AWS EC2 (eu-west-2 — London) | Active |
| AWS EC2 (eu-west-3 — Paris) | Active |
| AWS EC2 (sa-east-1 — São Paulo) | Active |
| AWS EC2 (us-east-1 — Virginia) | Active |
| AWS EC2 (us-east-2 — Ohio) | Active |
| AWS EC2 (us-west-1 — California) | Active |
| AWS EC2 (us-west-2 — Oregon) | Active |
Latest Aptible outages and incidents.
View the latest incidents for Aptible and check for official updates:
Host Provisioning Delays in us-east-1
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: Sept. 28, 2023, 11:31 p.m.
Updates:
-
Time: Sept. 29, 2023, 4:09 a.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: Sept. 29, 2023, 3:16 a.m.Status: MonitoringUpdate: We are again seeing successful deployment of new hosts in the affected single availability zone in the us-east-1 region. We will continue to monitor for an additional period before resolving the incident.
-
Time: Sept. 29, 2023, 1:12 a.m.Status: IdentifiedUpdate: AWS continues to work on recovering this issue in a single availability zone in the us-east-1 region. Running apps and databases continue not to be impacted by this failure.
-
Time: Sept. 28, 2023, 11:31 p.m.Status: InvestigatingUpdate: AWS is experiencing an issue preventing the timely deployment of new hosts in a single availability zone in the us-east-1 region. As a result, some app and database restart, scale, and deployment operations that result in a new host being provisioned may fail and roll back. Running apps and databases are not impacted by this failure.
EC2 Host Failure - us-east-1
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: July 29, 2023, 3:02 p.m.
Updates:
-
Time: July 29, 2023, 3:14 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: July 29, 2023, 3:02 p.m.Status: InvestigatingUpdate: We are investigating several EC2 dedicated host failures affecting some customers with apps and databases in us-east-1, related to an AWS incident. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).
Delayed Operations
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: July 20, 2023, 6:50 p.m.
Updates:
-
Time: July 20, 2023, 8:11 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: July 20, 2023, 7:34 p.m.Status: MonitoringUpdate: Our team has mitigated this issue, and newly created operations should now succeed. Customers may see long-delayed operations begin to fail. These failed operations will need to be restarted.
-
Time: July 20, 2023, 7:04 p.m.Status: IdentifiedUpdate: Our team has determined the root cause as an internal dependency causing operations to hang. We're currently beginning steps to remediate this issue—more updates to follow.
-
Time: July 20, 2023, 6:50 p.m.Status: InvestigatingUpdate: We are currently experiencing issues with operations being delayed. Our Engineering team is currently investigating—more updates to follow.
Metric Drains Interrupted for Some Dedicated Stacks
Description: # Incident Postmortem: Metric Drains Interrupted for Some Dedicated Stacks ## Executive Summary On June 20, 2023, our platform experienced a service degradation incident for Metric Drains while rolling out a new feature for Metric Drains. This was due to unexpected side effects of a new internal utility used to deploy the feature. Some of our customers experienced interruptions in their metric drains during this incident. All issues were subsequently addressed, and service has been fully restored. ## Detailed Incident Description Configuration Change Initiation: The rollout of the change relied on a two-step configuration process to update the software for the metric drain emitter and aggregator components within each dedicated stack. This process was initiated using a new utility that had been successfully deployed in the past but not at the scale required for this rollout. Utility Timeouts and Delays: During the rollout, the configuration utility started experiencing cascading timeouts as operations queued with increasing delays in executing the configuration changes. During this period of delay in having configuration uniformly updated for the rollout, this caused some customer stacks to be only partially configured for the updated metric drain software. Customer Impact: A small number of customers who were deploying or scaling services during this period had their metric drains interrupted due to the aforementioned configuration issues. Resolution: Our team immediately worked on fixing the configuration issues. By 16:24 EDT, we successfully restored the configuration state for the affected customers, and the service was resumed to its regular state. Follow-up Audit: On the following morning of June 21, a follow-up audit revealed that two additional customers still needed configuration updates for their metric drains. We immediately addressed these issues. ## Root Cause Analysis The root cause of this issue was a combination of the increased scale of the rollout and the relative novelty of the utility used for the configuration changes. Although this utility had performed successfully under previous workloads, it did not sufficiently scale to handle the increased demand of this particular rollout. ## Lessons Learned and Preventative Measures Testing Deployment Tools at Scale: testing new deployment tools and utilities under maximum practical loads is crucial to ensure they can handle expected full-scope workloads without disruption. Audit Processes: Though our follow-up audit process effectively identified additional affected customers, we will make such audits more timely to catch any lingering issues sooner. We sincerely apologize for any inconvenience caused to our customers during this incident. We take this issue seriously and are committed to ensuring that such incidents do not occur in the future.
Status: Postmortem
Impact: None | Started At: June 21, 2023, 3:28 p.m.
Updates:
-
Time: July 11, 2023, 4:08 p.m.Status: PostmortemUpdate: # Incident Postmortem: Metric Drains Interrupted for Some Dedicated Stacks ## Executive Summary On June 20, 2023, our platform experienced a service degradation incident for Metric Drains while rolling out a new feature for Metric Drains. This was due to unexpected side effects of a new internal utility used to deploy the feature. Some of our customers experienced interruptions in their metric drains during this incident. All issues were subsequently addressed, and service has been fully restored. ## Detailed Incident Description Configuration Change Initiation: The rollout of the change relied on a two-step configuration process to update the software for the metric drain emitter and aggregator components within each dedicated stack. This process was initiated using a new utility that had been successfully deployed in the past but not at the scale required for this rollout. Utility Timeouts and Delays: During the rollout, the configuration utility started experiencing cascading timeouts as operations queued with increasing delays in executing the configuration changes. During this period of delay in having configuration uniformly updated for the rollout, this caused some customer stacks to be only partially configured for the updated metric drain software. Customer Impact: A small number of customers who were deploying or scaling services during this period had their metric drains interrupted due to the aforementioned configuration issues. Resolution: Our team immediately worked on fixing the configuration issues. By 16:24 EDT, we successfully restored the configuration state for the affected customers, and the service was resumed to its regular state. Follow-up Audit: On the following morning of June 21, a follow-up audit revealed that two additional customers still needed configuration updates for their metric drains. We immediately addressed these issues. ## Root Cause Analysis The root cause of this issue was a combination of the increased scale of the rollout and the relative novelty of the utility used for the configuration changes. Although this utility had performed successfully under previous workloads, it did not sufficiently scale to handle the increased demand of this particular rollout. ## Lessons Learned and Preventative Measures Testing Deployment Tools at Scale: testing new deployment tools and utilities under maximum practical loads is crucial to ensure they can handle expected full-scope workloads without disruption. Audit Processes: Though our follow-up audit process effectively identified additional affected customers, we will make such audits more timely to catch any lingering issues sooner. We sincerely apologize for any inconvenience caused to our customers during this incident. We take this issue seriously and are committed to ensuring that such incidents do not occur in the future.
-
Time: June 21, 2023, 7:34 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: June 21, 2023, 3:57 p.m.Status: MonitoringUpdate: A fix has been implemented and we are monitoring the results.
-
Time: June 21, 2023, 3:28 p.m.Status: IdentifiedUpdate: Related to the 6/20 incident, some customers on Dedicated Stacks are seeing a continued disruption to their Metric Drains. We are implementing a fix for those customers/stacks and will follow up via support tickets to affected customers.
Metric Drains Interrupted for Some Dedicated Stacks
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: June 20, 2023, 8 p.m.
Updates:
-
Time: June 20, 2023, 8:24 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: June 20, 2023, 8 p.m.Status: IdentifiedUpdate: During a rollout of an updated version of Metric Drains in the past hour, some customers on dedicated stacks experienced an interruption in service for Metric Drains. The issue has been identified and is being addressed, with the issue expected to be resolved for all affected customers within the following 10 minutes.
