Time: June 13, 2023, 9:46 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: June 13, 2023, 8:59 p.m.

Status: Monitoring

Update: Backup copying has been re-enabled, and we are continuing to monitor for any other impact.

Time: June 13, 2023, 8:34 p.m.

Status: Monitoring

Update: We are aware of ongoing issues with AWS related to a failure of the Lambda service. Our assessment is that there is no direct impact to the availability or performance of any Aptible hosted services. One impact we are monitoring is that copying Database backups to a second region is not working at this time. When the incident resolves, the copies will be made automatically. If you are utilizing any AWS services directly in your own AWS account, you may monitor the status of those services at https://health.aws.amazon.com/health/status

Time: May 23, 2023, 9:18 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: May 23, 2023, 8:55 p.m.

Status: Identified

Update: Our email provider has reported delays in processing, resulting in delays in sending queued transactional emails sent by our platform. In particular, this means the following workflows might be delayed: - Email verifications - Password resets - Role invitations This does not affect any customer apps (unless you happen to be using the same email provider, of course): only transactional emails sent by Aptible itself are affected.

Time: Jan. 28, 2023, 2:56 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 27, 2023, 7:21 p.m.

Status: Monitoring

Update: At this time, all issues related to long queueing times should be resolved. We are leaving this incident in "monitoring" status until the underlying AWS issue is either resolved or acknowledged by AWS with an explicit resolution ETA.

Time: Jan. 27, 2023, 6:32 p.m.

Status: Monitoring

Update: Operations for resources in us-east-1 are queueing for up to 5 minutes before beginning to execute, as a result of an AWS issue causing certain operations (especially `CopySnapshot` operations) to be extremely delayed. (These operations ordinarily begin to execute within 1 or 2 seconds.) Currently, only resources in us-east-1 are affected. We are making changes to disable some of the affected AWS operations until the underlying AWS issue is resolved. In the meantime, operations may take longer than usual to start, but will eventually begin to execute if no action is taken (and if they are not cancelled).

Time: Dec. 13, 2022, 10:23 p.m.

Status: Resolved

Update: Calls to the EC2 API are responding normally, Operations in ap-southeast-1 have been resumed.

Time: Dec. 13, 2022, 9:54 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Dec. 13, 2022, 9:54 p.m.

Status: Investigating

Update: Due to the unavailability of the AWS EC2 API in the ap-southeast-1 region, Aptible Operations have been blocked in that region.

Time: Nov. 3, 2022, 8:32 p.m.

Status: Postmortem

Update: AWS has identified the root cause of the Endpoint unavailability: > Between 2:26 PM and 3:04 PM PDT\(9:26PM ~ 10:04 PM UTC\) we experienced increased packet loss for traffic destined to public endpoints in the US-EAST-1 Region, which affected Internet and public Direct Connect connectivity for endpoints in the US-EAST-1 Region. This is, unfortunately, essentially the same impact we've seen in two previous incidents, although AWS's description of the cause is slightly different: October 15th 2022: [https://status.aptible.com/incidents/grf6gdrrszf9](https://status.aptible.com/incidents/grf6gdrrszf9) > Between 12:20 AM and 11:28 AM PDT, we experienced intermittent failures in Route53 Health Checks impacting Target Health evaluation in US-EAST-1. The issue has been resolved and the service is operating normally. September 27th, 2021: \(Only a couple of Endpoints were impacted, so no incident was created\) > On September 27, 2021, between 8:45 AM and 2:09 PM PDT, Route53 experienced increased change propagation times for Health Check edits where unexpected failover to their secondary application load balancer \(ALB\) occurred despite their primary ALB targets being healthy. The issue has been resolved and the service is operating normally. While AWS describes these incidents as "increased change propagation times", "intermittent failures", and "increased packet loss", and apparently do not qualify as an incident to be posted to [https://status.aws.amazon.com,](https://status.aws.amazon.com,) the observed impact to our customers is very clear: the impacted Endpoints are totally unreachable for a period. As such, we will permanently implement the "temporary" change we made on October 15th: we will be disabling the Route53 health checks \(and the associated custom error page\) for all Endpoints, as this has been the root cause of these availability incidents. As we indicated to customers during the Oct 15th and Nov 3rd incidents, you may restart any App in order to immediately disable the Route53 health check. Any App which has been deployed, restarted, or scaled since October 15th will already have it disabled, and we will make another announcement when we intend to disable it globally on all Apps for which it remains enabled.

Time: Nov. 2, 2022, 10:28 p.m.

Status: Resolved

Update: We no longer see any impact, and will continue investigating for an RCA.

Time: Nov. 2, 2022, 10:18 p.m.

Status: Monitoring

Update: Based on a random sampling, and reported affected Endpoints, we are no longer seeing any impact. We will continue to monitor the situation.

Time: Nov. 2, 2022, 10:08 p.m.

Status: Identified

Update: We're seeing many Endpoints recover without action being taken, so we're looking into ways to identify Endpoints that remain impacted so that we can efficiently fix them. Restarting known impacted Apps remains the quickest solution that we know of.

Time: Nov. 2, 2022, 9:54 p.m.

Status: Identified

Update: We've observed that running `aptible restart --app $handle` can resolve the underlying issue with the ELB, and recommend restarting any of your impacted Apps at this time.

Time: Nov. 2, 2022, 9:45 p.m.

Status: Investigating

Update: We are currently investigating a large number of unreachable ELBs in AWS's us-east-1 region, and are wait for acknowledgement from AWS and trying to narrow the scope of the failures in order to provide failover/workarounds if possible.