Time: Nov. 3, 2022, 1:53 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 1, 2022, 5:21 p.m.

Status: Monitoring

Update: OpenSSL's pre-announcements of CVE-2022-3602 described this issue as CRITICAL but has since been downgraded to HIGH [0]. Aptible remains unaffected by this vulnerability. We still recommend every Aptible customer check the OpenSSL versions used in their apps to confirm they're unaffected. Please follow the aforementioned steps to check the version and update OpenSSL accordingly. Additional Context & Guidance from OpenSSL: https://www.openssl.org/blog/blog/2022/11/01/email-address-overflows/ [0] https://www.openssl.org/news/secadv/20221101.txt

Time: Oct. 31, 2022, 8:21 p.m.

Status: Monitoring

Update: OpenSSL has announced a critical vulnerability [0] for which a patch will be released tomorrow, November 1, 2022 between 13:00 and 17:00 UTC. The nature of the vulnerability has not been disclosed, but based on how it's being handled, Aptible expects it could be a serious vulnerability affecting data confidentiality for those running affected OpenSSL versions (>= 3.0.0, < 3.0.7). Aptible has reviewed all infrastructure components that we manage and have confirmed that all are unaffected by this vulnerability. These components include: - Our Managed TLS endpoints - The TLS endpoints for our REST API services (Auth and Deploy APIs) - All versions of our managed databases - Our log forwarding infrastructure - Our metrics collection infrastructure - Our SSH and Git server infrastructure Still, every Aptible customer should check the OpenSSL versions used in their apps to confirm they're unaffected. To do so, run: $ aptible ssh --app $APP_HANDLE openssl version If the version is >= 3.0.0, you should plan to upgrade your apps' Docker image(s) tomorrow as soon as OpenSSL 3.0.7 is released. We will continue to update this incident page as more information is revealed about the vulnerability. If the vulnerability is only exploitable for *server-side* OpenSSL functionality, the impact to Aptible customers would be significantly reduced. Only those customers who use plain TCP endpoints [1] with their own OpenSSL for TLS termination would be affected in this scenario. [0] https://mta.openssl.org/pipermail/openssl-announce/2022-October/000238.html [1] https://deploy-docs.aptible.com/docs/tcp-endpoints

Time: Oct. 17, 2022, 9:37 p.m.

Status: Resolved

Update: With an acknowledgement from AWS (copied below) we are marking this incident as resolved. > Between 12:20 AM and 11:28 AM PDT, [on October 15th] we experienced intermittent failures in Route53 Health Checks impacting Target Health evaluation in US-EAST-1. The issue has been resolved and the service is operating normally.

Time: Oct. 15, 2022, 6:32 p.m.

Status: Monitoring

Update: The issue should be resolved for all endpoints that have been affected. We've also updated the behavior of the platform so that if you see this issue, running `aptible restart` on the affected app will update the configuration of all the app's endpoints to ignore health checks, and should thereby resolve the issue.

Time: Oct. 15, 2022, 5:31 p.m.

Status: Monitoring

Update: All impacted endpoints that we have identified have been fixed by disabling their health checks. If you have an Endpoint that seems to be impacted, please email [email protected] and include the domain name of the Endpoint in question.

Time: Oct. 15, 2022, 3:42 p.m.

Status: Identified

Update: This is a continuation of https://status.aptible.com/incidents/b3xvn9tmzfjz It has been noticed that we can only detect the issue when HTTP requests are made to the Endpoint, and as the incident started in the middle of a weekend night, we were not able to identify Endpoints which were not in use. We're reviewing and remediating additional Endpoints now, and will make an update to the platform to remove health checks entirely if AWS cannot fix the Route53 issue.

Time: Oct. 15, 2022, 9:18 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Oct. 15, 2022, 8:26 a.m.

Status: Monitoring

Update: We are disabling health checks on the impacted Endpoints in order to bring those applications back online.

Time: Oct. 15, 2022, 8:18 a.m.

Status: Identified

Update: We have identified an AWS issue with Route 53 health checks where some records are failing over to Brickwall, our error server, despite the containers passing health checks. As far as we can tell, this is affecting a small number of Endpoints, unfortunately including dashboard.aptible.com, as well.

Time: Sept. 17, 2022, 8:36 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Sept. 17, 2022, 8:31 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Sept. 17, 2022, 8:20 p.m.

Status: Identified

Update: We are investigating an EC2 dedicated host failure affecting a small number of apps and databases. Affected apps are currently being restarted on healthy instances (apps scaled to 2 or more are automatically distributed across availability zones and will automatically failover).

Time: July 28, 2022, 6:46 p.m.

Status: Resolved

Update: Everything appears to be operational at this time. Please open a ticket with support if you continue to experience issues in us-east-2.

Time: July 28, 2022, 6:24 p.m.

Status: Monitoring

Update: It looks like AWS has recovered most services, and we are continuing to monitor operations in us-east-2 to ensure everything is working properly.

Time: July 28, 2022, 5:42 p.m.

Status: Identified

Update: AWS experienced a power failure in a single availability zone the us-east-2 Region. This affected networking in that AZ, and has also impacted load balancer registration times. We are working to ensure any resources we can identify as impact are moved to another AZ, but the impact to load balancer registration, and an issue creating/updating/deleting Route 53 records are both impacting our ability to mitigate availability issues.