-
Time: July 28, 2022, 8:24 p.m.
Status: Resolved
Update: After further monitoring, no further issues appear to be happening within the us-east-2 Ohio region. Everything has been resolved at this time.
-
Time: July 28, 2022, 7:18 p.m.
Status: Monitoring
Update: We've gone ahead and confirmed that all servers appear to be operational from within the us-east-2 Ohio region at this time. We will continue to monitor the region to ensure no other issues remain.
-
Time: July 28, 2022, 6:47 p.m.
Status: Identified
Update: At this time, most of the AWS issues that appeared to be the root cause of the issues has been cleared. We are still continuing to work on servers within the region that may still be unavailable to ensure they recover fully.
-
Time: July 28, 2022, 6:31 p.m.
Status: Identified
Update: Latest update from AWS:
Instance Impairments
11:25 AM PDT We continue to make progress in recovering the remaining EC2 instances and EBS volumes affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. The vast majority of EC2 instances are now healthy, but we continue to work on recovering the remaining EBS volumes affected by the issue. EC2 API error rates and latencies have returned to normal levels. Elastic Load Balancing remains weighted away from the affected Availability Zone. Error rates and latencies for Lambda function invocations have now returned to normal levels. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 30 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
-
Time: July 28, 2022, 6:31 p.m.
Status: Identified
Update: Latest update from AWS:
11:25 AM PDT We continue to make progress in recovering the remaining EC2 instances and EBS volumes affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. The vast majority of EC2 instances are now healthy, but we continue to work on recovering the remaining EBS volumes affected by the issue. EC2 API error rates and latencies have returned to normal levels. Elastic Load Balancing remains weighted away from the affected Availability Zone. Error rates and latencies for Lambda function invocations have now returned to normal levels. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 30 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
-
Time: July 28, 2022, 6:08 p.m.
Status: Identified
Update: Further update from AWS:
10:49 AM PDT We continue to see recovery of EC2 instances that were affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. At this stage, the vast majority of affected EC2 instances and EBS volumes have returned to a healthy state and we continue to work on the remaining EC2 instances and EBS volumes. Elastic Load Balancing has shifted traffic away from the affected Availability Zone. Single-AZ RDS databases were also affected and will recover as the underlying EC2 instance recovers. Multi-AZ RDS databases would have mitigated impact by failing away from the affected Availability Zone. While the vast majority of Lambda functions continue operating normally, some functions are experiencing invocation failures and latencies, but we expect this to improve over the next 30 minutes. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 45 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
-
Time: July 28, 2022, 5:31 p.m.
Status: Identified
Update: Our team is working on failover options while AWS works to resolve this issue.
Latest update from AWS:
Instance Impairments
10:25 AM PDT We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
-
Time: July 28, 2022, 5:12 p.m.
Status: Investigating
Update: We're currently investigating an issue with the AWS us-east-2 (Ohio) region causing an outage for servers within that region.
