Time: Nov. 3, 2022, 5:25 a.m.

Status: Postmortem

Update: The source of the outage was related to the maintenance to repair one of our network switches, which was also the source of the brief network issue experienced Tuesday morning. This minor maintenance was conducted within one of our SLA-provided maintenance windows. Our Operations Team is working with the hardware vendor to identify the root cause, along with a corrective action plan.

Time: Nov. 3, 2022, 5:05 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 3, 2022, 4:38 a.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Nov. 3, 2022, 4:24 a.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Nov. 3, 2022, 1:44 a.m.

Status: Investigating

Update: We are currently investigating this issue.

Time: Nov. 1, 2022, 4:43 p.m.

Status: Resolved

Update: Our production client-facing network experienced a hardware failure within one of our network devices. This event resulted in a brief loss in network connectivity. Network traffic automatically moved to a redundant network port. The Hyphen technology team is working with our network vendor to replace the failed hardware component. Update: We have been monitoring and are re-processing any integration messages that may have been impacted by this incident.

Time: Nov. 1, 2022, 4:43 p.m.

Status: Resolved

Update: Our production client-facing network experienced a hardware failure within one of our network devices. This event resulted in a brief loss in network connectivity. Network traffic automatically moved to a redundant network port. The Hyphen technology team is working with our network vendor to replace the failed hardware component. Update: We have been monitoring and are re-processing any integration messages that may have been impacted by this incident.

Time: Sept. 28, 2022, 9:19 p.m.

Status: Resolved

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 2:05 PM PDT As of 1:43 PM PDT, error rates and latencies for invoked on API Gateway endpoints in the US-WEST-2 Region are now at normal levels. The issue began at 9:20 AM PDT when error rates and latencies for API Gateway began to increase. Error rates began to improve at 10:38 AM PDT, when engineers took action to reduce contention within the subsystem that handles request processing for API Gateway. Error rates continued to improve until 1:10 PM PDT, when engineers applied a mitigation to resolve the contention within the affected subsystem. These actions accelerated recovery, and by 1:43 PM PDT, error rates and latencies had returned to normal levels. Affected AWS services have now recovered as well. The issue has been resolved and the service is operating normally.

Time: Sept. 28, 2022, 9:06 p.m.

Status: Monitoring

Update: 1:42 PM PDT Starting at 1:31 PM PDT, error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region are now close to pre-event levels, and we continue to work on the remaining hosts that are affected by the contention issue. Several AWS services, including AWS Connect and Lambda are seeing signs of strong recovery. We expect all services to recover as API Gateway error rates and latencies return to normal levels. Customers should be seeing recovery at these error levels as well. We will continue to provide updates until the error rates and latencies have returned to normal levels.

Time: Sept. 28, 2022, 8:52 p.m.

Status: Identified

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 1:42 PM PDT Starting at 1:31 PM PDT, error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region are now close to pre-event levels, and we continue to work on the remaining hosts that are affected by the contention issue. Several AWS services, including AWS Connect and Lambda are seeing signs of strong recovery. We expect all services to recover as API Gateway error rates and latencies return to normal levels. Customers should be seeing recovery at these error levels as well. We will continue to provide updates until the error rates and latencies have returned to normal levels.

Time: Sept. 28, 2022, 8:39 p.m.

Status: Identified

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 1:22 PM PDT Starting at 1:12 PM PDT, we saw a further reduction in error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. This was a result of the latest mitigation which addressed contention within the component in the subsystem responsible for request processing within API Gateway. Error rates are now at levels where some customers may begin to see recovery, and retries will begin to work more consistently. We will be applying the mitigation to the remaining hosts affected by the contention issue and expect to see further recovery from them in the next 30 minutes. 1:03 PM PDT Error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region, continue to hold steady. Engineers continue to work on resolving the contention affecting the subsystem responsible for request processing. We recently completed a mitigation that should help to reduce error rates and latencies to normal levels and will have further updates on the result of that change in the next update. Although Lambda function invocations are not affected by this issue, the Lambda console is experiencing some error rates which we are investigating. Other AWS services affected by this issue remain in much the same state, waiting on the recovery of API Gateway.

Time: Sept. 28, 2022, 7:38 p.m.

Status: Identified

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 12:26 PM PDT We continue to see an improvement in error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region, but have not fully resolved the issue. While our mitigations have improved error rates and latencies, we have also identified the root cause of the event. The subsystem responsible for request processing experienced increased load, which ultimately led to contention of a component within the affected subsystem. Engineers have been working to resolve the contention of the affected component, which has led to a reduction of error rates and latencies. The path to full recovery involves addressing the contention across the subsystem, which we are currently doing. As that progresses over the next two hours, we expect recovery to continue to improve. Customers with applications that use API Gateway will be experiencing elevated error rates and latencies as a result of this issue. Lambda is not affected by this event, but customers using API Gateway as an HTTP endpoint for Lambda will experience increased error rates and latencies. Other AWS services listed below are also experiencing elevated error rates as a result of this issue. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side. We do expect error rates to continue to improve as contention with the affected subsystem resides, and will provide further updates as recovery progresses.

Time: Sept. 28, 2022, 6:58 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: Sept. 28, 2022, 6:58 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: Sept. 28, 2022, 6:48 p.m.

Status: Identified

Update: As per AWS at 11:33 AM PDT We continue to work on resolving the elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. We continue to see a significant improvement in error rates, starting at 10:40 AM PDT, but are not seeing full recovery yet. The issue is caused by contention within the subsystem that is responsible for request processing within the API Gateway service. Engineers are engaged and have applied traffic filters as a precautionary measure, while they work to identify the root cause and resolve the issue. Engineers continue to work to reduce contention within the affected subsystem, which we believe will resolve the elevated error rates and latencies. Customers with applications that use API Gateway, or customers invoking Lambda functions via API Gateway, will be experiencing elevated error rates and latencies as a result of this issue. The AWS services listed below are also experiencing elevated error rates as a result of this issue. While we have seen improvements in error rates since 10:40 AM PDT, recovery has stalled and we do not have a clear ETA on full recovery. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side. We do expect error rates to continue to improve as contention with the affected subsystem resides, and will provide further updates as recovery progresses.

Time: Sept. 28, 2022, 6:17 p.m.

Status: Identified

Update: The issue has been identified on the AWS side of things. As per AWS. 10:59 AM PDT We continue to see elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. While engineers continue to work towards the root cause, we have deployed traffic filters from sources with significant increases in traffic prior to the event. As a result of these traffic filters, we are seeing a reduction in error rates and latencies, but continue to work towards full recovery. Although error rates are improving, we do not yet have an ETA for a full recovery. The issue is also affecting API requests to some AWS services, including those listed below. Amazon Connect is experiencing increased failures in handling new calls, chats, and tasks as well as issues with user login in the US-WEST-2 Region. We will continue to provide updates as we progress. We will continue to monitor and update our own status page. Here is the AWS status page where we are receiving updates: https://health.aws.amazon.com/health/status

Time: Sept. 28, 2022, 9:19 p.m.

Status: Resolved

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 2:05 PM PDT As of 1:43 PM PDT, error rates and latencies for invoked on API Gateway endpoints in the US-WEST-2 Region are now at normal levels. The issue began at 9:20 AM PDT when error rates and latencies for API Gateway began to increase. Error rates began to improve at 10:38 AM PDT, when engineers took action to reduce contention within the subsystem that handles request processing for API Gateway. Error rates continued to improve until 1:10 PM PDT, when engineers applied a mitigation to resolve the contention within the affected subsystem. These actions accelerated recovery, and by 1:43 PM PDT, error rates and latencies had returned to normal levels. Affected AWS services have now recovered as well. The issue has been resolved and the service is operating normally.

Time: Sept. 28, 2022, 9:06 p.m.

Status: Monitoring

Update: 1:42 PM PDT Starting at 1:31 PM PDT, error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region are now close to pre-event levels, and we continue to work on the remaining hosts that are affected by the contention issue. Several AWS services, including AWS Connect and Lambda are seeing signs of strong recovery. We expect all services to recover as API Gateway error rates and latencies return to normal levels. Customers should be seeing recovery at these error levels as well. We will continue to provide updates until the error rates and latencies have returned to normal levels.

Time: Sept. 28, 2022, 8:52 p.m.

Status: Identified

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 1:42 PM PDT Starting at 1:31 PM PDT, error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region are now close to pre-event levels, and we continue to work on the remaining hosts that are affected by the contention issue. Several AWS services, including AWS Connect and Lambda are seeing signs of strong recovery. We expect all services to recover as API Gateway error rates and latencies return to normal levels. Customers should be seeing recovery at these error levels as well. We will continue to provide updates until the error rates and latencies have returned to normal levels.

Time: Sept. 28, 2022, 8:39 p.m.

Status: Identified

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 1:22 PM PDT Starting at 1:12 PM PDT, we saw a further reduction in error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. This was a result of the latest mitigation which addressed contention within the component in the subsystem responsible for request processing within API Gateway. Error rates are now at levels where some customers may begin to see recovery, and retries will begin to work more consistently. We will be applying the mitigation to the remaining hosts affected by the contention issue and expect to see further recovery from them in the next 30 minutes. 1:03 PM PDT Error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region, continue to hold steady. Engineers continue to work on resolving the contention affecting the subsystem responsible for request processing. We recently completed a mitigation that should help to reduce error rates and latencies to normal levels and will have further updates on the result of that change in the next update. Although Lambda function invocations are not affected by this issue, the Lambda console is experiencing some error rates which we are investigating. Other AWS services affected by this issue remain in much the same state, waiting on the recovery of API Gateway.

Time: Sept. 28, 2022, 7:38 p.m.

Status: Identified

Update: (Source: AWS Status - https://health.aws.amazon.com/health/status) Increased Invoke Error Rates 12:26 PM PDT We continue to see an improvement in error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region, but have not fully resolved the issue. While our mitigations have improved error rates and latencies, we have also identified the root cause of the event. The subsystem responsible for request processing experienced increased load, which ultimately led to contention of a component within the affected subsystem. Engineers have been working to resolve the contention of the affected component, which has led to a reduction of error rates and latencies. The path to full recovery involves addressing the contention across the subsystem, which we are currently doing. As that progresses over the next two hours, we expect recovery to continue to improve. Customers with applications that use API Gateway will be experiencing elevated error rates and latencies as a result of this issue. Lambda is not affected by this event, but customers using API Gateway as an HTTP endpoint for Lambda will experience increased error rates and latencies. Other AWS services listed below are also experiencing elevated error rates as a result of this issue. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side. We do expect error rates to continue to improve as contention with the affected subsystem resides, and will provide further updates as recovery progresses.

Time: Sept. 28, 2022, 6:58 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: Sept. 28, 2022, 6:58 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: Sept. 28, 2022, 6:48 p.m.

Status: Identified

Update: As per AWS at 11:33 AM PDT We continue to work on resolving the elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. We continue to see a significant improvement in error rates, starting at 10:40 AM PDT, but are not seeing full recovery yet. The issue is caused by contention within the subsystem that is responsible for request processing within the API Gateway service. Engineers are engaged and have applied traffic filters as a precautionary measure, while they work to identify the root cause and resolve the issue. Engineers continue to work to reduce contention within the affected subsystem, which we believe will resolve the elevated error rates and latencies. Customers with applications that use API Gateway, or customers invoking Lambda functions via API Gateway, will be experiencing elevated error rates and latencies as a result of this issue. The AWS services listed below are also experiencing elevated error rates as a result of this issue. While we have seen improvements in error rates since 10:40 AM PDT, recovery has stalled and we do not have a clear ETA on full recovery. For customers that have dependencies on API Gateway and are experiencing error rates, we do not have any mitigations to recommend to address the issue on the customer side. We do expect error rates to continue to improve as contention with the affected subsystem resides, and will provide further updates as recovery progresses.

Time: Sept. 28, 2022, 6:17 p.m.

Status: Identified

Update: The issue has been identified on the AWS side of things. As per AWS. 10:59 AM PDT We continue to see elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. While engineers continue to work towards the root cause, we have deployed traffic filters from sources with significant increases in traffic prior to the event. As a result of these traffic filters, we are seeing a reduction in error rates and latencies, but continue to work towards full recovery. Although error rates are improving, we do not yet have an ETA for a full recovery. The issue is also affecting API requests to some AWS services, including those listed below. Amazon Connect is experiencing increased failures in handling new calls, chats, and tasks as well as issues with user login in the US-WEST-2 Region. We will continue to provide updates as we progress. We will continue to monitor and update our own status page. Here is the AWS status page where we are receiving updates: https://health.aws.amazon.com/health/status