Time: Jan. 26, 2022, 8:31 p.m.

Status: Resolved

Update: We have isolated the problem to cached connections in our caching layer and the issue has been mitigated.

Time: Jan. 26, 2022, 6:32 p.m.

Status: Investigating

Update: We are currently experiencing increase network errors in our infrastructure region. We are working with AWS team to isolate the issue. The impact might be that you are experiencing slower login times into the Demandbase platform.

Time: Dec. 22, 2021, 6:16 p.m.

Status: Resolved

Update: All Demandbase platforms are fully functional now. We will continue working on post-mortem and learning from this incident.

Time: Dec. 22, 2021, 5:13 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Dec. 22, 2021, 5:12 p.m.

Status: Investigating

Update: We have fully recovered from this incident in our Sales Intelligence Cloud and now running sanity checks

Time: Dec. 22, 2021, 4:16 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Dec. 22, 2021, 4:15 p.m.

Status: Investigating

Update: We are closely working with our AWS partner team. Below are the updates - We are investigating increased EC2 launch failures and networking connectivity issues for some instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Other Availability Zones within the US-EAST-1 Region are not affected by this issue. 05:18 AM PST: [NDA update] We continue to make progress in restoring power to the affected data center within the affected Availability Zone (USE1-AZ4) in the US-EAST-1 Region. We have now restored power to the majority of instances and networking devices within the affected data center and are starting to see some early signs of recovery. Customers experiencing connectivity or instance availability issues within the affected Availability Zone, should start to see some recovery as power is restored to the affected data center. RunInstances API error rates are returning to normal levels and we are working to recover affected EC2 instances and EBS volumes. While we would expect continued improvement over the coming hour, we would still recommend failing away from the Availability Zone if you are able to do so to mitigate this issue. 05:39 AM PST [NDA update] We have now restored power to all instances and network devices within the affected data center and are seeing recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone. Network connectivity within the affected Availability Zone has also returned to normal levels. While all services are starting to see meaningful recovery, services which were hosting endpoints within the affected data center - such as single-AZ RDS databases, ElastiCache, etc. - would have seen impact during the event, but are starting to see recovery now. Given the level of recovery, if you have not yet failed away from the affected Availability Zone, you should be starting to see recovery at this stage. 5:55 AM PST [NDA Update] We have now restored power to all instances and network devices within the affected data center and are seeing recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone. We continue to make progress in recovering the remaining EC2 instances and EBS volumes within the affected Availability Zone. If you are able to relaunch affected EC2 instances within the affected Availability Zone, that may help to speed up recovery. We have a small number of affected EBS volumes that are still experiencing degraded IO performance that we are working to recover. Given that some hardware lost power, there is some risk of impaired hardware, which may delay recovery for a very small number of EC2 instances, which we will message to affected customers if that happens.

Time: Dec. 8, 2021, 4:11 p.m.

Status: Resolved

Update: This incident has been resolved. We will be working with AWS on understanding the root cause of yesterday's network issues. All platforms are functional.

Time: Dec. 7, 2021, 11:03 p.m.

Status: Identified

Update: Update: AWS have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. They are seeing improvement in availability across most AWS services. All services are now independently working through service-by-service recovery. AWS will continue to work toward full recovery for all impacted AWS Services and API operations.

Time: Dec. 7, 2021, 10:33 p.m.

Status: Investigating

Update: Update: AWS have executed multiple shifts of internal service traffic to a separate set of network devices. They are now beginning to see significant recovery for AWS service availability in the US-EAST-1 Region. We are continuing to closely monitor the health and to continue identifying the specific traffic flows that were leading to congestion within these devices. This is complex, as the issue is not simply the volume of traffic, but rather the rate of new connections that is causing unexpectedly large load on the impacted devices. We expect to see continued improvement in service availability and are monitoring for full recovery.

Time: Dec. 7, 2021, 9:45 p.m.

Status: Investigating

Update: Update: AWS have identified that an increase in traffic between specific internal services is causing congestion within several network devices in the US-EAST-1 Region. While we continue to work across every service to investigate and mitigate impact, we are focusing our efforts on a specific mitigation for this traffic increase. We are taking action over the next 30-60 minutes that we expect will put us on the path to recovery, however we do not have an ETA for full recovery yet.

Time: Dec. 7, 2021, 8:06 p.m.

Status: Investigating

Update: Update: We are working closely with our infrastructure vendor AWS on resolving the widespread network outage in us-east-1. While we are still seeing incremental recovery, it is not consistent and we do not have an ETA for full recovery. All teams are actively engaged and working on resolving this impact as quickly as possible.

Time: Dec. 7, 2021, 5:27 p.m.

Status: Investigating

Update: We are currently experiencing service degradation as well as partial outage in our infrastructure (us-east-1) and this impacts users of Demandbase One, DB Classic and Sales Cloud platform. Multiple infrastructure services are impacted and we are currently working with our infrastructure partner to get resolution to this outage. We are attempting multiple paths to mitigating this issue, and have all teams engaged. Unfortunately at this time we do not have an ETA to when this issue will be mitigated

Time: Nov. 15, 2021, 1:04 a.m.

Status: Resolved

Update: The issue was resolved via a hotfix that invalidates the cached sessions in the web tier.

Time: Nov. 11, 2021, 8:07 p.m.

Status: Investigating

Update: We are seeing a recurring issue with delay in web page transition today

Time: Nov. 11, 2021, 6:09 p.m.

Status: Resolved

Update: We have isolated the slowness to a caching layer and mitigated the problem and we will be conducting post-mortem on the incident

Time: Nov. 11, 2021, 3:39 p.m.

Status: Investigating

Update: Demandbase One platform's web page transitions are taking longer today morning. The issue is currently under investigation