Time: Dec. 22, 2021, 4:30 p.m.

Status: Resolved

Update: Between 10:34 and 10:37 ET, we rolled out a scaling event to our API to mitigate some of the issues we are seeing with the AWS outage in one of the us-east-1 datacenters we use. This scaling event did not execute correctly due to the ongoing issues, making the situation temporarily worse. We immediately rolled back that change and are continuing to monitor the situation with AWS.

Time: Dec. 22, 2021, 3:36 p.m.

Status: Investigating

Update: Our API integration tests have encountered an increase in errors. We are currently investigating. Stay tuned for updates.

Time: Dec. 18, 2021, 12:45 a.m.

Status: Resolved

Update: Between 18:21 and 18:25 ET this evening, a change was made to our web application firewall (WAF) rules which caused some clients to experience a loss of service. This WAF change was suggested by Amazon to add an extra layer of security to our environment, but instead, we saw it was flagging most users as malicious. Once we saw that this occurred when the WAF rule was turned on, it was quickly rolled back. This change seemed low-risk due to the origination of the suggestion, but we should have collected more data on it before rolling it out to production. We will be making sure future WAF rules follow that process.

Time: Sept. 28, 2021, 11:38 a.m.

Status: Resolved

Update: Reason codes are now populating in the evaluations page for all new evaluations. This took longer than usual to patch due to a large schema migration we had to run overnight. We are auditing our data coming out of this to make sure all other similar instances are identified and fixed. We will be backfilling reason codes for yesterday's evaluations throughout the day today.

Time: Sept. 27, 2021, 7:07 p.m.

Status: Identified

Update: Starting at 13:20 ET today, one of our back-end services that populates reason code groups started malfunctioning. Because of this, we are currently unable to display reason codes and reason code groups in the Evaluations pages UI. No information is being lost, and we are working hard on getting this feature fully functional again.

Time: Sept. 7, 2021, 6:37 p.m.

Status: Postmortem

Update: Summary: On August 30th, our dashboard was inaccessible for about 30 minutes and slow to update for several hours. The API \(processing applications and data\) was not affected and no data was lost or put at risk. Operations teams performing manual reviews were the primary group impacted by this incident. Root Cause: The API server received thousands of requests with an unusual volume of entities in groups. This feature was not designed to support \(or defend against\) groups of the size received. The Groups feature is intended to manage situations where multiple entities belong to the same Application - such as 3 people applying for a joint checking account. Our dashboard uses a special database for displaying data in that dashboard. That database performs several queries that were not optimized to process a group of entities this large. The database slowed down, a backlog of queries developed, and eventually, the database went totally offline causing a dashboard outage. We were able to restore service by rate-limiting the client, removing the impacted entity links, and restarting the services that power the dashboard. This immediately restored service to the dashboard and after roughly another hour all of the backed-up queries were back to real-time. Actions taken or planned to avoid future incidents: In order to avoid this situation in the future we are: * Implementing rate limits on this feature to prevent unexpected behavior from misuse * Improving our proactive monitoring and alerting of the database that powers the dashboard to identify a potential issue more quickly * Implementing generalized guardrails and optimizations against intensive queries impacting the overall system

Time: Aug. 30, 2021, 5:02 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 30, 2021, 4:38 p.m.

Status: Monitoring

Update: The dashboard is functioning but some features are slow - continuing to work on this

Time: Aug. 30, 2021, 4:09 p.m.

Status: Identified

Update: There is a partial fix implemented and we're working to get everything back online

Time: Aug. 30, 2021, 3:45 p.m.

Status: Investigating

Update: The dashboard is currently unreachable - we are investigating and working to get it back up

Time: Aug. 30, 2021, 6:08 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 30, 2021, 5:18 p.m.

Status: Monitoring

Update: We're monitoring an increase in entity indexing latency. Since around 6am ET this morning, our queue of tasks to get new entity updates into our dashboard has been increasing, which has led to slower entity updates in the dashboard. We're currently adding more capacity to account for this as well as making some application changes to prioritize tasks.