Time: June 27, 2022, 7:21 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: June 26, 2022, 5:51 p.m.

Status: Monitoring

Update: The errors have been resolved. We are continuing to monitor this.

Time: June 24, 2022, 8:26 p.m.

Status: Investigating

Update: We are still seeing elevated 502 errors and are continuing to investigate. These issues are isolated to our Core API only and seem to be affecting ~0.1% of Core API requests only. Our Edge API continues to be fully operational.

Time: June 24, 2022, 2:18 p.m.

Status: Monitoring

Update: We have deployed a fix and are monitoring our container memory consumption.

Time: June 24, 2022, 1:54 p.m.

Status: Investigating

Update: Hi, We are experienced slightly elevated 502 errors returning from our Core API. We believe this is due to a recently introduced piece of code that is consuming more memory than it should be and are working on a fix.

Time: May 20, 2022, 7:28 a.m.

Status: Resolved

Update: API latency has returned to normal range following monitoring.

Time: May 19, 2022, 6:29 p.m.

Status: Monitoring

Update: Our fix has deployed to production and all key infrastructure metrics are returning to normal range. We will continue to monitor.

Time: May 19, 2022, 6:14 p.m.

Status: Investigating

Update: We're seeing elevated error rates to our API on account of a large influx of spurious traffic. We are deploying a fix to this which should help alleviate this.

Time: Feb. 3, 2022, 1:58 p.m.

Status: Postmortem

Update: ## Root Cause At 13:15 UTC on 03 Feb 2022, we began deploying a routine release of the Flagsmith application to our production SaaS environment. This release included a database migration which added a new unique index to one of our tables which holds information about multivariate values for features. When the migration was run in our other environments we noticed no ill effects from the addition of the index, however, in production where we have substantially more data this index took longer to add than anticipated and required a full table lock during that period.  ## Downtime Our monitoring shows that the application was unresponsive for a period of just under 2 minutes while the migration was running.  ## Long Term Remediation To improve on this in the future, we are planning to upgrade our version of Django to allow us to easily add indexes concurrently. We will also be monitoring more carefully for future index additions and checking whether they will require a table lock. Finally, we will be looking at making our staging environment more representative of production in terms of data so that we can catch issues such as this in the future.

Time: Feb. 3, 2022, 1:25 p.m.

Status: Resolved

Update: We've resolved an issue with a database migration that caused a temporary full table lock when modifying an index. Total outage was around 70 seconds. We're going to investigate the root cause and provide an update when it's ready.

Time: Sept. 10, 2021, 1:18 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Sept. 10, 2021, 12:33 p.m.

Status: Monitoring

Update: We have identified the issue as a large number of transactional database wait locks. We have increased the size of our API server cluster which has reduced these wait locks, as well as reducing overall API latency. We are monitoring the situation.

Time: Sept. 10, 2021, 11:34 a.m.

Status: Investigating

Update: Hi, We are seeing increased latency to our SDK endpoints for a small percentage of requests. We are investigating and will provide a further update as soon as we have more information. Apologies for the degradation in service.

Time: June 14, 2021, 1:09 p.m.

Status: Resolved

Update: This latency issue has been fixed.

Time: June 14, 2021, 12:16 p.m.

Status: Identified

Update: We recently migrated from Gitlab CI to Github actions. The build pipelines have introduced a change which is increasing the asset loading performance of the dashboard. We're deploying a fix now. Please note this is just causing a slowdown to loading of the site.