Time: Feb. 18, 2020, 8:23 a.m.

Status: Resolved

Update: This incident has been resolved. Post-mortem analysis will be published later this week.

Time: Feb. 17, 2020, 4:55 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: Feb. 17, 2020, 4:54 p.m.

Status: Monitoring

Update: We just reverted a minor update. Webtriggers are working again.

Time: Feb. 17, 2020, 4:13 p.m.

Status: Investigating

Update: We are currently investigating

Time: Dec. 18, 2019, 11:56 p.m.

Status: Postmortem

Update: ## What Happened On 13-December \(a Friday\), after midnight \(UTC\), an Intuiface employee using API Explorer reported an outage. The Intuiface Server Team also received automatic alerts about the lack of server response. These outages occurred several times until the morning of Monday, 16-December. On 16-December, the Server Team performed root cause analysis, during which it was discovered that the outage issue mainly affected Intuiface employees. Further analysis identified the root cause of performance degradation as an unused but resource-costly feature only available to Intuiface employees. Non-employees encountered API Explorer outages as a side effect. ## Corrective Actions On the morning of 17-December, a plan was executed to remove the unused feature and rebuild the server. The rebuilt server was tested and ready at 2PM UTC. In addition, to proactively compensate for possible performance issues in the future, the host supporting the server was accelerated by a factor of two. ## Remediation Plans Moving forward, five optimization options have been identified, with implementation occurring over the coming weeks. These optimizations will tune request storage to accommodate an ever-growing database. These requests are used by the machine learning recommendation algorithm in API Explorer.

Time: Dec. 17, 2019, 2:09 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 17, 2019, 1:57 p.m.

Status: Investigating

Update: We have updated the underlying software infrastructure and the outage issue appears to have been corrected. Testing is ongoing

Time: Dec. 17, 2019, 8:12 a.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Dec. 16, 2019, 7:11 p.m.

Status: Investigating

Update: API Explorer is currently subject to short duration outages of 2 to 3 minutes, occurring on an infrequent but repeat basis. Investigation is currently underway with a plan to implement a correction or workaround sometime tomorrow, Tuesday 17-Dec, CET time. In the meantime, if attempts to use API Explorer fail, what a minute or two then attempt a reconnect.

Time: Nov. 7, 2019, 8:14 a.m.

Status: Postmortem

Update: ## Post mortem analysis of the incident: * Immediately after the incident of 5-November between 8AM-11AM UTC, we needed to clean up the database and its storage on the disk. * We deleted old records contained within the largest table \(removing records older than 16 months instead of 18 months\). This deletion consumed a significant amount of journal storage \(approx. 15GB per million of records\), exhausting all available disk storage. * At around 1:30 PM UTC the database locked and entered automatic recovery mode. * We detected this issue three minutes later and immediately disabled the web server to limit the risk of paid Players and Composers stopping or refusing to launch. NB: Free Player and Composer remained unusable following remediation of the first incident * The database was repaired and placed back online after 30 minutes. * The web server was restarted and everything became fully operational. **NOTE**: At no time was any customer data – experiences or analytics – ever at risk. ## Remediation plans: * In the following days, the offending table will be carefully and slowly cleaned in small chunks to avoid the disk storage issue. * In the following weeks, we will migrate that table to a different database to avoid similar issues from occurring again.

Time: Nov. 5, 2019, 2:27 p.m.

Status: Resolved

Update: my.intuiface.com is now available. Free and paid Composers and Players can be launched again. We will monitor our servers in the following hours and days to prevent a future occurrence of this issue.

Time: Nov. 5, 2019, 1:49 p.m.

Status: Investigating

Update: my.intuiface.com is currently unavailable. Free Player and Composer cannot be launched. Paid Player and Composer continue to work properly.

Time: Nov. 7, 2019, 8:11 a.m.

Status: Postmortem

Update: ## Analysis of incident: * On 22-October we upgraded our database to enhance Intuiface user security. The upgrade went well but after the server was back in production, we detected some database timeouts. Investigation revealed they were due to some severely altered – thus fragmented – indexes, causing performance degradation. * We reindexed almost all but two tables on 24-October without any issue. Reindexing of the two largest tables was deferred to an already scheduled maintenance activity on 5-November. * On 5-November at around 6:15 AM UTC, the first table was reindexed without any issue. The second table was then reindexed and this operation generated a very large amount of data. At around 8:00 AM UTC the disk went full and the database locked, preventing any further updates to the database tables. At this point, we initiated a rollback of the index. * Until 8:50 AM UTC the web services for license check and activation responded with an error every time a Player or Composer attempted to retrieve or activate a license. This affected all free and licensed Players and Composers during startup. It also affected some running, licensed Players that happened to run a license check \(which normally occurs once a day\). It was also no longer possible to connect to the My Intuiface website \([my.intuiface.com](http://my.intuiface.com)\) or to the Intuiface Support site. * At 8:50 AM UTC, we neutralized the web services for license check and activation so they would no longer reply with an error. This enabled licensed Players and Composers to start correctly. Free Players and Composers could still not launch as they require a successful license check. * At 10:57 AM UTC, the table index was fully rolled-back, making the database accessible and enabling us to begin cleanup in order to free disk storage. * At 11:00 AM UTC the server was back online again. Player and Composer could check their licenses and users could again connect on [my.intuiface.com](http://my.intuiface.com) or support site. * After 10 minutes of monitoring, we considered the issue as fixed at 11:13 AM UTC **NOTE**: At no time was any customer data – experiences or analytics – ever at risk. ## Remediation plans: * We will ensure that database failure or inaccessibility – a very rare event that occurred only 5 hours in total over the last 3 years, an availability rate of better than 99.98% - will no longer impact running, paid Players nor the launch of paid Players and Composers. * We will enhance our ability to inform users and customers about issues and remediation steps in real-time, both via our website \([www.intuiface.com](http://www.intuiface.com)\) and through our [status.intuiface.com](http://status.intuiface.com) server monitoring website. \(NOTE: It is already possible to use our server monitoring site to sign up for notifications about ongoing issues.\) * We will also take action to better test maintenance operations, even if in this case it was hard to predict that it could lead to a major event. \(The faulty reindexing consumed more than 140GB of disk space for a single table!\) * An internal review of the incident is currently underway to identify and take the appropriate actions.

Time: Nov. 5, 2019, 11:12 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 5, 2019, 11:04 a.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Nov. 5, 2019, 11:03 a.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: Nov. 5, 2019, 11:03 a.m.

Status: Identified

Update: The database was recovered and the server is working correctly again. Any licensing issue should no longer occur.

Time: Nov. 5, 2019, 10:28 a.m.

Status: Identified

Update: The database rollback is still in progress. According to database progress information, the recovery should be finished in 40 minutes. We expect a complete return to normal operation after 11:00 AM UTC (12:00 AM CET - Central European Time) We will provide a full postmortem analysis of the problem later today or tomorrow.

Time: Nov. 5, 2019, 9:42 a.m.

Status: Identified

Update: We are still in progress of applying a rollback to our database. We are now expecting the database to be working again and the server to be fully functional around 10:20 AM UTC (11:20 AM CET - Central European Time).

Time: Nov. 5, 2019, 9:10 a.m.

Status: Identified

Update: We are currently applying a rollback to our database. We are expecting the database to be working again and the server to be fully functional around 9:38 AM UTC (10:38 AM CET - Central European Time).

Time: Nov. 5, 2019, 8:56 a.m.

Status: Investigating

Update: Due to server error, licensed player or composer could not start properly. This was fixed. at 8:50 AM UTC (9:50 AM CET - Central European Time). Paid player and composer can be restarted again. Please note that it is currently not possible to activate new licenses. We are currently continuing our investigation.

Time: Nov. 5, 2019, 8:41 a.m.

Status: Investigating

Update: we are currently investigating

Time: Sept. 24, 2019, 9:55 p.m.

Status: Resolved

Update: The issue was resolved.

Time: Sept. 24, 2019, 9:41 p.m.

Status: Investigating

Update: Sorry for the inconvenience, Explorer API server is currently unavailable. We are currently investigating this issue.