Time: Feb. 28, 2019, 9:01 a.m.

Status: Postmortem

Update: # The issue Due to the increase of users and the volume of information recorded on our server \(particularly device logs accessible from the Share and Deploy Console\), our database reached its storage limit. Feb 15, 2019 - 1:20 PM CET our database detected the approaching storage limit and started an automatic emergency backup. At that point, our Web servers were unable to add additional information in the database. Three minutes after this incident occurred, our IT Team was notified that the servers were no longer operational. The Team quickly determined that the underlying issue was with the database. It took around 20 minutes for the database to finish its automatic emergency backup, and another 20 minutes for the Team to delete old records to free sufficient storage. Meanwhile, we stopped our Web servers until the database was fully functional. Finally, at 1:51 PM CET \(after 41 minutes\) the system was fully restored with enough free storage to cover several weeks. # The fix Manually deleting device logs older than 12 months freed enough storage to restart our Web servers. # Remediation On 18-Feb-2019 we doubled our database storage limit. In addition, we implemented an automatic deletion process for the removal of device log records older than 12 months. In early March 2019, we will add new automatic alerts for predefined database storage thresholds. These new alerts will be checked every few months to ensure they are active and sufficient for our needs.

Time: Feb. 15, 2019, 12:55 p.m.

Status: Resolved

Update: The issue was resolved. We are now investigating why we were not notified far in advance of the disk shortage.

Time: Feb. 15, 2019, 12:48 p.m.

Status: Identified

Update: The problem is identified. Some disk storage were full preventing normal operations. We are currently fixing the issue. This incident prevents the use of my.intuiface.com and may prevent launching players or composers. We expect normal operation in 15 minutes.

Time: Feb. 15, 2019, 12:39 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Feb. 15, 2019, 12:39 p.m.

Status: Investigating

Update: We are currently investigating the issue affecting our server. This incident prevents the use of my.intuiface.com and may prevent launching players or composers.

Time: Nov. 22, 2018, 3:07 p.m.

Status: Postmortem

Update: ## The issue After deploying a server update, some minor errors were reported via email to the Intuiface staff. Due to a very high number of such notifications, our email servers were overloaded, causing us to reach the 24 hour email quota imposed by our email service provider, AWS. All emails were blocked, including all emails generated by the Hosted instance of Intuiface’s Share via Email Interface Asset. In total, the blockage affected the following services: * Hosted Share via e-mail Interface Asset, * Emails sent to warn users that Intuiface Players are going online / offline * Emails sent when sharing an experience. * Registration confirmation emails * Forgotten password emails * Invoice emails The affected e-mails were either delayed or lost. ## The fix We asked our cloud provider \(AWS\) to increase our email quota. At around 2PM UTC on the 21st of November, our email buffer was emptied and emails were once again correctly sent. ## Remediation To avoid this email overload problem, we have separated our email servers. An independent server is now used to send \(possibly very high volume\) notifications to the Intuiface staff. As a result, internal notification emails will no longer interfere with the quota for emails created for and by our users.

Time: Nov. 21, 2018, 4:08 p.m.

Status: Resolved

Update: The issue was solved. Emails created by the Hosted instance of the Share via Email Interface Asset are being sent again as of 2PM UTC.

Time: Nov. 21, 2018, 9:50 a.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Nov. 21, 2018, 9:49 a.m.

Status: Investigating

Update: Emails are not being sent to our customers (payment-related emails, registration confirmation, password reset...) or to third parties when using the Hosted version of the Share via Email Interface Asset. The issue has been diagnosed and is currently being corrected.

Time: Oct. 6, 2018, 7:19 a.m.

Status: Resolved

Update: This incident is resolved.

Time: Oct. 6, 2018, 7:10 a.m.

Status: Investigating

Update: The server issue was related to an underlying hardware issue as reported by our cloud provider (AWS). We will change the underlying hardware Monday 8 October early in the morning.

Time: Oct. 6, 2018, 7:08 a.m.

Status: Investigating

Update: We are currently investigating an infrastructure issue. We will provide more details shortly.

Time: Sept. 17, 2018, 1:06 p.m.

Status: Resolved

Update: A one minute outage for Management Console has been misrepresented as a 3+ hour outage.

Time: Sept. 6, 2018, 1:44 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Sept. 6, 2018, 1:42 p.m.

Status: Investigating

Update: A one minute outage for API Explorer has been misrepresented as a 4+ hour outage. We’ve got some bug fixing to do! Rest assured, all systems are go after what was really just a tiny hiccup.