Time: May 14, 2020, 4:21 p.m.

Status: Resolved

Update: This incident has been resolved

Time: May 12, 2020, 9:58 a.m.

Status: Monitoring

Update: Turtl is currently stable, but you may experience minor degradation of service. We are actively monitoring this issue to ensure service remains stable.

Time: May 11, 2020, 2:51 p.m.

Status: Identified

Update: Turtl is currently stable, but you may experience minor degradation of service. We are actively monitoring this issue to ensure service remains stable.

Time: May 11, 2020, 2:21 p.m.

Status: Investigating

Update: We are currently investigating an issue affecting all access to Turtl between 14:39 and 14:45 today.

Time: May 14, 2020, 4:21 p.m.

Status: Resolved

Update: This incident has been resolved

Time: May 12, 2020, 9:58 a.m.

Status: Monitoring

Update: Turtl is currently stable, but you may experience minor degradation of service. We are actively monitoring this issue to ensure service remains stable.

Time: May 11, 2020, 2:51 p.m.

Status: Identified

Update: Turtl is currently stable, but you may experience minor degradation of service. We are actively monitoring this issue to ensure service remains stable.

Time: May 11, 2020, 2:21 p.m.

Status: Investigating

Update: We are currently investigating an issue affecting all access to Turtl between 14:39 and 14:45 today.

Time: April 24, 2020, 6:27 p.m.

Status: Postmortem

Update: ### Postmortem - Incident of 24th April 2020 #### Summary: An application update released at 8:47 am UTC introduced a performance regression which caused very complex Turtl documents to use 100% of the available CPU for a long period of time, blocking all other ongoing requests reaching the affected server. Requests were routed to healthy servers by our load balancer. At 09:09 UTC all servers became affected by this and Turtl was unavailable for all uncached requests for 6 minutes \(09:09 - 09:15\). We were notified of the issue by automated alarms at 09:10 and restored service at 09:15, after which there was no further downtime, although until full resolution a few further requests timed out. We are currently gathering data regarding the exact number of requests that were affected. Once service was restored we continued to monitor the situation while investigating the root cause. A rollback was released at 10:11. At 12:12 we identified the application change which introduced the regression and implemented a fix which was released at 14:11. #### Details The personalisation feature which allows our customers to send documents tailored to each recipient uses markers such as \`%company\_name%\` to adapt each personalised version of a document. To make the replacements, when the document is being rendered on the server we first find all occurrences of these markers and then substitute them for the correct value. This is done by running a simple regex on the document content. On the 24th April, a change to this regex was released which introduced a high-performance penalty when it was run on very complex documents, documents over 100 pages with many large inline tables. Due to the specific circumstances in which the performance issue occurred, this was not identified in our staging quality assurance tests as there are only a handful of these types of documents at present within Turtl. #### Further steps During our investigation of the issue, we identified that the marker extraction logic was running for each document being rendered, regardless of it being personalised or not. If this wouldn't have been the case, the downtime wouldn't have happened until much later in the feature's lifetime. Although incidents like these are unpleasant, we are grateful it was caught early. As part of the fix released at 14:11, apart from optimising the regex in question, we have also removed the marker replacement logic from the standard document rendering, moving it to only those routes that require it. To ensure these kinds of issues will in the future be caught in our pre-release steps we will be adding to our automated tests suite a list of specially crafted potentially problematic documents, such as the ones described above, and ensure these are loaded in all possible scenarios. #### Incident timeline \(UTC\) \* 08:48 - application release \* 09:09 - uncached requests stop being delivered \* 09:10 - automated alarms trigger \* 09:15 - service restored \* 10:11 - rollback release \* 12:12 - code change introducing the issue identified \* 14:11 - permanent fix released

Time: April 24, 2020, 6:27 p.m.

Status: Postmortem

Update: ### Postmortem - Incident of 24th April 2020 #### Summary: An application update released at 8:47 am UTC introduced a performance regression which caused very complex Turtl documents to use 100% of the available CPU for a long period of time, blocking all other ongoing requests reaching the affected server. Requests were routed to healthy servers by our load balancer. At 09:09 UTC all servers became affected by this and Turtl was unavailable for all uncached requests for 6 minutes \(09:09 - 09:15\). We were notified of the issue by automated alarms at 09:10 and restored service at 09:15, after which there was no further downtime, although until full resolution a few further requests timed out. We are currently gathering data regarding the exact number of requests that were affected. Once service was restored we continued to monitor the situation while investigating the root cause. A rollback was released at 10:11. At 12:12 we identified the application change which introduced the regression and implemented a fix which was released at 14:11. #### Details The personalisation feature which allows our customers to send documents tailored to each recipient uses markers such as \`%company\_name%\` to adapt each personalised version of a document. To make the replacements, when the document is being rendered on the server we first find all occurrences of these markers and then substitute them for the correct value. This is done by running a simple regex on the document content. On the 24th April, a change to this regex was released which introduced a high-performance penalty when it was run on very complex documents, documents over 100 pages with many large inline tables. Due to the specific circumstances in which the performance issue occurred, this was not identified in our staging quality assurance tests as there are only a handful of these types of documents at present within Turtl. #### Further steps During our investigation of the issue, we identified that the marker extraction logic was running for each document being rendered, regardless of it being personalised or not. If this wouldn't have been the case, the downtime wouldn't have happened until much later in the feature's lifetime. Although incidents like these are unpleasant, we are grateful it was caught early. As part of the fix released at 14:11, apart from optimising the regex in question, we have also removed the marker replacement logic from the standard document rendering, moving it to only those routes that require it. To ensure these kinds of issues will in the future be caught in our pre-release steps we will be adding to our automated tests suite a list of specially crafted potentially problematic documents, such as the ones described above, and ensure these are loaded in all possible scenarios. #### Incident timeline \(UTC\) \* 08:48 - application release \* 09:09 - uncached requests stop being delivered \* 09:10 - automated alarms trigger \* 09:15 - service restored \* 10:11 - rollback release \* 12:12 - code change introducing the issue identified \* 14:11 - permanent fix released

Time: April 24, 2020, 10:19 a.m.

Status: Resolved

Update: This incident has been resolved and service has now returned to normal. We will continue to investigate the cause of this downtime and provide a post mortem shortly.

Time: April 24, 2020, 10:19 a.m.

Status: Resolved

Update: This incident has been resolved and service has now returned to normal. We will continue to investigate the cause of this downtime and provide a post mortem shortly.

Time: April 24, 2020, 9:31 a.m.

Status: Monitoring

Update: We have now implemented a fix for this issue and most users should have regular access again. We are monitoring the situation and will have a detailed post mortem once we have confirmed it is fully resolved.

Time: April 24, 2020, 9:31 a.m.

Status: Monitoring

Update: We have now implemented a fix for this issue and most users should have regular access again. We are monitoring the situation and will have a detailed post mortem once we have confirmed it is fully resolved.

Time: April 24, 2020, 9:29 a.m.

Status: Identified

Update: At 10:09 am we identified an incident resulting in Turtl being inaccessible for all users. We are currently resolving this issue and service should now be available again for most users. Please check back here for updates.

Time: April 24, 2020, 9:29 a.m.

Status: Identified

Update: At 10:09 am we identified an incident resulting in Turtl being inaccessible for all users. We are currently resolving this issue and service should now be available again for most users. Please check back here for updates.

Time: Nov. 12, 2019, 4:11 p.m.

Status: Resolved

Update: Between approximately 2019-11-12 13:10 UTC and 2019-11-12 16:10 UTC the analytics were unavailable for users with author or viewer permissions due to a bad deploy. This issue has now been fully resolved and normal service has been restored.

Time: Nov. 12, 2019, 3:20 p.m.

Status: Identified

Update: We are currently investigating an issue that is stopping viewers and authors from viewing analytics on any stories. The issue has been identified and we are working to fix it. Please check back here for updates.

Time: Nov. 12, 2019, 4:11 p.m.

Status: Resolved

Update: Between approximately 2019-11-12 13:10 UTC and 2019-11-12 16:10 UTC the analytics were unavailable for users with author or viewer permissions due to a bad deploy. This issue has now been fully resolved and normal service has been restored.

Time: Nov. 12, 2019, 3:20 p.m.

Status: Identified

Update: We are currently investigating an issue that is stopping viewers and authors from viewing analytics on any stories. The issue has been identified and we are working to fix it. Please check back here for updates.