Time: March 19, 2019, 5:03 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: March 19, 2019, 12:07 a.m.

Status: Monitoring

Update: As of approximately 4:45pm PDT, all functions are operational and performing normally. While we are continuing to monitor this service, please contact the GoFormz support team if you experience issues surrounding the functions mentioned below.

Time: March 18, 2019, 10:45 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: March 18, 2019, 10:44 p.m.

Status: Identified

Update: We are currently experiencing an outage with our communication service on our mobile and web platforms. This primarily impacts form completion events, form transfer events, workflows, PDF generation, and outgoing webhooks.

Time: Dec. 4, 2018, 5:43 a.m.

Status: Resolved

Update: GoFormz engineering has confirmed that all Workflow systems are operational and stable. Microsoft has issued the following statement: Intermittent Network Degradation - South Central US - Mitigated Summary of impact: Between 02:48 UTC and 04:52 UTC on 04 Dec 2018, a subset of customers in South Central US may have experienced degraded performance, network drops or time outs when accessing Azure resources hosted in this region. Applications/resources that retried connections or requests may have succeeded, as there are many redundant network devices in the impacted data center and retries are likely to have succeeded through different network devices. Preliminary root cause and mitigation: Two network devices in the South Central US region experienced an issue updating routing information for packet forwarding. The Azure Fabric failed to automatically remove these devices from service and continued to drop a fraction of the network packets passed through them. Once identified and manually removed from service by Azure engineers the network automatically recovered utilizing alternate network devices. Next steps: Engineers will continue to investigate to establish the full root cause. A full Root Cause Analysis will be provided in approximately 72 hours. To stay informed on any issues, maintenance events, or advisories, create service health alerts (https://www.aka.ms/ash-alerts) and you will be notified via your preferred communication channel(s): email, SMS, webhook, etc.

Time: Dec. 4, 2018, 5:24 a.m.

Status: Monitoring

Update: Microsoft Azure status has been updated to the following: Applying Mitigation and Validating Recovery - https://azure.microsoft.com/en-us/status/. GoFormz engineering has also confirmed that our test Workflows are performing as expected. Even though GoFormz customers should now experience normal Workflow behavior, we will continue monitoring until Microsoft provides updated status.

Time: Dec. 4, 2018, 4:42 a.m.

Status: Identified

Update: GoFormz engineering has identified the source as an outage with Microsoft Azure. They have published the following: Starting at approximately 03:30 UTC on 04 Dec 2018, customers in South Central US may experience degraded performance, network drops or time outs when accessing Azure resources hosted in this region. Engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant. https://azure.microsoft.com/en-us/status/ GoFormz engineering will continue to monitor Azure's progress and resolve any issues as services return to operating levels.

Time: Dec. 4, 2018, 4:24 a.m.

Status: Investigating

Update: Some workflows are not completing successfully. GoFormz engineering is investigating the issue and will update as soon as possible.

Time: Dec. 4, 2018, 5:43 a.m.

Status: Resolved

Update: GoFormz engineering has confirmed that all Workflow systems are operational and stable. Microsoft has issued the following statement: Intermittent Network Degradation - South Central US - Mitigated Summary of impact: Between 02:48 UTC and 04:52 UTC on 04 Dec 2018, a subset of customers in South Central US may have experienced degraded performance, network drops or time outs when accessing Azure resources hosted in this region. Applications/resources that retried connections or requests may have succeeded, as there are many redundant network devices in the impacted data center and retries are likely to have succeeded through different network devices. Preliminary root cause and mitigation: Two network devices in the South Central US region experienced an issue updating routing information for packet forwarding. The Azure Fabric failed to automatically remove these devices from service and continued to drop a fraction of the network packets passed through them. Once identified and manually removed from service by Azure engineers the network automatically recovered utilizing alternate network devices. Next steps: Engineers will continue to investigate to establish the full root cause. A full Root Cause Analysis will be provided in approximately 72 hours. To stay informed on any issues, maintenance events, or advisories, create service health alerts (https://www.aka.ms/ash-alerts) and you will be notified via your preferred communication channel(s): email, SMS, webhook, etc.

Time: Dec. 4, 2018, 5:24 a.m.

Status: Monitoring

Update: Microsoft Azure status has been updated to the following: Applying Mitigation and Validating Recovery - https://azure.microsoft.com/en-us/status/. GoFormz engineering has also confirmed that our test Workflows are performing as expected. Even though GoFormz customers should now experience normal Workflow behavior, we will continue monitoring until Microsoft provides updated status.

Time: Dec. 4, 2018, 4:42 a.m.

Status: Identified

Update: GoFormz engineering has identified the source as an outage with Microsoft Azure. They have published the following: Starting at approximately 03:30 UTC on 04 Dec 2018, customers in South Central US may experience degraded performance, network drops or time outs when accessing Azure resources hosted in this region. Engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant. https://azure.microsoft.com/en-us/status/ GoFormz engineering will continue to monitor Azure's progress and resolve any issues as services return to operating levels.

Time: Dec. 4, 2018, 4:24 a.m.

Status: Investigating

Update: Some workflows are not completing successfully. GoFormz engineering is investigating the issue and will update as soon as possible.

Time: Oct. 31, 2018, 4:43 p.m.

Status: Postmortem

Update: Analysis: After further investigation by GoFormz engineers, it was determined that this incident was triggered by an unusually large mobile Sync process initiated by a GoFormz customer that caused GoFormz’s Permission Service to become non-responsive. As designed, the Permission Service started an auto-heal process. However during this auto-heal process the Sync Service continued to receive 500 error codes which degraded performance until the Sync Service was then manually restarted. Mitigation: In order to mitigate potential recurrence of this specific type of incident, GoFormz’s engineering has scaled up our Permission Service to better handle such unusual usage spikes. We are also reconfiguring the Permission and Sync Service auto-heal policies to appropriately handle 500 error codes from different platform services.

Time: Oct. 20, 2018, 11:55 p.m.

Status: Resolved

Update: From approximately 2:20pm PST to 4:30pm PST, GoFormz mobile sync service suffered degraded performance which resulted in partial disruption of mobile sync and login to some customers. GoFormz engineering performed initial troubleshooting and was able to return the service to fully operational within 2 hours. GoFormz engineering will further investigate this disruption to determine necessary mitigation steps.

Time: Oct. 31, 2018, 4:43 p.m.

Status: Postmortem

Update: Analysis: After further investigation by GoFormz engineers, it was determined that this incident was triggered by an unusually large mobile Sync process initiated by a GoFormz customer that caused GoFormz’s Permission Service to become non-responsive. As designed, the Permission Service started an auto-heal process. However during this auto-heal process the Sync Service continued to receive 500 error codes which degraded performance until the Sync Service was then manually restarted. Mitigation: In order to mitigate potential recurrence of this specific type of incident, GoFormz’s engineering has scaled up our Permission Service to better handle such unusual usage spikes. We are also reconfiguring the Permission and Sync Service auto-heal policies to appropriately handle 500 error codes from different platform services.

Time: Oct. 20, 2018, 11:55 p.m.

Status: Resolved

Update: From approximately 2:20pm PST to 4:30pm PST, GoFormz mobile sync service suffered degraded performance which resulted in partial disruption of mobile sync and login to some customers. GoFormz engineering performed initial troubleshooting and was able to return the service to fully operational within 2 hours. GoFormz engineering will further investigate this disruption to determine necessary mitigation steps.