Time: Jan. 10, 2020, 11:33 p.m.

Status: Postmortem

Update: After review and consultation with Microsoft Azure support, it was determined that an Azure platform event, occurring shortly before GoFormz experienced degraded performance, triggered an application recycle leading to instability with our Forms, Template, and Auth services. Azure support has provided GoFormz engineering and ops teams with specific mitigation steps to more quickly resolve any impact to our services if a future similar platform event occurs.

Time: Jan. 10, 2020, 6:09 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 10, 2020, 12:54 a.m.

Status: Monitoring

Update: After troubleshooting with Azure Support, Forms and Template services have returned to normal operation. We will continue to work with Azure Support to determine the root cause and update this issue upon such findings.

Time: Jan. 10, 2020, 12:24 a.m.

Status: Investigating

Update: We are currently investigating an issue that is impacting Forms and Template pages. Our engineers are working with Azure support to determine the root cause.

Time: Jan. 10, 2020, 11:33 p.m.

Status: Postmortem

Update: After review and consultation with Microsoft Azure support, it was determined that an Azure platform event, occurring shortly before GoFormz experienced degraded performance, triggered an application recycle leading to instability with our Forms, Template, and Auth services. Azure support has provided GoFormz engineering and ops teams with specific mitigation steps to more quickly resolve any impact to our services if a future similar platform event occurs.

Time: Jan. 10, 2020, 6:09 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 10, 2020, 12:54 a.m.

Status: Monitoring

Update: After troubleshooting with Azure Support, Forms and Template services have returned to normal operation. We will continue to work with Azure Support to determine the root cause and update this issue upon such findings.

Time: Jan. 10, 2020, 12:24 a.m.

Status: Investigating

Update: We are currently investigating an issue that is impacting Forms and Template pages. Our engineers are working with Azure support to determine the root cause.

Time: Nov. 20, 2019, 4:53 p.m.

Status: Resolved

Update: This incident has been resolved. Please see below message from the Microsoft Azure support team: Multiple Services - Mitigated Summary of Impact: Between 00:57 and 03:40 UTC on 20 Nov 2019, customers and services utilizing Azure Front Door (AFD) services were impacted by an infrastructure service failure. This resulted in a loss of connectivity to multiple services reliant on AFD. Azure Front Door services provide Edge caching and network entry point services to the Microsoft global network. This issue impacted a large percentage of Microsoft Services, though not all services were impacted. Many impacted services were able to initiate fail over from the AFD platform, providing immediate mitigation to their customers. Preliminary Root Cause: During a recent periodic deployment, initial safety checks did not detect the issue and prevent the roll out. Monitoring detected once service failure was experienced and alerted engineers. Mitigation: Engineers immediately initiated deployment rollback procedures to correct the underlying Azure Front Door issue. This was completed at 02:40 UTC on 20 Nov 2019, at which point Impacted services began recovering. A detailed root cause analysis will be published within 72 hours.

Time: Nov. 20, 2019, 4:01 a.m.

Status: Monitoring

Update: At 5pm Pacific Time GoFormz was impacted by Azure CDN outages causing problems with login for approximately 45 minutes. The CDN outages have been traced to a current issue with Microsoft Azure, our cloud services provider. Azure engineers have identified outages of their Network Infrastructure across all regions. Even though GoFormz services remain stable, we will continue to monitor all services and will update if our status changes. Current Message from Azure Status as of 7:50pm Pacific Time: Multiple Services - Applying Mitigation Summary of Impact: Starting at 00:57 UTC on 20 Nov 2019, a subset of customers may experience failures when attempting to access Azure Resources globally. Engineers have identified a root cause and mitigation is in progress; customers may see signs of recovery as this progresses.

Time: Nov. 20, 2019, 4:53 p.m.

Status: Resolved

Update: This incident has been resolved. Please see below message from the Microsoft Azure support team: Multiple Services - Mitigated Summary of Impact: Between 00:57 and 03:40 UTC on 20 Nov 2019, customers and services utilizing Azure Front Door (AFD) services were impacted by an infrastructure service failure. This resulted in a loss of connectivity to multiple services reliant on AFD. Azure Front Door services provide Edge caching and network entry point services to the Microsoft global network. This issue impacted a large percentage of Microsoft Services, though not all services were impacted. Many impacted services were able to initiate fail over from the AFD platform, providing immediate mitigation to their customers. Preliminary Root Cause: During a recent periodic deployment, initial safety checks did not detect the issue and prevent the roll out. Monitoring detected once service failure was experienced and alerted engineers. Mitigation: Engineers immediately initiated deployment rollback procedures to correct the underlying Azure Front Door issue. This was completed at 02:40 UTC on 20 Nov 2019, at which point Impacted services began recovering. A detailed root cause analysis will be published within 72 hours.

Time: Nov. 20, 2019, 4:01 a.m.

Status: Monitoring

Update: At 5pm Pacific Time GoFormz was impacted by Azure CDN outages causing problems with login for approximately 45 minutes. The CDN outages have been traced to a current issue with Microsoft Azure, our cloud services provider. Azure engineers have identified outages of their Network Infrastructure across all regions. Even though GoFormz services remain stable, we will continue to monitor all services and will update if our status changes. Current Message from Azure Status as of 7:50pm Pacific Time: Multiple Services - Applying Mitigation Summary of Impact: Starting at 00:57 UTC on 20 Nov 2019, a subset of customers may experience failures when attempting to access Azure Resources globally. Engineers have identified a root cause and mitigation is in progress; customers may see signs of recovery as this progresses.

Time: Nov. 19, 2019, 11 p.m.

Status: Resolved

Update: After successful code rollback, all services have remained stable.

Time: Nov. 19, 2019, 7:46 p.m.

Status: Monitoring

Update: Account services and related services have returned to normal after a rollback from a failed code release. We will continue monitoring services throughout the day.

Time: Nov. 19, 2019, 7:12 p.m.

Status: Investigating

Update: We are currently experiencing an issue with our Account services that can impact login (mobile and web) and some website functions. We will update this message as the issue progresses.