Time: March 21, 2023, 6:51 p.m.

Status: Postmortem

Update: **Preliminary Root Cause:** SimpliGov utilizes Azure SQL Databases for database shard management within the application. The “Shard Manager” used within Azure SQL experienced communication issues, resulting in timeouts. SimpliGov moved the database to a different server and updated settings relating to connections to the shard manager. **Mitigation:** SimpliGov support moved the database to another server and updated settings relating to connections to the shard manager. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward and will work with Azure support to understand and resolve the connection errors noted. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.

Time: March 21, 2023, 6:50 p.m.

Status: Resolved

Update: SimpliGov OPS have monitored the behavior of the application for the last hour and can confirm that the application is responding as expected with usual calculation and dashboard performance. SimpliGov OPS are moving this to a resolved status with a post-mortem to follow for this incident.

Time: March 21, 2023, 5:26 p.m.

Status: Monitoring

Update: SimpliGov has identified and resolved the underlying cause of the issue. SimpliGov OPS have updated the required settings to bring operations back to normal. SimpliGov OPS will continue to monitor the situation before providing a final root cause analysis and resolution status.

Time: March 21, 2023, 5:14 p.m.

Status: Identified

Update: SimpliGov has identified the source of the issue and is working on resolving the issues noted.

Time: March 21, 2023, 5:10 p.m.

Status: Investigating

Update: SimpliGov has observed unusual slowness and calculation issues for some customers. Our team is working on this and will report back with further information as soon as possible.

Time: Dec. 2, 2022, 11:21 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 2, 2022, 11:21 a.m.

Status: Identified

Update: The issue has been resolved and SimpliGov IT are continuing to monitor the situation. The issue was caused by an underlying node failing to terminate successfully when auto-scaling down. SimpliGov IT is working with Azure support to identify the cause of this and prevent it going forward. Additional information will be supplied in the post-mortem attached to this incident.

Time: Dec. 2, 2022, 11:15 a.m.

Status: Identified

Update: 502 errors are being observed by some users at this time. SimpliGov IT have identified the issue and are resolving it.

Time: July 14, 2022, 4:34 p.m.

Status: Resolved

Update: The issue has been resolved. The cause of the inaccessible API was caused by a failure in a Microsoft shard routing library. SimpliGov has updated the library and the training and staging environments are now accessible.

Time: July 14, 2022, 3:19 p.m.

Status: Investigating

Update: The training and staging environments are inaccessible due to an orchestration services (Kubernetes) API being unavailable. SimpliGov IT/OPS is currently investigating and will resolve as soon as possible.

Time: June 17, 2022, 1:38 p.m.

Status: Postmortem

Update: **Summary:** _Cause:_ Azure introduced IPv6 addresses to the Azure Application Gateway that load balances traffic for SimpliSign. IPv6 addresses are not supported yet by Azure Application Gateway so redirection to these addresses resulted in a 502 status code for users _Manifestation:_ Users intermittently received 502 status code and error pages which prevented them from accessing SimpliSign signing pages, SimpliSign interactive pages etc. _Resolution:_ Microsoft Azure Support restarted backend services on their side and confirmed that DNS resolution does not occur for invalid IPv6 configurations. **Preliminary Root Cause:** SimpliGov utilizes Azure Application Gateway for load balancing SimpliSign services for all customers. This infrastructure component decides which application is used for each request and routes requests to different backend servers/applications based on the overall load within the system. It was noted that despite having healthy, available backend applications, the Azure Applicatoin Gateway was still throwing 502 status code errors even though all health checks and communication tests between the Azure Application Gateway and the backend SimpliSign applications were passing as expected. The outcome of the previous was that SimpliSign pages were intermittently unavailable. SimpliGov enlisted Azure Support and their Product Group to assist with resolving this issue. After debugging sessions with both Azure Support and the Azure Product Group, it was determined that a recent update by the Microsoft Azure Government team caused the issue. Azure Support released an update to apply IPv6 configurations to Azure Application Gateways within the Microsoft Azure Government cloud. During their update, the Azure Product Group restarted the components that make up the Azure Application Gateway but in one case, their restart processes did not have the desired effect in making sure that correct DNS resolution was applied for the backend applications behind the Application Gateway. This led to some components of the Application Gateway having multiple IPv6 and IPv4 backends. Azure Support confirmed that the presence of IPv6 backends was the direct cause of this issue as IPv6 is not supported for Azure Application Gateway backends. The presence of both IPv6 and IPv4 addresses explains the intermittent nature of this issue in that redirection to IPv4 backends resulted in successful interactions while interactions with IPv6 backends resulted in a 502 status code and error page being shown to the user. **Mitigation:** Azure Support restarted the components that held IPv6 IP addresses to make sure that they were removed and correct DNS resolution could take place. In simpler terms, Azure Support restarted the Application Gateway and made sure that only IPv4 addresses were present as IPv6 is not supported by the Azure Application Gateway currently. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward and will identify the underlying rotation stall issue. SimpliGov will continue working with Azure Support to try and prevent such occurences in the future. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.

Time: June 17, 2022, 1:37 p.m.

Status: Resolved

Update: Microsoft Azure support has resolved the issue on their end and provided a prelimiary root cause analysis for the cause of this issue. SimpliGov OPS have been monitoring the performance of SimpliSign since they pushed their update and the fix appears to have resolved all known issues regarding 502 status codes being shown to users. Additional information will be posted in the post-mortem for this incident.

Time: June 17, 2022, 5:22 a.m.

Status: Identified

Update: SimpliGov OPS have identified an issue within the Azure Application Gateway load balancer that routes requests to various SimpliSign services. The application gateway (load balancer) is recognizing that the applications behind it were up and running but the Azure Application Gateway still throws 502 errors and refuses to consistently route URLs correctly. SimpliGov OPS are working with Azure Support on this. We will provide an additional update when further information becomes available.

Time: June 16, 2022, 8:43 p.m.

Status: Investigating

Update: We are currently investigating a reported incident involving intermittent 502 Bad Gateway errors when accessing SimpliSign Interactive.

Time: March 14, 2022, 6:10 a.m.

Status: Postmortem

Update: Some customers were unable to save workflow templates and/or experienced slower than expected submission times on the preproduction environment on 03/10/2022. The SimpliGov IT/OPS team identified the cause of the behavior for affected customers and resolved the issue at approximately 7:30 AM PST. The cause of the issue was due to nightly maintenance jobs taking longer than expected for a small subset of customers. Restarting a step within the affected maintenance jobs allowed them to complete and customers were able to submit/save templates without issue.

Time: March 14, 2022, 6:10 a.m.

Status: Resolved

Update: Some customers were unable to save workflow templates and/or experienced slower than expected submission times on the preproduction environment on 03/10/2022. The SimpliGov IT/OPS team identified the cause of the behavior for affected customers and resolved the issue at approximately 7:30 AM PST. The cause of the issue was due to nightly maintenance jobs taking longer than expected for a small subset of customers. Restarting a step within the affected maintenance jobs allowed them to complete and customers were able to submit/save templates without issue.