Is there an SimpliGov outage?
SimpliGov status: Systems Active
Last checked: 7 minutes ago
Get notified about any outages, downtime or incidents for SimpliGov and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 7 minutes ago
Get notified about any outages, downtime or incidents for SimpliGov and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for SimpliGov.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| SendGrid API v3 | Active |
| Preproduction | Active |
| API | Active |
| Authorization | Active |
| Email Interaction | Active |
| eSignature | Active |
| Export | Active |
| File Conversion | Active |
| Metaquery | Active |
| Portal | Active |
| SimpliSign | Active |
| Submission | Active |
| Production | Active |
| API | Active |
| Authorization | Active |
| Email Interaction | Active |
| eSignature | Active |
| Export | Active |
| File Conversion | Active |
| Metaquery | Active |
| Portal | Active |
| SimpliSign | Active |
| Submission | Active |
| Staging | Active |
| API | Active |
| Authorization | Active |
| Email Interaction | Active |
| eSignature | Active |
| Export | Active |
| File Conversion | Active |
| Metaquery | Active |
| Portal | Active |
| SimpliSign | Active |
| Submission | Active |
| Training | Active |
| API | Active |
| Authorization | Active |
| Email Interaction | Active |
| eSignature | Active |
| Export | Active |
| File Conversion | Active |
| Metaquery | Active |
| Portal | Active |
| SimpliSign | Active |
| Submission | Active |
View the latest incidents for SimpliGov and check for official updates:
Description: **Preliminary Root Cause:** SimpliGov auto-submits rely upon backend submission services hosted on Azure Service Fabric and utilizing Azure Service Bus for processing, queuing etc. The SimpliGov developed submit processing services used to process these autosubmits experienced intermittent connectivity issues from 12/10/2021 to 12/12/2021. The batch sizes today were larger as a result of processing backlogs from the SimpliGov incident noted on 10/21/2021. During this period of time, intake submissions should have worked as expected for all async processes and most synchronous workflow configurations. In cases where chained auto-submits with zero delay would have provided users with a different user experience than they would be accustomed to. In such cases, users would have been shown a thank you page from the initial relationship rather than the last relationship in the autosubmit chain The root cause of the connectivity issue referenced was due to failed Azure maintenance procedures whereby DNS certificate in a US Government region failed automatic rotation before expiring. When the service was manually triggered to rotate, that operation failed as well. Azure engineering had to take down a DNS cluster that managed resolution for about 12 services in one capacity or another and manually swap certificates and redeploy the DNS cluster. Azure support are still determining the root cause of the rotation failure on their side and teams are actively working on it. Actions have been taken to increase the availability of DNS across Azure Gov at this time, though the details of that are not public and likely won't be due to security requirements around Azure topology. Whilst the backlog of auto-submits etc. was processing, customers may have experienced some lags on dashboard updates etc. as dashboard update services scaled to work with the immediate increase in dashboard update requests. **Mitigation:** SimpliGov worked with Azure support to determine the cause of the issue and resolve it during several support sessions. Additional capacity was added to expedite auto-submit processing for the backlog and dashboard sync processes were run to ensure that dashboards contained current and accurate data. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward with auto-submits and will be deploying additional updates / architectural changes as part of our upcoming production release scheduled for Jan 2022. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless they were directly notified to do so by the SimpliGov team.
Status: Postmortem
Impact: Minor | Started At: Dec. 13, 2021, 11:29 a.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Major | Started At: Nov. 19, 2021, 10:56 p.m.
Description: The issue has been identified and remediated. All services are operating as expected once again.
Status: Resolved
Impact: Minor | Started At: Nov. 15, 2021, 11:24 p.m.
Description: The issue has been identified and remediated. All services are operating as expected once again.
Status: Resolved
Impact: Major | Started At: Nov. 8, 2021, 9:13 p.m.
Description: **Preliminary Root Cause:** SimpliGov auto-submits rely upon backend submission services hosted on Azure Service Fabric and utilizing Azure Service Bus for processing, queuing etc. The SimpliGov developed submit processing services used to process these autosubmits stalled on previously incomplete auto-submits which were queued for batch processing between 6 AM PST and 7 AM PST \(batching occurred at different times for different customers\). The batch sizes today were larger as a result of processing backlogs from the SimpliGov incident noted on 10/21/2021. During this period of time, intake submissions should have worked as expected for all async processes and most synchronous workflow configurations. In cases where chained auto-submits with zero delay would have provided users with a different user experience than they would be accustomed to. In such cases, users would have been shown a thank you page from the initial relationship rather than the last relationship in the autosubmit chain SimpliGov identified the autosubmit records that caused this issue, removed them from the queue and restarted all affected customer processing services. From 9:15 AM PST, auto-submits began processing as expected and customer backlogs and batches were cleared within ~15 minutes depending on the number of auto-submits used within a customers tenants. The more auto-submits used by a customer, the longer it took for said auto-submit backlog to be processed. Whilst the backlog of auto-submits etc. was processing, customers may have experienced some lags on dashboard updates etc. whilst dashboard update services scaled to work with the immediate increase in dashboard update requests. **Mitigation:** In the immediate term, SimpliGov immediately restarted all backend processing services to remove any potentially hung SQL sessions and identified the problematic hung submissions in the database. After removing such records and restarting the submit servicesm they self-healed and began to process auto-submits as expected. Going forward, SimpliGov has implemented several new features within our upcoming Dec 2021/Jan 2022 Production release allowing for better failover and fault tolerance in such scenarios. SimpliGov will be hosted on Azure Kubernetes Services instead of Azure Service Fabric, split service bus queue architecture will be enabled and additional retry policies have been added to handle scenarios whereby Azure Services return transient errors. These 3 particular items, in addition to smaller individual fixes and improvements should reduce the propencity of such incidents going forward. **Next Steps:** We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward with auto-submits and will be deploying additional updates / architectural changes as part of our upcoming production release scheduled for Jan 2022. All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.
Status: Postmortem
Impact: Major | Started At: Oct. 22, 2021, 3:44 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
