Last checked: a minute ago
Get notified about any outages, downtime or incidents for Virtuoso and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Virtuoso.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
Application & API | Active |
Bridge | Active |
Cross browser and real device testing | Active |
Integrations | Active |
Jobs and bot cluster | Active |
View the latest incidents for Virtuoso and check for official updates:
Description: At 1647 BST on Wed 24 July we completed a routine release of the latest version of Virtuoso to our production environments. Part of this release removed an optional parameter on our web API that wraps responses with informational data \(“Envelopes”\). As part of the work to implement this, we consulted our monitoring and found that the parameter was omitted or set explicitly to false for all requests we had received over a substantial period. Removal of this parameter changed the response structure for requests that did not explicitly set the parameter as it is on by default, rather than off. This did not affect the Virtuoso interface as it disables envelopes. Prior to all releases, the Virtuoso platform runs through a set of unit, integration, and end-to-end UI tests. Several tests related to the Bridge component failed, but it was incorrectly determined that this was an unrelated issue with the infrastructure that runs our test clients, and on that basis we proceeded to release. At 1111 BST on Thu 24 July we were alerted to the issue and had a proposed fix deployed to our test environment at 1231 BST. This successfully fixed the issue, confirmed by running our whole test suite as well as further manual checks and this fix reached the production environment at 1328 BST. We are sorry for the inconvenience caused by the incident. We have concluded our review of the incident and identified improvements to our processes that will safeguard against repeats of this incident. Changes to our API will require additional scrutiny, and our release process will require that any failing test cases require further review and manual confirmation before a release may proceed.
Status: Postmortem
Impact: Major | Started At: July 25, 2024, 10:48 a.m.
Description: At 1647 BST on Wed 24 July we completed a routine release of the latest version of Virtuoso to our production environments. Part of this release removed an optional parameter on our web API that wraps responses with informational data \(“Envelopes”\). As part of the work to implement this, we consulted our monitoring and found that the parameter was omitted or set explicitly to false for all requests we had received over a substantial period. Removal of this parameter changed the response structure for requests that did not explicitly set the parameter as it is on by default, rather than off. This did not affect the Virtuoso interface as it disables envelopes. Prior to all releases, the Virtuoso platform runs through a set of unit, integration, and end-to-end UI tests. Several tests related to the Bridge component failed, but it was incorrectly determined that this was an unrelated issue with the infrastructure that runs our test clients, and on that basis we proceeded to release. At 1111 BST on Thu 24 July we were alerted to the issue and had a proposed fix deployed to our test environment at 1231 BST. This successfully fixed the issue, confirmed by running our whole test suite as well as further manual checks and this fix reached the production environment at 1328 BST. We are sorry for the inconvenience caused by the incident. We have concluded our review of the incident and identified improvements to our processes that will safeguard against repeats of this incident. Changes to our API will require additional scrutiny, and our release process will require that any failing test cases require further review and manual confirmation before a release may proceed.
Status: Postmortem
Impact: Major | Started At: July 25, 2024, 10:48 a.m.
Description: On Tuesday 16 July around 06:40 UTC, during a regular update of the platform on all environments we had a severe service degradation on the EU production environment. Usually this would trigger an automatic rollback of the update, but in this specific circumstance only a few of the services were able to return to the previous version, requiring manual intervention to restore full service availability. Alarms in place immediately warned the team that something unexpected occurred and some services were not being able to initialize the new versions. After analyzing the data available and impact of potential decisions, at 07:20 UTC the team triggered a manual downgrade of the service back to the previously known working version, restoring full availability at 08:10 UTC. We are sorry for the inconvenience caused to our customers affected by this outage. This falls below the standard of operational excellence that we wish to provide to our users. We have implemented and released a product change that hardened the automated capabilities of the platform to recover in the scenario described.
Status: Postmortem
Impact: Major | Started At: July 16, 2024, 6:50 a.m.
Description: On Tuesday 16 July around 06:40 UTC, during a regular update of the platform on all environments we had a severe service degradation on the EU production environment. Usually this would trigger an automatic rollback of the update, but in this specific circumstance only a few of the services were able to return to the previous version, requiring manual intervention to restore full service availability. Alarms in place immediately warned the team that something unexpected occurred and some services were not being able to initialize the new versions. After analyzing the data available and impact of potential decisions, at 07:20 UTC the team triggered a manual downgrade of the service back to the previously known working version, restoring full availability at 08:10 UTC. We are sorry for the inconvenience caused to our customers affected by this outage. This falls below the standard of operational excellence that we wish to provide to our users. We have implemented and released a product change that hardened the automated capabilities of the platform to recover in the scenario described.
Status: Postmortem
Impact: Major | Started At: July 16, 2024, 6:50 a.m.
Description: On Wednesday 15 May at 14:31 in the production environment the Flow integration was deleted in error by an internal team member with elevated privileges. This caused the integration to be removed from all instances, meaning users could not see Flows and jobs did not trigger. At 16:31 the integration was made available again, but this new instance did not contain existing Flows. The team then worked to restore the configurations from a backup. At 19:10 this was completed and configurations were restored for most affected users. Further to this, a small number of customers had some configurations that were later restored following further review on Friday 24 May at 13:45. We are sorry for the inconvenience caused to our customers affected by this outage. This falls below the standard of operational excellence that we wish to provide to our users. We have implemented and released a product change that removes the ability to completely delete integrations when they are installed in organizations.
Status: Postmortem
Impact: Major | Started At: May 15, 2024, 1:54 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.