Is there an Virtuoso outage?
Virtuoso status: Systems Active
Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for Virtuoso and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for Virtuoso and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Virtuoso.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Application & API | Active |
| Bridge | Active |
| Cross browser and real device testing | Active |
| Integrations | Active |
| Jobs and bot cluster | Active |
View the latest incidents for Virtuoso and check for official updates:
Description: The platform has cleared the backlog and queue times have returned to normal.
Status: Resolved
Impact: Minor | Started At: Oct. 23, 2024, 10:01 a.m.
Description: Between Wednesday 9th 2pm and Thursday 10th 1pm we had a partial outage on the job submission system. A small number of requests to run journeys did not progress and remained pending. Affected jobs will remain in the "pending" state with no journeys visible in the execution results. These will automatically time out, and will not execute. We advise customers to retry any jobs that are in this state. We have applied mitigations and are monitoring the platform closely as we work on further mitigations.
Status: Resolved
Impact: None | Started At: Oct. 9, 2024, 1 p.m.
Description: At 1647 BST on Wed 24 July we completed a routine release of the latest version of Virtuoso to our production environments. Part of this release removed an optional parameter on our web API that wraps responses with informational data \(“Envelopes”\). As part of the work to implement this, we consulted our monitoring and found that the parameter was omitted or set explicitly to false for all requests we had received over a substantial period. Removal of this parameter changed the response structure for requests that did not explicitly set the parameter as it is on by default, rather than off. This did not affect the Virtuoso interface as it disables envelopes. Prior to all releases, the Virtuoso platform runs through a set of unit, integration, and end-to-end UI tests. Several tests related to the Bridge component failed, but it was incorrectly determined that this was an unrelated issue with the infrastructure that runs our test clients, and on that basis we proceeded to release. At 1111 BST on Thu 24 July we were alerted to the issue and had a proposed fix deployed to our test environment at 1231 BST. This successfully fixed the issue, confirmed by running our whole test suite as well as further manual checks and this fix reached the production environment at 1328 BST. We are sorry for the inconvenience caused by the incident. We have concluded our review of the incident and identified improvements to our processes that will safeguard against repeats of this incident. Changes to our API will require additional scrutiny, and our release process will require that any failing test cases require further review and manual confirmation before a release may proceed.
Status: Postmortem
Impact: Major | Started At: July 25, 2024, 10:48 a.m.
Description: At 1647 BST on Wed 24 July we completed a routine release of the latest version of Virtuoso to our production environments. Part of this release removed an optional parameter on our web API that wraps responses with informational data \(“Envelopes”\). As part of the work to implement this, we consulted our monitoring and found that the parameter was omitted or set explicitly to false for all requests we had received over a substantial period. Removal of this parameter changed the response structure for requests that did not explicitly set the parameter as it is on by default, rather than off. This did not affect the Virtuoso interface as it disables envelopes. Prior to all releases, the Virtuoso platform runs through a set of unit, integration, and end-to-end UI tests. Several tests related to the Bridge component failed, but it was incorrectly determined that this was an unrelated issue with the infrastructure that runs our test clients, and on that basis we proceeded to release. At 1111 BST on Thu 24 July we were alerted to the issue and had a proposed fix deployed to our test environment at 1231 BST. This successfully fixed the issue, confirmed by running our whole test suite as well as further manual checks and this fix reached the production environment at 1328 BST. We are sorry for the inconvenience caused by the incident. We have concluded our review of the incident and identified improvements to our processes that will safeguard against repeats of this incident. Changes to our API will require additional scrutiny, and our release process will require that any failing test cases require further review and manual confirmation before a release may proceed.
Status: Postmortem
Impact: Major | Started At: July 25, 2024, 10:48 a.m.
Description: On Tuesday 16 July around 06:40 UTC, during a regular update of the platform on all environments we had a severe service degradation on the EU production environment. Usually this would trigger an automatic rollback of the update, but in this specific circumstance only a few of the services were able to return to the previous version, requiring manual intervention to restore full service availability. Alarms in place immediately warned the team that something unexpected occurred and some services were not being able to initialize the new versions. After analyzing the data available and impact of potential decisions, at 07:20 UTC the team triggered a manual downgrade of the service back to the previously known working version, restoring full availability at 08:10 UTC. We are sorry for the inconvenience caused to our customers affected by this outage. This falls below the standard of operational excellence that we wish to provide to our users. We have implemented and released a product change that hardened the automated capabilities of the platform to recover in the scenario described.
Status: Postmortem
Impact: Major | Started At: July 16, 2024, 6:50 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
