Is there an UpGuard outage?
UpGuard status: Systems Active
Last checked: 2 minutes ago
Get notified about any outages, downtime or incidents for UpGuard and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 2 minutes ago
Get notified about any outages, downtime or incidents for UpGuard and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for UpGuard.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| UpGuard CyberRisk | Active |
| Authentication | Active |
| External API | Active |
| Web App | Active |
View the latest incidents for UpGuard and check for official updates:
Description: # Incident Summary On June 21, 2022 at 06:40 UTC, the UpGuard CyberRisk, Web App & Authentication services experienced a partial outage for 46 minutes. We identified the cause to be related to the critical incident that Cloudflare reported. Some customers attempting to reach UpGuard CyberRisk observed a 500 error within their browser. # Fault Cloudflare suffered an outage that affected traffic in 19 of their data centres which handles a significant proportion of their global traffic. Depending upon your location in the world you may have been unable to access websites and services that rely on Cloudflare. # Detection Internal alerting systems notified internal channels of the service disruption to UpGuard CyberRisk, Web App & Authentication services. # Impact Outage: UpGuard CyberRisk, Web App & Authentication services were unavailable depending upon your location in the world for 46 minutes. There was intermittent performance within the product as a result. # Recovery Cloudflare fix implemented at 07:29 UTC, Cloudflare brought all data centres back online by 07:42 UTC. UpGuard CyberRisk, Web App & Authentication services were accessible. # Timeline June 21, 2022 at 06:27 UTC: Cloudflare incident flagged June 21, 2022 at 06:34 UTC: A critical incident was declared by Cloudflare June 21, 2022 at 06:40 UTC: UpGuard CyberRisk, Web App & Authentication services became inaccessible for some users June 21, 2022 at 06:48 UTC: An incident response group was formed June 21, 2022 at 07:14 UTC: Cloudflare identified the issue June 21, 2022 at 07:26 UTC: UpGuard CyberRisk, Web App & Authentication services became accessible June 21, 2022 at 07:29 UTC: Fix implemented by Cloudflare June 21, 2022 at 07:30 UTC: UpGuard monitoring of fix commenced June 21, 2022 at 07:48 UTC: UpGuard Customer communications sent June 21, 2022 at 08:06 UTC: Incident deemed closed, services restored # Root Cause Our authentication service provider experienced high error rates and timeouts across authentication and management API. Depending upon your location in the world, you may have been unable to access UpGuard CyberRisk as a result of our authentication provider being directly affected by the Cloudflare outage.
Status: Postmortem
Impact: Major | Started At: June 21, 2022, 6:40 a.m.
Description: # Incident Summary On June 21, 2022 at 06:40 UTC, the UpGuard CyberRisk, Web App & Authentication services experienced a partial outage for 46 minutes. We identified the cause to be related to the critical incident that Cloudflare reported. Some customers attempting to reach UpGuard CyberRisk observed a 500 error within their browser. # Fault Cloudflare suffered an outage that affected traffic in 19 of their data centres which handles a significant proportion of their global traffic. Depending upon your location in the world you may have been unable to access websites and services that rely on Cloudflare. # Detection Internal alerting systems notified internal channels of the service disruption to UpGuard CyberRisk, Web App & Authentication services. # Impact Outage: UpGuard CyberRisk, Web App & Authentication services were unavailable depending upon your location in the world for 46 minutes. There was intermittent performance within the product as a result. # Recovery Cloudflare fix implemented at 07:29 UTC, Cloudflare brought all data centres back online by 07:42 UTC. UpGuard CyberRisk, Web App & Authentication services were accessible. # Timeline June 21, 2022 at 06:27 UTC: Cloudflare incident flagged June 21, 2022 at 06:34 UTC: A critical incident was declared by Cloudflare June 21, 2022 at 06:40 UTC: UpGuard CyberRisk, Web App & Authentication services became inaccessible for some users June 21, 2022 at 06:48 UTC: An incident response group was formed June 21, 2022 at 07:14 UTC: Cloudflare identified the issue June 21, 2022 at 07:26 UTC: UpGuard CyberRisk, Web App & Authentication services became accessible June 21, 2022 at 07:29 UTC: Fix implemented by Cloudflare June 21, 2022 at 07:30 UTC: UpGuard monitoring of fix commenced June 21, 2022 at 07:48 UTC: UpGuard Customer communications sent June 21, 2022 at 08:06 UTC: Incident deemed closed, services restored # Root Cause Our authentication service provider experienced high error rates and timeouts across authentication and management API. Depending upon your location in the world, you may have been unable to access UpGuard CyberRisk as a result of our authentication provider being directly affected by the Cloudflare outage.
Status: Postmortem
Impact: Major | Started At: June 21, 2022, 6:40 a.m.
Description: PIR Date: 17th June, 2022 Incident Date: June 6th, 2022 Incident Time: 3:33 UTC Incident Number: INCI-159 Severity Level: 1 - Blocker Affected Services: UpGuard CyberRisk, Web App, External API, Authentication services Outage Duration: 30 Minutes # Incident Summary On Monday, June 6th at 3:33 UTC, the UpGuard CyberRisk, Web App, External API & Authentication services experienced an outage of 30 minutes, and recovery from this outage led to the loss of 18 hours of data affecting <1% of our customers. # Fault A database maintenance task commenced on UpGuard CyberRisk. The production database was incorrectly overwritten, halting access to UpGuard CyberRisk, Web App, External API & Authentication services. # Detection Internal alerting systems notified internal channels immediately of the service disruption across UpGuard CyberRisk, Web App, External API & Authentication services. # Impact 1. Outage: UpGuard CyberRisk, Web App, External API & Authentication services were unavailable for 30 minutes. All performance transactions within the product were halted as a result. 2. Data loss: The database backup restored was from the previous day Sunday, June 5th at 7:00 UTC. Data entered into UpGuard CyberRisk during the previous 18 hours was lost affecting <1% of our customers. # Recovery 1. Due to the low number of transactions, UpGuard CyberRisk, Web App, External API & Authentication services were restored and brought back online with the last available backup from Sunday, June 5th at 7:00 UTC. 2. Analysis was conducted to review changes that occurred between Sunday, June 5th at 7:00 UTC and Monday, June 6th at 3:33 UTC on UpGuard CyberRisk. # Timeline 3:30 UTC: Database maintenance commenced. 3:33 UTC: It was identified that the database maintenance was incorrectly carried out on the production database instead of the test database due to human error which halted access to UpGuard CyberRisk, Web App, External API & Authentication services. 3:45 UTC: An incident response group was formed. 3:55 UTC: A decision was made to restore from the last available full backup provided the low impact of transactions that were executed. 4:03 UTC: UpGuard CyberRisk, Web App, External API & Authentication services were restored from backup data as of Sunday, June 5th at 7:00 UTC was successful and within our Hosted Services Agreement. Data entered into UpGuard CyberRisk during the previous 18 hours was lost affecting <1% of our customers. # Root Cause It was concluded that the root cause was human error, along with insufficient testing and verification of the maintenance work. In addition, the change type \(restoring a database image into a non-production copy\) represents a unique case for our change control procedures and was not classified as a production change. Although performed from the production environment, the change classification as non-production was due to the destination being a non-production copy of the database rather than the source target. # Corrective Actions As a result of this incident, we have analyzed all of the transactions within UpGuard CyberRisk between Sunday, June 5th at 7:00 UTC and Monday, June 6th at 3:33 UTC to notify the customers affected with a description of the data loss. **Effective Immediately:** For this category of change, we will ensure it is aligned with all other types of change that potentially impact our customer and production data. This means that it will follow the formal change control process that requires review, testing, and approval. We will increase the frequency of our backups. We will require a backup before any major change to the production environment. **Targeting completion within 1 month:** We are reviewing all other types of changes that fall outside our regular change control process to verify coverage at the appropriate level of control. For any change that requires any manual process, we will ensure that: 1. A scripted solution is present that allows for review and testing. This will include a scripted and tested database restore function. 2. For changes that cannot be scripted a documented playbook is available that allows for peer-review and testing We will be reviewing our external communications plan for our customers to ensure that the relevant and active users are communicated with and ensuring there is an opt out function.
Status: Postmortem
Impact: Critical | Started At: June 6, 2022, 3:35 a.m.
Description: PIR Date: 17th June, 2022 Incident Date: June 6th, 2022 Incident Time: 3:33 UTC Incident Number: INCI-159 Severity Level: 1 - Blocker Affected Services: UpGuard CyberRisk, Web App, External API, Authentication services Outage Duration: 30 Minutes # Incident Summary On Monday, June 6th at 3:33 UTC, the UpGuard CyberRisk, Web App, External API & Authentication services experienced an outage of 30 minutes, and recovery from this outage led to the loss of 18 hours of data affecting <1% of our customers. # Fault A database maintenance task commenced on UpGuard CyberRisk. The production database was incorrectly overwritten, halting access to UpGuard CyberRisk, Web App, External API & Authentication services. # Detection Internal alerting systems notified internal channels immediately of the service disruption across UpGuard CyberRisk, Web App, External API & Authentication services. # Impact 1. Outage: UpGuard CyberRisk, Web App, External API & Authentication services were unavailable for 30 minutes. All performance transactions within the product were halted as a result. 2. Data loss: The database backup restored was from the previous day Sunday, June 5th at 7:00 UTC. Data entered into UpGuard CyberRisk during the previous 18 hours was lost affecting <1% of our customers. # Recovery 1. Due to the low number of transactions, UpGuard CyberRisk, Web App, External API & Authentication services were restored and brought back online with the last available backup from Sunday, June 5th at 7:00 UTC. 2. Analysis was conducted to review changes that occurred between Sunday, June 5th at 7:00 UTC and Monday, June 6th at 3:33 UTC on UpGuard CyberRisk. # Timeline 3:30 UTC: Database maintenance commenced. 3:33 UTC: It was identified that the database maintenance was incorrectly carried out on the production database instead of the test database due to human error which halted access to UpGuard CyberRisk, Web App, External API & Authentication services. 3:45 UTC: An incident response group was formed. 3:55 UTC: A decision was made to restore from the last available full backup provided the low impact of transactions that were executed. 4:03 UTC: UpGuard CyberRisk, Web App, External API & Authentication services were restored from backup data as of Sunday, June 5th at 7:00 UTC was successful and within our Hosted Services Agreement. Data entered into UpGuard CyberRisk during the previous 18 hours was lost affecting <1% of our customers. # Root Cause It was concluded that the root cause was human error, along with insufficient testing and verification of the maintenance work. In addition, the change type \(restoring a database image into a non-production copy\) represents a unique case for our change control procedures and was not classified as a production change. Although performed from the production environment, the change classification as non-production was due to the destination being a non-production copy of the database rather than the source target. # Corrective Actions As a result of this incident, we have analyzed all of the transactions within UpGuard CyberRisk between Sunday, June 5th at 7:00 UTC and Monday, June 6th at 3:33 UTC to notify the customers affected with a description of the data loss. **Effective Immediately:** For this category of change, we will ensure it is aligned with all other types of change that potentially impact our customer and production data. This means that it will follow the formal change control process that requires review, testing, and approval. We will increase the frequency of our backups. We will require a backup before any major change to the production environment. **Targeting completion within 1 month:** We are reviewing all other types of changes that fall outside our regular change control process to verify coverage at the appropriate level of control. For any change that requires any manual process, we will ensure that: 1. A scripted solution is present that allows for review and testing. This will include a scripted and tested database restore function. 2. For changes that cannot be scripted a documented playbook is available that allows for peer-review and testing We will be reviewing our external communications plan for our customers to ensure that the relevant and active users are communicated with and ensuring there is an opt out function.
Status: Postmortem
Impact: Critical | Started At: June 6, 2022, 3:35 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
