Admiral outages and incidents
Outage and incident data over the last 30 days for Admiral.
There have been 0 outages or incidents for Admiral in the last 30 days.
Severity Breakdown:
Tired of searching for status updates?
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowComponents and Services Monitored for Admiral
Outlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Consent | Active |
| Engage | Active |
| Measure | Active |
| Publisher Dashboard | Active |
| Reports | Active |
| Transact | Active |
Latest Admiral outages and incidents.
View the latest incidents for Admiral and check for official updates:
Dashboard Inaccessible
Description: ## Overview Between 13:29 ET and 13:51 ET on March 27, 2024, the Admiral Dashboard was inaccessible and customers saw an error message. ## Incident Summary During the timeframe customers trying to login or otherwise access the Admiral Dashboard were shown an error message. No workaround was identified. All of these issues were simultaneously corrected. ## Root Cause Analysis A misbehaving service led to instability across multiple servers that host the Dashboard. Since the instability affected all relevant servers a user-facing error message was served. ## Mitigation Steps Once the misbehaving service was identified as the cause the quickest resolution was to restart the relevant servers. As servers came back online the Dashboard quickly recovered. ## Preventive Measures Admiral will continue to improve redundancies surrounding our Dashboard and reevaluate our service architecture. We have already deployed a mitigation step and have outlined future steps to further harden our architecture.
Status: Postmortem
Impact: Critical | Started At: March 27, 2024, 5:46 p.m.
Updates:
-
Time: April 8, 2024, 3:35 p.m.Status: PostmortemUpdate: ## Overview Between 13:29 ET and 13:51 ET on March 27, 2024, the Admiral Dashboard was inaccessible and customers saw an error message. ## Incident Summary During the timeframe customers trying to login or otherwise access the Admiral Dashboard were shown an error message. No workaround was identified. All of these issues were simultaneously corrected. ## Root Cause Analysis A misbehaving service led to instability across multiple servers that host the Dashboard. Since the instability affected all relevant servers a user-facing error message was served. ## Mitigation Steps Once the misbehaving service was identified as the cause the quickest resolution was to restart the relevant servers. As servers came back online the Dashboard quickly recovered. ## Preventive Measures Admiral will continue to improve redundancies surrounding our Dashboard and reevaluate our service architecture. We have already deployed a mitigation step and have outlined future steps to further harden our architecture.
-
Time: March 27, 2024, 6:12 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: March 27, 2024, 5:54 p.m.Status: MonitoringUpdate: A fix has been implemented and we are monitoring the results.
-
Time: March 27, 2024, 5:51 p.m.Status: IdentifiedUpdate: The issue has been identified and a fix is being implemented.
-
Time: March 27, 2024, 5:46 p.m.Status: InvestigatingUpdate: We are currently investigating this issue.
Subscription site showing error
Description: ## Overview Between 9:56 ET and 10:34 ET on October 18, 2023, Engage adblock instructions, VRMBot, and the subscription management site experienced complete outages worldwide. ## Incident Summary During the timeframe visitors trying to subscribe or login were shown an error message. Visitors trying to load adblock disable instructions were also shown an error. Finally, for sites that have VRMBot enabled, it did not load correctly. All of these issues were simultaneously corrected. ## Root Cause Analysis A configuration change was deployed as part of routine maintenance led to the unavailability of part of our web serving platform. Admiral separates the serving web properties which limited the blast radius to only a few services. ## Mitigation Steps Once the change was identified as the cause it was corrected and re-deployed to the relevant infrastructure. The corrected deployment started at 10:24 ET and finished at 10:34 ET. ## Preventive Measures Admiral will continue to improve our deployment tooling and early-warning systems to catch these types of issues. We will continue migrating away from our legacy web serving infrastructure to improve consistency and reliability. Finally, we are reviewing our internal processes and will be adjusting them to better prevent cross-region outages going forward.
Status: Postmortem
Impact: Critical | Started At: Oct. 18, 2023, 2:05 p.m.
Updates:
-
Time: Oct. 23, 2023, 3:59 p.m.Status: PostmortemUpdate: ## Overview Between 9:56 ET and 10:34 ET on October 18, 2023, Engage adblock instructions, VRMBot, and the subscription management site experienced complete outages worldwide. ## Incident Summary During the timeframe visitors trying to subscribe or login were shown an error message. Visitors trying to load adblock disable instructions were also shown an error. Finally, for sites that have VRMBot enabled, it did not load correctly. All of these issues were simultaneously corrected. ## Root Cause Analysis A configuration change was deployed as part of routine maintenance led to the unavailability of part of our web serving platform. Admiral separates the serving web properties which limited the blast radius to only a few services. ## Mitigation Steps Once the change was identified as the cause it was corrected and re-deployed to the relevant infrastructure. The corrected deployment started at 10:24 ET and finished at 10:34 ET. ## Preventive Measures Admiral will continue to improve our deployment tooling and early-warning systems to catch these types of issues. We will continue migrating away from our legacy web serving infrastructure to improve consistency and reliability. Finally, we are reviewing our internal processes and will be adjusting them to better prevent cross-region outages going forward.
-
Time: Oct. 18, 2023, 2:53 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: Oct. 18, 2023, 2:30 p.m.Status: MonitoringUpdate: A fix has been implemented and we are monitoring the results.
-
Time: Oct. 18, 2023, 2:23 p.m.Status: IdentifiedUpdate: The issue has been identified and a fix is being implemented.
-
Time: Oct. 18, 2023, 2:05 p.m.Status: InvestigatingUpdate: We are currently investigating this issue.
Increased errors in certain regions
Description: ### Overview Between 9:48 ET and 14:40 ET on August 8, 2023, Admiral experienced degraded targeting capabilities leading to the unavailability of Measure, Engage, and Consent products for some visitors located in the Eastern United States of America. ### Incident Summary During the specified timeframe, for visitors located in the Eastern USA approximately 15% of requests to record Measure data failed and approximately 33% of of requests to target Engages and target Consents. When a targeting request fails, no Engage or Consent product is shown to the visitor. Admiral purposefully disabled targeting in the affected region from 12:23 ET to 12:39 ET to help recovery efforts. The cause of this incident was identified to be a an overload of a database cluster. ### Root Cause Analysis The root cause of the outage was a spike in abnormal traffic patterns that led to an overloaded database cluster. Admiral has found no evidence of unauthorized access and the traffic appears to be non-malicious in nature. ### Mitigation Steps Admiral operates many separate database clusters to reduce the impact of any individual cluster. Automated mitigations and configuration changes were made to return the cluster to normal operational status. ### Preventive Measures Admiral will continue to invest in our software to ensure platform stability despite all forms of traffic. We’ve identified several changes to targeting to further improve resiliency and performance. Additionally, we will be focused on improvements to further isolate Consent from many types of outages, ensuring compliance availability.
Status: Postmortem
Impact: Minor | Started At: Aug. 8, 2023, 4:05 p.m.
Updates:
-
Time: Aug. 16, 2023, 7:30 p.m.Status: PostmortemUpdate: ### Overview Between 9:48 ET and 14:40 ET on August 8, 2023, Admiral experienced degraded targeting capabilities leading to the unavailability of Measure, Engage, and Consent products for some visitors located in the Eastern United States of America. ### Incident Summary During the specified timeframe, for visitors located in the Eastern USA approximately 15% of requests to record Measure data failed and approximately 33% of of requests to target Engages and target Consents. When a targeting request fails, no Engage or Consent product is shown to the visitor. Admiral purposefully disabled targeting in the affected region from 12:23 ET to 12:39 ET to help recovery efforts. The cause of this incident was identified to be a an overload of a database cluster. ### Root Cause Analysis The root cause of the outage was a spike in abnormal traffic patterns that led to an overloaded database cluster. Admiral has found no evidence of unauthorized access and the traffic appears to be non-malicious in nature. ### Mitigation Steps Admiral operates many separate database clusters to reduce the impact of any individual cluster. Automated mitigations and configuration changes were made to return the cluster to normal operational status. ### Preventive Measures Admiral will continue to invest in our software to ensure platform stability despite all forms of traffic. We’ve identified several changes to targeting to further improve resiliency and performance. Additionally, we will be focused on improvements to further isolate Consent from many types of outages, ensuring compliance availability.
-
Time: Aug. 8, 2023, 4:56 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: Aug. 8, 2023, 4:45 p.m.Status: MonitoringUpdate: We have deployed a fix for the underlying cause to the affected regions.
-
Time: Aug. 8, 2023, 4:18 p.m.Status: IdentifiedUpdate: We will temporarily be disabling Engages and Consent modals in the regions affected.
-
Time: Aug. 8, 2023, 4:05 p.m.Status: IdentifiedUpdate: We've identified an issue causing increased errors when recording Measure data and targeting Engages in certain regions. Our team is actively working on resolving the issue.
Consent Engagements not Showing
Description: ### Overview Between 12:16 ET and 13:56 ET on May 18, 2023, Admiral experienced an outage with European Consent, which resulted in the failure to convey or prompt for consent for users subject to GDPR. ### Incident Summary During the specified timeframe, the TCF JS API would have not returned any consent and visitors would not have been prompted to give consent if they lived in a country the publisher deemed applicable to GDPR. The root cause of this outage was identified as a misconfiguration in a file, the Global Vendor List \(GVL\), provided by the Interactive Advertising Bureau \(IAB\). Admiral, along with other Consent Management Platforms \(CMPs\), relies on this configuration file to maintain a list of third-party vendors compliant with GDPR. This configuration file was periodically refreshed and served to visitors through Admiral’s bundled JavaScript \(JS\) file. ### Root Cause Analysis The root cause of the outage was a misconfiguration in the GVL causing an unhandled JS error, which disrupted the proper functioning of Admiral's Consent product. The exact reason for the misconfiguration is currently being investigated and will be addressed in collaboration with the IAB. ### Mitigation Steps To address the misconfiguration issue and mitigate the impact of the outage, Admiral has temporarily suspended the periodic, automatic, updates of the GVL. This decision was made to allow for a thorough evaluation of alternative methods to deliver an up-to-date list to visitors. ### Preventive Measures Admiral deeply regrets any disruptions caused by this incident and is committed to preventing similar failures in the future. We are working internally and with the IAB to add additional safeguards. Going forward, publishers can expect our JS will only contain GVL versions verified through our change control process. The outage highlighted the need for publishers and vendors relying on consent to include appropriate timeouts and safeguards when consent cannot be obtained. In the event of similar failures, it is crucial to have fallback mechanisms in place to assume a non-consenting visitor. Admiral recommends all affected parties to review their own consent handling processes and consider implementing safeguards to minimize interruptions.
Status: Postmortem
Impact: Major | Started At: May 18, 2023, 4:48 p.m.
Updates:
-
Time: May 22, 2023, 4:06 p.m.Status: PostmortemUpdate: ### Overview Between 12:16 ET and 13:56 ET on May 18, 2023, Admiral experienced an outage with European Consent, which resulted in the failure to convey or prompt for consent for users subject to GDPR. ### Incident Summary During the specified timeframe, the TCF JS API would have not returned any consent and visitors would not have been prompted to give consent if they lived in a country the publisher deemed applicable to GDPR. The root cause of this outage was identified as a misconfiguration in a file, the Global Vendor List \(GVL\), provided by the Interactive Advertising Bureau \(IAB\). Admiral, along with other Consent Management Platforms \(CMPs\), relies on this configuration file to maintain a list of third-party vendors compliant with GDPR. This configuration file was periodically refreshed and served to visitors through Admiral’s bundled JavaScript \(JS\) file. ### Root Cause Analysis The root cause of the outage was a misconfiguration in the GVL causing an unhandled JS error, which disrupted the proper functioning of Admiral's Consent product. The exact reason for the misconfiguration is currently being investigated and will be addressed in collaboration with the IAB. ### Mitigation Steps To address the misconfiguration issue and mitigate the impact of the outage, Admiral has temporarily suspended the periodic, automatic, updates of the GVL. This decision was made to allow for a thorough evaluation of alternative methods to deliver an up-to-date list to visitors. ### Preventive Measures Admiral deeply regrets any disruptions caused by this incident and is committed to preventing similar failures in the future. We are working internally and with the IAB to add additional safeguards. Going forward, publishers can expect our JS will only contain GVL versions verified through our change control process. The outage highlighted the need for publishers and vendors relying on consent to include appropriate timeouts and safeguards when consent cannot be obtained. In the event of similar failures, it is crucial to have fallback mechanisms in place to assume a non-consenting visitor. Admiral recommends all affected parties to review their own consent handling processes and consider implementing safeguards to minimize interruptions.
-
Time: May 18, 2023, 6:28 p.m.Status: ResolvedUpdate: This incident has been resolved.
-
Time: May 18, 2023, 5:56 p.m.Status: MonitoringUpdate: We've implemented a fix and we're monitoring as it rolls out to all users.
-
Time: May 18, 2023, 5:10 p.m.Status: IdentifiedUpdate: We've identified the issue with a vendor and we are working to remediate the issue.
-
Time: May 18, 2023, 4:48 p.m.Status: InvestigatingUpdate: We are currently investigating Consent Engagements not showing for most users.
Engagements not Showing up for Some Visitors
Description: We've confirmed that the incident is resolved.
Status: Resolved
Impact: Minor | Started At: April 24, 2023, 8:59 p.m.
Updates:
-
Time: April 24, 2023, 10:54 p.m.Status: ResolvedUpdate: We've confirmed that the incident is resolved.
-
Time: April 24, 2023, 10:27 p.m.Status: MonitoringUpdate: All components have returned to normal operation over the last 20 minutes. We are continuing to monitor.
-
Time: April 24, 2023, 9:55 p.m.Status: IdentifiedUpdate: We've identified an issue with an upstream provider and are working with them to resolve the issue.
