Is there an ChargeOver outage?
ChargeOver status: Systems Active
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for ChargeOver and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for ChargeOver and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for ChargeOver.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Developer Docs | Active |
| Email Sending | Active |
| Integrations | Active |
| Main Application | Active |
| Payment Processing | Active |
| Public Website | Active |
| Search | Active |
View the latest incidents for ChargeOver and check for official updates:
Description: This issue has been resolved. Please contact us if you continue to have any trouble.
Status: Resolved
Impact: Major | Started At: May 19, 2023, 1:40 a.m.
Description: The login issue has been resolved.
Status: Resolved
Impact: Minor | Started At: May 9, 2023, 3:22 p.m.
Description: ChargeOver experienced a DDoS (distributed denial of service) attack from a malicious actor using a large number of IP addresses and Digital Ocean ( https://www.digitalocean.com/ ) servers. An extremely large number of HTTP requests (hundreds of thousands of requests in just a few minutes) were directed at our servers in a very short amount of time, and this degraded the performance of ChargeOver's systems enough to make ChargeOver unavailable or slow for some users, for approximately 30 minutes. ChargeOver staff were immediately alerted of the attack, and responded by deploying firewall rules to block the attack. No intrusions made, no vulnerabilities were exploited, no data loss occurred, and scheduled payment processing was unaffected. ChargeOver is already in process of deploying additional DDoS mitigation tools into our network to mitigate against any future DDoS attacks.
Status: Resolved
Impact: None | Started At: Oct. 4, 2022, 8:30 p.m.
Description: At this time all scheduled payments have been processed, emails sent, and the platform is operating normally. We are investigating root cause. If you continue to experience any issues, please contact our support team via email at [email protected]
Status: Resolved
Impact: Minor | Started At: Aug. 1, 2022, 7 p.m.
Description: ## Root cause * A hardware failure in ChargeOver’s storage system caused severely degraded performance. * Automatic failover to redundant systems did not occur, which resulted in ChargeOver being completely unavailable for approximately 2 hours. ## Timeline * 06:10 CT on Jan 1 2022 - ChargeOver staff are alerted to abnormally slow invoice generation / payment processing times, and start investigating immediately. * 07:27 CT on Jan 1 2022 - Data storage for company logos is identified as causing performance impacts. * 08:16 CT on Jan 1 2022 - A hardware component of ChargeOver’s data storage system fails. Despite hardware redundancy being in place, the failure impacted other components and stops invoice generation / payment processing. Other components of ChargeOver remain operational. * 11:30 CT on Jan 1 2022 - Further hardware failures when trying to recover from the initial failure. Fail-over to the in-place redundant systems does not occur automatically, which causes ChargeOver to become entirely unavailable. * 13:15 CT on Jan 1 2022 - Access to most ChargeOver systems is restored. * 13:30 CT on Jan 1 2022 - ChargeOver staff immediately begin work on ensuring that invoice generation delays / payment processing delays are remediated. * 00:15 CT on Jan 2 2022 - ChargeOver staff notify affected customers of the need to check / update logos on their invoices. * Continued work - ChargeOver staff continue to check to ensure any data needing sync to third-party systems does get synced. ## Impact The following systems were known to be impacted by this failure: * **Some company logos may be unavailable - all ChargeOver customers should check to ensure their company logo is showing up correctly on their invoices / payment receipts.** * **ChargeOver, and all related components \(website, app, REST API, sync to external systems, customer portal, etc.\) was unavailable from approximately 11:30 CT to 13:30 CT on Jan 1 2022.** * Approximately 40% of ChargeOver customers experienced a delay in generating invoices. * Approximately 40% of ChargeOver customers experience a delay in processing payments. * Syncing of data to third-party integrations \(QuickBooks, Xero, Salesforce, etc.\) was delayed \(and in some limited cases, the sync to these systems may still be delayed as of Jan 3\). * Some emails \(invoice due notices, payment receipts\) may have failed to send. * Some webhooks may have failed to send. * Dunning / retry processes may not have run on Jan 1 and Jan 2 for a small subset of ChargeOver customers. ## Future planning ChargeOver recognizes the impact to our customers that this incident has caused. At this time we are assessing what needs to change to make the storage systems that failed more tolerant of failures, so that this sort of incident cannot be repeated.
Status: Postmortem
Impact: Critical | Started At: Jan. 1, 2022, 3:23 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
