Company Logo

Is there an ChargeOver outage?

ChargeOver status: Systems Active

Last checked: 6 minutes ago

Get notified about any outages, downtime or incidents for ChargeOver and 1800+ other cloud vendors. Monitor 10 companies, for free.

Subscribe for updates

ChargeOver outages and incidents

Outage and incident data over the last 30 days for ChargeOver.

There have been 0 outages or incidents for ChargeOver in the last 30 days.

Severity Breakdown:

Tired of searching for status updates?

Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!

Sign Up Now

Components and Services Monitored for ChargeOver

Outlogger tracks the status of these components for Xero:

Developer Docs Active
Email Sending Active
Integrations Active
Main Application Active
Payment Processing Active
Public Website Active
Search Active
Component Status
Developer Docs Active
Email Sending Active
Integrations Active
Main Application Active
Payment Processing Active
Public Website Active
Search Active

Latest ChargeOver outages and incidents.

View the latest incidents for ChargeOver and check for official updates:

Updates:

  • Time: July 30, 2024, 3:50 p.m.
    Status: Resolved
    Update: Email delivery is operating normally. A postmortem will follow.
  • Time: July 30, 2024, 12:24 p.m.
    Status: Monitoring
    Update: The email delays have been resolved. We are monitoring delivery of emails. Some users may still experience some delays and/or higher-than-normal bounce rates for a short amount of time. Further updates and a postmortem will follow.
  • Time: July 29, 2024, 8:49 p.m.
    Status: Identified
    Update: We have diverted all email delivery to a secondary email account. Some delivery delays are still occurring. While we are working to resolve this, some emails sent may not be sent with correctly aligned DKIM policies/signatures. We continue to work with our email provider to resolve this.
  • Time: July 29, 2024, 6:38 p.m.
    Status: Identified
    Update: We are diverting some email delivery to a secondary email account. Some delivery delays are still occurring. While we are working to resolve this, some emails sent may not be sent with correctly aligned DKIM policies/signatures.
  • Time: July 29, 2024, 5:27 p.m.
    Status: Identified
    Update: Emails of the following types are being delivered without any delays: * invites sent to new admin users * admin password resets * scheduled reports * any sort of admin notifications (e.g. notifications when a quote is accepted, when custom domains are configured, etc.) We are aware that many of our customers are seeing significant delivery delays for transactional emails (e.g. invoice due emails, payment receipt emails, etc.). We are working with our email provider to get the delays resolved as soon as possible. If your account is using your own SMTP server or your own SendGrid, Mailgun, or Mandrill account, your account is unaffected by the delivery delays.
  • Time: July 29, 2024, 12:38 p.m.
    Status: Identified
    Update: We have identified the problem, and are working with our email provider to resolve it. At this time, we expect all emails to be sent successfully, but some delivery delays may occur.
  • Time: July 28, 2024, 12:36 p.m.
    Status: Identified
    Update: We have identified the problem, and are working with our email provider to resolve it. At this time, we expect all emails to be sent successfully, but some delivery delays may occur.
  • Time: July 28, 2024, 3:09 a.m.
    Status: Investigating
    Update: Some customers are experiencing a delay in delivery of outgoing emails from ChargeOver. Our team is working with our email provider to investigate the delay. At this time, we expect all emails to be sent successfully, but some delivery delays may occur.

Updates:

  • Time: Dec. 5, 2023, 12:58 a.m.
    Status: Resolved
    Update: This issue has been resolved. A postmortem will follow.
  • Time: Dec. 4, 2023, 9:47 p.m.
    Status: Monitoring
    Update: We are continuing to monitor to ensure that all 3rd-party integration syncs catch up.
  • Time: Dec. 4, 2023, 8:14 p.m.
    Status: Monitoring
    Update: 3rd-party integrations sync is continuing to catch up, and now impacting only a small percentage of users. We are continuing to monitor the sync.
  • Time: Dec. 4, 2023, 5:23 p.m.
    Status: Identified
    Update: We have identified the cause of the degraded performance, and are working towards a resolution. Search and email sending is performing normally now. Sync to 3rd-party integrations is catching up, but still delayed.
  • Time: Dec. 4, 2023, 5:05 p.m.
    Status: Investigating
    Update: ChargeOver is experiencing delays when syncing data to 3rd-party platforms (Salesforce, QuickBooks, Xero, etc.) and some delays with search and email sending. We are investigating the delays.

Updates:

  • Time: July 19, 2023, 5:10 p.m.
    Status: Postmortem
    Update: ## Incident details The ChargeOver team follows an agile software development lifecycle for rolling out new features and updates. We routinely roll out new features and updates to the platform multiple times per week. Our typical continuous integration/continuous deployment roll-outs look something like this: * Developers make changes * Code review by other developers * Automated security, linting, and security testing is performed * Senior-level developer code review before deploying features to production * Automated deploy of new updates to production environment * If an error occurs, roll back the changes to a previously known-good configuration ChargeOver uses `Docker` containers and a redundant set of `Docker Swarm` nodes for production deployments. On `July 19th at 10:24am CT` we reviewed and deployed an update which, although passing all automated acceptance tests, errantly deployed an update which caused the deployment of `Docker` container images to silently fail. This caused the application to become unavailable, and a `503 Service Unavailable` error message was shown to all users. The deployment appeared to be successful to automated systems, but due to a syntax error actually only removed existing application servers rather than replacing them with the new software version. No automated roll back occurred, because the deployment appeared successful but had actually failed silently. ## Root cause A single extra space \(a single errant spacebar press!\) was accidentally added to the very beginning of a `docker-compose` `YAML` file, which caused the `docker-compose` file to be invalid `YAML` syntax. The single-space change was subtle enough to be missed when reviewing the code change. All automated tests passed, because the automated tests do not use the production `docker-compose` deployment configuration file. When deploying the service to `Docker Swarm`, `Docker Swarm` interpreted the invalid syntax in the `YAML` file as an empty set of services to deploy, rather than a set of valid application services to be deployed. This caused the deployment to look successful \(it successfully deployed, removing all existing application servers, and replacing them with nothing\) and thus automated roll-back to a known-good set of services did not happen. ## Incident timeline * 10:21am CT - Change was reviewed and merged from a staging branch, to our production branch. * 10:24am CT - Change was deployed to production, immediately causing an outage. * 10:29am CT - Our team posted a status update here, notifying affected customers. * 10:36am CT - Our team identified the related errant change, and started to revert to a known-good set of services. * 11:06am CT - All services became operational again after deploying to last known good configuration. At this time, all services were restored and operational. * 11:09am CT - Our team identified exactly what was wrong - an accidentally added single space character at the beginning of a configuration file, causing the file to be invalid `YAML` syntax. * 11:56am CT - Our team made a change to validate the syntax of the `YAML` configuration files, to ensure this failure scenario cannot happen again. ## Remediation plan There are several things that our team has identified as part of a remediation plan: * We have already deployed multiple checks to ensure that invalid `YAML` syntax and/or configuration errors cannot pass automated tests, and thus cannot reach testing/UAT or production environments. * Our team will work to improve the very generic `503 Service Unavailable` message that customers received, directing affected customers to [https://status.chargeover.com](https://status.chargeover.com) where they can see real-time updates regarding any system outages. * Customers logging in via [https://app.chargeover.com](https://app.chargeover.com) received generic `The credentials that you provided are not correct.` messages, instead of a notification of the outage. This will be improved. * Our team will do a review of our deployment pipelines, to see if we can identify any other similar potential failure points.
  • Time: July 19, 2023, 4:17 p.m.
    Status: Resolved
    Update: The incident has been resolved. A postmortem will be provided.
  • Time: July 19, 2023, 4:12 p.m.
    Status: Monitoring
    Update: All systems have been restored. We are monitoring the fix. A postmortem will be posted.
  • Time: July 19, 2023, 3:52 p.m.
    Status: Identified
    Update: We are continuing to work on a fix for this issue.
  • Time: July 19, 2023, 3:36 p.m.
    Status: Identified
    Update: We have identified the problem. ETA to resolution is less than 30 minutes.
  • Time: July 19, 2023, 3:29 p.m.
    Status: Investigating
    Update: We are aware of the problem, and are investigating. We will post further updates as we have them.

Updates:

  • Time: May 19, 2023, 1:58 a.m.
    Status: Resolved
    Update: This issue has been resolved. Please contact us if you continue to have any trouble.
  • Time: May 19, 2023, 1:50 a.m.
    Status: Monitoring
    Update: A fix has been implemented, and we are monitoring to make sure all services have recovered.
  • Time: May 19, 2023, 1:40 a.m.
    Status: Identified
    Update: We have identified an issue causing trouble logging in, and causing some pages to load incorrectly. ETA to fix is less than an hour.

Updates:

  • Time: May 9, 2023, 4:45 p.m.
    Status: Resolved
    Update: The login issue has been resolved.
  • Time: May 9, 2023, 4:26 p.m.
    Status: Identified
    Update: We have identified the login issue, and are working on a fix. You can continue to log in at https://(your-company-name).chargeover.com/admin/ while we work towards a fix. This issue does NOT affect any scheduled processes. Invoices were created, and payments were run as scheduled, without problem.
  • Time: May 9, 2023, 3:22 p.m.
    Status: Investigating
    Update: Some users are unable to log in at https://app.chargeover.com/ If you are having trouble, you can log in at your account-specific URL immediately instead. For example, log in at https://(your-company-name-here).chargeover.com/admin/ We are investigating, and will provide further updates soon.

Check the status of similar companies and alternatives to ChargeOver

Anaplan
Anaplan

Systems Active

SpotOn
SpotOn

Systems Active

impact.com
impact.com

Systems Active

Replicon
Replicon

Systems Active

Adjust
Adjust

Systems Active

Quantcast
Quantcast

Systems Active

Acoustic
Acoustic

Systems Active

Pantheon Operations
Pantheon Operations

Systems Active

DealerSocket
DealerSocket

Systems Active

Mixpanel
Mixpanel

Systems Active

Invoca
Invoca

Systems Active

Ceros
Ceros

Systems Active

Frequently Asked Questions - ChargeOver

Is there a ChargeOver outage?
The current status of ChargeOver is: Systems Active
Where can I find the official status page of ChargeOver?
The official status page for ChargeOver is here
How can I get notified if ChargeOver is down or experiencing an outage?
To get notified of any status changes to ChargeOver, simply sign up to OutLogger's free monitoring service. OutLogger checks the official status of ChargeOver every few minutes and will notify you of any changes. You can veiw the status of all your cloud vendors in one dashboard. Sign up here
What does ChargeOver do?
Manage recurring billing and subscriptions with ChargeOver software. Increase revenue and improve payment processing efficiency. Start free trial today.