Time: Nov. 18, 2024, 10:07 p.m.

Status: Resolved

Update: This incident has been resolved. A post-mortem containing further details to come.

Time: Nov. 18, 2024, 9:58 p.m.

Status: Identified

Update: We've identified the problem and are implementing a fix.

Time: Nov. 18, 2024, 9:44 p.m.

Status: Investigating

Update: We are aware of ongoing issues with access to our platform. We are currently investigating and will release details here as they become available.

Time: Nov. 12, 2024, 6:10 p.m.

Status: Resolved

Update: All connectivity issues are resolved now. Our primary data center (Flexential - Chaska) experienced a networking issue which impacted the ability to connect to ChargeOver services for some customers.

Time: Nov. 12, 2024, 5:50 p.m.

Status: Monitoring

Update: All ChargeOver services are operating normally, and are accessible to everyone now. A networking or connectivity incident occurred with our data center provider (Flexential) which prevented some customers from accessing ChargeOver. We are still waiting for full details, and then will post an update. We continue to monitor connectivity. We believe any impact is now resolved, and everyone should be able to access ChargeOver at this time. An update will follow with more details.

Time: Nov. 12, 2024, 5:35 p.m.

Status: Investigating

Update: All ChargeOver services are operating normally. Some customers are having trouble accessing ChargeOver. At this time, we believe this is due to a carrier network routing issue outside of ChargeOver. We are still investigating and working with our data center providers to get more information.

Time: Nov. 12, 2024, 5:05 p.m.

Status: Investigating

Update: We are receiving reports that some customers are having trouble accessing the ChargeOver app. We are investigating further. All ChargeOver services appear to be operating normally, but it looks like some web traffic may be having problems accessing ChargeOver services.

Time: July 30, 2024, 3:50 p.m.

Status: Resolved

Update: Email delivery is operating normally. A postmortem will follow.

Time: July 30, 2024, 12:24 p.m.

Status: Monitoring

Update: The email delays have been resolved. We are monitoring delivery of emails. Some users may still experience some delays and/or higher-than-normal bounce rates for a short amount of time. Further updates and a postmortem will follow.

Time: July 29, 2024, 8:49 p.m.

Status: Identified

Update: We have diverted all email delivery to a secondary email account. Some delivery delays are still occurring. While we are working to resolve this, some emails sent may not be sent with correctly aligned DKIM policies/signatures. We continue to work with our email provider to resolve this.

Time: July 29, 2024, 6:38 p.m.

Status: Identified

Update: We are diverting some email delivery to a secondary email account. Some delivery delays are still occurring. While we are working to resolve this, some emails sent may not be sent with correctly aligned DKIM policies/signatures.

Time: July 29, 2024, 5:27 p.m.

Status: Identified

Update: Emails of the following types are being delivered without any delays: * invites sent to new admin users * admin password resets * scheduled reports * any sort of admin notifications (e.g. notifications when a quote is accepted, when custom domains are configured, etc.) We are aware that many of our customers are seeing significant delivery delays for transactional emails (e.g. invoice due emails, payment receipt emails, etc.). We are working with our email provider to get the delays resolved as soon as possible. If your account is using your own SMTP server or your own SendGrid, Mailgun, or Mandrill account, your account is unaffected by the delivery delays.

Time: July 29, 2024, 12:38 p.m.

Status: Identified

Update: We have identified the problem, and are working with our email provider to resolve it. At this time, we expect all emails to be sent successfully, but some delivery delays may occur.

Time: July 28, 2024, 12:36 p.m.

Status: Identified

Update: We have identified the problem, and are working with our email provider to resolve it. At this time, we expect all emails to be sent successfully, but some delivery delays may occur.

Time: July 28, 2024, 3:09 a.m.

Status: Investigating

Update: Some customers are experiencing a delay in delivery of outgoing emails from ChargeOver. Our team is working with our email provider to investigate the delay. At this time, we expect all emails to be sent successfully, but some delivery delays may occur.

Time: Dec. 5, 2023, 12:58 a.m.

Status: Resolved

Update: This issue has been resolved. A postmortem will follow.

Time: Dec. 4, 2023, 9:47 p.m.

Status: Monitoring

Update: We are continuing to monitor to ensure that all 3rd-party integration syncs catch up.

Time: Dec. 4, 2023, 8:14 p.m.

Status: Monitoring

Update: 3rd-party integrations sync is continuing to catch up, and now impacting only a small percentage of users. We are continuing to monitor the sync.

Time: Dec. 4, 2023, 5:23 p.m.

Status: Identified

Update: We have identified the cause of the degraded performance, and are working towards a resolution. Search and email sending is performing normally now. Sync to 3rd-party integrations is catching up, but still delayed.

Time: Dec. 4, 2023, 5:05 p.m.

Status: Investigating

Update: ChargeOver is experiencing delays when syncing data to 3rd-party platforms (Salesforce, QuickBooks, Xero, etc.) and some delays with search and email sending. We are investigating the delays.

Time: July 19, 2023, 5:10 p.m.

Status: Postmortem

Update: ## Incident details The ChargeOver team follows an agile software development lifecycle for rolling out new features and updates. We routinely roll out new features and updates to the platform multiple times per week. Our typical continuous integration/continuous deployment roll-outs look something like this: * Developers make changes * Code review by other developers * Automated security, linting, and security testing is performed * Senior-level developer code review before deploying features to production * Automated deploy of new updates to production environment * If an error occurs, roll back the changes to a previously known-good configuration ChargeOver uses `Docker` containers and a redundant set of `Docker Swarm` nodes for production deployments. On `July 19th at 10:24am CT` we reviewed and deployed an update which, although passing all automated acceptance tests, errantly deployed an update which caused the deployment of `Docker` container images to silently fail. This caused the application to become unavailable, and a `503 Service Unavailable` error message was shown to all users. The deployment appeared to be successful to automated systems, but due to a syntax error actually only removed existing application servers rather than replacing them with the new software version. No automated roll back occurred, because the deployment appeared successful but had actually failed silently. ## Root cause A single extra space \(a single errant spacebar press!\) was accidentally added to the very beginning of a `docker-compose` `YAML` file, which caused the `docker-compose` file to be invalid `YAML` syntax. The single-space change was subtle enough to be missed when reviewing the code change. All automated tests passed, because the automated tests do not use the production `docker-compose` deployment configuration file. When deploying the service to `Docker Swarm`, `Docker Swarm` interpreted the invalid syntax in the `YAML` file as an empty set of services to deploy, rather than a set of valid application services to be deployed. This caused the deployment to look successful \(it successfully deployed, removing all existing application servers, and replacing them with nothing\) and thus automated roll-back to a known-good set of services did not happen. ## Incident timeline * 10:21am CT - Change was reviewed and merged from a staging branch, to our production branch. * 10:24am CT - Change was deployed to production, immediately causing an outage. * 10:29am CT - Our team posted a status update here, notifying affected customers. * 10:36am CT - Our team identified the related errant change, and started to revert to a known-good set of services. * 11:06am CT - All services became operational again after deploying to last known good configuration. At this time, all services were restored and operational. * 11:09am CT - Our team identified exactly what was wrong - an accidentally added single space character at the beginning of a configuration file, causing the file to be invalid `YAML` syntax. * 11:56am CT - Our team made a change to validate the syntax of the `YAML` configuration files, to ensure this failure scenario cannot happen again. ## Remediation plan There are several things that our team has identified as part of a remediation plan: * We have already deployed multiple checks to ensure that invalid `YAML` syntax and/or configuration errors cannot pass automated tests, and thus cannot reach testing/UAT or production environments. * Our team will work to improve the very generic `503 Service Unavailable` message that customers received, directing affected customers to [https://status.chargeover.com](https://status.chargeover.com) where they can see real-time updates regarding any system outages. * Customers logging in via [https://app.chargeover.com](https://app.chargeover.com) received generic `The credentials that you provided are not correct.` messages, instead of a notification of the outage. This will be improved. * Our team will do a review of our deployment pipelines, to see if we can identify any other similar potential failure points.

Time: July 19, 2023, 4:17 p.m.

Status: Resolved

Update: The incident has been resolved. A postmortem will be provided.

Time: July 19, 2023, 4:12 p.m.

Status: Monitoring

Update: All systems have been restored. We are monitoring the fix. A postmortem will be posted.

Time: July 19, 2023, 3:52 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: July 19, 2023, 3:36 p.m.

Status: Identified

Update: We have identified the problem. ETA to resolution is less than 30 minutes.

Time: July 19, 2023, 3:29 p.m.

Status: Investigating

Update: We are aware of the problem, and are investigating. We will post further updates as we have them.