Time: June 28, 2021, 8:55 p.m.

Status: Resolved

Update: This has been resolved.

Time: June 28, 2021, 8:36 p.m.

Status: Monitoring

Update: All syncing delays should be resolved now. We are monitoring to make sure all data has synced.

Time: June 28, 2021, 4:31 p.m.

Status: Identified

Update: ChargeOver is experiencing delays syncing some data to QuickBooks / Xero / Salesforce / some other integrations. Data WILL sync, but will be delayed. We are are working on a resolution.

Time: Sept. 10, 2020, 1:31 p.m.

Status: Postmortem

Update: ChargeOver primarily uses the MariaDB database for data storage. The MariaDB process crashed on our primary database server at 11:09 CST. The process crashed with the following error message which is still being investigated: `InnoDB: Failing assertion: templ->clust_rec_field_no != ULINT_UNDEFINED` ChargeOver staff were immediately notified, and we began to investigate the issue. The database was automatically restarted immediately, and the database began to run automated data integrity checks to ensure that no data was lost, and no data was corrupted before beginning to service requests again. Although we had the ability to fail-over to a secondary database server, a decision was made to let the process complete, as the estimated downtime was very short. This automated data integrity check took much longer than originally expected. Our estimated time to recovery was less than an hour, and instead the database server took approximately 3 hours to do integrity checks and restart safely. The data integrity checks took from 11:09 CST to 14:49 CST. After the data checks were complete, our team ran through our recovery checklist, and started the database server. Service was restored in a degraded state at 15:01 CST, and fully operational at 15:15 CST. We recognize that there are many things to be learned from this lesson, and are working towards putting pieces in place to be able to avoid the long check times and revise our fail-over processes in the future to better account for possible long check times. Please make sure to subscribe to updates at [https://status.ChargeOver.com](https://status.ChargeOver.com) to be notified of service disruptions in the future.

Time: Sept. 9, 2020, 8:39 p.m.

Status: Resolved

Update: This issue has been resolved and all services are operational. Root cause and postmortem will follow.

Time: Sept. 9, 2020, 8:25 p.m.

Status: Monitoring

Update: All ChargeOver services are operational now. We continue to monitor the situation. A postmortem will follow.

Time: Sept. 9, 2020, 8:17 p.m.

Status: Monitoring

Update: We continue to monitor the situation as services are being restored.

Time: Sept. 9, 2020, 8:16 p.m.

Status: Identified

Update: Services are being restored. We are continuing to monitor the situation and will provide further updates.

Time: Sept. 9, 2020, 6:43 p.m.

Status: Identified

Update: We are continuing to work towards restoration of service. More updates to follow.

Time: Sept. 9, 2020, 4:38 p.m.

Status: Identified

Update: We have identified the issue, and are working to resolve the outage as quickly as possible.

Time: Sept. 9, 2020, 4:15 p.m.

Status: Investigating

Update: We are aware of an issue, and are investigating. Further information to follow.

Time: Sept. 2, 2020, 11:58 p.m.

Status: Resolved

Update: First Data has informed us that the problem is resolved. First Data has not provided a root cause or any other information about the issue at this time.

Time: Sept. 2, 2020, 11:41 p.m.

Status: Monitoring

Update: The ChargeOver team is continuing to monitor this situation. First Data has not yet provided an ETA or status update. They are aware of the issue and are working to resolve it. As we get more information from First Data, we will provide further updates.

Time: Sept. 2, 2020, 8:29 p.m.

Status: Monitoring

Update: We continue to monitor this situation. First Data has not responded to our team with an ETA or status update at this time, but we are seeing much better connectivity and much improved processing with First Data now. As we get more information from First Data, we will provide further updates.

Time: Sept. 2, 2020, 5:09 p.m.

Status: Monitoring

Update: First Data (Payeezy, BluePay, and a backend to many other payment processors) is currently experiencing a network outage. Merchants using Payeezy as a payment processor may see sporadic gateway timeouts / connection errors / other errors until First Data resolves this issue. First Data was unable to give us an ETA at this time, but as we get more information from them, we will provide updates.

Time: June 3, 2020, 3:55 p.m.

Status: Postmortem

Update: **What Happened / Customer Impact** Our certificate authority issued ChargeOver a TLS/SSL certificate with a CA chain certificate which expired before our actual TLS/SSL certificate did. This caused automatic invoice generation and payment processing to be delayed for some customers. **Technical Details** ChargeOver secures internal and external connections with TLS/SSL certificates. The company ChargeOver purchases certificates from issued us an SSL/TLS certificate with a chain file which expired before our actual certificate. So even though our SSL/TLS certificate was valid and _not_ expired, a certificate in the chain needed to validate the certificate expired on May 30th. Most web browsers were unaffected, so access to the [ChargeOver.com](http://ChargeOver.com) website and app were unaffected. However, common libraries and tools like cURL and wget began rejecting connections due to the expired chain certificate. ChargeOver's scheduled invoice generation and scheduled payment processes depend on the cURL library, and so the system was unable to trigger invoice generation and payment processing for some accounts scheduled to generate invoices after the certificate had expired. Our monitoring picked up the issue, and alerted our engineering team. ChargeOver then removed the expired certificate from the chain, and invoices and payments began processing normally again. This caused a delay of invoice/payment processing of between 30 minutes and 4 hours for some ChargeOver accounts. **Ongoing Efforts** We are working on some additional validation to ensure that our CA cannot issue us certificates with a certificate chain that expires prior to the certificate we are being issued.

Time: May 31, 2020, 12:14 a.m.

Status: Resolved

Update: This issue has been resolved. Postmortem to follow.

Time: May 30, 2020, 9:29 p.m.

Status: Monitoring

Update: We continue to monitor things, as things catch up -- invoices are generating, emails are sending, and payments are processing.

Time: May 30, 2020, 7:23 p.m.

Status: Monitoring

Update: We have resolved the issue, and services are being restored. Invoices are generating, emails are sending, and payments are processing as normal now. Some ChargeOver accounts may see delayed generation of invoices while things catch back up. A further update will follow.

Time: May 30, 2020, 6:09 p.m.

Status: Investigating

Update: Some ChargeOver accounts are experiencing delayed invoice generation, invoice sending, and automated payment processing. We are investigating.

Time: April 3, 2020, 4:25 p.m.

Status: Resolved

Update: Some ChargeOver accounts were unavailable for less than 2 minutes due to an internal routing issue. The routing issue was related to an infrastructure-related upgrade deployment. Engineering identified the issue immediately after the deploy, and the issue was resolved in under 2 minutes. Engineering has taken steps to ensure that this class of errors can no longer occur on future routing-system upgrades.