Time: Sept. 10, 2024, 5:17 a.m.

Status: Resolved

Update: We identified that the incident impacting APAC users was caused by a network connectivity issue specific to this region. The issue affected traffic from certain end-user networks in the Asia Pacific region when accessing services hosted in the US-EAST-1 AWS region. Users in other regions were not affected by this disruption. The root cause was traced to a network-level issue within AWS’s infrastructure, which impacted connectivity for APAC users. As of 04:30 UTC, the issue has been resolved, and there are no further signs of latency or dropped requests. The incident is now considered resolved. If you have any additional concerns or need further assistance, please contact our support team.

Time: Sept. 10, 2024, 4:38 a.m.

Status: Monitoring

Update: AWS has confirmed that the traffic for APAC customers accessing services hosted on the US-East data centre has been successfully rerouted through a secondary network, restoring full access. As of 04:30 UTC, network connectivity has been fully restored, and all services are functioning normally. We continue to actively monitor the network to ensure long-term stability and performance.

Time: Sept. 10, 2024, 4:23 a.m.

Status: Identified

Update: We have identified the issue as external to the Chargebee platform, related to connectivity between the APAC and US regions. Specifically, it is causing timeouts and login failures for users attempting to access app.chargebee.com. We have contacted AWS and they confirmed that it is due to ISP and they are rerouting the requests from APAC.

Time: Sept. 10, 2024, 4:10 a.m.

Status: Investigating

Update: Some customers are facing difficulties while logging in to the Chargebee Application. Our best minds are looking into the issue and we will be providing updates as we make progress.

Time: Sept. 5, 2024, 11:52 a.m.

Status: Resolved

Update: We have not observed any errors after 07:49 AM UTC, we are proceeding with resolving the incident.

Time: Sept. 5, 2024, 8:20 a.m.

Status: Monitoring

Update: We have observed a decrease in errors starting from 06:31 UTC, indicating potential improvement in the issue affecting merchants during checkout and add payment methods. We are continuing to monitor and will provide further updates as we receive more information from Adyen.

Time: Sept. 5, 2024, 8:06 a.m.

Status: Identified

Update: We have identified a problem external to our platform that has caused an increase in error rates for transactions and add payment methods processed through Adyen starting at 03:43 UTC. We have reached out to Adyen and will provide an update as soon as we have more information. For more details, please follow Adyen’s official Update: https://status.adyen.com

Time: Aug. 28, 2024, 9:18 p.m.

Status: Resolved

Update: Between 18:57 UTC and 20:17 UTC, we observed an increased error rate for merchants using Anrok as their tax provider. This issue affected various actions, including subscription estimates and subscription creation, and may have caused errors during page loading. Upon identifying the issue, we promptly notified Anrok, who took corrective actions to revert the changes responsible. As a result, all functionality has now returned to normal.

Time: Aug. 22, 2024, 9:15 a.m.

Status: Postmortem

Update: **Incident Summary:**  We experienced an issue affecting some customer-initiated transactions processed through the Adyen payment gateway. The problem was identified within the temporary token workflow after a recent upgrade to our payment integration. The issue was directly linked to the transition from Adyen's Classic integration to the Checkout integration. As a result, certain transactions were failing due to unsupported flows in the new integration. After identifying the problem, we reverted the changes, and transaction processing has since returned to normal. **Services Impacted:** Customer transactions processed via the Adyen payment gateway, specifically those involving the temporary \(CSE\) token workflow, were impacted. **Root Cause:** The root cause of the issue was the transition from Adyen's Classic Integration to the Checkout Integration executed on Aug 15th,2024. During this upgrade, we discovered that one of [**Adyen's existing payment flows**](https://docs.adyen.com/online-payments/classic-integrations/api-integration-ecommerce/cse-integration-ecommerce/), specifically the process of converting a temporary \(CSE\) token into a permanent token, was not supported in the Checkout integration. This incompatibility between the old and new integration flows led to the failure of some customer transactions, resulting in 4xx errors occurring during payment processing. While we had monitoring systems in place for server-side 5xx errors, we lacked appropriate alerts for 4xx errors, which delayed our identification and resolution of the issue. **Remediation & Follow-up Steps:** To resolve the issue, we reverted to the previous integration, which immediately restored normal transaction processing. We also implemented new alerts to quickly detect and address similar issues in the future. Looking ahead, we have added automated tests to cover the scenarios that led to this issue, ensuring they are included in future testing cycles. Additionally, we are working closely with Adyen to find an alternative solution for the unsupported CSE token workflow in the Checkout Integration. Once a viable solution is identified, we will proceed with implementing the necessary changes.

Time: Aug. 16, 2024, 6:48 p.m.

Status: Resolved

Update: We identified an issue affecting some customer-initiated transactions processed through the Adyen gateway, specifically within the temporary token workflow. This was caused by recent changes on our end, which we have since reverted. Transactions are now processing correctly, and the issue has been resolved. We apologize for any inconvenience this may have caused.

Time: Sept. 4, 2024, 9:45 a.m.

Status: Postmortem

Update: **Incident Summary:** On August 15, 2024, at 3:36 PM AEST, a new version of the platform was deployed to production by the **PayFurl** vendor. This release included stricter order validation rules that inadvertently disrupted transaction processing, specifically affecting integrations with Chargebee. The issue led to failed transaction validations, temporarily preventing the processing of payments. PayFurl was alerted to the issue by Chargebee on August 16, 2024, at 4:46 AM AEST. A fix was successfully deployed by 6:26 AM AEST on the same day, restoring normal transaction processing. **Services Impacted:** Merchants integrated with Chargebee who use the NMI, Ecentric, and Dlocal gateways experienced disrupted payment services during the incident. Other merchants not using these specific gateways were unaffected. **Root Cause:** The incident was caused by a new validation rule introduced in PayFurl’s latest release. This rule required that order items be present whenever an order number was provided, in line with gateway validation standards. However, this stricter validation conflicted with Chargebee’s existing logic, which allowed an order number to be populated without associated order items. This mismatch led to transaction failures originating from Chargebee, resulting in 4xx errors during payment processing. Although our monitoring systems effectively tracked server-side 5xx errors, there were no alerts configured for 4xx errors, delaying our detection and resolution of the issue. **Remediations & Follow-Up Steps:** A fix was deployed on August 16, 2024, at 6:26 AM AEST, relaxing the validation rule to allow transactions to proceed even when only the order number is populated. An automated test has been added to the PayFurl platform to specifically cover the interaction between PayFurl’s validation rules and Chargebee’s integration. This will prevent similar issues from occurring in future updates. Monitoring has been expanded to include 4xx errors, with alerts set to notify the team if no successful transactions occur within an hour, enabling quicker identification and resolution of similar incidents.

Time: Sept. 3, 2024, 8:43 a.m.

Status: Resolved

Update: We discovered that a recent update from PayFurl introduced a new validation rule that conflicted with our existing transaction processing logic, causing transaction failures. PayFurl quickly reverted the update, restoring the previous version and resolving the issue. Transaction processing has since returned to normal.