Time: July 12, 2024, 8:06 a.m.

Status: Postmortem

Update: **Incident Summary:** The Account Hierarchy feature experienced an interruption, resulting in customers facing difficulties accessing the Account Hierarchy Dashboard. This interruption affected the ability of customers to manage and view their account hierarchies. **Business Impact:** Customers were unable to manage and view their account hierarchies due to the interruption. We have identified the affected customers and communicated the next steps to them. **Root Cause:** The root cause was traced back to a recent migration performed by Chargebee. This migration inadvertently caused conflicts with the processing capabilities of the Account Hierarchy service leading to a feature lock on the service. **Steps Taken & Way Forward:** The feature settings were manually updated for all the impacted customers, restoring the functionality of the Account Hierarchy feature. To prevent similar issues in the future, the erroneous behaviour has been fixed, and similar workflows will be tested in the automation suite. Additionally, entitlements import functionality is being implemented to avoid service disruptions from migrations.

Time: July 8, 2024, 12:03 p.m.

Status: Resolved

Update: The Account Hierarchy feature experienced an interruption, causing difficulties for customers accessing the Account Hierarchy Dashboard. A fix has been implemented to restore services. We apologize for any inconvenience this may have caused.

Time: July 3, 2024, 1:30 p.m.

Status: Resolved

Update: Between 11:43 UTC and 11:47 UTC, we experienced intermittent API service degradation in the US region due to server timeout issues. We apologize for any inconvenience. Note – This event occurred on TEST sites, no LIVE sites were impacted.

Time: July 23, 2024, 1:32 p.m.

Status: Resolved

Update: We discovered an issue where expired coupons on active subscriptions were being incorrectly removed during any modifications, such as adding new coupons or add-ons via UI. The change has been reverted, restoring the correct behaviour. We apologize for any inconvenience caused.

Time: July 3, 2024, 7:47 a.m.

Status: Resolved

Update: We experienced intermittent API service interruption on Jun 30th 2024, between 13:33 and 13:35 UTC. This disruption was due to an issue on the AWS platform, leading to an unexpected increase in API failures in the US region. Please note that the impact was very minimal, We apologize for any inconvenience caused.

Time: July 2, 2024, 7:47 a.m.

Status: Postmortem

Update: **Incident Summary:** On 26th June at 05:57 UTC, we experienced a significant surge in traffic targeting one of our customers, including the payment URLs managed by Chargebee. This unexpected increase in requests led to service interruptions as our systems, including AWS Application Load Balancers \(ALBs\), were unable to scale adequately. **Current Setup:** Our current infrastructure is designed with multiple clusters to effectively minimize the impact radius during both internal changes and external load events. We utilize AWS's Layer7 functionality to manage overall routing to these clusters. We also have configured WAF with robust rules to block malicious traffic. **Root Cause:** The root cause of the downtime was a targeted attack involving requests from different IPs directed at one of our customer’s websites and their payment URLs \(Chargebee domain\). The high volume of the attack overwhelmed the system's capacity to scale, particularly affecting our AWS Load Balancers. Although AWS WAFs were enabled, the magnitude and speed of the attack within a short duration still caused a significant problem in the AWS Load Balancers, preventing the WAF from even evaluating the incoming requests. This resulted in an impact on all the clusters even though the underlying instances were healthy. **Steps Taken & Way Forward:** * A tool has been provided to the 24/7 operations team along with SOP to quickly implement the honeypot-based routing mitigation to be done during DDoS attacks. * Evaluating additional DDoS mitigation providers to replace/supplement AWS WAF. * We have planned to enhance our mitigation strategy by having traffic distribution with multiple load balancers in the routing layer. We aim to enhance our system's resilience against similar attacks and ensure uninterrupted service for our customers. We will continue to monitor and adapt our strategies as needed to address evolving security threats.

Time: June 26, 2024, 6:50 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: June 26, 2024, 6:45 a.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: June 26, 2024, 6:37 a.m.

Status: Monitoring

Update: On June 26, 2024, between 05:57 and 06:17 UTC, we experienced brief server timeout issues for customers in the US region. We apologize for any inconvenience this may have caused.