Time: July 21, 2023, 7:43 p.m.

Status: Postmortem

Update: ## Summary A planned operating system upgrade on July 12th resulted in slightly higher ongoing CPU usage than the prior OS. This compounded so that by July 19th, we began to exhaust CPU credits, which reduced our ability to successfully process a number of transactions. ## What Happened As prologue, on Wednesday July 12th, we exercised a planned-for operating system upgrade on the servers running the cryptography service. This involved our usual and practiced behavior of gracefully moving traffic from one cluster of servers \(i.e.: “blue” to “green”\), patching the offline cluster and then performing the same in reverse. Testing illustrated that the upgrade was a success and the system continued processing ~100m\+ internal service API requests daily without issue or incident. On July 18th at 18:26 UTC, internal monitoring alerted on an increased number of failures being generated from our cryptography service. Teams were immediately assembled to work the issue. Logs indicated a resource contention `(limiting connections by zone "default"`\) as individual service nodes exceeded the configured limit of simultaneous connections. We began the effort to alter the configuration of these limits and the task of quieting one cluster, effecting the repair, bringing the cluster online, and then performing the steps for the other cluster. Unfortunately, once reconfigured, a different resource contention was found, this time at the operating system level. We began the effort to alter that configuration and percolate the change through both “blue” and “green” clusters but the errors merely moved to another type of resource constraint. We reverted potentially relevant changes from earlier in the day, without positive effect on the system. Finally, we reverted the operating system upgrade from the July 12th activity which ultimately resolved the issue. By July 19th at 00:29 UTC, the error rate on all cryptography service nodes returned to zero and we left the incident in a monitor state for approximately 12 hours before declaring the incident resolved. A subsequent root cause analysis has determined that we overran the “CPU Credits” allocated to our AWS EC2 instance type on which the cryptography service was running. We believe that the OS upgrade incrementally consumed enough CPU capacity to eventually exhaust CPU credits, as credits are utilized at relatively small CPU usage, in excess of 20% ## Next Steps We are re-provisioning this service to use an AWS EC2 instance type that has no CPU credit limits which should prevent the problem from happening again. ## Conclusion We want to apologize to all customers who were impacted by failed transactions due to this incident. We understand how much you rely on our systems being fully operational in order to successfully operate your business. We also appreciate your patience while this issue was being investigated and resolved, thank you again for the trust you place on us every day.

Time: July 19, 2023, 2:08 p.m.

Status: Resolved

Update: After closely monitoring our Core Transactional API and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. We are completing our investigation with regard to the causes of the incident and any residual impact. A post-incident review will be published. We apologize for any inconvenience or disruption.

Time: July 19, 2023, 12:56 a.m.

Status: Monitoring

Update: Our Engineering team has identified and implemented some changes to address the issue. As we actively monitor the fix, we are seeing a reduction in error rates indicating the fix is stabilizing the errors and addressing the issue. We’ll continue to monitor the fix closely and provide updates.

Time: July 18, 2023, 10:20 p.m.

Status: Identified

Update: Our team is actively investigating an issue with one of the libraries that feeds our Core component resulting in some transactions receiving errors. Some customers may have experienced 500 errors. We are still investigating a potential solution. Thank you for your patience.

Time: July 18, 2023, 7:38 p.m.

Status: Identified

Update: Since 2023-07-18 3:01 ET, there has been an issue detected causing intermittent errors on the Spreedly API. Our team is actively working towards a solution to rectify the problem, and we will keep you updated on any progress made in resolving the matter. We value your patience as we work to resolve this issue.

Time: June 29, 2023, 9:15 p.m.

Status: Resolved

Update: After a period of monitoring, we have confirmed that the issue has been resolved.

Time: June 29, 2023, 8:08 p.m.

Status: Monitoring

Update: The intermittent errors with the Spreedly iFrame have subsided. We are continuing to monitor to ensure that the issue is fully resolved

Time: June 29, 2023, 7:37 p.m.

Status: Identified

Update: We have identified an issue with an upstream service provider causing intermittent errors on the Spreedly iFrame. We are communicating with our service provider as they work to resolve the issue.

Time: June 29, 2023, 5:22 p.m.

Status: Investigating

Update: We are currently investigating an issue with the Spreedly iFrame. Users may experience that iFrame will intermittently not load when requested.

Time: June 29, 2023, 9:15 p.m.

Status: Resolved

Update: After a period of monitoring, we have confirmed that the issue has been resolved.

Time: June 29, 2023, 8:08 p.m.

Status: Monitoring

Update: The intermittent errors with the Spreedly iFrame have subsided. We are continuing to monitor to ensure that the issue is fully resolved

Time: June 29, 2023, 7:37 p.m.

Status: Identified

Update: We have identified an issue with an upstream service provider causing intermittent errors on the Spreedly iFrame. We are communicating with our service provider as they work to resolve the issue.

Time: June 29, 2023, 5:22 p.m.

Status: Investigating

Update: We are currently investigating an issue with the Spreedly iFrame. Users may experience that iFrame will intermittently not load when requested.

Time: June 12, 2023, 10:14 p.m.

Status: Resolved

Update: We have received confirmation from Mastercard that a service disruption, which was identified for their Smart Interface and caused customers to experience intermittent transaction failures during the impact period, has now been restored. We appreciate your patience while we tracked this issue with our partners.

Time: June 12, 2023, 10:14 p.m.

Status: Resolved

Update: We have received confirmation from Mastercard that a service disruption, which was identified for their Smart Interface and caused customers to experience intermittent transaction failures during the impact period, has now been restored. We appreciate your patience while we tracked this issue with our partners.

Time: June 12, 2023, 7:24 p.m.

Status: Identified

Update: Our team is still tracking an issue that started June 12th, 2023, at 07:03 am EST (12:03 UTC). Customers may continue to experience transaction failures within our 3DS2 Global Solution. Our Global 3DS2 provider and Mastercard have both confirmed this is an ongoing issue impacting Mastercard's card types. We understand that the Mastercard team is working to mitigate the issue and restore service to all customers as soon as possible. We will provide an update as soon as more details are available. Thank you for your patience at this time.

Time: June 12, 2023, 7:24 p.m.

Status: Identified

Update: Our team is still tracking an issue that started June 12th, 2023, at 07:03 am EST (12:03 UTC). Customers may continue to experience transaction failures within our 3DS2 Global Solution. Our Global 3DS2 provider and Mastercard have both confirmed this is an ongoing issue impacting Mastercard's card types. We understand that the Mastercard team is working to mitigate the issue and restore service to all customers as soon as possible. We will provide an update as soon as more details are available. Thank you for your patience at this time.

Time: June 12, 2023, 3:04 p.m.

Status: Identified

Update: We have identified an issue which is impacting some Mastercard 3DS2 transactions for a subset of our customers using our 3DS2 Global solution. These transactions are failing with the following error "Remote Error: Intermediate layer error - error in intermediate layer (message received invalid. could not read the message. authentication response)". We have logged a ticket with our third party 3DS2 provider and they are actively investigating. We will provide updates as these become available to us.

Time: June 12, 2023, 3:04 p.m.

Status: Identified

Update: We have identified an issue which is impacting some Mastercard 3DS2 transactions for a subset of our customers using our 3DS2 Global solution. These transactions are failing with the following error "Remote Error: Intermediate layer error - error in intermediate layer (message received invalid. could not read the message. authentication response)". We have logged a ticket with our third party 3DS2 provider and they are actively investigating. We will provide updates as these become available to us.

Time: June 7, 2023, 5:50 p.m.

Status: Resolved

Update: After deploying a fix, our system is now stabilized and functioning. This incident is being considered resolved. We apologize for any inconvenience and disruption to service this caused to our customers transacting via Stripe.

Time: June 7, 2023, 5:32 p.m.

Status: Identified

Update: Spreedly is actively addressing an issue with our Stripe integration which is impacting customers transacting through this end point. We are expected to be in this degraded performance state for about 20 minutes, while we work to correct this issue.