Time: Aug. 24, 2022, 2:07 a.m.

Status: Resolved

Update: After a period of monitoring, service to the Spreedly Insights Dashboard and docs.spreedly.com has remained stable, and the affected service provider has confirmed that a fix has been applied. We are considering the issue to be resolved.

Time: Aug. 24, 2022, 1:15 a.m.

Status: Monitoring

Update: Service to the Spreedly Insights Dashboard and docs.spreedly.com has been restored. We are continuing to monitor status with our service provider until we can ensure that the issue is fully resolved.

Time: Aug. 24, 2022, 12:47 a.m.

Status: Identified

Update: We are currently experiencing an outage on the Spreedly Insights Dashboard and docs.spreedly.com due to an outage with one of Spreedly's service providers. Dashboard and Spreedly Docs may be unavailable for users. We will provide updates as they become available.

Time: Aug. 24, 2022, 2:07 a.m.

Status: Resolved

Update: After a period of monitoring, service to the Spreedly Insights Dashboard and docs.spreedly.com has remained stable, and the affected service provider has confirmed that a fix has been applied. We are considering the issue to be resolved.

Time: Aug. 24, 2022, 1:15 a.m.

Status: Monitoring

Update: Service to the Spreedly Insights Dashboard and docs.spreedly.com has been restored. We are continuing to monitor status with our service provider until we can ensure that the issue is fully resolved.

Time: Aug. 24, 2022, 12:47 a.m.

Status: Identified

Update: We are currently experiencing an outage on the Spreedly Insights Dashboard and docs.spreedly.com due to an outage with one of Spreedly's service providers. Dashboard and Spreedly Docs may be unavailable for users. We will provide updates as they become available.

Time: July 28, 2022, 8:36 p.m.

Status: Postmortem

Update: As part of a major framework update to the Core API, a code dependency was updated. This dependency was not backwards compatible and another codebase responsible for sending transactions to gateways failed to properly send transactions for a single gateway type. Because this behavior presented in only a single gateway type and traffic at the time was very low, automated alerting failed to detect the regression. Upon being alerted by a customer of the issue, Spreedly engineering downgraded a code dependency to a backwards compatible version and the issue was resolved. ## What Happened As part of a major framework update for Core, a number of code dependencies were updated, including one for XML parsing. The newer version of this dependency parsed and generated XML in a way that was not fully backwards compatible. The mapping code in the TransFirstTransactionExpress gateway in the Active Merchant adapter improperly formatted a nested XML attribute, causing the gateway to reject requests. While Spreedly actively monitors for anomalous gateway failures, due to the nature of typical transaction flows, Gateway rejections are common. To avoid excessive alerting, Spreedly uses “squelching”; in other words, we require a minimum number of overall transactions to be present in any given time period of monitoring before applying anomaly detection. Unfortunately, this particular gateway's transaction count fell below the “squelch” minimum, so the failure rates were not detected in an automated fashion. After a customer notification was escalated to Engineering, the team quickly identified the cause of the issue and downgraded the XML parsing dependency to a stable version that was backwards compatible with the existing Core and Active Merchant codebases. Upon deployment of this dependency version, transactions were once again accepted by the gateway. ## Next Steps * Spreedly will update Active Merchant to support the latest XML parsing dependency * Spreedly will investigate better “squelch” control for low volume gateways

Time: July 14, 2022, 12:22 p.m.

Status: Resolved

Update: After a period of monitoring, we have observed that issues with the TransFirstTransactionExpress Gateway have been resolved. We apologize for any inconvenience and disruption to service.

Time: July 14, 2022, 12:58 a.m.

Status: Monitoring

Update: We have implemented a change and are seeing successful transactions being sent to TransFirstTransactionExpress gateway. We are monitoring to ensure the issue is fully resolved.

Time: July 14, 2022, 12:58 a.m.

Status: Monitoring

Update: We have implemented a change and are seeing successful transactions being sent to TransFirstTransactionExpress gateway. We are monitoring to ensure the issue is fully resolved.

Time: July 14, 2022, 12:15 a.m.

Status: Identified

Update: We have identified an issue with transactions being sent to TransFirstTransactionExpress gateway. Our Engineering team is reviewing a potential resolution. We will be updating as this investigation continues.

Time: July 14, 2022, 12:15 a.m.

Status: Identified

Update: We have identified an issue with transactions being sent to TransFirstTransactionExpress gateway. Our Engineering team is reviewing a potential resolution. We will be updating as this investigation continues.

Time: July 27, 2022, 4:57 p.m.

Status: Resolved

Update: We are happy to report that as of this morning, our July 15th Account Updater cycle successfully completed and all customer responses have been sent. We will continue to work with our provider to monitor and optimize future cycles.

Time: July 25, 2022, 4:19 p.m.

Status: Monitoring

Update: We are happy to report that our July 15th Account Updater cycle has been progressing successfully and we anticipate it completing by this Wednesday, July 27th, and in time for our next run to begin as scheduled. We will continue to monitor this run until completion and will provide a status update then.

Time: July 14, 2022, 9:01 p.m.

Status: Monitoring

Update: As of today we’ve had a partially successful run of the July 1st Account Updater service, with approximately 30% of the run completed and those updates returned to customers. We identified a slow down in responses over the last two days caused by our service provider not receiving expected updates from the credit card networks, which has created a significant delay in results being returned. Since we do not estimate receiving those pending results prior to starting our July 15th run, we’ve worked with our Account Updater provider to force cancel any batches that are stuck in processing. As a result, users who did not receive complete responses as expected from the July 1st cycle of Account Updater will not receive any additional updates for that run. Any cards that were not updated will be sent in the July 15th run. It is our top priority to ensure our Account Updater future cycles run seamlessly for our customers. Therefore, Spreedly and our provider have taken steps to adjust our Account Updater application to reduce the batch volume and throughput to our provider. These changes have been implemented in order to provide the highest chance of success for the July 15th Account Updater cycle. We will continue to work on longer term solutions jointly with our partner to ensure further optimization for this service going forward. We will monitor the July 15th run closely with our provider and update as it progresses.

Time: July 13, 2022, 10:06 p.m.

Status: Monitoring

Update: We continue monitoring the ongoing July 1st Account Updater run jointly with our provider. We expect to receive the pending responses soon and will notify of any updates.

Time: July 12, 2022, 11:23 p.m.

Status: Monitoring

Update: Our July 1st Account Updater run is still progressing and we expect to receive the pending responses soon. We continue to monitor closely with our provider and will update if anything changes.

Time: July 11, 2022, 3:50 p.m.

Status: Monitoring

Update: We are closely monitoring the July 1st Account Updater cycle along with our provider and we have been receiving responses, as expected. We will continue to monitor progress and will update if there are any significant changes.

Time: July 8, 2022, 7:30 p.m.

Status: Monitoring

Update: We are continuing to monitor the July 1st Account Updater cycle along with our provider. Everything is progressing as expected and we anticipate receiving responses starting tonight or early tomorrow. We will continue to monitor progress and will provide an update on Monday, unless there is a significant change.

Time: July 7, 2022, 7:40 p.m.

Status: Monitoring

Update: We are continuing to monitor the July 1st Account Updater cycle along with our provider. We are expecting our first batch of Account Updater responses to arrive tomorrow. We will continue to monitor progress and provide updates daily.

Time: July 6, 2022, 8:07 p.m.

Status: Monitoring

Update: In continuing to work closely with our Account Updater partner, we've identified a 50% reduction in processing throughput which is resulting in delays sending cards to the networks for updates. This in turn will add additional time to the end to end account updater cycle processing time. As such, customers should expect additional delays in the receipt of your Account Updater updates. Both Spreedly and our partner have dedicated engineering resources working on making improvements to throughput and processing efficiency and are also working with the networks to track and monitor the progress of card processing. We will continue daily status updates and will keep you apprised of additional changes if and when they arise.

Time: July 5, 2022, 8:02 p.m.

Status: Monitoring

Update: We continue to work with our Account Updater service provider to monitor the July 1st run. At present, the process continues to proceed as expected. We will continue to closely monitor the July 1st run along with our Account Updater vendor, and we will provide daily updates until the process is complete.

Time: July 1, 2022, 9:38 p.m.

Status: Monitoring

Update: We have kicked off our July 1st Account Updater cycle and it is progressing as expected so far. We will continue to closely monitor progress with our Account Updater provider daily. Unless there is a significant update to share over the holiday weekend, we will provide our next status page update on Tuesday, July 5th.

Time: June 30, 2022, 9:31 p.m.

Status: Monitoring

Update: We’ve worked with our Account Updater provider to identify the root cause and determined this incident to be related to the volume of updates being sent to the Account Updater service, which resulted in significant batch processing delays and records being dropped. In order to move forward with the July 1st Account Updater cycle in a timely and reliable way, we are proceeding with a cancellation of any June 15 pending batches that are stuck in processing. This means that users who did not receive complete results as expected from the June 15th cycle will not receive further updates until the July 1st cycle. Spreedly is taking short term steps to adjust our Account Updater application to reduce the batch volume and throughput to our provider so that our service remains resilient and successfully processes subsequent cycles. However, these adjustments will result in longer processing times as we continue to work on longer term solutions with our partner for improved speed and efficiency of the service. We will continue to closely monitor the status of the July 1 cycle along with our Account Updater service provider and provide daily updates until we can assure its success.

Time: June 30, 2022, 4:36 p.m.

Status: Identified

Update: Spreedly has determined a path forward working with our Account Updater (AU) service provider. A formal customer communication is being developed with details on how Spreedly will be addressing the June 15 AU issue, and this will be shared with customers using Spreedly's Account Updater service as soon as possible.

Time: June 30, 2022, 1:53 p.m.

Status: Identified

Update: Spreedly continues to actively work with our Account Updater (AU) vendor to troubleshoot the issue being seen with a subset of Spreedly customers not receiving their June 15 account updates. We have identified the root cause of the issue with our AU vendor, and we are discussing several options for resolution. We are continuing to work with our AU vendor to resolve the issue and will provide additional updates later today.

Time: June 29, 2022, 8:54 p.m.

Status: Identified

Update: We have identified an issue with our upstream Account Updater provider, and we are working with them to resolve the issue. We will provide the next update no later than 10AM ET tomorrow or earlier if additional, significant progress is made prior to this time.

Time: June 29, 2022, 6:22 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 29, 2022, 5:17 p.m.

Status: Investigating

Update: Spreedly is aware that some customers using the Account Updater service have reported results lower than historical trends. We are actively investigating this issue.

Time: July 27, 2022, 4:57 p.m.

Status: Resolved

Update: We are happy to report that as of this morning, our July 15th Account Updater cycle successfully completed and all customer responses have been sent. We will continue to work with our provider to monitor and optimize future cycles.

Time: July 25, 2022, 4:19 p.m.

Status: Monitoring

Update: We are happy to report that our July 15th Account Updater cycle has been progressing successfully and we anticipate it completing by this Wednesday, July 27th, and in time for our next run to begin as scheduled. We will continue to monitor this run until completion and will provide a status update then.

Time: July 14, 2022, 9:01 p.m.

Status: Monitoring

Update: As of today we’ve had a partially successful run of the July 1st Account Updater service, with approximately 30% of the run completed and those updates returned to customers. We identified a slow down in responses over the last two days caused by our service provider not receiving expected updates from the credit card networks, which has created a significant delay in results being returned. Since we do not estimate receiving those pending results prior to starting our July 15th run, we’ve worked with our Account Updater provider to force cancel any batches that are stuck in processing. As a result, users who did not receive complete responses as expected from the July 1st cycle of Account Updater will not receive any additional updates for that run. Any cards that were not updated will be sent in the July 15th run. It is our top priority to ensure our Account Updater future cycles run seamlessly for our customers. Therefore, Spreedly and our provider have taken steps to adjust our Account Updater application to reduce the batch volume and throughput to our provider. These changes have been implemented in order to provide the highest chance of success for the July 15th Account Updater cycle. We will continue to work on longer term solutions jointly with our partner to ensure further optimization for this service going forward. We will monitor the July 15th run closely with our provider and update as it progresses.

Time: July 13, 2022, 10:06 p.m.

Status: Monitoring

Update: We continue monitoring the ongoing July 1st Account Updater run jointly with our provider. We expect to receive the pending responses soon and will notify of any updates.

Time: July 12, 2022, 11:23 p.m.

Status: Monitoring

Update: Our July 1st Account Updater run is still progressing and we expect to receive the pending responses soon. We continue to monitor closely with our provider and will update if anything changes.

Time: July 11, 2022, 3:50 p.m.

Status: Monitoring

Update: We are closely monitoring the July 1st Account Updater cycle along with our provider and we have been receiving responses, as expected. We will continue to monitor progress and will update if there are any significant changes.

Time: July 8, 2022, 7:30 p.m.

Status: Monitoring

Update: We are continuing to monitor the July 1st Account Updater cycle along with our provider. Everything is progressing as expected and we anticipate receiving responses starting tonight or early tomorrow. We will continue to monitor progress and will provide an update on Monday, unless there is a significant change.

Time: July 7, 2022, 7:40 p.m.

Status: Monitoring

Update: We are continuing to monitor the July 1st Account Updater cycle along with our provider. We are expecting our first batch of Account Updater responses to arrive tomorrow. We will continue to monitor progress and provide updates daily.

Time: July 6, 2022, 8:07 p.m.

Status: Monitoring

Update: In continuing to work closely with our Account Updater partner, we've identified a 50% reduction in processing throughput which is resulting in delays sending cards to the networks for updates. This in turn will add additional time to the end to end account updater cycle processing time. As such, customers should expect additional delays in the receipt of your Account Updater updates. Both Spreedly and our partner have dedicated engineering resources working on making improvements to throughput and processing efficiency and are also working with the networks to track and monitor the progress of card processing. We will continue daily status updates and will keep you apprised of additional changes if and when they arise.

Time: July 5, 2022, 8:02 p.m.

Status: Monitoring

Update: We continue to work with our Account Updater service provider to monitor the July 1st run. At present, the process continues to proceed as expected. We will continue to closely monitor the July 1st run along with our Account Updater vendor, and we will provide daily updates until the process is complete.

Time: July 1, 2022, 9:38 p.m.

Status: Monitoring

Update: We have kicked off our July 1st Account Updater cycle and it is progressing as expected so far. We will continue to closely monitor progress with our Account Updater provider daily. Unless there is a significant update to share over the holiday weekend, we will provide our next status page update on Tuesday, July 5th.

Time: June 30, 2022, 9:31 p.m.

Status: Monitoring

Update: We’ve worked with our Account Updater provider to identify the root cause and determined this incident to be related to the volume of updates being sent to the Account Updater service, which resulted in significant batch processing delays and records being dropped. In order to move forward with the July 1st Account Updater cycle in a timely and reliable way, we are proceeding with a cancellation of any June 15 pending batches that are stuck in processing. This means that users who did not receive complete results as expected from the June 15th cycle will not receive further updates until the July 1st cycle. Spreedly is taking short term steps to adjust our Account Updater application to reduce the batch volume and throughput to our provider so that our service remains resilient and successfully processes subsequent cycles. However, these adjustments will result in longer processing times as we continue to work on longer term solutions with our partner for improved speed and efficiency of the service. We will continue to closely monitor the status of the July 1 cycle along with our Account Updater service provider and provide daily updates until we can assure its success.

Time: June 30, 2022, 4:36 p.m.

Status: Identified

Update: Spreedly has determined a path forward working with our Account Updater (AU) service provider. A formal customer communication is being developed with details on how Spreedly will be addressing the June 15 AU issue, and this will be shared with customers using Spreedly's Account Updater service as soon as possible.

Time: June 30, 2022, 1:53 p.m.

Status: Identified

Update: Spreedly continues to actively work with our Account Updater (AU) vendor to troubleshoot the issue being seen with a subset of Spreedly customers not receiving their June 15 account updates. We have identified the root cause of the issue with our AU vendor, and we are discussing several options for resolution. We are continuing to work with our AU vendor to resolve the issue and will provide additional updates later today.

Time: June 29, 2022, 8:54 p.m.

Status: Identified

Update: We have identified an issue with our upstream Account Updater provider, and we are working with them to resolve the issue. We will provide the next update no later than 10AM ET tomorrow or earlier if additional, significant progress is made prior to this time.

Time: June 29, 2022, 6:22 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 29, 2022, 5:17 p.m.

Status: Investigating

Update: Spreedly is aware that some customers using the Account Updater service have reported results lower than historical trends. We are actively investigating this issue.