Time: Aug. 16, 2024, 7:54 p.m.

Status: Postmortem

Update: ## Kindful Fundraising Tool Service Outage August 13th, 2024 ### Overview and Timeline On August 13th at 8:00 pm EDT, the Kindful Fundraising Tool experienced an infrastructure-level issue that impacted customers’ ability to use KFT and utilize the Kindful API.  Our engineering operations team planned a window for updating the SSL certificate without the correct time zone considerations resulting in a gap in certification. The team identified the impact and started addressing the root issue at 9:13 pm EDT. The SSL certificate did not propagate to systems at the correct time leading to customer browser TLS issues. During the incident, customers attempting to utilize KFT via the web experienced a message noting a connection error. Kindful API calls failed on the request.  By 9:25 pm EDT, the underlying SSL certificate was properly propagated to all infrastructure systems and restored service. The team continued monitoring to ensure new requests were successfully received and processed.  Bloomerang utilizes monitoring tools to help detect system errors and outages before customer impact.  It was later discovered that internal alerting procedures had failed to properly notify the engineering team responsible for Kindful services preventing more expedited identification and remediation.  ### Root Cause\(s\) **An updated SSL certificate utilized in Kindful systems did not propagate to all systems preventing infrastructure from properly functioning.**   ### Action Item\(s\)

Time: Aug. 14, 2024, 1:37 a.m.

Status: Resolved

Update: Kindful experienced a service outage on 8/13 between 8 PM and 9:25 PM EDT impacting all users attempting to use the CRM and the API within this time frame. We have implemented a fix and all services are restored. We apologize for any inconvenience and do not expect any further interruptions.

Time: Aug. 6, 2024, 3:37 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 2, 2024, 2:16 p.m.

Status: Monitoring

Update: We have implemented a fix for the issue causing Kindful admin pages to not load for some users. We are actively monitoring the situation to ensure that no further problems arise.

Time: Aug. 2, 2024, 1:52 p.m.

Status: Investigating

Update: We're aware of an issue preventing Kindful admin pages from loading for some users. Our engineering team is actively working to resolve this. If you encounter this problem, please try refreshing the page. We apologize for any inconvenience and appreciate your patience as we work to restore normal functionality.

Time: Aug. 1, 2024, 9:31 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 1, 2024, 6:05 p.m.

Status: Monitoring

Update: We have deployed a fix for this issue. We are monitoring to ensure no further issues occur.

Time: Aug. 1, 2024, 3:31 p.m.

Status: Identified

Update: We're aware of an issue preventing Kindful admin pages from loading for some users. Our engineering team is actively working to resolve this. If you encounter this problem, please try refreshing the page. We apologize for any inconvenience and appreciate your patience as we work to restore normal functionality.

Time: July 17, 2024, 4:56 p.m.

Status: Resolved

Update: The issue with removing billing profiles from donor profiles has been resolved. Thank you for your patience and understanding while we worked to fix this. We apologize for any inconvenience this may have caused.

Time: July 15, 2024, 1:58 p.m.

Status: Identified

Update: We're still working on the issue with removing donor's payment methods. Our team is actively working on a solution, and we'll update you as soon as we have more information.

Time: July 12, 2024, 2:30 p.m.

Status: Identified

Update: We've identified the issue that's preventing users from deleting billing profiles. Our team is actively working on a solution, and we'll update you as soon as we have more information.

Time: July 11, 2024, 10:14 p.m.

Status: Investigating

Update: We have identified an issue with deleting payment methods from a donor's profile. We are working to resolve this issue and apologize for any inconvenience this may cause. Users are still able to add new payment methods and change the payment methods used for existing recurring donations. We will provide an update as soon as we have more information.

Time: Aug. 1, 2024, 6:12 p.m.

Status: Resolved

Update: We have released an update to correct the issue with Kindful's BCC mail feature. Please contact support if you have any issues with this feature going forward.

Time: July 31, 2024, 7:24 p.m.

Status: Monitoring

Update: We have released an update to correct the issue with Kindful's BCC mail feature. Please contact support if you have any issues with this feature going forward.

Time: July 26, 2024, 9:54 p.m.

Status: Identified

Update: We have identified the issue with Kindful's BCC feature and are working to restore full functionality for all users. We apologize for any inconvenience and we will provide more updates for this issue here as we learn more.

Time: July 15, 2024, 1:56 p.m.

Status: Investigating

Update: We're still looking into the issue with Kindful's BCC feature. We apologize for any inconvenience this may cause and will keep you updated as we learn more. Thank you for your patience.

Time: July 10, 2024, 9:43 p.m.

Status: Investigating

Update: The Kindful's BCC feature which automatically tracks email communication between Admins and Contact is not functioning at this time. We understand the importance of this feature and are actively investigating the issue. While we work to resolve this issue, please manually add emails to the contact's profile using the "Add New Note" option found within the contact's profile, under the notes tab. We apologize for any inconvenience and will provide another update as soon as we have more information.