Time: June 21, 2024, 7:43 p.m.

Status: Resolved

Update: We are considering this issue fully resolved. We have monitored our incoming logs and confirmed the disconnects have stopped occurring since we rolled out our fix to all customers. The root cause was confirmed to be the update in the "Summer 24" release from Salesforce that blocked simultaneous refresh tokens. We resolved the issue by adding a more robust retry strategy in addition to better logic in how we handle refresh tokens. As these changes were not trivial and required significant engineering resources, the fix took longer than usual to develop. As an outcome, the retry mechanism will remain in place and our notification emails related to dropped connections were adjusted to better guide admins should any unrelated disconnect issues occur in the future.

Time: June 21, 2024, 3:14 p.m.

Status: Monitoring

Update: Our fix is now deployed to our production environments, and we are now monitoring in realtime to validate that the fix is working as expected. Please make our team aware of any additional disconnects you experience via support@chilipiper.com so that we can act quickly to investigate it further.

Time: June 21, 2024, 1:05 p.m.

Status: Identified

Update: We have successfully replicated and validated the issue's root cause through testing. Our solution is confirmed to resolve the disconnects and will be deployed to our production environment shortly. Once it has deployed to all customers, we will provide another update to monitor the fix before confirming it is fully resolved.

Time: June 20, 2024, 7:56 p.m.

Status: Identified

Update: We have begun deploying some solutions to our testing environments to start validating them internally. Once we have completed tests we will start rolling it out to customers to monitor the fix. We have also updated our automated email sent to admins in order to make it clearer where to go in order to reconnect. Previously it had indicated the personal-level connection, but please be sure to reconnect the global admin-level connection as the new notification indicates.

Time: June 20, 2024, 3:10 p.m.

Status: Investigating

Update: We understand the urgency and concern this incident has caused. The resolution is taking longer than anticipated due to the need for significant changes in our authentication infrastructure rather than if it was a more trivial bug. This issue has been escalated internally and is our top priority. We've allocated additional engineering resources to expedite the fix so we can get it rolled out as soon as possible. We aim to have a solution deployed to our testing environment before the end of today. Once verified through quality assurance, it will be rolled out to all customers. In the meantime, please continue to monitor your integration user email for any Salesforce disconnects. Thanks again for your patience and understanding.

Time: June 18, 2024, 6:21 p.m.

Status: Investigating

Update: We are currently investigating an issue affecting the global Salesforce connection (integration user) for a small subset of customers using our legacy product line. Please note that our Demand Conversion Platform products and HubSpot CRM customers are not impacted. While disconnected, there may be issues with creating or updating Lead records or Events. Additionally, routing will be limited to form data and will not function properly based on existing Lead/Contact data. Our preliminary findings suggest that recent updates from Salesforce, which block specific refresh token strategies, may be the underlying cause. We are working on a fix to implement a more robust refresh token logic. We have confirmed that a notification email is sent to the integration user email account when a disconnection occurs, prompting them to reconnect as soon as possible. We are actively working on a solution. In the meantime, please be aware that the service may disconnect again even after reconnecting. To mitigate this, we have implemented active monitoring to detect disconnections promptly and may reach out directly if needed. We urge admins to monitor their integration user inbox closely to ensure timely action in the event of a disconnection. Thank you for your understanding and patience as we resolve this issue.

Time: Dec. 16, 2023, 4:47 p.m.

Status: Resolved

Update: Our team identified an issue that's preventing Chili Piper from successfully trigger upon form submission. We have quickly resolved the issue by applying a hotfix. Chili Piper show now be triggering properly and the calendar would appear when a prospect successfully qualifies for a booking.

Time: Dec. 6, 2023, 3:09 p.m.

Status: Resolved

Update: Our team identified an issue with a new security policy that was preventing Chili Piper assets from loading. Our engineers have already deployed a fix for this and we should see it resolved. While this was impacting Chili Piper, routing and booking links were impacted as well as internal pages and they were instead presenting a blank page or could not be loaded. We have resolved the issue by reverting the change and have corrected the underlying issue moving forward.

Time: Oct. 19, 2023, 9:01 p.m.

Status: Resolved

Update: We are marking this issue as resolved since incoming data shows services are recovered, error rates have dropped, and loading times are stabilized. Our team will continue to monitor services closely to ensure they remain stable moving forward, and actions have already been taken to ensure this stability is maintained.

Time: Oct. 19, 2023, 8:14 p.m.

Status: Monitoring

Update: Services have been restored, and several sub-services were restarted in order to further support the recovery of all instances. We are seeing load times gradually return to normal with only a minor performance impact reported. We will continue to monitor the recovery closely to ensure our services remain stable.

Time: Oct. 19, 2023, 8:14 p.m.

Status: Monitoring

Update: Services have been restored, and several sub-services were restarted in order to further support the recovery of all instances. We are seeing load times gradually return to normal with only a minor performance impact reported. We will continue to monitor the recovery closely to ensure our services remain stable.

Time: Oct. 19, 2023, 7:59 p.m.

Status: Identified

Update: We have begun to restore services and have reduced the load on the impacted databases. Databases instances are being restarted to help mitigate the issue. This is showing improvement on load times, however users may still experience long delays or see an "Error: Server Error" 502 message with degraded performance of all Chili Piper pages, links, Instant Booker, and Concierge. We should start seeing more services gradually become healthy in time, and the loading speed will continue to increase, however due to the increased load times we are still considering this a partial outage.

Time: Oct. 19, 2023, 7:59 p.m.

Status: Identified

Update: We have begun to restore services and have reduced the load on the impacted databases. Databases instances are being restarted to help mitigate the issue. This is showing improvement on load times, however users may still experience long delays or see an "Error: Server Error" 502 message with degraded performance of all Chili Piper pages, links, Instant Booker, and Concierge. We should start seeing more services gradually become healthy in time, and the loading speed will continue to increase, however due to the increased load times we are still considering this a partial outage.

Time: Oct. 19, 2023, 7:29 p.m.

Status: Investigating

Update: We are currently investigating an issue impacting our database which is preventing Chili Piper assets from loading. Instant Booker as well as Chili Piper's dashboard are both impacted. Some booking links appear to load, but performance is extremely degraded. Pages may load after a long delay. Our Site Reliability infrastructure team is actively investigating this with priority, and we will provide an update once this issue is resolved.

Time: Oct. 19, 2023, 7:29 p.m.

Status: Investigating

Update: We are currently investigating an issue impacting our database which is preventing Chili Piper assets from loading. Instant Booker as well as Chili Piper's dashboard are both impacted. Some booking links appear to load, but performance is extremely degraded. Pages may load after a long delay. Our Site Reliability infrastructure team is actively investigating this with priority, and we will provide an update once this issue is resolved.

Time: Sept. 29, 2023, 11:01 a.m.

Status: Resolved

Update: This incident is now considered resolved. We have not received any additional reports of long loading times or performance issues since our last update. We have documented the issue and taken the necessary measures to prevent this in the future. Thank you for your patience while we worked on this!

Time: Sept. 29, 2023, 9:50 a.m.

Status: Monitoring

Update: All systems have returned to a stable status. Our engineers are monitoring now to ensure that this is maintained. Please continue to contact our Customer Love team at support@chilipiper.com if you are still experiencing any slowness or performance issues with Chili Piper.

Time: Sept. 29, 2023, 9:25 a.m.

Status: Identified

Update: Having identified the issue, our engineers are making significant progress working on a fix. We apologize for the extended duration of this incident and sincerely appreciate your patience. We understand its impact on your experience and are fully committed to resolving it swiftly.

Time: Sept. 29, 2023, 8:36 a.m.

Status: Investigating

Update: We are seeing instances of concierge and booking link pages loading endlessly and eventually timing out. Our engineering team is currently investigating. The impact appears to be in a smaller subset of databases. We will provide additional updates here as we have them.