Time: Oct. 9, 2024, 6:58 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Oct. 9, 2024, 6:27 p.m.

Status: Investigating

Update: Our engineers are investigating reports of the Distro routers screen in the admin center showing an error message: "Could not load Routers" followed by a blank page. We do not expect routing itself to be impacted by this issue. Additionally, not all customers are impacted by the issue - if you are able to view your list of Distro routers, you are not currently impacted and can use Distro services as usual. No other products are impacted.

Time: Sept. 19, 2024, 4:36 p.m.

Status: Resolved

Update: We have reverted the change that initiated the Outlook add-in issues and will ensure future deployments are stable. All impacted customers should be back up and running.

Time: Sept. 19, 2024, 2:11 p.m.

Status: Investigating

Update: We are investigating issues with our production Outlook Add-in within our classic product suite for Microsoft customers. This may impact your ability to open the Instant Booker app within Outlook Desktop or Web. Using Instant Booker from within Chili Piper directly will still function as expected. The standard Chrome Extension can still be used to book meetings directly form the extension itself, just not as buttons within Outlook. Existing meetings will not be impacted. This does not impact Concierge form bookings, or meetings booked via booking links. Google customers are not impacted at this time. Our Site reliability team is looking into this, and we expect a fairly rapid solution. Thank you for your patience in the meantime!

Time: June 21, 2024, 7:43 p.m.

Status: Resolved

Update: We are considering this issue fully resolved. We have monitored our incoming logs and confirmed the disconnects have stopped occurring since we rolled out our fix to all customers. The root cause was confirmed to be the update in the "Summer 24" release from Salesforce that blocked simultaneous refresh tokens. We resolved the issue by adding a more robust retry strategy in addition to better logic in how we handle refresh tokens. As these changes were not trivial and required significant engineering resources, the fix took longer than usual to develop. As an outcome, the retry mechanism will remain in place and our notification emails related to dropped connections were adjusted to better guide admins should any unrelated disconnect issues occur in the future.

Time: June 21, 2024, 3:14 p.m.

Status: Monitoring

Update: Our fix is now deployed to our production environments, and we are now monitoring in realtime to validate that the fix is working as expected. Please make our team aware of any additional disconnects you experience via [email protected] so that we can act quickly to investigate it further.

Time: June 21, 2024, 1:05 p.m.

Status: Identified

Update: We have successfully replicated and validated the issue's root cause through testing. Our solution is confirmed to resolve the disconnects and will be deployed to our production environment shortly. Once it has deployed to all customers, we will provide another update to monitor the fix before confirming it is fully resolved.

Time: June 20, 2024, 7:56 p.m.

Status: Identified

Update: We have begun deploying some solutions to our testing environments to start validating them internally. Once we have completed tests we will start rolling it out to customers to monitor the fix. We have also updated our automated email sent to admins in order to make it clearer where to go in order to reconnect. Previously it had indicated the personal-level connection, but please be sure to reconnect the global admin-level connection as the new notification indicates.

Time: June 20, 2024, 3:10 p.m.

Status: Investigating

Update: We understand the urgency and concern this incident has caused. The resolution is taking longer than anticipated due to the need for significant changes in our authentication infrastructure rather than if it was a more trivial bug. This issue has been escalated internally and is our top priority. We've allocated additional engineering resources to expedite the fix so we can get it rolled out as soon as possible. We aim to have a solution deployed to our testing environment before the end of today. Once verified through quality assurance, it will be rolled out to all customers. In the meantime, please continue to monitor your integration user email for any Salesforce disconnects. Thanks again for your patience and understanding.

Time: June 18, 2024, 6:21 p.m.

Status: Investigating

Update: We are currently investigating an issue affecting the global Salesforce connection (integration user) for a small subset of customers using our legacy product line. Please note that our Demand Conversion Platform products and HubSpot CRM customers are not impacted. While disconnected, there may be issues with creating or updating Lead records or Events. Additionally, routing will be limited to form data and will not function properly based on existing Lead/Contact data. Our preliminary findings suggest that recent updates from Salesforce, which block specific refresh token strategies, may be the underlying cause. We are working on a fix to implement a more robust refresh token logic. We have confirmed that a notification email is sent to the integration user email account when a disconnection occurs, prompting them to reconnect as soon as possible. We are actively working on a solution. In the meantime, please be aware that the service may disconnect again even after reconnecting. To mitigate this, we have implemented active monitoring to detect disconnections promptly and may reach out directly if needed. We urge admins to monitor their integration user inbox closely to ensure timely action in the event of a disconnection. Thank you for your understanding and patience as we resolve this issue.

Time: Dec. 16, 2023, 4:47 p.m.

Status: Resolved

Update: Our team identified an issue that's preventing Chili Piper from successfully trigger upon form submission. We have quickly resolved the issue by applying a hotfix. Chili Piper show now be triggering properly and the calendar would appear when a prospect successfully qualifies for a booking.

Time: Dec. 6, 2023, 3:09 p.m.

Status: Resolved

Update: Our team identified an issue with a new security policy that was preventing Chili Piper assets from loading. Our engineers have already deployed a fix for this and we should see it resolved. While this was impacting Chili Piper, routing and booking links were impacted as well as internal pages and they were instead presenting a blank page or could not be loaded. We have resolved the issue by reverting the change and have corrected the underlying issue moving forward.