Time: April 15, 2022, 10:36 p.m.

Status: Resolved

Update: After continuous monitoring, we are confirming this incident is resolved. Our team will continue to monitor the status of Google connections across our infrastructure to ensure stability is maintained. Users who were impacted by the initial incident may find that their calendars are still disconnected. The resolution of this incident does not retroactively restore connections. If you have a user that is disconnected, please ensure they reconnect by logging into Chili Piper with their Google credentials.

Time: April 15, 2022, 1 p.m.

Status: Monitoring

Update: We are working to implement fixes for the identified issues and are monitoring the results. Most users should have at least attempted to reconnect their Calendar and we have noticed a significant drop in error rates. Please continue to reconnect your calendars and the calendars of assignees / bookers if trailing reports come in. Once reconnected, the issues should be fully mitigated for those users. If you are experiencing critical issues as a result of the incident, or experiencing difficulty reconnecting, please reach out to our Department of Customer Love support team at [email protected] or your dedicated Customer Success Manager so that we can investigate.

Time: April 14, 2022, 8:56 p.m.

Status: Identified

Update: Our engineering team is working to get the identified issues resolved as quickly as possible, and progress is being made. We urge customers who are impacted to continue to attempt to reconnect their Calendar if they are experiencing unexpected behaviors while using Chili Piper, even if the UI indicates you are fully connected. Login with Google credentials: https://apps.chilipiper.com/login.html?success=true to attempt to reconnect. We have learned that multiple attempts may be required in order for the new credentials to take hold. The impacting issues include, but are not limited to, calendars showing no availability or not loading at all, Instant Booker showing issues sharing or viewing calendars, and issues where the Booker of the meeting receives the event (as organizer) rather than the Assignee.

Time: April 13, 2022, 10:37 p.m.

Status: Identified

Update: We have identified some of the underlying root causes of Google calendars disconnects and are continuing to investigate a resolution. It is still recommended that impacted users reconnect their calendars as this resolves the problem in most cases. This can be done by simply logging into Chili Piper and using your Google calendar as the login option here: https://apps.chilipiper.com/login.html?success=true We will continue to provide the latest updates here as they become available.

Time: April 11, 2022, 7:11 p.m.

Status: Investigating

Update: We are continuing to investigate the issue and are narrowing down the root cause. Impact: - You may see issues displaying the proper availability on booking links or within Instant Booker. - If you are the "Default Booker" within a Concierge queue, you may get more meetings booked to your calendar. - Updates to Google events such as reschedules or cancellations may not happen automatically. As noted previously, reconnecting Google on a user-level renews disconnection and permanently fixes the issue. If you are impacted, check out our help article for the steps to reconnect: https://help.chilipiper.com/hc/en-us/articles/360053800193 Please stay tuned here for updates.

Time: April 11, 2022, 3:57 p.m.

Status: Investigating

Update: Our engineering team is still investigating the Gcal token disconnects - the issue seems to be related to Google API's permissions scopes. Reconnecting Google on an individual level renews the token and permanently fixes the issue. If you are impacted, check out our help article for the steps to reconnect: https://help.chilipiper.com/hc/en-us/articles/360053800193 We'll update this incident with more details.

Time: April 11, 2022, 2:40 p.m.

Status: Investigating

Update: We are currently investigating an issue with some Google Calendar users not able to book meetings due to Gcal being disconnected. Only a minority of Gcal customers are experiencing this issue, O365 customers are not affected. If booking via Instant Booker, the booker will see the message "You can't book meetings in this calendar'. If booking via a personal or queue link, the prospect will see the message "Select a different slot" To fix this immediately, please change the default booker to a different user with a valid calendar connection, or have the default booker reconnect their calendar under https://apps.chilipiper.com/meetings/personal-settings/integrations Check out this help article for detailed steps: https://help.chilipiper.com/hc/en-us/articles/360053800193 We'll keep you updated via this thread.

Time: April 15, 2022, 10:36 p.m.

Status: Resolved

Update: After continuous monitoring, we are confirming this incident is resolved. Our team will continue to monitor the status of Google connections across our infrastructure to ensure stability is maintained. Users who were impacted by the initial incident may find that their calendars are still disconnected. The resolution of this incident does not retroactively restore connections. If you have a user that is disconnected, please ensure they reconnect by logging into Chili Piper with their Google credentials.

Time: April 15, 2022, 1 p.m.

Status: Monitoring

Update: We are working to implement fixes for the identified issues and are monitoring the results. Most users should have at least attempted to reconnect their Calendar and we have noticed a significant drop in error rates. Please continue to reconnect your calendars and the calendars of assignees / bookers if trailing reports come in. Once reconnected, the issues should be fully mitigated for those users. If you are experiencing critical issues as a result of the incident, or experiencing difficulty reconnecting, please reach out to our Department of Customer Love support team at [email protected] or your dedicated Customer Success Manager so that we can investigate.

Time: April 14, 2022, 8:56 p.m.

Status: Identified

Update: Our engineering team is working to get the identified issues resolved as quickly as possible, and progress is being made. We urge customers who are impacted to continue to attempt to reconnect their Calendar if they are experiencing unexpected behaviors while using Chili Piper, even if the UI indicates you are fully connected. Login with Google credentials: https://apps.chilipiper.com/login.html?success=true to attempt to reconnect. We have learned that multiple attempts may be required in order for the new credentials to take hold. The impacting issues include, but are not limited to, calendars showing no availability or not loading at all, Instant Booker showing issues sharing or viewing calendars, and issues where the Booker of the meeting receives the event (as organizer) rather than the Assignee.

Time: April 13, 2022, 10:37 p.m.

Status: Identified

Update: We have identified some of the underlying root causes of Google calendars disconnects and are continuing to investigate a resolution. It is still recommended that impacted users reconnect their calendars as this resolves the problem in most cases. This can be done by simply logging into Chili Piper and using your Google calendar as the login option here: https://apps.chilipiper.com/login.html?success=true We will continue to provide the latest updates here as they become available.

Time: April 11, 2022, 7:11 p.m.

Status: Investigating

Update: We are continuing to investigate the issue and are narrowing down the root cause. Impact: - You may see issues displaying the proper availability on booking links or within Instant Booker. - If you are the "Default Booker" within a Concierge queue, you may get more meetings booked to your calendar. - Updates to Google events such as reschedules or cancellations may not happen automatically. As noted previously, reconnecting Google on a user-level renews disconnection and permanently fixes the issue. If you are impacted, check out our help article for the steps to reconnect: https://help.chilipiper.com/hc/en-us/articles/360053800193 Please stay tuned here for updates.

Time: April 11, 2022, 3:57 p.m.

Status: Investigating

Update: Our engineering team is still investigating the Gcal token disconnects - the issue seems to be related to Google API's permissions scopes. Reconnecting Google on an individual level renews the token and permanently fixes the issue. If you are impacted, check out our help article for the steps to reconnect: https://help.chilipiper.com/hc/en-us/articles/360053800193 We'll update this incident with more details.

Time: April 11, 2022, 2:40 p.m.

Status: Investigating

Update: We are currently investigating an issue with some Google Calendar users not able to book meetings due to Gcal being disconnected. Only a minority of Gcal customers are experiencing this issue, O365 customers are not affected. If booking via Instant Booker, the booker will see the message "You can't book meetings in this calendar'. If booking via a personal or queue link, the prospect will see the message "Select a different slot" To fix this immediately, please change the default booker to a different user with a valid calendar connection, or have the default booker reconnect their calendar under https://apps.chilipiper.com/meetings/personal-settings/integrations Check out this help article for detailed steps: https://help.chilipiper.com/hc/en-us/articles/360053800193 We'll keep you updated via this thread.

Time: April 7, 2022, 4:38 p.m.

Status: Resolved

Update: Issue is resolved and Chili Piper is loading as expected. Our engineering has deployes further infrastructures changes to make sure this doesn't happen in the future. Thank you for your patience!

Time: April 7, 2022, 4:20 p.m.

Status: Monitoring

Update: We have identified the issue and we are now seeing a decrease in loading issues - this is now resolved. We are monitoring and working on a Root Cause Analysis.

Time: April 7, 2022, 3:47 p.m.

Status: Investigating

Update: Some customers are experiencing slowness in loading Chili Piper, our Reliability Engineering Team are looking into it. Routing functionality is not impacted. We'll update this post with updates

Time: April 7, 2022, 4:38 p.m.

Status: Resolved

Update: Issue is resolved and Chili Piper is loading as expected. Our engineering has deployes further infrastructures changes to make sure this doesn't happen in the future. Thank you for your patience!

Time: April 7, 2022, 4:20 p.m.

Status: Monitoring

Update: We have identified the issue and we are now seeing a decrease in loading issues - this is now resolved. We are monitoring and working on a Root Cause Analysis.

Time: April 7, 2022, 3:47 p.m.

Status: Investigating

Update: Some customers are experiencing slowness in loading Chili Piper, our Reliability Engineering Team are looking into it. Routing functionality is not impacted. We'll update this post with updates

Time: Jan. 6, 2022, 2:15 p.m.

Status: Resolved

Update: After an hour of monitoring, we confirm the database issue has been resolved. The booking links and queues links are properly working on the Canary cluster.

Time: Jan. 6, 2022, 2:15 p.m.

Status: Resolved

Update: After an hour of monitoring, we confirm the database issue has been resolved. The booking links and queues links are properly working on the Canary cluster.

Time: Jan. 6, 2022, 1:13 p.m.

Status: Monitoring

Update: A fix has been implemented on the database. Booking links and queue links are operational again for the Canary Cluster. Our engineering team is currently monitoring it.

Time: Jan. 6, 2022, 1:13 p.m.

Status: Monitoring

Update: A fix has been implemented on the database. Booking links and queue links are operational again for the Canary Cluster. Our engineering team is currently monitoring it.

Time: Jan. 6, 2022, 12:50 p.m.

Status: Identified

Update: Our engineering team has identified the issue as stemming from the database of the Canary cluster. We are currently working on restoring the functionality.

Time: Jan. 6, 2022, 12:50 p.m.

Status: Identified

Update: Our engineering team has identified the issue as stemming from the database of the Canary cluster. We are currently working on restoring the functionality.

Time: Jan. 6, 2022, 12:27 p.m.

Status: Investigating

Update: Our engineering team is still investigating the issue. Booking links and queue links are affected for the Canary cluster, a small subset of users - production customers are not affected. We'll continue to keep you updated, thank you for your patience!

Time: Jan. 6, 2022, 12:27 p.m.

Status: Investigating

Update: Our engineering team is still investigating the issue. Booking links and queue links are affected for the Canary cluster, a small subset of users - production customers are not affected. We'll continue to keep you updated, thank you for your patience!

Time: Jan. 6, 2022, 11:59 a.m.

Status: Investigating

Update: We are investigating an issue with booking links not working - only our Canary cluster is affected. Dashboard is loading as expected, but individual booking link or links inserted via Instant Booker are affected. We'll keep you updated.