Time: July 3, 2022, 12:09 p.m.

Status: Resolved

Update: The newest version of the Chrome Extension was approved by the Chrome store and this is available to everyone (version 3.0.486): https://chrome.google.com/webstore/detail/chili-piper-automate-sche/digckgagekofoajjlaeceojmdokllfmf If you have uninstalled the extension, please just install it back and it will be with the proper version. If you have only disabled it, please follow these steps: 1. Go to the extension settings at chrome://extensions/?id=digckgagekofoajjlaeceojmdokllfmf 2. At the top-right of the page, click on the toggle to enable the extension again; 3. At the top-left, click on the "Update" button. As soon as you confirm that the extension is now active and have the v3.0.486 installed, you can remove the local copy of our extension previously installed: 1. Go to this page: chrome://extensions/?id=kdmalpdadhpjffdocncnecbpkcgkngge 2. At the end of the page, you can click on "Remove extension". Thank you for your understanding and patience on this and we are sorry for any inconvenience this issue might have caused you.

Time: July 1, 2022, 4:21 p.m.

Status: Monitoring

Update: We continue to wait on the Chrome store approval process to publish the changes we have made to the Chrome Extension. Though we have no ETA on this, we expect it should be any moment now. For some additional information into next steps - We have since fixed the pipeline to ensure this is built correctly in the future and have additionally assigned resources to manually test these builds across all environments before release. Any users who have installed the local copy of our extension will eventually need to remove it and install the public copy again. We will provide steps for this once the extension is fixed in the store, and this will not urgently need to happen immediately. In the meantime, please continue to follow the steps listed in our knowledge base article for a temporary solution: https://help.chilipiper.com/hc/en-us/articles/7406298455059

Time: June 30, 2022, 7:08 p.m.

Status: Monitoring

Update: Per our previous update we are still waiting on the Chrome app review process. Unfortunately, this can take some time for Google to complete and although we don't have an exact ETA, it can take up to 24 hours to approve. In the meantime, we have released a local download copy of our fix that can be deployed immediately. Steps are provided in our helpdesk article as well as a video highlighting the step-by-step process. We have confirmed that this local copy resolves the issue once installed. To view these steps and download a local copy, review this page here: https://help.chilipiper.com/hc/en-us/articles/7406298455059 If you opt not to install the local copy the Chrome extension will eventually update automatically once approved on Google's end with no action needed on your end. We will continue to monitor the fix and ensure we provide an update here once it has rolled out to the majority of customers via the Chrome web store.

Time: June 30, 2022, 2:35 p.m.

Status: Monitoring

Update: Our engineering team identified the cause of the issue and a fix was created. The new version was sent to Chrome Review, which can one day or more. In the meantime, please follow the steps described in the following article to make the extension operational again: https://help.chilipiper.com/hc/en-us/articles/7406298455059

Time: June 30, 2022, 1:55 p.m.

Status: Investigating

Update: The Chrome Extension is currently not working for customers with instances on the production cluster. The issue is not affecting instances running on the canary cluster. Our engineering team is already analyzing the issue to provide a fix.

Time: June 4, 2022, 12:05 a.m.

Status: Resolved

Update: We are considering this incident resolved as the overall remaining database refresh is no longer impacting major services. Some sub-services may still experience temporary issues including the sending of reminders, and email notifications. If these issues occur, please wait 24 hours and they should be resolved.

Time: June 4, 2022, 12:05 a.m.

Status: Resolved

Update: We are considering this incident resolved as the overall remaining database refresh is no longer impacting major services. Some sub-services may still experience temporary issues including the sending of reminders, and email notifications. If these issues occur, please wait 24 hours and they should be resolved.

Time: June 3, 2022, 10:07 p.m.

Status: Identified

Update: The impact of this incident has been minimized as we continue to make progress on the database recovery. We are monitoring this closely and will provide another update once services have returned to a healthy status for the subset of customers impacted.

Time: June 3, 2022, 10:07 p.m.

Status: Identified

Update: The impact of this incident has been minimized as we continue to make progress on the database recovery. We are monitoring this closely and will provide another update once services have returned to a healthy status for the subset of customers impacted.

Time: June 3, 2022, 5:27 p.m.

Status: Identified

Update: Our engineers identified the root cause quickly and have restored most services, including booking flows for the subset of impacted customers. Some features will remain offline such as reminders, reports, and automated services until the databases are fully operational.

Time: June 3, 2022, 5:27 p.m.

Status: Identified

Update: Our engineers identified the root cause quickly and have restored most services, including booking flows for the subset of impacted customers. Some features will remain offline such as reminders, reports, and automated services until the databases are fully operational.

Time: June 3, 2022, 5:23 p.m.

Status: Investigating

Update: We are seeing new instances of booking links, and Chili Piper admin pages timing out with "502 Errors'. Out engineering team is investigating currently. The impact appears to be in a smaller subset of databases than yesterday, so the overall resolution will be faster. We will provide additional updates here as we have them.

Time: June 3, 2022, 5:23 p.m.

Status: Investigating

Update: We are seeing new instances of booking links, and Chili Piper admin pages timing out with "502 Errors'. Out engineering team is investigating currently. The impact appears to be in a smaller subset of databases than yesterday, so the overall resolution will be faster. We will provide additional updates here as we have them.

Time: June 3, 2022, 2:04 a.m.

Status: Resolved

Update: We have restored all services and databases to fully operational status. Our team will continue to monitor the results to make certain these services remain healthy and have provided additional resources to ensure stability moving forward.

Time: June 2, 2022, 8:16 p.m.

Status: Identified

Update: Our team is making good progress in restoring the database services that are still impacted and this may take take an extended amount of time to complete. While this is in progress you may experience issues with reporting, reminders, and some automated services, however booking flows and most admin functions should work normally. There will be no loss of data. Stability of these services is extremely important to us and we are working to ensure this is more robust moving forward. We will provide another update once all services have been restored.

Time: June 2, 2022, 5:15 p.m.

Status: Identified

Update: Impact has been minimized and we have a clear plan in place to get back to fully operational mode without any loss of data - we are executing on it right now. Once we're fully healthy we'll re-enable all less essential services such as reminders and we'll investigate what happened in detail.

Time: June 2, 2022, 4:42 p.m.

Status: Identified

Update: Our engineers restored all database instances to a 'partial mode' - Chili Piper is now loading as expected and booking flows are back to normal, but reminders, reporting and some automated processes still need to be restored. We are working on restoring all functionality, we will update you once the process is finalized.

Time: June 2, 2022, 4:20 p.m.

Status: Identified

Update: Our Team has identified the issue as stemming from the primary database. We are working on a plan to restore functionality without losing any data. We will keep you update!

Time: June 2, 2022, 4:10 p.m.

Status: Investigating

Update: We are continuing to investigate the root cause with our Reliability Team. The issue manifests in not being able to log into the Dashboard or book meetings. We will keep you updated as new details emerge.

Time: June 2, 2022, 3:47 p.m.

Status: Investigating

Update: We are seeing instances of the Dashboard loading slowly and timing out with 502 errors. Our Engineering Team is investigating

Time: June 3, 2022, 2:04 a.m.

Status: Resolved

Update: We have restored all services and databases to fully operational status. Our team will continue to monitor the results to make certain these services remain healthy and have provided additional resources to ensure stability moving forward.

Time: June 2, 2022, 8:16 p.m.

Status: Identified

Update: Our team is making good progress in restoring the database services that are still impacted and this may take take an extended amount of time to complete. While this is in progress you may experience issues with reporting, reminders, and some automated services, however booking flows and most admin functions should work normally. There will be no loss of data. Stability of these services is extremely important to us and we are working to ensure this is more robust moving forward. We will provide another update once all services have been restored.

Time: June 2, 2022, 5:15 p.m.

Status: Identified

Update: Impact has been minimized and we have a clear plan in place to get back to fully operational mode without any loss of data - we are executing on it right now. Once we're fully healthy we'll re-enable all less essential services such as reminders and we'll investigate what happened in detail.

Time: June 2, 2022, 4:42 p.m.

Status: Identified

Update: Our engineers restored all database instances to a 'partial mode' - Chili Piper is now loading as expected and booking flows are back to normal, but reminders, reporting and some automated processes still need to be restored. We are working on restoring all functionality, we will update you once the process is finalized.

Time: June 2, 2022, 4:20 p.m.

Status: Identified

Update: Our Team has identified the issue as stemming from the primary database. We are working on a plan to restore functionality without losing any data. We will keep you update!

Time: June 2, 2022, 4:10 p.m.

Status: Investigating

Update: We are continuing to investigate the root cause with our Reliability Team. The issue manifests in not being able to log into the Dashboard or book meetings. We will keep you updated as new details emerge.

Time: June 2, 2022, 3:47 p.m.

Status: Investigating

Update: We are seeing instances of the Dashboard loading slowly and timing out with 502 errors. Our Engineering Team is investigating

Time: May 27, 2022, 9:54 p.m.

Status: Resolved

Update: This incident has been resolved. We are continuing to monitor, but have seen a significantly steep decline in error rates that are now within normal limits. For most customers, no further action is required. If you notice issues with routing, 500 errors, or issues connecting to the CRM, please login as admin and reconnect the sync user at https://apps.na.chilipiper.com/admin-center/integrations. Doing so should put you back in business.

Time: May 27, 2022, 9:54 p.m.

Status: Resolved

Update: This incident has been resolved. We are continuing to monitor, but have seen a significantly steep decline in error rates that are now within normal limits. For most customers, no further action is required. If you notice issues with routing, 500 errors, or issues connecting to the CRM, please login as admin and reconnect the sync user at https://apps.na.chilipiper.com/admin-center/integrations. Doing so should put you back in business.

Time: May 27, 2022, 9:10 p.m.

Status: Monitoring

Update: We have deployed a few hotfixes and are monitoring the results now. Customer accounts appear to be recovering after a short wait. In the meantime we will continue to keep an eye on error rates to ensure they are dropping. Once we have confirmed the resolution, we will provide another update.

Time: May 27, 2022, 9:10 p.m.

Status: Monitoring

Update: We have deployed a few hotfixes and are monitoring the results now. Customer accounts appear to be recovering after a short wait. In the meantime we will continue to keep an eye on error rates to ensure they are dropping. Once we have confirmed the resolution, we will provide another update.

Time: May 27, 2022, 8 p.m.

Status: Identified

Update: Please be aware that HubSpot CRM-connected customers may experience intermittent issues related to their connection. The scope of this issue can cause problems routing via web form, logging in with HubSpot, setting queue rules, refreshing fields, or doing Contact update/search. One primary symptom is hitting a "500 error" when attempting to submit a form or do admin tasks. Our engineers have identified several root causes and are working to mitigate this. Additionally, we are refactoring the way these requests happen to ensure this does not impact connections in the future. We will have more information on a resolution soon.

Time: May 27, 2022, 8 p.m.

Status: Identified

Update: Please be aware that HubSpot CRM-connected customers may experience intermittent issues related to their connection. The scope of this issue can cause problems routing via web form, logging in with HubSpot, setting queue rules, refreshing fields, or doing Contact update/search. One primary symptom is hitting a "500 error" when attempting to submit a form or do admin tasks. Our engineers have identified several root causes and are working to mitigate this. Additionally, we are refactoring the way these requests happen to ensure this does not impact connections in the future. We will have more information on a resolution soon.