Time: Nov. 11, 2022, 9:19 a.m.

Status: Resolved

Update: Kustomer has resolved an event affecting Searches and Inbox Views in PROD2 instances. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 11, 2022, 8:35 a.m.

Status: Monitoring

Update: Kustomer has implemented an update to address an event impacting Searches and Inbox Views and seeing events are now working as normal. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further half-hourly updates and reach out to Support team at [email protected] if you have additional questions or concerns.

Time: Nov. 11, 2022, 8:14 a.m.

Status: Identified

Update: Kustomer is aware of an event affecting Searches and Inbox Views in PROD2 instances not updating within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect further half-hourly and reach out to Support team at [email protected] if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:41 p.m.

Status: Resolved

Update: Kustomer has resolved an event affecting Knowledge Bases using custom domains in Prod 1 and Prod 2 throwing 502 errors. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:41 p.m.

Status: Resolved

Update: Kustomer has resolved an event affecting Knowledge Bases using custom domains in Prod 1 and Prod 2 throwing 502 errors. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:34 p.m.

Status: Monitoring

Update: Kustomer has implemented an update to address an event affecting Knowledge Bases using custom domains on Prod 1 and Prod 2 throwing 502 errors. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 3 hours and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:34 p.m.

Status: Monitoring

Update: Kustomer has implemented an update to address an event affecting Knowledge Bases using custom domains on Prod 1 and Prod 2 throwing 502 errors. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 3 hours and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:16 p.m.

Status: Identified

Update: Kustomer has identified an event affecting Knowledge Bases using custom domains on Prod 1 and Prod 2 that may cause 502 Errors to be thrown. Our team is currently working to design and implement a resolution. Please expect further updates within 30 minutes and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:16 p.m.

Status: Identified

Update: Kustomer has identified an event affecting Knowledge Bases using custom domains on Prod 1 and Prod 2 that may cause 502 Errors to be thrown. Our team is currently working to design and implement a resolution. Please expect further updates within 30 minutes and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:12 p.m.

Status: Investigating

Update: Kustomer is aware of an event affecting Knowledge Bases on a custom domain that may cause 502 Error within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. If you need to access your KB you can do so by clicking on the internal link in Knowledge Base -> Configurations and then under the Internal KB domain section. Please expect further updates within 30 minutes and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:12 p.m.

Status: Investigating

Update: Kustomer is aware of an event affecting Knowledge Bases on a custom domain that may cause 502 Error within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. If you need to access your KB you can do so by clicking on the internal link in Knowledge Base -> Configurations and then under the Internal KB domain section. Please expect further updates within 30 minutes and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:11 p.m.

Status: Investigating

Update: Kustomer is aware of an event affecting Knowledge Bases on a custom domain that may cause 502 Error within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. If you need to access your KB you can do so by clicking on the internal link in Knowledge Base -> Configurations and then under the Internal KB domain section. Please expect further updates within 30 minutes and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 8, 2022, 10:11 p.m.

Status: Investigating

Update: Kustomer is aware of an event affecting Knowledge Bases on a custom domain that may cause 502 Error within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. If you need to access your KB you can do so by clicking on the internal link in Knowledge Base -> Configurations and then under the Internal KB domain section. Please expect further updates within 30 minutes and reach out to support at [email protected] or over chat if you have additional questions or concerns.

Time: Nov. 11, 2022, 5:47 p.m.

Status: Postmortem

Update: # **Summary** From 1:54 PM to 3:32 PM on 2022-11-02, assisted conversations disappeared from timelines exactly one hour after the time of their creation. All data was subsequently restored to timelines. **Root Cause** This incident came about during the development of an assistant preview feature, which involves the “soft removal” of preview conversations after time has elapsed. The affected conversations were mislabeled as previewed conversations. # **Timeline** **11/02 ~1:54 PM -** Assisted conversations start disappearing from timelines **11/02  3:12 PM** - First reports of missing conversations begin to appear in internal channels. **11/02 3:30 PM** - Rollback of affected services begins **11/02 3:32 PM** - Conversations stop disappearing from timelines **11/03 ~4:30 PM** - Timeline restoration completes for EU orgs **11/03 7:00 PM** - Prod1 data restoration completes for Non-EU orgs **11/03 10:46 PM** - Search restored for Non-EU orgs **11/03 10:52 PM** - Search restored for EU orgs **11/03 11:12 PM** - Incident is marked resolved ‌ **Lessons/Improvements** The engineering team is responding to this incident by: * Building testing guardrails to prevent similar changes from entering production in the future * Implementing event monitoring to more quickly identify and respond to similar incidents

Time: Nov. 4, 2022, 3:15 a.m.

Status: Resolved

Update: Our engineers have restored all conversations back to their original timeline and have reindexed the conversations so they are now available in searches. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 4, 2022, 12:02 a.m.

Status: Monitoring

Update: Our engineers have restored all conversations back to their original timeline. We are now working to reindex those affected conversations so they are searchable in Kustomer. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 6:11 p.m.

Status: Monitoring

Update: Our engineers continue working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 11:05 a.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers continue working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 5:56 a.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers are currently working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 2:02 a.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers are currently working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 9:38 p.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers are currently working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 8:37 p.m.

Status: Identified

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineering team is working to restore all conversations and messages that disappeared from the customer timeline. Please expect further updates within 1 hour, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 7:57 p.m.

Status: Identified

Update: Kustomer is aware of an event affecting Conversational Assistants that may cause conversations disappearing from the timeline within the platform. Our team has identified the cause of this issue and is working to implement a resolution. Please expect further updates within 1 hour, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 7:43 p.m.

Status: Investigating

Update: Kustomer has identified an event affecting Timelines on Prod 1 and 2 that may cause Assistant Conversations to not appear in the timeline. Our team is currently working to design and implement a resolution. Please expect further updates within 30 minutes, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 11, 2022, 5:47 p.m.

Status: Postmortem

Update: # **Summary** From 1:54 PM to 3:32 PM on 2022-11-02, assisted conversations disappeared from timelines exactly one hour after the time of their creation. All data was subsequently restored to timelines. **Root Cause** This incident came about during the development of an assistant preview feature, which involves the “soft removal” of preview conversations after time has elapsed. The affected conversations were mislabeled as previewed conversations. # **Timeline** **11/02 ~1:54 PM -** Assisted conversations start disappearing from timelines **11/02  3:12 PM** - First reports of missing conversations begin to appear in internal channels. **11/02 3:30 PM** - Rollback of affected services begins **11/02 3:32 PM** - Conversations stop disappearing from timelines **11/03 ~4:30 PM** - Timeline restoration completes for EU orgs **11/03 7:00 PM** - Prod1 data restoration completes for Non-EU orgs **11/03 10:46 PM** - Search restored for Non-EU orgs **11/03 10:52 PM** - Search restored for EU orgs **11/03 11:12 PM** - Incident is marked resolved ‌ **Lessons/Improvements** The engineering team is responding to this incident by: * Building testing guardrails to prevent similar changes from entering production in the future * Implementing event monitoring to more quickly identify and respond to similar incidents

Time: Nov. 4, 2022, 3:15 a.m.

Status: Resolved

Update: Our engineers have restored all conversations back to their original timeline and have reindexed the conversations so they are now available in searches. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 4, 2022, 12:02 a.m.

Status: Monitoring

Update: Our engineers have restored all conversations back to their original timeline. We are now working to reindex those affected conversations so they are searchable in Kustomer. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 6:11 p.m.

Status: Monitoring

Update: Our engineers continue working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 11:05 a.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers continue working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to Support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 5:56 a.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers are currently working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 3, 2022, 2:02 a.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers are currently working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 9:38 p.m.

Status: Monitoring

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineers are currently working on restoring all conversations back to their original timeline. Please expect further updates within 4 hours, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 8:37 p.m.

Status: Identified

Update: Kustomer has resolved the issue with Conversational Assistant conversations disappearing from the timeline. Our engineering team is working to restore all conversations and messages that disappeared from the customer timeline. Please expect further updates within 1 hour, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 7:57 p.m.

Status: Identified

Update: Kustomer is aware of an event affecting Conversational Assistants that may cause conversations disappearing from the timeline within the platform. Our team has identified the cause of this issue and is working to implement a resolution. Please expect further updates within 1 hour, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Nov. 2, 2022, 7:43 p.m.

Status: Investigating

Update: Kustomer has identified an event affecting Timelines on Prod 1 and 2 that may cause Assistant Conversations to not appear in the timeline. Our team is currently working to design and implement a resolution. Please expect further updates within 30 minutes, and reach out to support at [email protected] if you have additional questions or concerns.

Time: Oct. 31, 2022, 8:58 p.m.

Status: Postmortem

Update: ## Summary At 11:57am ET on October 28th, 2022, a deployment in Kustomer’s public chat SDK caused an issue with the UI on all consuming pages. This issue added an invisible overlay on the bottom corner \(left or right depending on implementation\) of 400px wide by 650px tall on all pages where Kustomer’s chat was hosted. This invisible overlay prevented end users from interacting with the hosted page under the overlay. This problem was caught by our Customer Experience team internally paired with reports from external customers. The issue was assessed by our engineering team, creating a revert that was merged and deployed, fixing the issue by 12:49pm. The root cause of this issue has been addressed and fixed, with added tests to ensure the sizing functionality used to show and hide Kustomer Chat does not perform in a similar manner in the future. ## Timeline of Events * 10/28 11:57 pm - UI change is deployed to production * 10/28 12:15pm - Support reaches out to engineering team to assess the problem on the internal Kustomer instance * 10/28 12:21pm - The issue is diagnosed by our engineering team * 10/28 12:32pm - A revert of the erroneous code is created * 10/28 12:49pm - The reverted code is deployed to production ## Lessons/Improvements * Add more robust end-to-end tests around resizing Chat \[IN PROGRESS\] * Add end-to-end tests for triggering Chat and also interacting with the parent page * Investigate creation of internal deployment canaries around UI interactions to potentially catch issues like this one sooner

Time: Oct. 31, 2022, 8:58 p.m.

Status: Resolved

Update: At 11:57am ET on October 28th, 2022, a deployment in Kustomer’s public chat SDK caused an issue with the UI on all consuming pages. This issue added an invisible overlay on the bottom corner (left or right depending on implementation) of 400px wide by 650px tall on all pages where Kustomer’s chat was hosted. This invisible overlay prevented end users from interacting with the hosted page under the overlay. This problem was caught by our Customer Experience team internally paired with reports from external customers. The issue was assessed by our engineering team, creating a revert that was merged and deployed, fixing the issue by 12:49pm. The root cause of this issue has been addressed and fixed, with added tests to ensure the sizing functionality used to show and hide Kustomer Chat does not perform in a similar manner in the future.