Time: Feb. 3, 2022, 5:42 p.m.

Status: Resolved

Update: This is a retrospective about an incident we discovered at 9:50am PT and immediately recovered from (the incident had started at 7:57am PT). A small subset of Tours with a specific legacy configuration [1] were displayed on pages that did not match the URL Rules. This resulted in Tours being erroneously displayed to users who would not have otherwise have seen the Tour. All other rules were correctly applied including Segmentation, Element matching etc. Upon learning of this issue we immediately rolled back the change and invalidated all relevant caches. We are following up with our customers directly to provide details about how this incident impacted their Tours and Users. Please feel free to reach out for more information. [1] We added support for multiple URL matching conditions (~11 months ago) and continued to support the legacy configuration of having only one URL match, when we deployed an update to how URL matching was handled the legacy configuration was omitted from the new code (and should not have been). Upon learning of this issue we immediately rolled back the change and invalidated all relevant caches.

Time: Jan. 3, 2022, 10:11 p.m.

Status: Resolved

Update: Chameleon nor any of our vendors are vulnerable; we have evaluated our systems and the systems of key vendors to check for the presence of Apache Log4j in reference to the Log4Shell vulnerabilities (CVE-2021-45046, CVE-2021-44228). We will continue to use best in class tools and continue to monitor the situation for any updates.

Time: Jan. 3, 2022, 10:11 p.m.

Status: Resolved

Update: Chameleon nor any of our vendors are vulnerable; we have evaluated our systems and the systems of key vendors to check for the presence of Apache Log4j in reference to the Log4Shell vulnerabilities (CVE-2021-45046, CVE-2021-44228). We will continue to use best in class tools and continue to monitor the situation for any updates.

Time: Dec. 8, 2021, 5:15 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 7, 2021, 10:50 p.m.

Status: Monitoring

Update: The main impact of this incident is now Resolved for the Chameleon Data API. For a period of ~6 hours today the autoscaler (and and turn the manual scaler) was not able to add additional resources to meet the typical morning/afternoon needs of the Data API. Any data that was received by Chameleon is now available for display on the dashboard and for access via API. As mentioned previously, all other aspects of Chameleon remained online during this incident (we used Chameleon to update our customers in the Chameleon dashboard 🎉 http://is.am/4qxa). We will hold this incident open until AWS and Heroku mark it as resolved, but since the impact to Chameleon is resolved, this will be the last email/slack alert you receive about this. Thank you for your patience today.

Time: Dec. 7, 2021, 10:13 p.m.

Status: Identified

Update: The referenced issue on heroku is here => https://status.heroku.com/incidents/2390 - we're still keeping up with updates as they come and will report back here when we have a resolution.

Time: Dec. 7, 2021, 5:04 p.m.

Status: Identified

Update: Due to the API and console outage at AWS us-east-1 and downstream for the Heroku API etc. the Data API will likely remain unavailable until those issues are resolved. Data sent to Chameleon, your Chameleon Experiences (live to your end-users), editing and publishing Experiences are not impacted at this time.

Time: Dec. 7, 2021, 4:31 p.m.

Status: Investigating

Update: We are currently seeing Segmentation and Analytics data fail to load on the Chameleon dashboard and via API requests. We're currently investigating the cause of the issue, and will continue to provide updates as more information becomes available. URLs affected but this: https://api.trychameleon.com/v3/analyze/* and https://ultraviolet.trychameleon.com/*

Time: Dec. 8, 2021, 5:15 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 7, 2021, 10:50 p.m.

Status: Monitoring

Update: The main impact of this incident is now Resolved for the Chameleon Data API. For a period of ~6 hours today the autoscaler (and and turn the manual scaler) was not able to add additional resources to meet the typical morning/afternoon needs of the Data API. Any data that was received by Chameleon is now available for display on the dashboard and for access via API. As mentioned previously, all other aspects of Chameleon remained online during this incident (we used Chameleon to update our customers in the Chameleon dashboard 🎉 http://is.am/4qxa). We will hold this incident open until AWS and Heroku mark it as resolved, but since the impact to Chameleon is resolved, this will be the last email/slack alert you receive about this. Thank you for your patience today.

Time: Dec. 7, 2021, 10:13 p.m.

Status: Identified

Update: The referenced issue on heroku is here => https://status.heroku.com/incidents/2390 - we're still keeping up with updates as they come and will report back here when we have a resolution.

Time: Dec. 7, 2021, 5:04 p.m.

Status: Identified

Update: Due to the API and console outage at AWS us-east-1 and downstream for the Heroku API etc. the Data API will likely remain unavailable until those issues are resolved. Data sent to Chameleon, your Chameleon Experiences (live to your end-users), editing and publishing Experiences are not impacted at this time.

Time: Dec. 7, 2021, 4:31 p.m.

Status: Investigating

Update: We are currently seeing Segmentation and Analytics data fail to load on the Chameleon dashboard and via API requests. We're currently investigating the cause of the issue, and will continue to provide updates as more information becomes available. URLs affected but this: https://api.trychameleon.com/v3/analyze/* and https://ultraviolet.trychameleon.com/*