Time: Jan. 8, 2022, 6:58 a.m.

Status: Resolved

Update: The SSL certificate is renewed and now the issue has been resolved.

Time: Jan. 7, 2022, 10:29 p.m.

Status: Identified

Update: Since we are now in the process of getting certificates updated.

Time: Jan. 7, 2022, 6:41 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Jan. 7, 2022, 6:39 p.m.

Status: Investigating

Update: Presto clusters are seeing degraded performance and failing queries. Some clusters may not start as expected. The internal teams have identified a potential issue with an expiring SSL certificate and are investigating.

Time: Jan. 8, 2022, 6:58 a.m.

Status: Resolved

Update: The SSL certificate is renewed and now the issue has been resolved.

Time: Jan. 7, 2022, 10:29 p.m.

Status: Identified

Update: Since we are now in the process of getting certificates updated.

Time: Jan. 7, 2022, 6:41 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Jan. 7, 2022, 6:39 p.m.

Status: Investigating

Update: Presto clusters are seeing degraded performance and failing queries. Some clusters may not start as expected. The internal teams have identified a potential issue with an expiring SSL certificate and are investigating.

Time: Jan. 5, 2022, 2:07 p.m.

Status: Resolved

Update: DevOps provided a workaround by disabling the offline edit option via asterix in the account level and the jupyter seems to be working fine without any issues.

Time: Jan. 5, 2022, 1:01 p.m.

Status: Identified

Update: DevOps team identified there is an issue with the Asterix cluster while processing the request of the jupyter notebooks. they are suggesting a workaround to disable the Asterix flag.

Time: Jan. 4, 2022, 10:48 p.m.

Status: Investigating

Update: We have identified a Jupyter Notebook launch issue on the us and api qubole environments. Our DevOps team is investigating this issue and will be providing progress updates.

Time: Jan. 5, 2022, 2:07 p.m.

Status: Resolved

Update: DevOps provided a workaround by disabling the offline edit option via asterix in the account level and the jupyter seems to be working fine without any issues.

Time: Jan. 5, 2022, 1:01 p.m.

Status: Identified

Update: DevOps team identified there is an issue with the Asterix cluster while processing the request of the jupyter notebooks. they are suggesting a workaround to disable the Asterix flag.

Time: Jan. 4, 2022, 10:48 p.m.

Status: Investigating

Update: We have identified a Jupyter Notebook launch issue on the us and api qubole environments. Our DevOps team is investigating this issue and will be providing progress updates.

Time: March 10, 2022, 3:39 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 31, 2021, 6:46 p.m.

Status: Monitoring

Update: Update - Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The Qubole control plane does not utilize a version of log4j affected by the vulnerability. While Qubole does not use an affected version of log4j, we do provide AMIs intended for deployment within customer-controlled hardware (i.e. the Data Plane) that include third-party software such as Spark, Hive, and Presto that contain potentially vulnerable versions of the log4j libraries. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. Customers can immediately refer to the Apache website for mitigation steps and will be able to contact Qubole support to apply new AMIs once these are available. We are providing new AMIs for AWS currently. We are working on remediation steps for GCP and OCI currently. We will continue to update here when those will be available. Monitoring - NOTE: This incident is no longer considered active, but is being maintained as Monitoring for short-term visibility. Qubole’s investigation of the Log4j vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure. Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. Customers will need to contact support to apply new AMIs Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.

Time: Dec. 23, 2021, 7:13 a.m.

Status: Monitoring

Update: Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The Qubole control plane does not utilize a version of log4j affected by the vulnerability. While Qubole does not use an affected version of log4j, we do provide AMIs intended for deployment within customer-controlled hardware (i.e. the Data Plane) that include third-party software such as Spark, Hive, and Presto that contain potentially vulnerable versions of the log4j libraries. The Apache foundation has provided general mitigation strategies that you may apply to your Data Plane clusters to ensure you are not impacted by these vulnerabilities. IMPORTANT NOTE: The following guidance is provided as general guidance that you can apply to your Data Plane clusters based on the Apache Foundation tech bulletin. You will need to customize your individual steps according to your environment. We have tested these strategies with sample Data Plane clusters and they prove effective at eliminating the vulnerabilities identified in Log4J 2.x versions. 1. Customers that have Java 7 and/or Java 8 installed on their clusters will need to add a line similar to the one below to their bootstrap.sh script. The cluster will then need to be restarted. This will remove the offending JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class 2. Based on newer CVEs published, customers with Java 8 are also vulnerable if Context Lookups are being used in the property files. Upon investigation, we have determined that there are no Context Lookups being used in the Qubole provided AMIs. There is no action required to respond to these newer CVEs. Note: Apache has also reported a vulnerability with the 1.x version of Log4j. They have advised to investigate the use of JMSAppender. Log4j 1.x configurations without JMSAppender are not impacted by this vulnerability. Qubole does not use JMSAppender in either the Control nor the Data planes therefore Qubole is not impacted by this issue.

Time: Dec. 16, 2021, 2:55 p.m.

Status: Monitoring

Update: NOTE: This incident is no longer considered active, but is being maintained as Monitoring for short-term visibility. Qubole’s investigation of the CVE-2021-44228 vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure. Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. This immediate action can be achieved by following the mitigation instructions as published by Apache on the Apache website (https://logging.apache.org/log4j/2.x/security.html). Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.