Time: March 10, 2022, 3:39 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 31, 2021, 6:46 p.m.

Status: Monitoring

Update: Update - Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The Qubole control plane does not utilize a version of log4j affected by the vulnerability. While Qubole does not use an affected version of log4j, we do provide AMIs intended for deployment within customer-controlled hardware (i.e. the Data Plane) that include third-party software such as Spark, Hive, and Presto that contain potentially vulnerable versions of the log4j libraries. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. Customers can immediately refer to the Apache website for mitigation steps and will be able to contact Qubole support to apply new AMIs once these are available. We are providing new AMIs for AWS currently. We are working on remediation steps for GCP and OCI currently. We will continue to update here when those will be available. Monitoring - NOTE: This incident is no longer considered active, but is being maintained as Monitoring for short-term visibility. Qubole’s investigation of the Log4j vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure. Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. Customers will need to contact support to apply new AMIs Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.

Time: Dec. 23, 2021, 7:13 a.m.

Status: Monitoring

Update: Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The Qubole control plane does not utilize a version of log4j affected by the vulnerability. While Qubole does not use an affected version of log4j, we do provide AMIs intended for deployment within customer-controlled hardware (i.e. the Data Plane) that include third-party software such as Spark, Hive, and Presto that contain potentially vulnerable versions of the log4j libraries. The Apache foundation has provided general mitigation strategies that you may apply to your Data Plane clusters to ensure you are not impacted by these vulnerabilities. IMPORTANT NOTE: The following guidance is provided as general guidance that you can apply to your Data Plane clusters based on the Apache Foundation tech bulletin. You will need to customize your individual steps according to your environment. We have tested these strategies with sample Data Plane clusters and they prove effective at eliminating the vulnerabilities identified in Log4J 2.x versions. 1. Customers that have Java 7 and/or Java 8 installed on their clusters will need to add a line similar to the one below to their bootstrap.sh script. The cluster will then need to be restarted. This will remove the offending JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class 2. Based on newer CVEs published, customers with Java 8 are also vulnerable if Context Lookups are being used in the property files. Upon investigation, we have determined that there are no Context Lookups being used in the Qubole provided AMIs. There is no action required to respond to these newer CVEs. Note: Apache has also reported a vulnerability with the 1.x version of Log4j. They have advised to investigate the use of JMSAppender. Log4j 1.x configurations without JMSAppender are not impacted by this vulnerability. Qubole does not use JMSAppender in either the Control nor the Data planes therefore Qubole is not impacted by this issue.

Time: Dec. 16, 2021, 2:55 p.m.

Status: Monitoring

Update: NOTE: This incident is no longer considered active, but is being maintained as Monitoring for short-term visibility. Qubole’s investigation of the CVE-2021-44228 vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure. Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. This immediate action can be achieved by following the mitigation instructions as published by Apache on the Apache website (https://logging.apache.org/log4j/2.x/security.html). Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.

Time: Dec. 16, 2021, 2:39 p.m.

Status: Resolved

Update: Qubole’s investigation of the CVE-2021-44228 vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure. Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. This immediate action can be achieved by following the mitigation instructions as published by Apache on the Apache website (https://logging.apache.org/log4j/2.x/security.html). Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.

Time: Dec. 16, 2021, 2:39 p.m.

Status: Resolved

Update: Qubole’s investigation of the CVE-2021-44228 vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure. Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. This immediate action can be achieved by following the mitigation instructions as published by Apache on the Apache website (https://logging.apache.org/log4j/2.x/security.html). Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.

Time: Dec. 14, 2021, 8:39 p.m.

Status: Investigating

Update: This incident will be used to track information about the Log4J Issue and its impact on the Qubole infrastructure. Below is the current message from the Idera Security and Compliance Team: Customer Update to Log4J Issue (CVE-2021-44228) This is an update of Idera's internal review of the Log4J Issue (CVE-2021-44228). At this time we are still reviewing the following products and we expect to complete this by the end of day today. Qubole - Still under review. XBlend / XRay On Prem - Still under review. (SaaS version complete with no issues) All other Idera family of products have been reviewed and confirmed to have no issues with Log4J Issue (CVE-2021-44228) at this time. If you have any questions or concerns please contact us. Idera Security and Compliance Team.

Time: Dec. 14, 2021, 8:39 p.m.

Status: Investigating

Update: This incident will be used to track information about the Log4J Issue and its impact on the Qubole infrastructure. Below is the current message from the Idera Security and Compliance Team: Customer Update to Log4J Issue (CVE-2021-44228) This is an update of Idera's internal review of the Log4J Issue (CVE-2021-44228). At this time we are still reviewing the following products and we expect to complete this by the end of day today. Qubole - Still under review. XBlend / XRay On Prem - Still under review. (SaaS version complete with no issues) All other Idera family of products have been reviewed and confirmed to have no issues with Log4J Issue (CVE-2021-44228) at this time. If you have any questions or concerns please contact us. Idera Security and Compliance Team.

Time: Nov. 28, 2021, 7:18 a.m.

Status: Resolved

Update: The azure-eu.qubole.com seems to be back online after the azure cluster upgrade.

Time: Nov. 27, 2021, 6:29 a.m.

Status: Investigating

Update: We are currently experiencing site availability issue, while accessing https://azure-eu.qubole.com/ . This is currently being investigated and we will update back at the earliest.

Time: Nov. 27, 2021, 2:58 a.m.

Status: Investigating

Update: azure-eu.qubole.com is currently seeing some performance issue and is returning errors during access. At this time failures appear to be partial and intermittent, but Devops is investigating.

Time: Nov. 28, 2021, 7:18 a.m.

Status: Resolved

Update: The azure-eu.qubole.com seems to be back online after the azure cluster upgrade.

Time: Nov. 27, 2021, 6:29 a.m.

Status: Investigating

Update: We are currently experiencing site availability issue, while accessing https://azure-eu.qubole.com/ . This is currently being investigated and we will update back at the earliest.

Time: Nov. 27, 2021, 2:58 a.m.

Status: Investigating

Update: azure-eu.qubole.com is currently seeing some performance issue and is returning errors during access. At this time failures appear to be partial and intermittent, but Devops is investigating.

Time: Aug. 5, 2021, 5:26 p.m.

Status: Resolved

Update: DevOps has confirmed that after replacing the bad tunnel node the us.qubole.com seems to be working fine.

Time: Aug. 5, 2021, 4:54 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: Aug. 5, 2021, 3:07 p.m.

Status: Monitoring

Update: There seems to be an issue with one of the tunnel servers node in us.qubole.com, we have removed the bad tunnel node. The team continues to monitor the environment's stability.

Time: Aug. 5, 2021, 2:59 p.m.

Status: Identified

Update: DevOps is continuing to work on this issue. We have identified that one of the tunnel server node is having an issue.

Time: Aug. 5, 2021, 2:42 p.m.

Status: Investigating

Update: us.qubole.com is currently seeing some degraded performance and is returning errors during cluster start and command processing. At this time failures appear to be partial and intermittent. Currently, DevOps is investigating.