Time: June 4, 2021, 8:58 p.m.

Status: Resolved

Update: New nodes were launched with upgraded software that should eliminate the issue. To avoid further disruption, these new nodes were created in parallel to the old ones, and the migration to new nodes will be complete once you perform a redeploy of your services. If this issue is occurring in your environment, please perform a redeploy.

Time: May 27, 2021, 5:49 p.m.

Status: Identified

Update: We’ve observed a subtle increase in these rare events of pod sandbox changed errors. Many have been confirmed through customer reports as well so we are currently investigating to see if there’s any correlation between them and to fix the root cause. Impact: This error ultimately causes a self-healing restart of the service. With High Availability (HA) environments, this should cause very minimal downtime as requests will be routed to the healthy instance. With Standard (STD) environments, this restart will cause a 5-10 minute downtime to occur as the single instance will not be available as it is being restarted. Details: A Docker process can become unstable or crash. Those events can happen due to low-level activities on the underlying infrastructure like network errors, system cleanup, unanswered requests, or other operating-system issues. It is the role of the self-healing capabilities of the Kubernetes system to spot crashed containers and automatically restart them. In this case, the Docker process crashed, Kubernetes noticed the problem, killed the broken process, and restarted a new container. In setups without High Availability, during the restart process, the application will be unavailable. In HA setups, you may notice minimal downtime. Workaround: Currently, there is no workaround for this issue. We are working diligently to resolve this issue and prevent any similar issues from appearing in the future.

Time: May 8, 2021, 5:36 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: May 8, 2021, 4:01 p.m.

Status: Monitoring

Update: The fix has been applied and is now propagating. All users should now be able to log in. If any issues regarding log-in persist, please create a support ticket.

Time: May 8, 2021, 3:30 p.m.

Status: Identified

Update: A certificate related to our platform login operation has expired. This is blocking users from logging in with a warning from their browser. We are actively working to resolve the issue and will provide an update soon.

Time: April 30, 2021, 12:23 a.m.

Status: Resolved

Update: The read-only error experienced with your DB is related to an issue with the legacy HA mode for Cloud SQL. Cloud SQL checks the availability of the master instance using different checkpoints to know if the instance is healthy and in particular, one of them is the writability of the disk; in case the master is not ok because of the writability, the traffic will be routed to the failover replica causing the state of the DB to be "read-only. The long-term solution is to set up an HA mode that does not use an explicit failover replica. The new HA replication mode has some added benefits such as increased replication performance and will have no cost implications for our customers The Liferay team will coordinate with customers to provide a detailed path to perform this migration. Meanwhile, should the behavior happen the recommended workaround is to restart the Liferay service.

Time: April 28, 2021, 2:34 p.m.

Status: Identified

Update: Some customers are experiencing an error that limits their database service to read-only operations. The root cause of this error is related to a change in the underlying HA configuration by our infrastructure partner. This error can be fixed by restarting the Liferay Service. We are working to prevent this from happening again and new information will be posted soon. If you have questions or need further assistance, please open a Support ticket.

Time: Feb. 24, 2021, 7:32 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 24, 2021, 5:39 p.m.

Status: Identified

Update: One of our cloud infrastructure providers has reported an issue creating external HTTP(S) load balancers. This could cause errors or delays when deploying to an environment for the first time. Below is the related incident: https://status.cloud.google.com//incident/cloud-networking/21003

Time: Feb. 20, 2021, 7:27 p.m.

Status: Resolved

Update: This incident has been resolved. For information regarding deploying/restarting the Search service in your production environment, please read the previous updates to this incident. We appreciate your patience and understand throughout this incident. If you have any remaining questions or issues, please open a Support ticket.

Time: Feb. 19, 2021, 11:51 p.m.

Status: Monitoring

Update: This issue should be completely resolved for all environments, except for production environments where the Search service was deployed between Feb 11, 2021, 1:31:12 PM PST and Feb 18, 2021, 1:50:53 PM PST. If you did not deploy the Search service to your production environment within this time frame, your environment should be completely fixed of this issue and you can continue normal operations. If you deployed within this time period, we have released a fix that will take effect on your next deployment or when you restart the search service (no version upgrade is required), which you can do at your earliest convenience. DO NOT delete the search service or rename the search data volume BEFORE deploying/restarting. If you choose not to deploy/restart for some time, your Liferay and search service will continue operating smoothly, so you can deploy/restart when you are ready. To summarize, if you performed a deployment of your Search service to Production within this time period, simply perform a regular deployment/restart at your earliest convenience. If you have any concerns, please open a Support ticket.

Time: Feb. 19, 2021, 2:05 a.m.

Status: Monitoring

Update: The initial fix which should resolve the issue from occurring for the majority of customers has been successfully completed. For customers that have clustered Search services and that renamed their volume, we still suggest that you continue to temporarily avoid deployments while we conduct further tests. For all other customers, you are safe to proceed with your deployment operations. Further updates will be provided as new information is available. If you have questions or issues, please open a Support ticket.

Time: Feb. 18, 2021, 8:48 p.m.

Status: Identified

Update: We have identified the issue and are beginning to roll out a fix for all customers. For customers whose Search service was restored by following the support *instructions, we will create replica shards for primary shards that do not have replicas in their search service. This is necessary to ensure that Elasticsearch indices are successfully replicated across nodes during only the next rolling update. We will be conducting a rolling update of the search service in order to realize this solution. No customer action is required. We suggest that all customers temporarily avoid deployments to affected environments until the rollout of the solution is complete. We will post another update here when this process has concluded (approx. 4 hours). *https://help.liferay.com/hc/en-us/articles/360056704432

Time: Feb. 17, 2021, 6 p.m.

Status: Investigating

Update: We have noticed an increase in errors when initializing search services, including an unhealthy search cluster status and unsuccessful deployments. We believe this is due to a Volume mounting regression in a recent release. Our engineers are actively working on a non-disruptive resolution. We have prepared a workaround for this issue so you can remove the cluster errors from your environment. Please see this documentation for details regarding the workaround: https://help.liferay.com/hc/en-us/articles/360056704432. In case of any issues or questions, please contact our support team by opening a ticket.