Time: July 25, 2024, 2:03 a.m.

Status: Postmortem

Update: # Summary Beginning at approximately 10:17am PDT and lasting until service restoration at approximately 11:52am, the Lucidworks Signals Beacon on client sites was not functional due to an error in retrieving its configuration from the Lucidworks Platform. During this time, some signals from new site visits were not collected. # Root Cause A database change that was intended to only be applied in our development environment for testing purposes was inadvertently applied to the production database, which resulted in the creation of new, empty tables for storing Beacon configurations and authorization information. For the duration of this incident, when the Beacon initialized and attempted to retrieve its configuration from the Lucidworks Platform at page load time, the call failed with an HTTP 401 Unauthorized response, and the Beacon would silently fail to load. This behavior, while not site-impacting in any way, does result in the inability to send user behavior signals to Lucidworks Analytics, and so for the duration of this incident some analytics data for affected clients will be incomplete. The database change was rolled back, which allowed the Platform to once again successfully return Beacon configurations, which had persisted in the previously-used table throughout the duration of the incident.  No configuration data was lost, and Signals Beacon functionality has been fully restored. # Lucidworks Actions We will update our Beacon and Signals APIs to return HTTP 500 errors when the configuration cannot be loaded, in order to generate a higher-severity alert for our teams so that we can more urgently react to similar issues in the future. We will also enhance the Beacon to cache the most recently retrieved configuration in local browser storage so that there is a fallback in addition to a potential backoff strategy to minimize throwing errors. # Recommended Client Actions There are no recommended client actions as a result of this event. Affected customers with additional questions or concerns may contact us via[ Lucidworks Support](https://support.lucidworks.com/).

Time: July 25, 2024, 2:02 a.m.

Status: Resolved

Update: The fix has been verified and we've confirmed that full functionality has been restored. No Signals Beacon configurations were lost during this incident.

Time: July 25, 2024, 2 a.m.

Status: Monitoring

Update: We identified the source of the error as being a database configuration update that was inadvertently applied to our production site, rather than in our development environment as intended. We have rolled back that change, and the Beacon is able to successfully load its configuration again. We're continuing to monitor to ensure there are no ongoing effects of this incident.

Time: July 25, 2024, 1:58 a.m.

Status: Investigating

Update: The Lucidworks Signals Beacon is currently unable to load its configuration due to a server-side error on the Lucidworks Platform. As a result, while existing site visits will continue to send signals to Lucidworks Analytics, new page loads will not be able to initialize the Beacon and some signals will not be collected. We are working to restore full functionality as soon as possible, and will provide timely updates as this incident is addressed.

Time: Oct. 5, 2023, 8 p.m.

Status: Postmortem

Update: A configuration change in our [ArgoCD](https://argo-cd.readthedocs.io/en/stable/#what-is-argo-cd) continuous deployment platform caused nondeterministic behavior when applied to our Kubernetes [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress) controllers.  Though this change was successfully applied in our development environment, when the same configuration was deployed to the production [Springboard UI](https://platform.lucidworks.com/), it put the cluster into an inaccessible state.  We manually intervened to roll back the change as quickly as possible, but found that ArgoCD was still intermittently mis-applying the configuration in this environment.  Further troubleshooting efforts identified a stale cache as the culprit; once we forced a hard reset of ArgoCD’s cache, the correct configuration was applied, and the configuration UI returned to a fully functional state. No Connected Search indexing or query functionality was affected during this incident; the extent of the impact was limited to the web-based configuration interface, and no data loss occurred.

Time: Sept. 29, 2023, 6:24 p.m.

Status: Resolved

Update: We have confirmed that reverting our earlier change has corrected the root cause of this incident, and platform.lucidworks.com is once again fully functional.

Time: Sept. 29, 2023, 5:55 p.m.

Status: Investigating

Update: The platform.lucidworks.com interface is once again available and functional. No indexing or search query outage was experienced during this incident. We're continuing to monitor to ensure there are no lingering issues.

Time: Sept. 29, 2023, 5:29 p.m.

Status: Investigating

Update: We have identified the source of the issue and have reverted the recent change that caused it. The fix is actively rolling out. Some users will see recovery and be able to log in to platform.lucidworks.com already, while for others it may not show full recovery until the rollout has fully completed.

Time: Sept. 29, 2023, 5:14 p.m.

Status: Investigating

Update: The configuration UI at platform.lucidworks.com is currently unavailable and returning errors when users attempt to access it. Lucidworks engineers are actively investigating the problem. This incident does NOT affect any indexing or search functionality; it is only preventing configuration changes to existing Connected Search applications. We will post updates as soon as we have more details to share.

Time: Oct. 5, 2023, 8 p.m.

Status: Postmortem

Update: A configuration change in our [ArgoCD](https://argo-cd.readthedocs.io/en/stable/#what-is-argo-cd) continuous deployment platform caused nondeterministic behavior when applied to our Kubernetes [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress) controllers.  Though this change was successfully applied in our development environment, when the same configuration was deployed to the production [Springboard UI](https://platform.lucidworks.com/), it put the cluster into an inaccessible state.  We manually intervened to roll back the change as quickly as possible, but found that ArgoCD was still intermittently mis-applying the configuration in this environment.  Further troubleshooting efforts identified a stale cache as the culprit; once we forced a hard reset of ArgoCD’s cache, the correct configuration was applied, and the configuration UI returned to a fully functional state. No Connected Search indexing or query functionality was affected during this incident; the extent of the impact was limited to the web-based configuration interface, and no data loss occurred.

Time: Sept. 29, 2023, 6:24 p.m.

Status: Resolved

Update: We have confirmed that reverting our earlier change has corrected the root cause of this incident, and platform.lucidworks.com is once again fully functional.

Time: Sept. 29, 2023, 5:55 p.m.

Status: Investigating

Update: The platform.lucidworks.com interface is once again available and functional. No indexing or search query outage was experienced during this incident. We're continuing to monitor to ensure there are no lingering issues.

Time: Sept. 29, 2023, 5:29 p.m.

Status: Investigating

Update: We have identified the source of the issue and have reverted the recent change that caused it. The fix is actively rolling out. Some users will see recovery and be able to log in to platform.lucidworks.com already, while for others it may not show full recovery until the rollout has fully completed.

Time: Sept. 29, 2023, 5:14 p.m.

Status: Investigating

Update: The configuration UI at platform.lucidworks.com is currently unavailable and returning errors when users attempt to access it. Lucidworks engineers are actively investigating the problem. This incident does NOT affect any indexing or search functionality; it is only preventing configuration changes to existing Connected Search applications. We will post updates as soon as we have more details to share.

Time: April 18, 2023, 3:57 p.m.

Status: Postmortem

Update: Customers affected by this incident may request a detailed Event Analysis Report via their TAM or through Lucidworks Support.

Time: April 18, 2023, 3:57 p.m.

Status: Postmortem

Update: Customers affected by this incident may request a detailed Event Analysis Report via their TAM or through Lucidworks Support.

Time: April 18, 2023, 3:57 p.m.

Status: Resolved

Update: Our upstream vendor has deployed a fix for the bug that was impacting some Connected Search customers, and Lucidworks has verified that the issue is now fully resolved.

Time: April 18, 2023, 3:57 p.m.

Status: Resolved

Update: Our upstream vendor has deployed a fix for the bug that was impacting some Connected Search customers, and Lucidworks has verified that the issue is now fully resolved.

Time: April 13, 2023, 10:49 p.m.

Status: Monitoring

Update: Our upstream vendor has identified the cause of the bug and is working on an urgent fix. In the meantime, we are keeping our mitigation in place and continuing to monitor to ensure continued operations and functionality of Connected Search for all customers. We will provide further updates as soon as we have them.

Time: April 13, 2023, 10:49 p.m.

Status: Monitoring

Update: Our upstream vendor has identified the cause of the bug and is working on an urgent fix. In the meantime, we are keeping our mitigation in place and continuing to monitor to ensure continued operations and functionality of Connected Search for all customers. We will provide further updates as soon as we have them.

Time: April 11, 2023, 9:14 p.m.

Status: Monitoring

Update: We have confirmed the underlying cause of this issue, and it has been mitigated at this time. A particularly resource-intensive query was found to be impacting the Connected Search backend for all customers by triggering a bug in the underlying data store service. Disabling this query has returned service to normal, and all Connected Search and Springboard UI functionality should be fully restored at this time. Lucidworks will follow up on a root-cause analysis with the upstream vendor in question, and will update our own findings here as we have more details to share.

Time: April 11, 2023, 9:14 p.m.

Status: Monitoring

Update: We have confirmed the underlying cause of this issue, and it has been mitigated at this time. A particularly resource-intensive query was found to be impacting the Connected Search backend for all customers by triggering a bug in the underlying data store service. Disabling this query has returned service to normal, and all Connected Search and Springboard UI functionality should be fully restored at this time. Lucidworks will follow up on a root-cause analysis with the upstream vendor in question, and will update our own findings here as we have more details to share.

Time: April 11, 2023, 8:26 p.m.

Status: Identified

Update: We continue to work to fully resolve the source of the query timeouts, while minimizing the impact as much as possible. We believe we have identified the root cause and are now pursuing corrective actions. We'll continue to provide updates as soon as more information is available.

Time: April 11, 2023, 8:26 p.m.

Status: Identified

Update: We continue to work to fully resolve the source of the query timeouts, while minimizing the impact as much as possible. We believe we have identified the root cause and are now pursuing corrective actions. We'll continue to provide updates as soon as more information is available.

Time: April 11, 2023, 6:04 p.m.

Status: Investigating

Update: We have identified the source of intermittent query failures and occasional timeouts, and are working with an upstream vendor to resolve them. We do not have an ETA for full resolution at this time, but believe that our initial troubleshooting has already lessened both the frequency and severity of these query issues. We will continue to provide updates as the situation develops.

Time: April 11, 2023, 6:04 p.m.

Status: Investigating

Update: We have identified the source of intermittent query failures and occasional timeouts, and are working with an upstream vendor to resolve them. We do not have an ETA for full resolution at this time, but believe that our initial troubleshooting has already lessened both the frequency and severity of these query issues. We will continue to provide updates as the situation develops.

Time: April 11, 2023, 4:45 p.m.

Status: Investigating

Update: Some Connected Search requests to the Query API are intermittently timing out, which results in them returning either empty or inconsistent results. This issue is affecting all Connected Search customers, as well as the Experience Optimizer section of the Springboard UI (platform.lucidworks.com). Lucidworks engineers are investigating the cause of this issue and updates will provided here as soon as we have them.

Time: Jan. 12, 2023, 3:28 a.m.

Status: Postmortem

Update: Connected Search customers who were affected by this incident may request a detailed Event Analysis via Lucidworks Support.

Time: Jan. 9, 2023, 11:23 p.m.

Status: Resolved

Update: An internal routing component of Connected Search had gotten into a stuck state where it could not validate a newly-rotated TLS certificate used to route traffic to the search backend. By restarting this component, we were able to mitigate the issue. We have updated our tooling to force a restart of this component every time the TLS certificate is rotated, in order to avoid a recurrence.

Time: Jan. 9, 2023, 11:21 p.m.

Status: Investigating

Update: We have mitigated the issue, and all search functionality (including both API-based search requests, as well as the ability to use Experience Optimizer in our platform UI) has been restored.

Time: Jan. 9, 2023, 11:20 p.m.

Status: Investigating

Update: The Lucidworks Connected Search platform is experiencing an issue with routing traffic that is resulting in failed search query requests. This issue is also affecting the ability to use Experience Optimizer from within the platform web interface.