Time: Aug. 14, 2024, 8:27 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 14, 2024, 8:10 a.m.

Status: Monitoring

Update: Availability of the user documentation is restored. We continue monitoring.

Time: Aug. 14, 2024, 7:56 a.m.

Status: Identified

Update: Our user documentation, typically accessible via e.g. docs.leanix.net or docs-eam.leanix.net, is currently unavailable. The team has identified the issue and implemented a mitigation. However, due to a necessary DNS change, it will take some time before full availability is restored. We appreciate your patience and understanding.

Time: Aug. 15, 2024, 8:20 a.m.

Status: Postmortem

Update: ## Incident Description Between \(UTC\) Aug 7, 2024, 2:00:00 PM and 7:00 PM, there was an outage in the ServiceNow integration access to the configurations for mappings, making the integration unavailable for customers. After fixing the root cause, the integration configurations were again available, making the whole integration available without any data loss. ## Incident Resolution * An immediate attempt to revert to the previous configuration failed. Then, a more extended analysis of the root cause led to a definitive fix released, making the ServiceNow integration configurations accessible again. * No data loss occurred since the problem was a temporary change in the service’s internal data format that was reverted, restoring the original data again \(only ServiceNow integration configurations were affected and then fixed; no workspace data was involved\). ## Root Cause Analysis * ServiceNow integration configuration was not readable during the incident because a planned change in the structure of those configurations had technical issues during its execution. * Reverting that change as mitigation did not immediately solve the issue because the configurations had already been changed to the new format. ## Preventative Measures * Now the system handles both formats correctly, and configuration data is preserved using the expected format \(the internal format will no longer be used in API communication but still supported for configs that may still present this format\). * Improve internal handling of the tools involved in this issue and move away from the one that caused unexpected technical issues when this problem was introduced. * Improve internal alerts for this service availability. The service was running in this case, but the responses were not useful for the User Interface to work properly. Those cases will be considered as services unavailable for alerting purposes.

Time: Aug. 7, 2024, 6:52 p.m.

Status: Resolved

Update: This incident has been resolved. We appreciate your patience and understanding.

Time: Aug. 7, 2024, 5:33 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented to restore the loading of the configuration. No customer data has been lost or affected. Only the loading of the Configuration UI and starting of new synchronization runs for the ServiceNow integration is affected.

Time: Aug. 7, 2024, 4:30 p.m.

Status: Investigating

Update: We have identified a partial outage for the ServiceNow integration. There seems to be an issue preventing loading of the integration configuration. The teams are working on the investigation and will keep this page updated.

Time: Aug. 5, 2024, 2:36 p.m.

Status: Resolved

Update: Our Hyperscaler experienced an unexpected host failure at 14:08 UTC today and automatically initiated the auto-recovery process to start the affected virtual machine on a different host. The affected instances were up and running again by 14:18 UTC.

Time: Aug. 12, 2024, 1:58 p.m.

Status: Postmortem

Update: ## Incident Description ### Integration API On July 31, 2024, from 07:52 UTC to 12:29 UTC, Integration API executions for some customers failed with an error message similar to `"External ID's type 'externalId' field is not found in Data Model"`. Besides some requests not being successfully executed, some customers also had `DeletionScope` configured in the Integration API processor, which caused the deletion of some factsheets, relations, and subscriptions. It affected the Reference Catalog service. ### Reference Catalogs Due to the faulty validation logic, the Reference Catalog in all regions showed only limited catalog entries between July 31st, 07:52 UTC, and Aug 3rd, 23:56 UTC. This had an impact on the following use cases: * [Linking an Application to a Catalog Item](https://docs-eam.leanix.net/docs/saas-catalog#linking-an-application-to-a-catalog-item) * [Linking an IT Component to a Catalog Item](https://docs-eam.leanix.net/docs/lifecycle-catalog#linking-an-it-component-to-a-catalog-item) The incomplete Reference Catalog affected the following SAP LeanIX integrations: * [SAP Landscape Discovery](https://docs-eam.leanix.net/docs/sap-landscape-discovery) * [SaaS Discovery](https://docs-eam.leanix.net/docs/saas-discovery-1) * [ServiceNow Discovery](https://docs-eam.leanix.net/docs/servicenow) Those integrations were unable to link discovered software items to the Reference Catalog. The issue will be automatically corrected with the next run of the related integration. All other functionalities within the SAP LeanIX application have not been impacted. Customer data was not affected. ## Root Cause Analysis On Wednesday, July 31st, at 07:52 UTC, the Integration API was updated with a change that validated the External IDs in the Integration API processors against the External IDs configured in the data model. The change had faulty logic, which caused the validation to fail even if the processor's External ID was present in the Data Model. The change triggered the actions defined in the `DeletionScope` for the Reference Catalog sync processes. These actions are responsible for the incomplete sync of the Reference Catalog to all regions. ## Incident Resolution The change was reverted on July 31, 2024, 12:29 UTC, and the deleted factsheets, relations, and subscriptions of the majority of the customers were restored by subsequent executions of the Integration API. We reached out to the few remaining customers where the subsequent executions did not take place. It took until Aug 3rd, 23:56 UTC, to completely restore all regional Reference Catalogs. ## Preventative Measures In addition to the existing tests, we will introduce additional testing and monitoring for such changes in the Integration API. The Reference Catalog sync logic now contains a safeguard that will prevent changes to the regional Reference Catalogs from being applied when the change affects more than the expected number of catalog items.

Time: Aug. 4, 2024, 12:36 p.m.

Status: Resolved

Update: Monitoring has concluded, and the incident has been successfully resolved.

Time: Aug. 4, 2024, 7:06 a.m.

Status: Monitoring

Update: The restoration process is now complete, and all features are fully operational again. We will follow up with a comprehensive postmortem in the next days and will also continue to work on preventive measures for the future. We appreciate your patience and understanding

Time: Aug. 2, 2024, 8:18 p.m.

Status: Identified

Update: Our engineers have started the restoration process in all regions that will recover the reference-catalogs. It will take several hours. Until the restoration is completed, some reference-catalog features (please see below) might still experience issues. We will give another update once the process is completed. Again, all customer data is unaffected, and the main functionalities, including Reports and Fact Sheet Inventory, are operational.

Time: Aug. 1, 2024, 8:37 p.m.

Status: Identified

Update: We want to highlight that all customer data is unaffected, and the main functionalities, including Reports and Fact Sheet Inventory, are operational. Only the following reference-catalog features are experiencing issues: - Certain catalog items are not available for single and bulk linking. - Certain already linked catalog items can not be viewed. - New integration runs for, e.g., SaaS and SAP Discovery, won't be able to link newly found software items to the catalog. Our engineers are still working on a fix to restore the functionality.

Time: Aug. 1, 2024, 7:33 p.m.

Status: Identified

Update: The tests failed, and the team continues to fix the reference catalog issue. We will give another update when there is news.

Time: Aug. 1, 2024, 5:49 p.m.

Status: Identified

Update: The team has prepared a fix to restore the reference catalogs and is performing tests. We will give another update in 60 minutes.

Time: Aug. 1, 2024, 4:18 p.m.

Status: Identified

Update: The team has prepared a fix to restore the reference catalogs and is performing tests. We will give another update in 60 minutes.

Time: Aug. 1, 2024, 3 p.m.

Status: Identified

Update: The team has prepared a fix to restore the reference catalogs and is performing tests. We will give another update in 60 minutes.

Time: Aug. 1, 2024, 1:07 p.m.

Status: Identified

Update: The team is still working to fix the issue with the reference catalog. We will give another update in 120 minutes.

Time: Aug. 1, 2024, 11:16 a.m.

Status: Identified

Update: The team is still working to fix the issue with the reference catalog. We will give another update in 120 minutes.

Time: Aug. 1, 2024, 8:50 a.m.

Status: Identified

Update: We have observed degraded performance in reference-catalogs which affects fact sheet linking and the integrations connected to it. Our team has identified the cause and is working on a solution. We will give another update in 30 minutes.

Time: Aug. 12, 2024, 11:33 a.m.

Status: Postmortem

Update: ## Incident Description Between 16:06 UTC on 30 Jul 2024 and 10:47 UTC on 31 Jul 2024, the metrics service in our AE region was affected by a degradation in the availability of our hyperscaler’s database management service. ## Incident Resolution At 10:47 UTC the degradation in the availability of the database management service was resolved by our hyperscaler and the metrics service in our AE region was available again. ## Root Cause Analysis The hyperscaler’s automatic monitoring detected a thermal anomaly in one datacenter. As a proactive measure, resources were powered down, causing a drop in availability. Affected cooling units were then restored to reduce temperatures in the datacenter to normal operational thresholds. Once those normal operational thresholds were reached the restoration of storage and compute resources began. ## Preventative Measures We will analyze our database management services configuration and implement necessary improvements to mitigate similar future incidents faster.

Time: July 31, 2024, 1:44 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: July 31, 2024, 11:40 a.m.

Status: Monitoring

Update: Our Hyperscaler restored the database connectivity and we are monitoring the results.

Time: July 31, 2024, 5:15 a.m.

Status: Identified

Update: Our Hyperscaler still works actively to restore the database connectivity and has provided us with the following update. "We have successfully restored cooling systems, and temperatures have returned to normal operating thresholds. We have commenced a structured power-up sequence for previously powered-down compute and storage resources and are actively monitoring as we work through this extended process. We have recovered all Azure Network and Storage resources and are currently running consistency and confirmation checks for these resources. The recovery of Compute resources is now progressing successfully." We will send additional updates as soon as we receive more information from our Hyperscaler.

Time: July 30, 2024, 11:27 p.m.

Status: Identified

Update: Our Hyperscaler still works actively to restore the database connectivity and has provided us with the following update. "Automatic monitors detected a thermal anomaly. As a proactive measure, resources were powered down, causing a drop in availability. We have made progress in restoring some of the affected cooling units and are continuing to work to restore the remaining units in parallel. The restoration of cooling units is helping with reducing temperatures towards normal operation thresholds in the impacted areas of the datacenter. Once operational threshold temperatures have stabilized, we will begin work on the structured restoration of Compute and Storage resources which were proactively shut down." We will send additional updates as soon as we receive more information from our Hyperscaler.

Time: July 30, 2024, 7:20 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: July 30, 2024, 6:26 p.m.

Status: Identified

Update: Our Hyperscaler still works actively on a mitigation to recover database connectivity. We will send additional updates as soon as we receive more information from our Hyperscaler.

Time: July 30, 2024, 5:18 p.m.

Status: Identified

Update: Our Hyperscaler still works actively on a mitigation to recover database connectivity. We will send an additional update in 120 minutes.

Time: July 30, 2024, 4:18 p.m.

Status: Identified

Update: Our Hyperscaler is currently experiencing a service disruption in our UAE region affecting our databases. We will send an additional update in 30 minutes.