Time: Nov. 3, 2023, 3:13 p.m.

Status: Postmortem

Update: # Overview A small subset of pipelines encountered VM initialization failures on Cloud Mac builds. ## Timeline \(PST\) | Time | Event | | --- | --- | | 1:17 AM | Customer reported this issue via Support ticket | | 1:29 AM | Internal incident created | | 1:32 AM | Identified that we hit full capacity in Mac pools | | 2:11 AM | We completed a manual cleanup of stale VM’s before the scheduled cleanup, making sure customer pipelines were able to proceed | | 2:30 AM | Mac pool was returned to normal state | ‌ ## Resolution Manual cleanup of stale VM’s before the scheduled cleanup to allow the active pipelines to complete. ## Affected accounts One of the customers using Mac hosted builds in Prod2 was impacted. ## RCA During the VM provisioning process, the VMs were not cleaned up as they should have been when an abort call was made. This issue arose from a recent modification we made to address other cleanup issues. The situation would have been managed effectively if appropriate error handling and cleanup had been incorporated into the Anka driver \(Driver for mac OS deployments\) to VM creation logic. However, as the Anka driver returned an error, we didn't create a database entry for cleanup, which in turn led to the dlite instance not performing the necessary cleanup due to the absence of a database entry. ## Action Items * Anka create API needs to be made Idempotent * Additional monitoring/alerting for stale VMs

Time: Nov. 2, 2023, 4:14 p.m.

Status: Resolved

Update: The incident has been resolved and the pool capacity is freed up. We will provide additional details including RCA in postmortem. We regret the impact this disruption might have caused to few customers.

Time: Nov. 2, 2023, 4:11 p.m.

Status: Identified

Update: Pipelines with infrastructure as Cloud->MacOS are impacted due to pool running at full capacity.

Time: Nov. 2, 2023, 4:10 p.m.

Status: Investigating

Update: We are currently investigating this issue.

Time: Nov. 1, 2023, 8:35 p.m.

Status: Postmortem

Update: # Overview We experienced a service disruption that affected the pipeline functionality, specifically impacting the pipelines with Test Intelligence steps responsible for uploading call graphs. ## Timeline \(PST\) | **Time** | **Event** | | --- | --- | | 7:37 AM Nov 1 2023 | Problematic Lite Engine service version got deployed | | 11:06 AM Nov 1 2023 | Internal customer reported issue | | 11:43 AM Nov 1 2023 | Lite Engine service rolled back | ## Resolution We have rolled back the recent hotfix to restore the service to its previous state. ## Affected accounts A total of 2 customers and 12 pipelines were impacted by this incident for CIE - Cloud Builds. ## RCA The hotfix aimed to improve the Test Intelligence service by skipping the upload of empty call graphs. However, this change led to the unintended consequence of not serializing empty call graphs that were expected to be present by subsequent steps in the pipeline. The testing for the hotfix was conducted in an environment that did not accurately reflect the production infrastructure where the Test Intelligence service operates. This oversight resulted in the failure to capture the issue before the hotfix was deployed to production. ## Action Items * We are reviewing our testing environments and procedures to ensure they accurately mirror our production settings. * Thoroughly review and test the change required for skip upload callgraph functionality

Time: Nov. 1, 2023, 8:34 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 1, 2023, 8:32 p.m.

Status: Investigating

Update: The incident has been resolved. Will post details in the post mortem.

Time: Nov. 1, 2023, 8:31 p.m.

Status: Investigating

Update: We are currently investigating this issue.

Time: Oct. 25, 2023, 8:55 p.m.

Status: Postmortem

Update: ## Overview Post deployment of 63xx release for ci-manager we started observing failures in pipelines where clone codebase properties was disabled. ## Timeline | **Time** | **Event** | | --- | --- | | 24 Oct 2023 4 AM IST | Internally identified the issue | | 24 Oct 2023 4.20 AM IST | Reverted the release to 62xx | ‌ ## Resolution Reverted CI Manager to previous release in Prod 2 to 62xx release ## RCA & Action Items As part of a error handling improvement we added a check that codebase input variable shouldn't be passed as null. But this check should have been applied to only stages where clone is enabled and the input was not passed. Due to a code bug we applied it to all stages. This caused the failure for customer pipelines. Automation for pipeline with stage with and without clone codebase in the same pipeline will be added to catch such scenarios.

Time: Oct. 25, 2023, 8:53 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Oct. 25, 2023, 8:53 p.m.

Status: Investigating

Update: We are currently investigating this issue.

Time: Oct. 25, 2023, 10:26 p.m.

Status: Postmortem

Update: # Overview There was a delay in newly published artifacts showing up in Harness. Impact was limited to few customers in one of our clusters \(prod-1\). ## Timeline \(PST\) | **Time** | **Event** | | --- | --- | | 1:19 AM | Incident declared, and the team got engaged in Incident response. | | 8:55 AM | Issue identified and working on mitigation. | | 10:25 AM | Mitigated and Monitoring. | | 12:17 PM | Issue got resolved | ## Resolution To mitigate, we purged old redundant background job records which were slowing down the collection process. ## RCA & Action Items There was up to a 30 minute delay in our system for collecting newly published artifacts. The typical delay is less than 4 minutes. The slow collection was due to a spike in the backlog of background job requests in our database. To mitigate, we purged old redundant background job records, and the system caught up automatically. The actual cause of build up of the background jobs is unknown. We have added more aggressive alerts in the system for the job queue build up.

Time: Oct. 18, 2023, 7:17 p.m.

Status: Resolved

Update: The issue has been resolved and we will continue to monitor the situation.

Time: Oct. 18, 2023, 5:25 p.m.

Status: Monitoring

Update: We have successfully mitigated this issue and are actively monitoring the situation. We discovered a significant number of database records not getting processed and we've manually cleared these records to mitigate the issue

Time: Oct. 18, 2023, 3:55 p.m.

Status: Identified

Update: Issue identified and we are currently working on mitigation

Time: Oct. 18, 2023, 1:52 p.m.

Status: Investigating

Update: We have reproduced the issue with Harness Internal account and we are continuously investigating the Artifact Collection task lifecycle to understand the issue further. As a Workaround for this issue customers can manually pull the artifacts.

Time: Oct. 18, 2023, 11:50 a.m.

Status: Investigating

Update: Some of harness customers are still facing the issue. We are continuously investigating the issue.

Time: Oct. 18, 2023, 10:34 a.m.

Status: Monitoring

Update: The service has been restored, We have enabled Additional Monitoring as part of the investigation. We are actively monitoring this issue.

Time: Oct. 18, 2023, 9:01 a.m.

Status: Investigating

Update: We have noticed degraded performance with Automatic artifact collection. Team is currently Investigating the issue.

Time: Oct. 25, 2023, 9:56 p.m.

Status: Postmortem

Update: This incident is the first instance of this incident. [https://status.harness.io/incidents/w047bt2yqjm5](https://status.harness.io/incidents/w047bt2yqjm5) Please find the RCA here.

Time: Oct. 18, 2023, 1:23 a.m.

Status: Resolved

Update: We will keep a close watch on the situation.

Time: Oct. 18, 2023, 12:08 a.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: Oct. 18, 2023, 12:03 a.m.

Status: Monitoring

Update: We will continue to monitor the situation

Time: Oct. 17, 2023, 11:46 p.m.

Status: Investigating

Update: While the service has been resorted, we're currently in the process of identifying its root cause. We initially observed delays in the Artifact collection, and we're diligently working to pinpoint the exact reason behind this delay.

Time: Oct. 17, 2023, 10:47 p.m.

Status: Investigating

Update: Team has noticed degraded performance with artifact collection. Investigation is in progress.