Is there an Harness outage?
Harness status: Systems Active
Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Harness and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 8 minutes ago
Get notified about any outages, downtime or incidents for Harness and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Harness.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Service Reliability Management - Error Tracking FirstGen (fka OverOps) | Active |
| Software Engineering Insights FirstGen (fka Propelo) | Active |
| Prod 1 | Active |
| Chaos Engineering | Active |
| Cloud Cost Management (CCM) | Active |
| Continuous Delivery (CD) - FirstGen - EOS | Active |
| Continuous Delivery - Next Generation (CDNG) | Active |
| Continuous Error Tracking (CET) | Active |
| Continuous Integration Enterprise(CIE) - Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Linux Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Self Hosted Runners | Active |
| Continuous Integration Enterprise(CIE) - Windows Cloud Builds | Active |
| Custom Dashboards | Active |
| Feature Flags (FF) | Active |
| Infrastructure as Code Management (IaCM) | Active |
| Internal Developer Portal (IDP) | Active |
| Security Testing Orchestration (STO) | Active |
| Service Reliability Management (SRM) | Active |
| Software Engineering Insights (SEI) | Active |
| Software Supply Chain Assurance (SSCA) | Active |
| Prod 2 | Active |
| Chaos Engineering | Active |
| Cloud Cost Management (CCM) | Active |
| Continuous Delivery (CD) - FirstGen - EOS | Active |
| Continuous Delivery - Next Generation (CDNG) | Active |
| Continuous Error Tracking (CET) | Active |
| Continuous Integration Enterprise(CIE) - Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Linux Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Self Hosted Runners | Active |
| Continuous Integration Enterprise(CIE) - Windows Cloud Builds | Active |
| Custom Dashboards | Active |
| Feature Flags (FF) | Active |
| Infrastructure as Code Management (IaCM) | Active |
| Internal Developer Portal (IDP) | Active |
| Security Testing Orchestration (STO) | Active |
| Service Reliability Management (SRM) | Active |
| Software Engineering Insights (SEI) | Active |
| Software Supply Chain Assurance (SSCA) | Active |
| Prod 3 | Active |
| Chaos Engineering | Active |
| Cloud Cost Management (CCM) | Active |
| Continuous Delivery (CD) - FirstGen - EOS | Active |
| Continuous Delivery - Next Generation (CDNG) | Active |
| Continuous Error Tracking (CET) | Active |
| Continuous Integration Enterprise(CIE) - Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Linux Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Self Hosted Runners | Active |
| Continuous Integration Enterprise(CIE) - Windows Cloud Builds | Active |
| Custom Dashboards | Active |
| Feature Flags (FF) | Active |
| Infrastructure as Code Management (IaCM) | Active |
| Internal Developer Portal (IDP) | Active |
| Security Testing Orchestration (STO) | Active |
| Service Reliability Management (SRM) | Active |
| Software Supply Chain Assurance (SSCA) | Active |
| Prod 4 | Active |
| Chaos Engineering | Active |
| Cloud Cost Management (CCM) | Active |
| Continuous Delivery - Next Generation (CDNG) | Active |
| Continuous Error Tracking (CET) | Active |
| Continuous Integration Enterprise(CIE) - Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Linux Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Self Hosted Runners | Active |
| Continuous Integration Enterprise(CIE) - Windows Cloud Builds | Active |
| Custom Dashboards | Active |
| Feature Flags (FF) | Active |
| Infrastructure as Code Management (IaCM) | Active |
| Internal Developer Portal (IDP) | Active |
| Security Testing Orchestration (STO) | Active |
| Service Reliability Management (SRM) | Active |
| Prod Eu1 | Active |
| Chaos Engineering | Active |
| Cloud Cost Management (CCM) | Active |
| Continuous Delivery - Next Generation (CDNG) | Active |
| Continuous Error Tracking (CET) | Active |
| Continuous Integration Enterprise(CIE) - Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Linux Cloud Builds | Active |
| Continuous Integration Enterprise(CIE) - Self Hosted Runners | Active |
| Continuous Integration Enterprise(CIE) - Windows Cloud Builds | Active |
| Custom Dashboards | Active |
| Feature Flags (FF) | Active |
| Infrastructure as Code Management (IaCM) | Active |
| Internal Developer Portal (IDP) | Active |
| Security Testing Orchestration (STO) | Active |
| Service Reliability Management (SRM) | Active |
View the latest incidents for Harness and check for official updates:
Description: We can confirm normal operation. Get Ship Done! We will continue to monitor and ensure stability.
Status: Resolved
Impact: Minor | Started At: Aug. 16, 2024, 5:15 p.m.
Description: ## **Summary** After the Redis isolation Maintenance on Prod1, internal monitoring tools showed the pipelines were running slower. ## **What was the issue?** Harness platform uses a set of services including producers and consumers for the redis streams. The order in which these services were brought up caused some of the streams to not be consumed. ## **Timeline** | **Time** | **Event** | | --- | --- | | 9:55AM PT | Noticed intermittent slowness in Pipelines | | 10:00AM PT | Core services were rolled out again | | 10:10AM PT | Pipeline performance improved and services were running well | ## **Resolution** Restarting the services in the correct order made the redis producers/consumers available. The pipeline performance also improved and returned to normal latency.
Status: Postmortem
Impact: Minor | Started At: July 20, 2024, 4:40 p.m.
Description: We can confirm normal operation. Get Ship Done! We will continue to monitor and ensure stability.
Status: Resolved
Impact: Minor | Started At: July 19, 2024, 4:11 p.m.
Description: **What was the issue?** Few customer pipelines with a high number of secrets were failing due to timeout errors. | **Time \(UTC\)** | **Event** | | --- | --- | | 11:21 AM UTC | The issue started | | 11:21 AM UTC | Alert for `Name resolution delay` error messages received. | | 11:35 AM UTC | Team started investigating | | 11:41 AM UTC | Incident was started | | 11:45 AM UTC | **Issue got resolved without any intervention at 11:41 AM UTC.** | **RCA:** Pipelines containing a large number of secrets, combined with system load, led to thread pool exhaustion during the decryption process. This caused timeouts for pipeline task execution. As the thread load decreased, the pipeline tasks recovered on their own. **Action Items:** * **Thread Pool Optimization:** Fine-tune the thread pool size to better match the current production load. * **Configurability:** Make the thread pool size configurable to allow for gradual increases based on demand.
Status: Postmortem
Impact: None | Started At: July 8, 2024, 12:28 p.m.
Description: ## Summary The GitOps project overview page was unable to load for select customers following the upgrade. ## Root Cause One customer with many applications led to an API call which initiated the synchronisation leading to a heavy load on our database. This resulted in some locks in our database which impacted the rollback of the changes introduced by the upgrade. ## Timeline | **TIME** | **EVENTS** | | --- | --- | | 1st July 2024 10:00 AM UTC | Received a customer escalation. | | 1st July 2024 10:20 AM UTC | Rolled back the upgrade, but GitOps remained down. | | 1st July 2024 10:31 AM UTC | Identified the issue and initiated a new deployment. | | 1st July 2024 10:31 AM UTC | The GitOps service has been restored. | ## Resolution We extended the synchronisation duration and manually removed the locks, allowing the pods to start. This action restored the GitOps services. ## Follow-up action items 1. Increase the duration of AppSyncs. 2. Add a control to prevent the AppSync when necessary. 3. Implement the capability to stop traffic from a specific customer, if required.
Status: Postmortem
Impact: None | Started At: July 1, 2024, 10:43 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
