Is there an InfluxDB outage?
InfluxDB status: Systems Active
Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for InfluxDB and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for InfluxDB and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for InfluxDB.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| AWS: Sydney (Discontinued) | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Dedicated | Active |
| API Reads | Active |
| API Writes | Active |
| Management API | Active |
| Cloud Serverless: AWS, EU-Central | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Serverless: AWS, US-East-1 | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Serverless: AWS, US-West-2-1 | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Serverless: AWS, US-West-2-2 | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Serverless: Azure, East US | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Serverless: Azure, W. Europe | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Cloud Serverless: GCP | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Google Cloud: Belgium (Discontinued) | Active |
| API Queries | Active |
| API Writes | Active |
| Compute | Active |
| Other | Active |
| Persistent Storage | Active |
| Tasks | Active |
| Web UI | Active |
| Other Services | Active |
| Auth0 User Authentication | Active |
| Marketplace integrations | Active |
| Web UI Authentication (Auth0) | Active |
View the latest incidents for InfluxDB and check for official updates:
Description: RCA - Write and Query failures on prod102-us-west-2 on September 7, 2023 **Summary** We upgraded nginx from 1.3 to 1.7 in prod102-us-west-2 and the upgrade appeared to be fine, with the cluster functioning normally immediately after the upgrade. We had already done the same upgrade of nginx in other Cloud 2 clusters, with no negative impact. Approximately 8 hours later, we were alerted by a high rate of write and query errors in the cluster. We ultimately resolved the issue by reverting the nginx upgrade. **Cause of the Incident** We believe that the cause of the incident was an interaction between the nginx upgrade and stale connection handling, although we were not able to definitively prove this, as we could not cause the problem to occur in later testing. We also did not see this interaction in any of the other clusters where we upgraded nginx. Eight hours after nginx was upgraded, we did a normal deployment which caused all of our gateway pods to restart \(normal behavior\). When they restarted, we got a very high rate \(though not quite 100%\) of connection failures on the gateway pods. This was visible to our customers, who saw 100% failures for queries, and between 80% and 90% failures for writes. **Recovery** On receiving the alerts, we investigated and identified the high failure rate to the gateway pods. As nginx had already been running for 8 hours, we didn’t immediately assume that nginx was at fault, and instead we investigated any other potential causes \(including the changes that had been deployed in the deployment that had just occurred\). When we could not identify any other potential source, we rolled back the nginx upgrade, and the cluster recovered. We have since upgraded nginx again, on this cluster, with no negative impact. Our assumption is not that there is a problem with the particular version of nginx, but that the action of upgrading nginx caused it to hold onto stale connections, so after the gateway pods restarted they could no longer connect successfully. **Timeline** Sept 7, 2023 18:15 UTC - Alerted that we are seeing a high rate of write and query errors. Sept 7, 2023 18:20 UTC - Engineering team began investigating. Sept 7, 2023 18:20 UTC to 21:10 UTC - Our early investigation showed that all the gateway pods had restarted. We investigated any changes that had been in that deployment, but there was nothing in the updated code that could have caused an issue of this magnitude \(impacting queries and writes\). We manually restarted one gateway pod and it did not recover. We also investigated whether this could have been caused by something external \(e.g. some network issue within the AWS infrastructure\) but could not identify any cause there. As we could not find a root cause, we chose to undo all recent changes in that cluster, including the nginx upgrade. Sept 7, 21:15 UTC - Rolled back nginx to v1.3. Sept 7, 21:20 UTC - All gateway connections recovered. Query and Write latency was still high as there was a backlog of failed requests to catch up. Sept 7, 22:30 UTC - Cluster fully recovered. Sept 8, 10:00 UTC - We upgraded nginx to 1.7, and restarted all the gateway pods, without any negative consequences. **Future mitigations** We will force a deployment directly after each nginx upgrade, to ensure that all connections are refreshed, to avoid any potential interaction with stale connections.
Status: Postmortem
Impact: None | Started At: Sept. 7, 2023, 7:21 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: Sept. 7, 2023, 2:27 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: Sept. 7, 2023, 2:27 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: Sept. 6, 2023, 7:28 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: Sept. 6, 2023, 7:28 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
