Time: Aug. 21, 2024, 5:02 p.m.

Status: Resolved

Update: From 9:14 AM to 9:35 AM PT, users experienced increased error rates while trying to log into ChatGPT and the API Platform due to an outage from a third-party authentication provider. This issue has now been resolved.

Time: Aug. 21, 2024, 4:27 p.m.

Status: Investigating

Update: We are investigating an issue impacting logins to ChatGPT and the API Platform.

Time: Aug. 21, 2024, 1:42 a.m.

Status: Resolved

Update: Between 5:48pm and 6:04pm PT, users experienced elevated error rates when attempting to log into ChatGPT and the API Platform due to an outage from a third-party auth provider. This issue is now resolved.

Time: Aug. 21, 2024, 1:34 a.m.

Status: Monitoring

Update: This has been mitigated and logins are operational on both ChatGPT and the API Platform.

Time: Aug. 21, 2024, 1:04 a.m.

Status: Identified

Update: We have identified an issue impacting logins to ChatGPT and the API Platform.

Time: Aug. 19, 2024, 10:02 p.m.

Status: Resolved

Update: File uploads should be working as expected. This issue is resolved.

Time: Aug. 19, 2024, 9:32 p.m.

Status: Monitoring

Update: The engineering team has made a change to alleviate the file uploads process. File uploads should be working as expected. Engineers are monitoring the change.

Time: Aug. 19, 2024, 8:04 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Aug. 19, 2024, 7:28 p.m.

Status: Investigating

Update: We are continuing to investigate reports that file uploads in both ChatGPT and our API products are returning errors for customers.

Time: Aug. 19, 2024, 6:51 p.m.

Status: Investigating

Update: We are currently investigating reports that certain file uploads are returning error messages for customers.

Time: Aug. 23, 2024, 8:03 p.m.

Status: Postmortem

Update: On August 16, 2024 from 11:38 AM to 1:16 PM PT, a significant issue impacted the reliability of the OpenAI primary API, resulting in degraded service for users. This incident led to reduced success rates for ChatGPT conversations and affected login and account creation processes. The incident occurred in two waves, lasting 44 and 15 minutes respectively. ‌ The root cause was a combination of factors. A scheduled maintenance and an upgrade to the ingress of the OpenAI user-facing clusters introduced a networking control plane regression. This manifested itself in a short-lived data plane outage. As a result of the momentary loss of connectivity, a set of services became unhealthy and were automatically restarted. The restarts, however, took much longer than expected as the services starting up overwhelmed a backend persistence store with a heavy first-start query. The backend persistence store required additional time to catch up and recover. ‌ As part of the incident response, we have already implemented the following measures: ‌ 1. We have mitigated the networking control plane regression, and validated that control plane restarts do not interfere with the clusters' data plane 2. We implemented software changes to improve services' start time and remove first-start query alleviating pressure on the persistence layer and speeding up start and restart of services. 3. Deployed configuration changes to optimize networking control plane's effect on clusters ability to handle traffic 4. Removed the expensive database query from critical startup paths 5. Implemented additional monitoring and alerting for networking control-plane related issues ‌ Additionally, we will be implementing the following changes to prevent future incidents altogether: ‌ 1. We are introducing staged rollouts for infrastructure changes with longer soak time to ensure regressions are caught as early as possible and affect as few systems and users as possible. 2. We are auditing our systems for other slow queries that may affect service start time. ‌ We are continuing to improve our infrastructure to ensure greater resilience and faster recovery in the event of future incidents. ‌ We know that extended API outages affect our customers’ products and business, and outages of this magnitude are particularly damaging. While we came up short here, we are committed to preventing such incidents in the future and improving our service reliability.

Time: Aug. 17, 2024, 12:35 a.m.

Status: Resolved

Update: From 11:38am PT to 12:33pm PT and then from 1:02pm PT to 1:16pm PT, some users saw elevated errors across ChatGPT and Platform API services. Engineers have now completed a series of updates to ensure this issue doesn't recur. ChatGPT and Platform API services are fully operational. This issue has been fully resolved.

Time: Aug. 16, 2024, 10:56 p.m.

Status: Monitoring

Update: Engineers are continuing to make updates to ensure the issue does not reoccur. No impact is expected to ChatGPT and API services.

Time: Aug. 16, 2024, 9:21 p.m.

Status: Monitoring

Update: ChatGPT and API services continue to be operational. Engineers are making some additional updates to ensure the issue does not reoccur.

Time: Aug. 16, 2024, 7:28 p.m.

Status: Monitoring

Update: Engineers have taken measures to mitigate the elevated errors and are monitoring for additional changes.

Time: Aug. 16, 2024, 6:54 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Aug. 16, 2024, 6:53 p.m.

Status: Investigating

Update: We are currently investigating elevated error rates on ChatGPT and Platform API.

Time: Aug. 15, 2024, 4:48 p.m.

Status: Resolved

Update: Between 8:40AM and 9:23AM PDT, all requests to the Assistants API were failing. Assistants in the OpenAI Playground were also impacted. This issue is now resolved.