Time: Nov. 30, 2023, 7:46 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 30, 2023, 7:06 p.m.

Status: Monitoring

Update: We're pleased to inform you that we have successfully implemented a resolution for the multidev environment creation and site creation issue. Our team is diligently monitoring the results to ensure continued stability. Should you have any further questions or encounter ongoing concerns, please don't hesitate to reach out to our support team by initiating a support chat. We appreciate your understanding and patience throughout this process.

Time: Nov. 30, 2023, 6:48 p.m.

Status: Identified

Update: We've identified the issue and are actively implementing a fix. We apologize for any inconvenience caused and appreciate your patience. If you need immediate assistance, please open a support chat with us, or email [email protected].

Time: Nov. 30, 2023, 6:35 p.m.

Status: Investigating

Update: We are currently experiencing an infrastructure issue that is affecting several workflows on Pantheon, including creating new sites or environments. There is no impact on sites or downtime due to this issue. We apologize for the interruption to your work and appreciate your patience as we work towards a fix.

Time: Dec. 6, 2023, 6:13 p.m.

Status: Postmortem

Update: On November 24th 2023, between 05:35 and 07:00 UTC, a small percentage of Pantheon sites started reporting downtime per internal monitoring. Upon further investigation it was determined that an attacker was affecting Pantheon's routing infrastructure. This lead to an intermittent drop in other traffic, with 503 responses being served from customer sites. We are actively working to reduce our exposure to this sort of incident in the future by tuning rate limiting as well as improving tooling and processes to address malicious activity.

Time: Nov. 25, 2023, 8:06 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 25, 2023, 7:53 a.m.

Status: Investigating

Update: We have detected degraded application performance on the platform where sites are showing "Page Could Not Be Loaded" 503 errors. Rest assured, our team is currently investigating the issue. For urgent issues, please contact support via [email protected] or by opening a support chat.

Time: Nov. 28, 2023, 7:14 p.m.

Status: Postmortem

Update: On November 14, 2023, starting at 20:19 UTC, a surge of workflow failures alerted Pantheon engineering to an issue pulling container images from an upstream vendor registry. Confirmation of the upstream issue was quickly confirmed via the vendor status page site detailing the external service outage impacting container image push and pull operations. Upon upstream remediations, the workflow failure rate returned below threshold on November 14, 2023 at 23:25 UTC. We have identified areas to improve our infrastructure to prevent issues like this from occurring in the future.

Time: Nov. 15, 2023, 2:48 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Nov. 15, 2023, 11:36 a.m.

Status: Monitoring

Update: We're pleased to report that all systems are currently functioning correctly. Our team is actively monitoring performance to ensure optimal service. If you have any questions or remaining issues with workflows, please open a support chat with us, or email [email protected]. We'll provide another status update within the next 6 hours or sooner if the situation is resolved. Thank you for your patience and understanding.

Time: Nov. 15, 2023, 5:11 a.m.

Status: Monitoring

Update: All systems are functioning correctly. However, we are currently in the process of closely monitoring to ensure optimal performance. Please feel free to reach out if you have any further questions. We will provide another update within 6 hours or sooner if the problem is resolved.

Time: Nov. 15, 2023, 2:36 a.m.

Status: Monitoring

Update: We are still in the monitoring phase to confirm the solution's effectiveness. The fix to address the recent workflow issues on Pantheon has been implemented for more than 2 hours now, and we haven't received any additional reports. Further updates will be provided as the situation progresses. Your patience and understanding are appreciated. If you have any questions or encounter workflow issues, please don't hesitate to contact support.

Time: Nov. 15, 2023, 12:09 a.m.

Status: Monitoring

Update: Our upstream provider has implemented a fix that resolves the recent workflow issues on Pantheon. We are now in the monitoring phase to ensure that the solution is effective. We will provide further updates as the situation progresses. Thank you for your patience and understanding, and please reach out to support if you have any questions or remaining issues with workflows.

Time: Nov. 14, 2023, 11:12 p.m.

Status: Identified

Update: The issue continues to affect various workflows on Pantheon. Our team is diligently working on addressing the current challenge, exploring all possible solutions while coordinating with our service provider. We will share the next update in 3 hours, or sooner if more information becomes available. Your patience and understanding during this time are greatly appreciated.

Time: Nov. 14, 2023, 10:03 p.m.

Status: Identified

Update: The issue is still impacting workflows on Pantheon. Our team is actively exploring all available options to alleviate the current issue. We will provide an update in 1 hour or as more information becomes available. Thank you for your ongoing patience.

Time: Nov. 14, 2023, 9:14 p.m.

Status: Identified

Update: We are currently experiencing an infrastructure issue that is affecting several workflows on Pantheon, including code synchronization, cache clearing, enabling Solr, and creating new sites or environments. The current investigation points to an issue with one of our Docker image service providers. We are working with the provider to resolve the issue as quickly as possible. We apologize for any inconvenience caused and appreciate your patience as we work towards a fix.

Time: Nov. 16, 2023, 6:45 p.m.

Status: Postmortem

Update: On November 12, 2023, at 13:33 UTC, we encountered an infrastructure issue affecting routing, which led to temporary service disruptions on multiple Pantheon-hosted sites. Our team quickly identified and addressed the root cause, restoring all services by 14:42 UTC. A brief recurrence of the issue occurred on November 13 at 02:06 UTC, which was rapidly resolved by our engineering team. We apologize for the impact caused to our customers and assure you that we are actively taking steps to enhance our systems to prevent similar incidents in the future. These improvements include strategic upgrades to our traffic management protocols, set to be implemented in November 2023, and refinement of our response strategies to ensure swift resolution of potential issues. At Pantheon, we remain committed to providing reliable and secure hosting services.

Time: Nov. 12, 2023, 4:35 p.m.

Status: Resolved

Update: The issue has been successfully resolved, and we anticipate no further disruptions. If you encounter any unexpected issues, please feel free to reach out to our support team for immediate assistance. We sincerely appreciate your patience and continued support during this incident.

Time: Nov. 12, 2023, 3:59 p.m.

Status: Monitoring

Update: A fix has been implemented and we are actively monitoring the situation as we observe a complete recovery in progress.

Time: Nov. 12, 2023, 3:29 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Nov. 12, 2023, 2:54 p.m.

Status: Investigating

Update: We are currently investigating an infrastructure failure impacting customer sites. The identified symptoms include encountering 503 errors, resulting in site unavailability. For urgent issues, please contact [email protected]

Time: Nov. 5, 2023, 12:10 a.m.

Status: Resolved

Update: The issue has been successfully resolved and there should be no further disruptions. If you come across any unexpected issues, please don't hesitate to get in touch with our support team for immediate assistance.

Time: Nov. 4, 2023, 1:13 a.m.

Status: Monitoring

Update: While Pantheon's Front End Sites infrastructure remains available and stable, our team will remain vigilant around the clock to ensure the stability of our service.

Time: Nov. 3, 2023, 1:28 p.m.

Status: Monitoring

Update: While our infrastructure is running normally and remains stable, we're actively keeping an eye out for any possible issues.

Time: Nov. 3, 2023, 10:01 a.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems

Time: Nov. 3, 2023, 5:48 a.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems

Time: Nov. 3, 2023, 2:19 a.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems

Time: Nov. 3, 2023, 1:19 a.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems

Time: Nov. 3, 2023, 12:19 a.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems.

Time: Nov. 2, 2023, 11:18 p.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems.

Time: Nov. 2, 2023, 10:14 p.m.

Status: Monitoring

Update: While our infrastructure remains stable, we are actively monitoring for any possible problems.

Time: Nov. 2, 2023, 7:52 p.m.

Status: Monitoring

Update: While we are observing stability on our platform, we will continue to monitor for any potential issues.

Time: Nov. 2, 2023, 6:29 p.m.

Status: Monitoring

Update: The infrastructure remains in a stable. We are actively monitoring for any potential issues.

Time: Nov. 2, 2023, 5:19 p.m.

Status: Monitoring

Update: The fix has been implemented, and we are currently monitoring for stability. Should you have any concerns, please open a support chat or ticket and we will be happy to assist.

Time: Nov. 2, 2023, 4:25 p.m.

Status: Identified

Update: We are continuing to work on a fix for this issue, and we have identified that it specifically impacts Gatsby sites on Pantheon's Front End Sites. Rest assured, we will continue to provide you with frequent updates until the issue is completely resolved. We sincerely appreciate your patience as we work to mitigate the problem.

Time: Nov. 2, 2023, 3:20 p.m.

Status: Identified

Update: Work on an infrastructure fix is still in progress. We have narrowed down the impact scope and can confirm that the problem is limited to Gatsby sites on Pantheon's Front End Sites infrastructure. Next.js sites and all other services are operating normally. We are making progress and will continue to provide updates as we move closer to a resolution.

Time: Nov. 2, 2023, 1:49 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented. We will keep providing regular updates until the issue is resolved, thank you for your patience as we work on mitigating the problem.

Time: Nov. 2, 2023, 1:28 p.m.

Status: Investigating

Update: We are actively investigating an infrastructure issue leading to occasional error displays on Front End Sites. This issue does not impact Drupal or WordPress sites but is specific to Next.js and Gatsby sites on Pantheon's Front End Sites infrastructure. The error may also be visible on docs.pantheon.io.