Time: July 19, 2024, 3:48 p.m.

Status: Resolved

Update: Dear valued customers, We understand that you may have concerns regarding the recent CrowdStrike incident (https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts). We want to assure you that Opti has conducted a thorough audit of our cloud environment, and we can confirm that none of our systems or systems of our customer data handling sub-processors rely on the CrowdStrike software that is currently experiencing an outage. As part of our commitment to providing a reliable platform, we have implemented extensive monitoring measures. In light of the CrowdStrike incident, we have performed spot checks on our customer dashboards and are pleased to inform you that there have been no indications of data loss or delay over the past 12 hours attributable to this incident. Rest assured that we are closely monitoring the situation as it evolves. We remain vigilant in ensuring the stability and security of our systems. We will continue to provide updates through this channel as necessary. Thank you for your trust in Opti. If you have any further questions or concerns, please do not hesitate to reach out to our support team.

Time: May 17, 2024, 6:36 p.m.

Status: Resolved

Update: All systems have returned to normal operating conditions. We have created new platform monitoring alerts to help us react faster to this issue in the future, and are considering other changes to further mitigate future occurrences of this event.

Time: May 17, 2024, 3:13 p.m.

Status: Monitoring

Update: Disaster recovery procedures worked as expected, and we are continuing to see signs of recovery of Watershed Operations service latency towards the pre-event baseline. We are monitoring the platform and continuing to explore possible root causes.

Time: May 17, 2024, 1:59 p.m.

Status: Identified

Update: We have identified an issue with our cloud platform leading to delayed processing of inbound events that began on 5/16/2024 and did not automatically recover. As of 5/17/2024 at 9 am EDT, Watershed Operations service was making decisions based on information approximately one and a half hours old. No data loss or failsafe operations have occurred at this time. Our engineers are implementing our disaster recovery procedures, and we will update this thread again when we are able to see signs of recovery.

Time: April 22, 2024, 2:28 p.m.

Status: Resolved

Update: We have confirmation from NOAA that the incident is resolved.

Time: April 19, 2024, 6:19 p.m.

Status: Monitoring

Update: We continue to see no errors requesting data from the National Weather Service. We have opened a ticket with the NOAA Senior Duty Meteorologist, which has been escalated to a specialist group within NWS to investigate. NOAA has not yet provided details on this outage. We continue to monitor for any regressions in service to be responsive to any further changes and will update this page accordingly.

Time: April 19, 2024, 3:03 p.m.

Status: Monitoring

Update: For the last hour, we have seen no errors requesting data from the National Weather Service. All impacted sites have received an updated forecast and have transitioned back to automatic operation. We have had no updates from NOAA on this outage event. We continue to monitor for any regressions in service to be responsive to any further changes and will update this page accordingly.

Time: April 19, 2024, 1:41 p.m.

Status: Investigating

Update: We are investigating degraded forecast reporting from the National Weather Service that began overnight. This outage may be related to a reported issue at NOAA's College Park data center, impacting National Weather Service operations, and with no estimated time of resolution provided: https://www.nco.ncep.noaa.gov/status/messages/. Some systems have already moved to their failsafe positions due to a lack of up-to-date forecast information.  We will continue to update this page with our progress. ---- Copy of NOAA's latest status as of this posting: SENIOR DUTY METEOROLOGIST NWS ADMINISTRATIVE MESSAGE NWS NCEP CENTRAL OPERATIONS COLLEGE PARK MD 0955Z THU APR 18 2024 ...RECOVERY EFFORTS IN THE COLLEGE PARK DATA CENTER... No significant updates during overnight restoration efforts. Still no ETR. Current known impacts include: - NCEP Centers' websites hosted in CP that remain inaccessible include EMC and NCWCP intranet sites. - WPC, OPC, and CPC's operational product suites' status, range from being degraded to down. - NCO operations personnel are still unable monitor NWS networks and circuits. - CONUS QPE data is not updating on MRMS (https://mrms.ncep.noaa.gov/data/) - Several layers are not updating on NWS Cloud Services (GIS and Map Viewer) - Multiple outside datasets are not available/delayed (UKMET data, ECMWF data, Canadian METARS) Liddick/SDM/NCO/NCEP

Time: April 1, 2024, 7:33 p.m.

Status: Resolved

Update: Due to an account management issue, weather forecasts from Tomorrow.io were unavailable between 2:30 AM EDT on March 30th, 2024, and 11:27 AM EDT on April 1st, 2024. Sites that depend on Tomorrow.io for their forecast source and were in automatic mode transitioned to failsafe after 8 hours without an updated forecast. All impacted systems have since recovered. While we do not expect this specific issue to occur again, we are updating our internal systems to diagnose and respond to Tomorrow.io integration issues more quickly.

Time: Jan. 25, 2024, 12:37 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Jan. 25, 2024, 12:12 a.m.

Status: Monitoring

Update: A fix has been released. Manual Control on the Opti Portal is working normally again. We are monitoring the platform for other issues.

Time: Jan. 24, 2024, 10:38 p.m.

Status: Identified

Update: We have identified an issue causing the Manual Control functionality on Opti Portal dashboards to not send commands or indicate which mode (Automatic or Manual) the system is presently in. We have identified the regression and are implementing a fix. No data loss or impacts to automated site operations occurred as a result of this regression. We expect to have service fully restored within the next 3 hours.