Time: Aug. 3, 2023, 6:42 p.m.

Status: Resolved

Update: Between 08/03/2023 00:49 and 08/03/2023 03:11 EDT, customers’ sites in North America that are serviced by AT&T towers experienced 3 periods of temporary cellular outages. The longest of these outages lasted 33 minutes. Customers can expect brief periods of missing data during this period of time. Watershed Operations were degraded during this period, however, all impacted sites have resumed normal operations without transitioning to failsafe. Our cellular provider’s Status Page attributes this to AT&T network maintenance: https://status.particle.io/incidents/tyjmvj50qj91.

Time: July 17, 2023, 8:08 p.m.

Status: Resolved

Update: We have received confirmation from NOAA that the service issue should be resolved on their end, independently validated that their service is now operational, and switched all impacted customer deployments back to the production endpoints.

Time: July 5, 2023, 7:54 p.m.

Status: Monitoring

Update: Weather forecasts from the National Weather Service for many areas of North America stopped being updated on 7/2/2023. Starting at 7/4/2023 at 6pm EDT, impacted sites began to transition to their pre-configured failsafe positions. By 7/5/2023 at 11 AM EDT almost 50% of our control fleet had made that transition, and Opti’s engineering team had determined a workaround to acquire current forecasts, restoring service to impacted sites. We are in contact with NOAA operations teams to assist with their efforts at resolution and ensure our data needs are continually met. During this event, site data was not lost, however, sites may have released stored water when they transitioned to failsafe. Impacted customers can see the period of time their sites were in Failsafe mode in the System Status visualization in the left sidebar of their site dashboards.

Time: May 12, 2023, 10:05 p.m.

Status: Resolved

Update: All systems have been upgraded, which has mitigated the likelihood of a false offline alert email being sent. This issue is resolved.

Time: May 12, 2023, 8:42 p.m.

Status: Identified

Update: Upgrades are continuing to roll out. We expect all impacted deployments will be upgraded around 6pm EDT, at which point this issue will be resolved.

Time: May 12, 2023, 5:05 p.m.

Status: Identified

Update: We have identified a subset of our offline alert deployments were not upgraded to the latest version, which already includes a patch for this potential source of false alarms. We are rolling out this upgrade to the remaining offline alert deployments, which will correct this issue. No other alarms appear to be impacted by this issue.

Time: May 12, 2023, 2 p.m.

Status: Investigating

Update: We are aware of an issue with requests to one of our databases that has caused several customers to experience false offline alarms this morning. Data from impacted sites is not lost, control operations appear unaffected, and dashboards correctly reflect site operations. We are analyzing the issue and will continue to update this incident as we understand an ETA for resolution. Please reach out to [email protected] if you have received alarms about your site that you suspect may be erroneous today.

Time: April 12, 2023, 4:07 p.m.

Status: Resolved

Update: From 12:39 UTC to 13:43 UTC (54 minutes) all DNS queries to OptiRTC services were unavailable due to an issue with our DNS provider. During this time, the company home page at optirtc.com, the Opti Portal at portal.onopti.com, and the Device Telemetry and Public APIs were unavailable. Other than Device Telemetry API integrations, this issue did not impact watershed operations or cause data loss: automated control was functioning normally. To prevent this type of issue from occurring again, we are eliminating single points of failure in our DNS management we identified during this outage.

Time: Jan. 24, 2023, 3:01 a.m.

Status: Resolved

Update: This incident has been resolved. Exports are running smoothly with the mitigations we have put in place. We will be making future changes to address stability issues with some exceptionally long and large exports.

Time: Dec. 2, 2022, 7:40 p.m.

Status: Investigating

Update: We have reset impacted servers and data exports are working as normal again. If you have requested data in the last day and have not received it yet, retry your export now to receive your data. We are continuing to investigate this issue and will keep updating this case as we identify a root cause we can address. In the meantime, we are monitoring data exports and will reach out if your export appears to be stuck in the system.

Time: Dec. 2, 2022, 4:32 p.m.

Status: Investigating

Update: We are aware of an issue impacting at least several customers where data export jobs are taking a long time to start, appearing to be "stalled" in the Opti Portal. We are investigating possible root causes and adjusting service configuration to increase throughput on this service and get these exports to run.