Time: July 2, 2024, 7:39 p.m.

Status: Postmortem

Update: # Summary On July 2nd at 6:47 AM MT, TrackVia Support reported an increase in support tickets related to application scripts. A quick investigation attributed the issue to a recent software release. Reverting the application change corrected the issue and full functionality was restored by 8:00 AM MT. Full timeline detailed below. # Impact Analysis A number of accounts encountered failed transactions corresponding error messages during the incident timeline. No data corruption was identified as a result of the incident. ### Scope: | **Scope** | **Value** | | --- | --- | | Count of Accounts Impacted | 84 | | Count of Failed Transactions | 3212 | | % Total Traffic Impacted | 0.04% | # Root Cause A new application script threading model introduced in the 23.43 release was designed to manage app script compilation and execution in a single-threaded manner, per application server. This design was intended to mitigate a long-standing issue. As part of the architectural changes, we also provided a fallback mechanism through an application configuration value that would allow us to utilize the legacy threading logic should any unexpected errors occur. The ultimate root cause is still under investigation. We believe we are running into a contention scenario related to a singleton value shared across application server instances and threads. # Resolution and Recovery TrackVia Operations added the config parameter, forcing the threading logic to revert to legacy handling routines, which remained unchanged in the codebase. New application servers were deployed utilizing the configuration override, restoring full service. # Corrective and preventative measures * TrackVia applications will investigate any error anomaly monitor to identify these type of issues * The overridden configuration value will active until such time we have fully identified and mitigated the root cause in a future release. * QA Testing methodology will be evaluated and adapted to more accurately reproduce contention-at-scale scenarios that were difficult to replicate in our testing environments. ## Incident Timeline * July 1 2024 1:01PM - Initial application deployment * July 1 2024 3:00PM - First error reported to SEIM * July 2 2024 6:47AM - Support team escalated customer reported issue to Operations team * July 2 2024 7:38AM - Parameter config changed and application deploy begins * July 2 2024  8:00AM - Functionality restored

Time: July 2, 2024, 3:06 p.m.

Status: Resolved

Update: On July 2nd at 6:47 AM MT, TrackVia Support reported an increase in support tickets related to application scripts. A quick investigation attributed the issue to a recent software release. Reverting the application change corrected the issue and full functionality was restored by 8:00 AM MT.

Time: July 2, 2024, 7:39 p.m.

Status: Postmortem

Update: # Summary On July 2nd at 6:47 AM MT, TrackVia Support reported an increase in support tickets related to application scripts. A quick investigation attributed the issue to a recent software release. Reverting the application change corrected the issue and full functionality was restored by 8:00 AM MT. Full timeline detailed below. # Impact Analysis A number of accounts encountered failed transactions corresponding error messages during the incident timeline. No data corruption was identified as a result of the incident. ### Scope: | **Scope** | **Value** | | --- | --- | | Count of Accounts Impacted | 84 | | Count of Failed Transactions | 3212 | | % Total Traffic Impacted | 0.04% | # Root Cause A new application script threading model introduced in the 23.43 release was designed to manage app script compilation and execution in a single-threaded manner, per application server. This design was intended to mitigate a long-standing issue. As part of the architectural changes, we also provided a fallback mechanism through an application configuration value that would allow us to utilize the legacy threading logic should any unexpected errors occur. The ultimate root cause is still under investigation. We believe we are running into a contention scenario related to a singleton value shared across application server instances and threads. # Resolution and Recovery TrackVia Operations added the config parameter, forcing the threading logic to revert to legacy handling routines, which remained unchanged in the codebase. New application servers were deployed utilizing the configuration override, restoring full service. # Corrective and preventative measures * TrackVia applications will investigate any error anomaly monitor to identify these type of issues * The overridden configuration value will active until such time we have fully identified and mitigated the root cause in a future release. * QA Testing methodology will be evaluated and adapted to more accurately reproduce contention-at-scale scenarios that were difficult to replicate in our testing environments. ## Incident Timeline * July 1 2024 1:01PM - Initial application deployment * July 1 2024 3:00PM - First error reported to SEIM * July 2 2024 6:47AM - Support team escalated customer reported issue to Operations team * July 2 2024 7:38AM - Parameter config changed and application deploy begins * July 2 2024  8:00AM - Functionality restored

Time: July 2, 2024, 3:06 p.m.

Status: Resolved

Update: On July 2nd at 6:47 AM MT, TrackVia Support reported an increase in support tickets related to application scripts. A quick investigation attributed the issue to a recent software release. Reverting the application change corrected the issue and full functionality was restored by 8:00 AM MT.

Time: May 10, 2024, 5:57 p.m.

Status: Postmortem

Update: # Incident Report May 10, 2024 ## Summary On May 8 at 8:07 AM MT, TrackVia Support notified the TrackVia Operations team that integration connections were being dropped. An investigation of the issue determined that a TrackVia vendor was reporting issues with their API service.TrackVia’s Vendor first reported an issue at 7:21 MT and resolved the issue. Full functionality was restored by 7:59 MT. ## Root cause TrackVia’s Vendor reports unexpected behavior of the network controller managing load balancers recreated some resources with invalid configurations. ## Resolution and recovery TrackVia’s Vendor corrected the mis-configuration and restored API operations by 7:59 MT. ## Corrective and preventive measures * TrackVia’s Vendor replaced the network controller which caused the mis-configuration. * TrackVia added additional monitoring to identify vendor issues in a more timely manner.  ## Incident timeline * May 8, 2023 6:58 MT: Vendor update of a production cluster. * May 8, 2023 7:17 MT: Vendor started receiving alerts for the internal API, Echo API and ActiveDocs Proxy. * May 8, 2023 7:58 MT: Root cause identified and solution deployed. * May 8, 2023 8:00 MT: Service is restored.

Time: May 10, 2024, 5:57 p.m.

Status: Postmortem

Update: # Incident Report May 10, 2024 ## Summary On May 8 at 8:07 AM MT, TrackVia Support notified the TrackVia Operations team that integration connections were being dropped. An investigation of the issue determined that a TrackVia vendor was reporting issues with their API service.TrackVia’s Vendor first reported an issue at 7:21 MT and resolved the issue. Full functionality was restored by 7:59 MT. ## Root cause TrackVia’s Vendor reports unexpected behavior of the network controller managing load balancers recreated some resources with invalid configurations. ## Resolution and recovery TrackVia’s Vendor corrected the mis-configuration and restored API operations by 7:59 MT. ## Corrective and preventive measures * TrackVia’s Vendor replaced the network controller which caused the mis-configuration. * TrackVia added additional monitoring to identify vendor issues in a more timely manner.  ## Incident timeline * May 8, 2023 6:58 MT: Vendor update of a production cluster. * May 8, 2023 7:17 MT: Vendor started receiving alerts for the internal API, Echo API and ActiveDocs Proxy. * May 8, 2023 7:58 MT: Root cause identified and solution deployed. * May 8, 2023 8:00 MT: Service is restored.

Time: May 7, 2024, 2:48 p.m.

Status: Resolved

Update: A TrackVia vendor experienced a service disruption in the ActiveDocs Proxy which may have caused interruption with TrackVia api services. The disruption lasted for approximately 38 minutes and was operational by 7:59 AM MT.

Time: May 7, 2024, 2:48 p.m.

Status: Resolved

Update: A TrackVia vendor experienced a service disruption in the ActiveDocs Proxy which may have caused interruption with TrackVia api services. The disruption lasted for approximately 38 minutes and was operational by 7:59 AM MT.

Time: May 10, 2024, 10:10 p.m.

Status: Resolved

Update: Web Forms were unavailable on April 26,2024 due to a deploy issue. This outage affected the Private, HIPAA and Gov clouds

Time: May 10, 2024, 10:10 p.m.

Status: Resolved

Update: Web Forms were unavailable on April 26,2024 due to a deploy issue. This outage affected the Private, HIPAA and Gov clouds

Time: Sept. 19, 2023, 4:07 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Sept. 19, 2023, 12:36 a.m.

Status: Monitoring

Update: TrackVia's cloud provider has successfully applied an update to the subsystem responsible for network mapping propagation to address resource contention. They have seen network mapping propagation times stabilize but they have not yet begun to trend towards normal levels. They expect that to begin over the next 30 minutes, at which time we expect latencies and error rates to improve. We will continue to keep you updated on our progress towards full recovery.

Time: Sept. 18, 2023, 11:31 p.m.

Status: Identified

Update: TrackVia's cloud provider continues to progress toward resolving the increased networking latencies and errors. At this time, they are approximately 50% completed with the update to address resource contention within the subsystem responsible for network mappings propagation. Their current expectation is to have the problem resolved within the next 60 to 90 minutes, and we will continue to provide updates as recovery progresses.

Time: Sept. 18, 2023, 9:33 p.m.

Status: Identified

Update: While TrackVia's cloud provider continues to make progress in addressing the issue, we wanted to provide some more details on the issue. Starting at 10:00 AM PDT this morning, TrackVia's cloud provider has been experiencing a delay in the propagation of network configuration changes to the underlying hardware ensuring network packets can flow between source and destination. The root cause appears to be increased load to the subsystem responsible for the handling of these network mappings. Once load has been reduced on the subsystem responsible for network mapping propagation, we would expect full recovery. We will continue to keep you updated as we make progress towards full recovery.

Time: Sept. 18, 2023, 7:46 p.m.

Status: Identified

Update: TrackVia's cloud provider is continuing to investigate increased networking latencies and errors affecting Availability Zones in the US-WEST-2 Region. The issue affects some instances in each of these Availability Zones, where network mappings are not being propagated to the underlying hardware. Any changes to network configurations would see delayed mappings, which will affect network connectivity. For other TrackVia services, such as micro services and backup/recovery, delays in function as well as increased error rates may occur due to this issue. AWS is working to resolve the issue and is seeing improvements as a result of these efforts. We will continue to keep you updated as we make progress towards full recovery.

Time: Sept. 18, 2023, 7:10 p.m.

Status: Identified

Update: TrackVia's cloud provider is continuing to work on a fix for this issue.

Time: Sept. 18, 2023, 7:09 p.m.

Status: Identified

Update: [11:57 AM PDT] TrackVia's cloud provider confirmed increased networking latencies and errors affecting multiple Availability Zones in the US-WEST-2 Region. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates and latencies. We continue to work towards a full root case and recovery. [11:43 AM PDT] TrackVia's cloud provider is investigating increased networking latencies and errors affecting multiple Availability Zones in the US-WEST-2 Region.