Time: Feb. 17, 2023, 4:17 p.m.

Status: Resolved

Update: Invoca Availability for January 2023: 100% Invoca Availability for Q1 2023: 100%

Time: Jan. 9, 2023, 6:18 p.m.

Status: Resolved

Update: Invoca Availability for December 2022: 100% Invoca Availability for Q4 2022: 100%

Time: Dec. 13, 2022, 3:59 p.m.

Status: Resolved

Update: Invoca Availability for November 2022: 100% Invoca Availability for Q4 2022: 100%

Time: Nov. 4, 2022, 1:27 p.m.

Status: Resolved

Update: Invoca Availability for October 2022: 100% Invoca Availability for Q4 2022: 100%

Time: Oct. 27, 2022, 1:18 p.m.

Status: Postmortem

Update: **Summary** Between 12:13 and 18:55 PT on October 19th, 2022 the Invoca platform experienced disruptions to the following services: * Reporting availability * Reporting data freshness * Signal processing * Post-call webhooks * Invoca web portal login * Transaction Events API requests and ordering While reporting data was delayed, no data was lost. There were no impacts to call routing or other Invoca services not listed. Availability and functionality varied per service and are detailed further below. ‌ **Sequence of events** Our metrics quickly alerted us to a problem and our engineering team was able to begin coordination and mitigation right away. The timeline below provides a high-level overview of key events, service availability,  and mitigation efforts: ‌ **12:25:** Automated monitoring notifies us of a problem in the data processing pipeline. **12:27:** Engineering teams start our incident response process. **12:31:** Isolation to a specific component in the data processing pipeline. **13:32:** First change deployed to attempt mitigation. Service for Invoca portal & Transaction Events API restored. Reports are still failing and new report data is delayed. **15:03:** Second change deployed for mitigation. This change enabled reports to be displayed on the portal, but new report data was still delayed. **16:01:** Incident fully mitigated. Final change deployed for mitigation. This change allowed the system to fully recover and report data started to catch up from this point. **18:55:** All reporting data was caught up, and the incident was fully resolved. ‌ **Why it happened** The main failure occurred in the backend used to move data for post-call processing and reporting. A recent change to a software library used in the backend introduced a defect in the logic used to re-queue failed data processing jobs. The retry logic would work for a typical amount of expected failures but could not slow down or stop retrying failed jobs when they accumulated. ‌ **What we did and what we’re doing** During the incident, we made a change to how the retry logic operates. This was key to resolving this incident both now and in the future. Post-incident, we’re planning additional changes: * Improvements in our ability to detect and troubleshoot issues in the data processing pipeline, allowing us to find problems sooner and resolve incidents faster * Removing the potential for a “snowball effect” to overwhelm the system with additional changes to retry logic for data processing * Speeding up our ability to deploy changes to the Invoca platform during incidents while maintaining our commitments to software lifecycle best practices

Time: Oct. 19, 2022, 11:33 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Oct. 19, 2022, 11:12 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Oct. 19, 2022, 10:39 p.m.

Status: Investigating

Update: The change has been deployed and is currently updated. New data is still delayed and we're continuing to work on this update.

Time: Oct. 19, 2022, 9:06 p.m.

Status: Investigating

Update: The change has been deployed and is currently updated. New data is still delayed and we're continuing to work on this update.

Time: Oct. 19, 2022, 8:44 p.m.

Status: Identified

Update: The change has been deployed and we are currently assessing the results. Thank you for your patience.

Time: Oct. 19, 2022, 8:17 p.m.

Status: Identified

Update: We have identified the issue and are prepping to deploy a fix. Targeting to deploy in the next 5-10 minutes.

Time: Oct. 19, 2022, 8:11 p.m.

Status: Investigating

Update: We are currently investigating issues with the Invoca platform reporting not loading as expected. Call records and associated data are being captured, but are taking longer than normal to become available in Invoca reporting after the call is completed. Updates will be provided as they become available.