Time: April 4, 2023, 7:12 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: April 4, 2023, 7:07 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: April 4, 2023, 6:07 p.m.

Status: Investigating

Update: We are actively investigating the issue with users unable to view loans in their pipeline. We understand the critical nature of this issue and will provide an update as soon as we have identified the issue.

Time: March 20, 2023, 11:01 p.m.

Status: Resolved

Update: The incident has been identified and resolved. MeridianLink (LQB) recently updated their API password requirements without providing prior notification of this change. This caused the Floify API user to be locked out in customers LQB instances. This issue has been resolved for most instances where the customer provided us their updated password. If you are still experiencing issues, please reach out to [email protected] with your updated password.

Time: March 20, 2023, 9:08 p.m.

Status: Investigating

Update: We are currently investigating an issue with MeridianLink (LQB) webhook failures some customers are experiencing. We will update this page once we've identified the issue and have begun working on a fix.

Time: March 20, 2023, 4:15 p.m.

Status: Resolved

Update: This incident has been resolved and there are no longer issues with disclosure downloads.

Time: March 20, 2023, 3:48 p.m.

Status: Identified

Update: Encompass was updated to version 23.1 and has caused some issues with disclosure downloads. This is not impacting all Encompass integrated dashboards, this is only intermittent. Our engineering team has determined a fix and this is currently being tested. An update will be posted when this is ready for release.

Time: Feb. 25, 2023, 3:52 p.m.

Status: Resolved

Update: The fix to route disclosures to the appropriate workflow depending on when they were submitted has been released to production and is working as expected. We will plan to turn on the 4506-C feature for all disclosure companies on Monday, February 27th.

Time: Feb. 24, 2023, 6:26 p.m.

Status: Identified

Update: Due to the timing of the 4506-C deployment some disclosures in flight were impacted by the 4506-C release. As of 9:40am MT we have turned this feature off and any disclosures submitted after this time are not impacted. Around 50 pending disclosures were impacted this morning, our team will be notifying disclosure contacts and uploading these documents to assigned eFolders throughout the day. Our engineering team is working on a fix to route disclosures to the appropriate workflow depending on when they were submitted. An update will be posted when this is ready for release.

Time: Aug. 25, 2022, 9:25 p.m.

Status: Postmortem

Update: ### Floify Service Interruption August 25, 2022 This incident report details the Floify outage that occurred on August 25, 2022. ‌ **Issue Summary** From 7:45 AM to 12:18 PM MT, Floify experienced a system outage causing users to lose access to the Floify service. The root cause of this outage was a database performance issue that triggered excessive resource usage on the Floify server. ‌ **Timeline \(all times Mountain Time\)** * 7:45 AM - 12:18 PM Service outage * 7:45 AM Pagers alerted the Engineering team * 8:00 AM Engineering identified Hazelcast exceptions in the logs. * 8:00 AM - 8.30 AM Engineering restarted the Floify apps * 8.30 AM - 9:00 AM The Engineering team noticed SQL exceptions in the logs. Analysis of the exceptions in progress. * 9:00 AM - 11:00 AM Several database indexes were proposed to resolve the issue however none of them solved the issue * 11:00 AM - 12 PM The Engineering team archived records from a key Floify table. Queries now started to perform efficiently. * 12:00 PM - 12:18 PM Floify apps were restarted and the fix was verified ‌ **Root Cause** An increase in the number of records in a key Floify database table caused the queries on the table to run abnormally slow. This increase caused inordinate resource usage and concurrent stability issues. Slow-running queries consumed available resources on servers and prevented them from receiving other requests, which then triggered automatic application restarts. The first occurrence of this issue was detected on the morning of August 25, 2022, and was triggered as a result of higher volume in the database table. ‌ **Resolution and recovery** At 7:45 AM MT, monitoring systems alerted our engineers who investigated and escalated the issue. By 12:00 PM, engineers identified the root cause of the problem and developed a fix. At 12:18 PM, access to the Floify service was restored. ‌ **Corrective and Preventative Measures** The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: Short term: * Manually archive records from the Floify table that was causing query slowness \(Completed\) * Implement an automated solution for automatic archivals for the affected table once a week \(In Progress\) * Use indexes or other mechanisms to improve query performance going forward  \(In Progress\) Longer-term strategies being explored: * Explore database strategies to identify hot spots and opportunities to scale to mitigate downtime * Design and implement architectural changes to application task management to mitigate downtime risk * Implement additional automation to archive tables * Instrument performance analysis and profiling tools to identify areas of risk Floify is committed to continually adapting and improving our technology and process to prevent service interruptions. We appreciate your patience and again apologize for the impact of this outage. We thank you for your business and continued support. ‌ Sincerely, The Floify Team

Time: Aug. 25, 2022, 8:42 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 25, 2022, 5:20 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Aug. 25, 2022, 4:59 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Aug. 25, 2022, 2:21 p.m.

Status: Investigating

Update: We are currently investigating this issue.