Is there an Floify outage?
Floify status: Systems Active
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Floify and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Floify and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Floify.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| System | Active |
View the latest incidents for Floify and check for official updates:
Description: ### Floify Service Interruption August 15, 2023 to August 16, 2023 This incident report details the intermittent issues with Floify that occurred August 15-16, 2023. **Issue Summary** From 1 PM MT on August 15 to 4:21 PM MT on August 16, Floify experienced intermittent database exceptions causing some loan creation and conversion interruptions. The root causes were determined to be a database query performance issue and an AWS database failover incident. **Timeline \(all times Mountain Time\)** August 15, 2023 * 1:00 PM Pagers alerted the Engineering team about abnormal database exceptions * 1:05 PM Hibernate exceptions were identified in the logs. * 1:10 PM Investigation of the root cause began in partnership with AWS. * 1:30 PM Engineering was able to identify and cancel a few blocking SQL queries to improve stabilization. * 3:30 PM Pagers alerted Engineering about abnormal database exceptions * 3:44 PM and 5:04 PM Rolling refreshes were performed as part of standard procedures. August 16, 2023 * 8 AM - 12 PM An issue was identified with the database locking mechanism used in one query. Code updates to fix the issue were made. AWS advised to update the database lock configuration parameter that would positively affect query performance and system stability. * 12 PM - 2 PM QA tested and certified the code patch. * 1:55 PM A rolling refresh performed with the code fix and the database parameter update. * 2:00 PM Engineering noticed dynamic database parameter update caused high CPU usage on the writer database instance. * 2:00 PM Engineering team partnered with AWS to address the issue. * 4:05 - 4:16 PM An automatic database failover was performed by AWS which caused loss of connectivity to the database and thereby intermittent issues in the Floify application. AWS confirmed their platform was the cause of this issue. * 4:21 PM A rolling refresh was performed on to refresh database connections. After monitoring database usage, the incident has been resolved. **Root Cause** A database SELECT FOR UPDATE mechanism was used in Floify code to synchronize changes to the loan custom field data. However, due to an issue with MySQL 5.x this query was holding a significant number of gap locks under certain load conditions. This locking caused concurrency issues with multiple transactions and led to database query timeouts. The first occurrence of this issue was detected on August 15, 2023. **Resolution and recovery** At 1:00 PM on August 15, 2023, monitoring systems alerted our engineers who investigated and escalated the issue. For the rest of the day Engineers were able to alleviate the effect of the issue by performing rolling refreshes of the Floify application and selectively terminating certain locked queries. The Engineering team quickly contacted AWS to troubleshoot the issue and develop a fix. At 4:21 PM on August 16, 2023, Engineers were able to deploy fixes to address the root cause and access to the Floify service was restored. **Corrective and Preventative Measures** The following are actions we are taking to address the underlying causes of the issue and to help prevent recurrence and improve response times: Short term: * Manually perform Floify application refreshes to temporarily resolve the database locking issues \(Completed\) * Deploy code changes and database configuration updates to resolved the issue \(Completed\) Longer term strategies: * Explore database strategies in partnership with AWS to identify hot spots and opportunities to scale to mitigate downtime \(In Progress\) Floify is committed to continually adapting and improving our technology and process to prevent service interruptions. We appreciate your patience and again apologize for the impact of this issue. We thank you for your business and continued support. Sincerely, The Floify Team
Status: Postmortem
Impact: Critical | Started At: Aug. 15, 2023, 9:49 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Aug. 10, 2023, 10:41 p.m.
Description: The root cause of the ongoing issue has been identified as a lock in the database. We have a short-term solution for any such occurrences. Our team is actively working on implementing a comprehensive solution to permanently resolve this problem. While we work towards a full resolution, we will continue to closely monitor the situation and address any occurrences with urgency.
Status: Resolved
Impact: Major | Started At: July 17, 2023, 9:22 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: July 17, 2023, 4:50 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: None | Started At: July 17, 2023, 3:18 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
