Time: Feb. 4, 2022, 2:30 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 14, 2021, 10:08 p.m.

Status: Monitoring

Update: FMX is completing an investigation to discover if any of the services we provide or third-party services we use are impacted by the Apache log4j vulnerability, log4shell. We have verified that the data you’ve entered into the FMX application (<hostname>.gofmx.com) is not impacted by log4shell. We do use a third-party service to provide our reporting dashboards. Like many services, they are impacted by log4shell. However, the part of this service impacted by the vulnerability is well guarded in the FMX infrastructure and is only accessible by FMX employees. Thus, we consider it highly unlikely for this vulnerability to ever have been exploited. Some of the internal services we use to provide customer support, sales, and marketing services do contain organizational-level customer data and are impacted by the vulnerability. Many of our vendors have since mitigated the vulnerability and we expect the rest to do so very soon. We are continuing to monitor our vendors’ responses and if we become aware of any unauthorized access to this organizational-level data, we will notify any impacted customers without delay. Organizational-level data can include organization name and contact information, contract value, product-use data, meeting notes, customer support conversations, etc. Updates will be posted to our status page as any additional information becomes available. Please reach out to [email protected] with any questions. For more information about log4shell please review CVE-2021-44228 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228) and the Apache Log4j2 (https://logging.apache.org/log4j/2.x/security.html) post.

Time: May 17, 2021, 9:11 p.m.

Status: Resolved

Update: A fix has been implemented and we are currently monitoring results. Please reach out to [email protected] if you are still unable to access your reporting dashboards.

Time: May 17, 2021, 7:03 p.m.

Status: Identified

Update: Reporting dashboards have been partially restored at this time. You may still be experiencing issues loading dashboards on your site. We will update periodically as more information is available.

Time: May 17, 2021, 5:59 p.m.

Status: Identified

Update: The issue has been identified and a fix has been implemented. Reporting dashboards may be offline for a few hours. We apologize for the inconvenience. We will update the status of this incident periodically and as we have more information.

Time: May 17, 2021, 9:11 p.m.

Status: Resolved

Update: A fix has been implemented and we are currently monitoring results. Please reach out to [email protected] if you are still unable to access your reporting dashboards.

Time: May 17, 2021, 7:03 p.m.

Status: Identified

Update: Reporting dashboards have been partially restored at this time. You may still be experiencing issues loading dashboards on your site. We will update periodically as more information is available.

Time: May 17, 2021, 5:59 p.m.

Status: Identified

Update: The issue has been identified and a fix has been implemented. Reporting dashboards may be offline for a few hours. We apologize for the inconvenience. We will update the status of this incident periodically and as we have more information.

Time: Feb. 18, 2021, 3:33 p.m.

Status: Postmortem

Update: We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **The root cause of the outage** On the morning of February 9th, we discovered that the page response time for the FMX application was higher than normal. For reference, “page response time” refers to the length of time it takes to load an individual page in an application. After investigating we determined that the cause of this high page response time was due to a lack of available hardware in the Microsoft Azure datacenter where our application is hosted. At any given time, FMX can tap into additional server capacity available in our datacenter to handle the load of increased traffic. On Tuesday morning as our traffic began to increase, we were unable to autoscale to a higher number of servers because there was no additional hardware available for us to scale up to.  **Solution** To solve the problem, we manually removed all of our servers and redeployed our application in a new cluster of servers. This allowed us to access additional hardware elsewhere in Azure. During this period, users were unable to access FMX at all. Once the application was fully re-deployed, users were once more able to access the application, and the application’s page response time was once more back in its normal range. **Future mitigations** After we confirmed the root cause of the outage, we alerted Microsoft to the problem and asked for additional guidance. They alerted us that although this problem is quite rare, it is possible that we may run into it again. For context, we have used Azure as our hosting provider for 8 years and have only encountered this issue once.  * Microsoft provided us with some advice on how to mitigate the problem without taking the application offline should we encounter it again.  * Additionally, we have added alerts for when an autoscale failure occurs so that we can respond more quickly to the problem should it ever recur.  * Lastly, we’re currently exploring options to improve our page response time during the time it would take for us to mitigate this problem if it reoccurs. Once more, we deeply apologize for this outage and we will be taking steps to ensure that in the unlikely event we have this problem in the future, that we will limit the amount of disruption as much as possible. ‌ Regards, FMX Team

Time: Feb. 9, 2021, 3:21 p.m.

Status: Resolved

Update: This issue has now been resolved. Normal site performance should be restored. We will continue to monitor the situation closely.

Time: Feb. 9, 2021, 2:48 p.m.

Status: Monitoring

Update: A fix has been implemented and we're currently monitoring the results. We will post a post-mortem when we have more details about the cause of the issue. We apologize for any inconvenience.

Time: Feb. 9, 2021, 2:05 p.m.

Status: Investigating

Update: We are continuing to investigate this issue. We apologize for the inconvenience.

Time: Feb. 9, 2021, 1:41 p.m.

Status: Investigating

Update: We are currently investigating an issue that is causing degraded performance. We apologize for any inconvenience.

Time: Feb. 18, 2021, 3:33 p.m.

Status: Postmortem

Update: We want to thank you for using FMX to manage your operations. We understand that FMX is often a critical component of running your organization and therefore we take any service disruptions seriously. This postmortem report will help you to better understand what caused the interruption in service as well as how we plan to avoid issues like these in the future. **The root cause of the outage** On the morning of February 9th, we discovered that the page response time for the FMX application was higher than normal. For reference, “page response time” refers to the length of time it takes to load an individual page in an application. After investigating we determined that the cause of this high page response time was due to a lack of available hardware in the Microsoft Azure datacenter where our application is hosted. At any given time, FMX can tap into additional server capacity available in our datacenter to handle the load of increased traffic. On Tuesday morning as our traffic began to increase, we were unable to autoscale to a higher number of servers because there was no additional hardware available for us to scale up to.  **Solution** To solve the problem, we manually removed all of our servers and redeployed our application in a new cluster of servers. This allowed us to access additional hardware elsewhere in Azure. During this period, users were unable to access FMX at all. Once the application was fully re-deployed, users were once more able to access the application, and the application’s page response time was once more back in its normal range. **Future mitigations** After we confirmed the root cause of the outage, we alerted Microsoft to the problem and asked for additional guidance. They alerted us that although this problem is quite rare, it is possible that we may run into it again. For context, we have used Azure as our hosting provider for 8 years and have only encountered this issue once.  * Microsoft provided us with some advice on how to mitigate the problem without taking the application offline should we encounter it again.  * Additionally, we have added alerts for when an autoscale failure occurs so that we can respond more quickly to the problem should it ever recur.  * Lastly, we’re currently exploring options to improve our page response time during the time it would take for us to mitigate this problem if it reoccurs. Once more, we deeply apologize for this outage and we will be taking steps to ensure that in the unlikely event we have this problem in the future, that we will limit the amount of disruption as much as possible. ‌ Regards, FMX Team

Time: Feb. 9, 2021, 3:21 p.m.

Status: Resolved

Update: This issue has now been resolved. Normal site performance should be restored. We will continue to monitor the situation closely.

Time: Feb. 9, 2021, 2:48 p.m.

Status: Monitoring

Update: A fix has been implemented and we're currently monitoring the results. We will post a post-mortem when we have more details about the cause of the issue. We apologize for any inconvenience.

Time: Feb. 9, 2021, 2:05 p.m.

Status: Investigating

Update: We are continuing to investigate this issue. We apologize for the inconvenience.

Time: Feb. 9, 2021, 1:41 p.m.

Status: Investigating

Update: We are currently investigating an issue that is causing degraded performance. We apologize for any inconvenience.