Is there an Files.com outage?
Files.com status: Systems Active
Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for Files.com and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for Files.com and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Files.com.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Australia Region | Active |
| Background Jobs, including Sync and Webhooks | Active |
| Canada Region | Active |
| Core Services / API | Active |
| EU (Germany) Region | Active |
| Files Tools | Active |
| FTP/FTPS | Active |
| Japan Region | Active |
| Remote Server Integrations (Sync and Mount) | Active |
| SFTP | Active |
| Singapore Region | Active |
| UK Region | Active |
| USA Region | Active |
| WebDAV | Active |
| Web Interface | Active |
View the latest incidents for Files.com and check for official updates:
Description: All services have been restored and are operating normally. We have resolved a performance degradation on Files.com which affected all Files.com core and auxiliary services in all regions. This incident occurred between the times of 12:17 pm PT to 1:01 pm PT. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.
Status: Resolved
Impact: Major | Started At: Jan. 12, 2024, 9:01 p.m.
Description: All services have been restored and are operating normally. We have resolved a performance degradation on Files.com which affected all Files.com core and auxiliary services in all regions. This incident occurred between the times of 12:17 pm PT to 1:01 pm PT. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.
Status: Resolved
Impact: Major | Started At: Jan. 12, 2024, 9:01 p.m.
Description: On December 5th, 2023, at 4:06 AM PST, [Files.com](http://Files.com) correlated multiple customer tickets indicating _‘authentication errors when logging into SFTP’_, which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. The _‘authentication errors when logging into SFTP’_ issue was resolved on December 5th, 2023 at 6:29 AM PST, returning the platform to full functionality. In this incident, our SFTP servers became unstable and failed to process requests for certain customers due to a bad configuration file that was applied on 12-04-2023 at 10:41 PM to our SFTP servers via our automated configuration management system. This failure only affected a small number of customers. Specifically, it only affected customers where our API was required to authenticate the provenance of the origin IP of the connecting SFTP user. This includes customers who use IP Whitelisting or IP Geolocation \(such as country whitelist/blacklisting\). We use a sophisticated system to cryptographically authenticate the origin IP of the connectiing SFTP user when making upstream calls to our internal API, and it was a configuration related to this system that was inadvertently misapplied. The reason for the bad configuration file being deployed is as follows: A separate configuration change was correctly and successfully made to another system \(our HTTP servers\) via our configuration management systems. Due to a logic error in the code of the change, the change also inadvertently targeted our SFTP systems as well. This change should not have been deployed to our SFTP systems, but was inadvertently deployed to them anyway. Internally, [Files.com](http://Files.com) runs SFTP services on several dedicated servers in each service region. Our configuration management system deploys changes to servers one at a time, checking to ensure correct operation prior to continuing forward with the rollout of configuration changes. The contents of this document are for general release and classified PUBLIC Unfortunately, while this check did validate proper operation of SFTP in general, it did not specifically validate proper operation of the subsystem that provides for cryptographic authentication of IP addresses. Upon discovery of the incident, [Files.com](http://Files.com) reverted the inappropriate configuration change on the SFTP servers. The root cause of this incident is twofold. Firstly, [Files.com](http://Files.com) failed to automatically monitor and validate the correct operation of the subsystem that provides for cryptographic authentication of IP addresses on SFTP servers. While a downtime of this system doesn’t cause a full downtime of SFTP, it causes a functional equivalent of that if customers require IP Whitelisting or IP Geolocation. Secondly, [Files.com](http://Files.com) failed to provide feedback to the engineers who developed and deployed the original configuration change targeted at the HTTP servers to let them know that the change would also be applied to SFTP servers. [Files.com](http://Files.com) will be developing two major improvements to its processes as a result of this incident. First, [Files.com](http://Files.com) will implement additional detection and monitoring around the subsystem that provides for cryptographic authentication of IP addresses on SFTP servers. Second, [Files.com](http://Files.com) will develop a system to provide feedback to its infrastructure engineers about exactly which servers will be affected by a configuration change before that change will be approved. Both of these improvements will require substantial engineering work and are not completed yet. We look forward to completing them in the coming quarter. We are hugely disappointed by the downtime, and we will work hard to implement the additional layers of protection needed to avoid similar incidents in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: Major | Started At: Dec. 5, 2023, 12:51 p.m.
Description: On December 5th, 2023, at 4:06 AM PST, [Files.com](http://Files.com) correlated multiple customer tickets indicating _‘authentication errors when logging into SFTP’_, which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. The _‘authentication errors when logging into SFTP’_ issue was resolved on December 5th, 2023 at 6:29 AM PST, returning the platform to full functionality. In this incident, our SFTP servers became unstable and failed to process requests for certain customers due to a bad configuration file that was applied on 12-04-2023 at 10:41 PM to our SFTP servers via our automated configuration management system. This failure only affected a small number of customers. Specifically, it only affected customers where our API was required to authenticate the provenance of the origin IP of the connecting SFTP user. This includes customers who use IP Whitelisting or IP Geolocation \(such as country whitelist/blacklisting\). We use a sophisticated system to cryptographically authenticate the origin IP of the connectiing SFTP user when making upstream calls to our internal API, and it was a configuration related to this system that was inadvertently misapplied. The reason for the bad configuration file being deployed is as follows: A separate configuration change was correctly and successfully made to another system \(our HTTP servers\) via our configuration management systems. Due to a logic error in the code of the change, the change also inadvertently targeted our SFTP systems as well. This change should not have been deployed to our SFTP systems, but was inadvertently deployed to them anyway. Internally, [Files.com](http://Files.com) runs SFTP services on several dedicated servers in each service region. Our configuration management system deploys changes to servers one at a time, checking to ensure correct operation prior to continuing forward with the rollout of configuration changes. The contents of this document are for general release and classified PUBLIC Unfortunately, while this check did validate proper operation of SFTP in general, it did not specifically validate proper operation of the subsystem that provides for cryptographic authentication of IP addresses. Upon discovery of the incident, [Files.com](http://Files.com) reverted the inappropriate configuration change on the SFTP servers. The root cause of this incident is twofold. Firstly, [Files.com](http://Files.com) failed to automatically monitor and validate the correct operation of the subsystem that provides for cryptographic authentication of IP addresses on SFTP servers. While a downtime of this system doesn’t cause a full downtime of SFTP, it causes a functional equivalent of that if customers require IP Whitelisting or IP Geolocation. Secondly, [Files.com](http://Files.com) failed to provide feedback to the engineers who developed and deployed the original configuration change targeted at the HTTP servers to let them know that the change would also be applied to SFTP servers. [Files.com](http://Files.com) will be developing two major improvements to its processes as a result of this incident. First, [Files.com](http://Files.com) will implement additional detection and monitoring around the subsystem that provides for cryptographic authentication of IP addresses on SFTP servers. Second, [Files.com](http://Files.com) will develop a system to provide feedback to its infrastructure engineers about exactly which servers will be affected by a configuration change before that change will be approved. Both of these improvements will require substantial engineering work and are not completed yet. We look forward to completing them in the coming quarter. We are hugely disappointed by the downtime, and we will work hard to implement the additional layers of protection needed to avoid similar incidents in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: Major | Started At: Dec. 5, 2023, 12:51 p.m.
Description: We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning around 5:00 am PST and ending by 6:45 am PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.
Status: Resolved
Impact: Major | Started At: Nov. 21, 2023, 1:55 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
