Is there an Files.com outage?
Files.com status: Systems Active
Last checked: a minute ago
Get notified about any outages, downtime or incidents for Files.com and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: a minute ago
Get notified about any outages, downtime or incidents for Files.com and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Files.com.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Australia Region | Active |
| Background Jobs, including Sync and Webhooks | Active |
| Canada Region | Active |
| Core Services / API | Active |
| EU (Germany) Region | Active |
| Files Tools | Active |
| FTP/FTPS | Active |
| Japan Region | Active |
| Remote Server Integrations (Sync and Mount) | Active |
| SFTP | Active |
| Singapore Region | Active |
| UK Region | Active |
| USA Region | Active |
| WebDAV | Active |
| Web Interface | Active |
View the latest incidents for Files.com and check for official updates:
Description: On April 22nd, 2023, at 10:30 AM PST, [Files.com](http://Files.com) received automated alerts of delays on batch and scheduled operations in the Canada region which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. [Files.com](http://Files.com) released an initial Status Page posting on April 22nd, 2023, at 11:04 AM stating: _“**Canada Region Only: Delays and Errors Related to Certain Background Processing**: Canada only: We are investigating elevated error rates related to certain background processing performed as part of the core_ [_Files.com_](http://Files.com) _file transfer pipeline in the Canada region. Impacted functions of_ [_Files.com_](http://Files.com) _include file transformations such as zip/unzip, GPG, preview generation, and file statistics calculation such as MD5. This situation should not impact customers at all unless they have files stored in the Canada region. This situation should not affect real-time operations such as the_ [_Files.com_](http://Files.com) _API, FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_ [Files.com](http://Files.com) released an updated Status Page posting on April 22nd, 2023, at 11:31 AM PST stating: _“We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_ The delays on batch and scheduled operations in the Canada region was resolved on April 22nd, 2023, at 11:32 AM PST, returning the platform to full functionality. [Files.com](http://Files.com) released a resolution Status Page posting on April 22nd, 2023, at 11:38 AM PST stating: _“We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 10:30 AM PST and ending by 11:32 AM PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_ This incident was triggered by a customer-initiated regional migration of a fairly large amount of data from our USA region to our Canada region. We typically process hundreds or thousands of such migrations daily without incident. During this process our Canada region worker servers became overloaded. As a result, all regional background jobs in Canada began failing for customers using our Canada region. At the time the incident actually occurred, we incorrectly identified the root cause, but were able to resolve the issue anyway by resubmitting the failed jobs. On May 2nd, another similar incident occurred also in Canada where we discovered the true root cause of this incident. In that investigation, we determined the overload to be caused by a design flaw in our checksum calculation code which failed to properly use all available CPU cores on the machine, and instead only attempted to use a single CPU core. Basically, the machine locked up because dozens of jobs were attempting to use the same core, rather than spreading out to all available cores. As part of that incident’s resolution, [Files.com](http://Files.com) pushed an update to introduce more parallelism to this calculation and allowed all available CPU cores to be used. Additionally, one CPU core is now reserved for communication with our job scheduling system, which will prevent the communication problems in high load situations in the future. As a result of “back pressure” caused by very high error rates, other jobs on [Files.com](http://Files.com)’s background job scheduling system outside of the Canada region were also impacted with delays. The root cause of the broader \(non-Canada\) was a failure of [Files.com](http://Files.com)’s internal job scheduling system to probably route around the failed Canada workers and prevent their failure from causing broader impact. Ultimately this was caused by a design failure internal job scheduling system, which we have now redesigned to avoid this type of issue. \(See next paragraph.\) As a result of this incident and several other recent incidents, [Files.com](http://Files.com) worked on dramatic improvements to its internal job scheduling code during the last week of April and first week of May, and those improvements have been tested in staging and are now in production. These improvements provide multiple new protection mechanisms to prevent issues with specific customers, job types, or regions from “backflowing” and impacting other customers, job types, or regions. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: Major | Started At: April 22, 2023, 6:04 p.m.
Description: On April 22nd, 2023, at 10:30 AM PST, [Files.com](http://Files.com) received automated alerts of delays on batch and scheduled operations in the Canada region which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. [Files.com](http://Files.com) released an initial Status Page posting on April 22nd, 2023, at 11:04 AM stating: _“**Canada Region Only: Delays and Errors Related to Certain Background Processing**: Canada only: We are investigating elevated error rates related to certain background processing performed as part of the core_ [_Files.com_](http://Files.com) _file transfer pipeline in the Canada region. Impacted functions of_ [_Files.com_](http://Files.com) _include file transformations such as zip/unzip, GPG, preview generation, and file statistics calculation such as MD5. This situation should not impact customers at all unless they have files stored in the Canada region. This situation should not affect real-time operations such as the_ [_Files.com_](http://Files.com) _API, FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_ [Files.com](http://Files.com) released an updated Status Page posting on April 22nd, 2023, at 11:31 AM PST stating: _“We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_ The delays on batch and scheduled operations in the Canada region was resolved on April 22nd, 2023, at 11:32 AM PST, returning the platform to full functionality. [Files.com](http://Files.com) released a resolution Status Page posting on April 22nd, 2023, at 11:38 AM PST stating: _“We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 10:30 AM PST and ending by 11:32 AM PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_ This incident was triggered by a customer-initiated regional migration of a fairly large amount of data from our USA region to our Canada region. We typically process hundreds or thousands of such migrations daily without incident. During this process our Canada region worker servers became overloaded. As a result, all regional background jobs in Canada began failing for customers using our Canada region. At the time the incident actually occurred, we incorrectly identified the root cause, but were able to resolve the issue anyway by resubmitting the failed jobs. On May 2nd, another similar incident occurred also in Canada where we discovered the true root cause of this incident. In that investigation, we determined the overload to be caused by a design flaw in our checksum calculation code which failed to properly use all available CPU cores on the machine, and instead only attempted to use a single CPU core. Basically, the machine locked up because dozens of jobs were attempting to use the same core, rather than spreading out to all available cores. As part of that incident’s resolution, [Files.com](http://Files.com) pushed an update to introduce more parallelism to this calculation and allowed all available CPU cores to be used. Additionally, one CPU core is now reserved for communication with our job scheduling system, which will prevent the communication problems in high load situations in the future. As a result of “back pressure” caused by very high error rates, other jobs on [Files.com](http://Files.com)’s background job scheduling system outside of the Canada region were also impacted with delays. The root cause of the broader \(non-Canada\) was a failure of [Files.com](http://Files.com)’s internal job scheduling system to probably route around the failed Canada workers and prevent their failure from causing broader impact. Ultimately this was caused by a design failure internal job scheduling system, which we have now redesigned to avoid this type of issue. \(See next paragraph.\) As a result of this incident and several other recent incidents, [Files.com](http://Files.com) worked on dramatic improvements to its internal job scheduling code during the last week of April and first week of May, and those improvements have been tested in staging and are now in production. These improvements provide multiple new protection mechanisms to prevent issues with specific customers, job types, or regions from “backflowing” and impacting other customers, job types, or regions. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: Major | Started At: April 22, 2023, 6:04 p.m.
Description: On April 20th, 2023, at 11:57 AM PST, [Files.com](http://Files.com) received customer reports of 2FA login errors which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. [Files.com](http://Files.com) released an initial Status Page posting on April 20th, 2023, at 11:57 AM PST, stating: _“**Reports of Elevated Errors - 2FA and SSO Logins:** We are investigating reports of errors on the_ [_Files.com_](http://Files.com) _service. This is affecting user logins on the web portal that use two-factor authentication or single-sign on.”_ The 2FA login errors were resolved on April, 20th, 2023, at 11:58 AM PST, returning the platform to full functionality. [Files.com](http://Files.com) released a resolution Status Page posting on April 20th, 2023, at 12:08 PM PST stating: _“All services have been restored and are operating normally. We have resolved an issue that was causing web logins to fail when using single sign-on or two-factor authentication. This incident occurred between the times of 11:28 to 11:58 a.m.”_ This incident began when our frontend team deployed a code change designed to fix a bug in the login process in the [Files.com](http://Files.com) web interface. Unfortunately, the code change actually made things worse and prevented 2FA or SSO logins from working at all. This change was rolled back as soon as the incident was identified. The root cause of this incident was [Files.com](http://Files.com)’s failure to properly identify a change to the login page as a high-risk change and implement a sufficient level of automated and manual testing prior to deployment. [Files.com](http://Files.com) will be conducting remedial training on change risk identification at its upcoming Engineering meeting. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: Major | Started At: April 20, 2023, 6:57 p.m.
Description: On April 20th, 2023, at 11:57 AM PST, [Files.com](http://Files.com) received customer reports of 2FA login errors which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. [Files.com](http://Files.com) released an initial Status Page posting on April 20th, 2023, at 11:57 AM PST, stating: _“**Reports of Elevated Errors - 2FA and SSO Logins:** We are investigating reports of errors on the_ [_Files.com_](http://Files.com) _service. This is affecting user logins on the web portal that use two-factor authentication or single-sign on.”_ The 2FA login errors were resolved on April, 20th, 2023, at 11:58 AM PST, returning the platform to full functionality. [Files.com](http://Files.com) released a resolution Status Page posting on April 20th, 2023, at 12:08 PM PST stating: _“All services have been restored and are operating normally. We have resolved an issue that was causing web logins to fail when using single sign-on or two-factor authentication. This incident occurred between the times of 11:28 to 11:58 a.m.”_ This incident began when our frontend team deployed a code change designed to fix a bug in the login process in the [Files.com](http://Files.com) web interface. Unfortunately, the code change actually made things worse and prevented 2FA or SSO logins from working at all. This change was rolled back as soon as the incident was identified. The root cause of this incident was [Files.com](http://Files.com)’s failure to properly identify a change to the login page as a high-risk change and implement a sufficient level of automated and manual testing prior to deployment. [Files.com](http://Files.com) will be conducting remedial training on change risk identification at its upcoming Engineering meeting. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: Major | Started At: April 20, 2023, 6:57 p.m.
Description: On April 21st, 2023, at 10:26 AM PST, [Files.com](http://Files.com) received customer reports of elevated errors rates with wildcard searches on FTP, which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. The elevated error rates with wildcard searches on FTP was resolved on April 21st, 2023, at 10:36 AM PST, returning the platform to full functionality. [Files.com](http://Files.com) released a resolution Status Page posting on April 21st, 2023, at 11:00 AM PST, stating: _“**FTP Service Only: Elevated Error Rates On Wildcard Searches:** FTP only: We identified elevated error rates on the FTP service on_ [_Files.com_](http://Files.com) _in all regions. Attempts to list files using \* \(wildcard\) searches were not returning results. This issue affected wildcard search results from 11:00 a.m. PDT on 4/20 until 10:36 a.m. PDT on 4/21. All services have been restored and are operating normally. This incident did not impact other network services such as API, SFTP, WebDAV, AS2, and others.”_ This incident began when a code change was deployed to the [Files.com](http://Files.com) API to make the operation of filtering operate consistently with the API documentation at [http://developers.files.com/](http://developers.files.com/). However, it turns out that FPS-FTP, [Files.com](http://Files.com)’s FTP server, was relying on undocumented API behavior in order to implement filtering, and therefore broke when the API change was deployed. The incident was resolved by updating FPS-FTP's use of the API to follow the documented API interface. The root cause of this issue was the [Files.com](http://Files.com) FTP team’s failure to rely on only documented API behavior. As an additional note, although the [Files.com](http://Files.com) FTP interface uses the [Files.com](http://Files.com) API internally, it does not use a [Files.com](http://Files.com) SDK as a wrapped around that API connection. We recommend to our customers to always use a [Files.com](http://Files.com) SDK because it provides protection against incorrect API usage, and if we used it ourselves for FTP, it may have prevented this incident. We are tracking a long-term project to have FPS always use a [Files.com](http://Files.com) SDK rather than implementing API methods directly. We hope to have this project accomplished this year. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.
Status: Postmortem
Impact: None | Started At: April 20, 2023, 6 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
