Time: May 15, 2023, 7:49 p.m.

Status: Postmortem

Update: On April 21st, 2023, at  10:26 AM PST, [Files.com](http://Files.com) received customer reports of elevated errors rates with wildcard searches on FTP, which resulted in an incident being declared.  The Incident Management Team \(IMT\) convened and immediately began investigation.  The elevated error rates with wildcard searches on FTP was resolved on April 21st, 2023, at 10:36 AM PST, returning the platform to full functionality.  [Files.com](http://Files.com) released a resolution Status Page posting on April 21st, 2023, at  11:00 AM PST,  stating:    _“**FTP Service Only: Elevated Error Rates On Wildcard Searches:** FTP only: We identified elevated error rates on the FTP service on_ [_Files.com_](http://Files.com) _in all regions. Attempts to list files using \* \(wildcard\) searches were not returning results. This issue affected wildcard search results from 11:00 a.m. PDT on 4/20 until 10:36 a.m. PDT on 4/21.  All services have been restored and are operating normally.   This incident did not impact other network services such as API, SFTP, WebDAV, AS2, and others.”_  This incident began when a code change was deployed to the [Files.com](http://Files.com) API to make the operation of filtering operate consistently with the API documentation at [http://developers.files.com/](http://developers.files.com/).  However, it turns out that FPS-FTP, [Files.com](http://Files.com)’s FTP server, was relying on undocumented API behavior in order to implement filtering, and therefore broke when the API change was deployed.  The incident was resolved by updating FPS-FTP's use of the API to follow the documented API interface.  The root cause of this issue was the [Files.com](http://Files.com) FTP team’s failure to rely on only documented API behavior.  As an additional note, although the [Files.com](http://Files.com) FTP interface uses the [Files.com](http://Files.com) API internally, it does not use a [Files.com](http://Files.com) SDK as a wrapped around that API connection.   We recommend to our customers to always use a [Files.com](http://Files.com) SDK because it provides protection against incorrect API usage, and if we used it ourselves for FTP, it may have prevented this incident.  We are tracking a long-term project to have FPS always use a [Files.com](http://Files.com) SDK rather than implementing API methods directly.  We hope to have this project accomplished this year.  Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future.    We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: April 21, 2023, 5:47 p.m.

Status: Resolved

Update: FTP only: We identified elevated error rates on the FTP service on Files.com in all regions. Attempts to list files using * (wildcard) searches were not returning results. This issue affected wildcard search results from 11:00 a.m. PDT on 4/20 until 10:36 a.m. PDT on 4/21. All services have been restored and are operating normally. This incident did not impact other network services such as API, SFTP, WebDAV, AS2, and others. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.

Time: April 20, 2023, 11:25 a.m.

Status: Resolved

Update: We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 3:58 AM PST and ending by 4:22 AM PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.

Time: April 20, 2023, 11:25 a.m.

Status: Resolved

Update: We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 3:58 AM PST and ending by 4:22 AM PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.

Time: May 15, 2023, 5:40 p.m.

Status: Postmortem

Update: On April 19th, 2023, at 6:52 AM PST, [Files.com](http://Files.com) received automated alerting of delays to batch and scheduled operations, which resulted in an incident being declared.  The Incident Management Team \(IMT\) convened and immediately began investigation.   [Files.com](http://Files.com) released an initial Status Page posting on April 19th, 2023, at 7:22 AM PST, stating:  _“**Delays To Batch and Scheduled Operations: ** We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_   The delays to batch and scheduled operations were resolved on April 19th, 2023, at 7:51 AM PST, returning the platform to full functionality.  [Files.com](http://Files.com) released a resolution Status Page posting on April 19th, 2023, at 7:52 AM PST, stating:  _“We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  Operations were delayed beginning at 6:35 AM PST and ending by 7:51 AM PST.  All operations did successfully complete despite delays.  This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_   [Files.com](http://Files.com) regularly reboots machines during the early AM hours to install security updates.  This is performed on a routine basis and typically occurs without incident.    This incident started when a Search cluster used for searching and sorting files by name on [Files.com](http://Files.com) failed to come back up after a routine reboot.    After investigation, it was determined that the failure was caused by a configuration change that was meant to target a different Search cluster at [Files.com](http://Files.com) but was inadvertently installed on this cluster.  The change was actually made over a month ago, but due to the nature of the configuration change, it did not cause any issues until the reboot.   After reverting the inadvertent configuration change, the Search cluster came back online and service was restored.    As a result of “back pressure” caused by very high error rates on search, other jobs on [Files.com](http://Files.com)’s background job scheduling system unrelated to Search were also impacted with delays lasting about an hour.    The root cause of the broader backup \(i.e. jobs other than Search\) was a failure of [Files.com](http://Files.com)’s internal job scheduling system to probably route around the failed search indexing jobs and prevent their failure from causing broader impact.   Part of this was caused by a design failure internal job scheduling system, which we have now redesigned to avoid this type of issue. \(See next paragraph.\).  Further contributing to the problem was a software design flaw where the [Files.com](http://Files.com) search code used a 1 second delay prior to retrying a failed query to the Search cluster.  1 second is an eternity on a service that normally serves thousands requests per second. This has also been fixed.   As a result of this incident and several other recent incidents, [Files.com](http://Files.com) worked on dramatic improvements to its internal job scheduling code during the last week of April and first week of May, and those improvements have been tested in staging and are now in production.   These improvements provide multiple new protection mechanisms to prevent issues with specific customers, job types, or regions from “backflowing” and impacting other customers, job types, or regions.   Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: April 19, 2023, 2:53 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: April 19, 2023, 2:52 p.m.

Status: Investigating

Update: We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 6:35 AM PST and ending by 7:51 AM PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.

Time: April 19, 2023, 2:22 p.m.

Status: Investigating

Update: We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.

Time: May 15, 2023, 5:40 p.m.

Status: Postmortem

Update: On April 19th, 2023, at 6:52 AM PST, [Files.com](http://Files.com) received automated alerting of delays to batch and scheduled operations, which resulted in an incident being declared.  The Incident Management Team \(IMT\) convened and immediately began investigation.   [Files.com](http://Files.com) released an initial Status Page posting on April 19th, 2023, at 7:22 AM PST, stating:  _“**Delays To Batch and Scheduled Operations: ** We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_   The delays to batch and scheduled operations were resolved on April 19th, 2023, at 7:51 AM PST, returning the platform to full functionality.  [Files.com](http://Files.com) released a resolution Status Page posting on April 19th, 2023, at 7:52 AM PST, stating:  _“We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.  Operations were delayed beginning at 6:35 AM PST and ending by 7:51 AM PST.  All operations did successfully complete despite delays.  This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where_ [_Files.com_](http://Files.com) _acts as a server.”_   [Files.com](http://Files.com) regularly reboots machines during the early AM hours to install security updates.  This is performed on a routine basis and typically occurs without incident.    This incident started when a Search cluster used for searching and sorting files by name on [Files.com](http://Files.com) failed to come back up after a routine reboot.    After investigation, it was determined that the failure was caused by a configuration change that was meant to target a different Search cluster at [Files.com](http://Files.com) but was inadvertently installed on this cluster.  The change was actually made over a month ago, but due to the nature of the configuration change, it did not cause any issues until the reboot.   After reverting the inadvertent configuration change, the Search cluster came back online and service was restored.    As a result of “back pressure” caused by very high error rates on search, other jobs on [Files.com](http://Files.com)’s background job scheduling system unrelated to Search were also impacted with delays lasting about an hour.    The root cause of the broader backup \(i.e. jobs other than Search\) was a failure of [Files.com](http://Files.com)’s internal job scheduling system to probably route around the failed search indexing jobs and prevent their failure from causing broader impact.   Part of this was caused by a design failure internal job scheduling system, which we have now redesigned to avoid this type of issue. \(See next paragraph.\).  Further contributing to the problem was a software design flaw where the [Files.com](http://Files.com) search code used a 1 second delay prior to retrying a failed query to the Search cluster.  1 second is an eternity on a service that normally serves thousands requests per second. This has also been fixed.   As a result of this incident and several other recent incidents, [Files.com](http://Files.com) worked on dramatic improvements to its internal job scheduling code during the last week of April and first week of May, and those improvements have been tested in staging and are now in production.   These improvements provide multiple new protection mechanisms to prevent issues with specific customers, job types, or regions from “backflowing” and impacting other customers, job types, or regions.   Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: April 19, 2023, 2:53 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: April 19, 2023, 2:52 p.m.

Status: Investigating

Update: We have resolved a situation causing delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. Operations were delayed beginning at 6:35 AM PST and ending by 7:51 AM PST. All operations did successfully complete despite delays. This situation was only a delay of scheduled and batch processing and did not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.

Time: April 19, 2023, 2:22 p.m.

Status: Investigating

Update: We are investigating reports of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations. This situation should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server.