Time: April 7, 2023, 10:07 p.m.

Status: Postmortem

Update: On March 23rd, 2023 at 3:24 PM PST, [Files.com](http://Files.com) received automated alerting of delays to critical file transfer and batch operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations, which resulted in an incident being declared. The Incident Management Team \(IMT\) convened and immediately began investigation. [Files.com](http://Files.com) released an initial Status Page posting on March 23rd, 2023 at 4:00 PM PST, stating “We are currently investigating reports of delays to critical file transfer and batch operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.” The Delays to Critical File Transfer and Batch Operations was resolved on March 23rd, 2023 at 4:04 PM PST, returning the platform to full functionality. [Files.com](http://Files.com) released a resolution Status Page posting on March 23rd, 2023 at 4:07 PM PST, stating “From 3:24 PM to 4:04 PM PST, we experienced delays processing critical file transfer and batch operations. Operations such as scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations were considerably delayed during this time period. As of 4:04 PM PST, all batch operations are performing normally.” This incident began when our infrastructure team deployed a routine software update to Apache Arrow \(a dependency used by our systems as part of logging pipelines\) to our production environment. Like all updates, this update was first tested on our staging environment. When the update was deployed to our production environment, a library incompatibility prevented our application from restarting on certain servers in our production environment. Upon investigation, we discovered that certain servers in our staging environment were running an OS version that was different from their counterparts in the production environment. Our team quickly resolved the issue in production by manually replacing the problematic library. The root cause of this issue is [Files.com](http://Files.com)'s failure to ensure that servers in the staging environment exactly match their counterparts in the production environment. [Files.com](http://Files.com) has already corrected the staging environment servers to exactly match their production environment equivalents and has added new monitoring tasks to also ensure that OS versions match between staging and production. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: March 23, 2023, 11:07 p.m.

Status: Resolved

Update: From 3:24 PM to 4:04 PM PST, we experienced delays processing critical file transfer and batch operations. Operations such as scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations were considerably delayed during this time period. As of 4:04 PM PST, all batch operations are performing normally.

Time: March 23, 2023, 11 p.m.

Status: Investigating

Update: We are currently investigating reports of delays to critical file transfer and batch operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations.

Time: March 23, 2023, 3:59 a.m.

Status: Resolved

Update: On March 22, 2023 from 7:28-7:55 PM PST and then from 8:28-8:39 PM PST a small number of users incorrectly received error messages telling them that their country was not authorized to sign into their account. This was due to a software error relating to our IP geolocation databases. The issue was quickly recognized by automated testing and resolved. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: March 23, 2023, 3:59 a.m.

Status: Resolved

Update: On March 22, 2023 from 7:28-7:55 PM PST and then from 8:28-8:39 PM PST a small number of users incorrectly received error messages telling them that their country was not authorized to sign into their account. This was due to a software error relating to our IP geolocation databases. The issue was quickly recognized by automated testing and resolved. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: April 7, 2023, 10:05 p.m.

Status: Postmortem

Update: On March 13th, 2023, at 2:18 PM PST, [Files.com](http://Files.com) received internal alerting of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations, which should not affect real-time operations such as FTP, SFTP, AS2, and other operations where [Files.com](http://Files.com) acts as a server. An incident was called due to the potential impact on customers. The Incident Management Team \(IMT\) immediately convened and started troubleshooting. The cause was narrowed down and the system was returned to normal functionality on March 13th, 2023 at 3:41 PM PST. This incident began when our customer support team initiated a file restoration request in response to a request from a customer who had inadvertently deleted some of their own files. That customer had been recently migrated from ExaVault, and as a result their file data lacked the required checksums to be able to process a restoration, so our systems automatically kicked off a job to recalculate checksums for all of the impacted files. We use a proprietary technology called files-rpc to ensure that background job processing capacity is fairly allocated between customers and no customer can overwhelm the system to the point that other customers are affected. However, the job that processes checksum calculation is on an older version of files-rpc and didn't have all of the protections implemented. This is the first time a checksum job has resulted in delays to other customers. The root cause is [Files.com](http://Files.com)'s failure to utilize its internal stack for multi-tenant job management in a consistent manner across all job types \(specifically the checksum job\). [Files.com](http://Files.com) has made immediate changes to increase the efficiencies of the checksum job and will be implementing the latest version of files- rpc in all job types in the coming weeks. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: April 7, 2023, 10:05 p.m.

Status: Postmortem

Update: On March 13th, 2023, at 2:18 PM PST, [Files.com](http://Files.com) received internal alerting of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations, which should not affect real-time operations such as FTP, SFTP, AS2, and other operations where [Files.com](http://Files.com) acts as a server. An incident was called due to the potential impact on customers. The Incident Management Team \(IMT\) immediately convened and started troubleshooting. The cause was narrowed down and the system was returned to normal functionality on March 13th, 2023 at 3:41 PM PST. This incident began when our customer support team initiated a file restoration request in response to a request from a customer who had inadvertently deleted some of their own files. That customer had been recently migrated from ExaVault, and as a result their file data lacked the required checksums to be able to process a restoration, so our systems automatically kicked off a job to recalculate checksums for all of the impacted files. We use a proprietary technology called files-rpc to ensure that background job processing capacity is fairly allocated between customers and no customer can overwhelm the system to the point that other customers are affected. However, the job that processes checksum calculation is on an older version of files-rpc and didn't have all of the protections implemented. This is the first time a checksum job has resulted in delays to other customers. The root cause is [Files.com](http://Files.com)'s failure to utilize its internal stack for multi-tenant job management in a consistent manner across all job types \(specifically the checksum job\). [Files.com](http://Files.com) has made immediate changes to increase the efficiencies of the checksum job and will be implementing the latest version of files- rpc in all job types in the coming weeks. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: March 24, 2023, 1:29 p.m.

Status: Resolved

Update: On March 13th, 2023, at 2:18 PM PST, Files.com received internal alerting of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations, which should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server. An incident was called due to the potential impact on customers. The Incident Management Team (IMT) immediately convened and started troubleshooting. The cause was narrowed down and the system was returned to normal functionality on March 13th, 2023 at 3:41 PM PST. This incident began when our customer support team initiated a file restoration request in response to a request from a customer who had inadvertently deleted some of their own files. That customer had been recently migrated from ExaVault, and as a result their file data lacked the required checksums to be able to process a restoration, so our systems automatically kicked off a job to recalculate checksums for all of the impacted files. We use a proprietary technology called files-rpc to ensure that background job processing capacity is fairly allocated between customers and no customer can overwhelm the system to the point that other customers are affected. However, the job that processes checksum calculation is on an older version of files-rpc and didn't have all of the protections implemented. This is the first time a checksum job has resulted in delays to other customers. The root cause is Files.com's failure to utilize its internal stack for multi-tenant job management in a consistent manner across all job types (specifically the checksum job). Files.com has made immediate changes to increase the efficiencies of the checksum job and will be implementing the latest version of files- rpc in all job types in the coming weeks. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: March 24, 2023, 1:29 p.m.

Status: Resolved

Update: On March 13th, 2023, at 2:18 PM PST, Files.com received internal alerting of delays to batch and scheduled operations, including but not limited to scheduled and ad-hoc syncs, moves, copies, file transformations, automations, webhooks, batch deletes, email delivery, previews, and similar batch operations, which should not affect real-time operations such as FTP, SFTP, AS2, and other operations where Files.com acts as a server. An incident was called due to the potential impact on customers. The Incident Management Team (IMT) immediately convened and started troubleshooting. The cause was narrowed down and the system was returned to normal functionality on March 13th, 2023 at 3:41 PM PST. This incident began when our customer support team initiated a file restoration request in response to a request from a customer who had inadvertently deleted some of their own files. That customer had been recently migrated from ExaVault, and as a result their file data lacked the required checksums to be able to process a restoration, so our systems automatically kicked off a job to recalculate checksums for all of the impacted files. We use a proprietary technology called files-rpc to ensure that background job processing capacity is fairly allocated between customers and no customer can overwhelm the system to the point that other customers are affected. However, the job that processes checksum calculation is on an older version of files-rpc and didn't have all of the protections implemented. This is the first time a checksum job has resulted in delays to other customers. The root cause is Files.com's failure to utilize its internal stack for multi-tenant job management in a consistent manner across all job types (specifically the checksum job). Files.com has made immediate changes to increase the efficiencies of the checksum job and will be implementing the latest version of files- rpc in all job types in the coming weeks. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: March 8, 2023, 9:23 p.m.

Status: Postmortem

Update: On March 6th, 2023, [Files.com](http://Files.com) received customer reports of Box integration syncs and mounts not working. An incident was called on March 6th, 2023, at 9:27 AM PST, due to the impact on customers. The Incident Management Team \(IMT\) immediately gathered and began investigation. A software bug deployed on March 1st, 2023, at 10:58 AM PST was the issue. A fix was created, tested, and deployed which returned full functionality for Box integrations on March 6th, 2023, at 9:57 AM PST. The root cause was identified as the inability to perform automated testing on Box integrations due to constraints placed on the integration by Box. Additional automated alerting was created and deployed on March 6th, 2023, at 10:28 AM PST to identify this issue should it occur again in the future. Extensive review and testing was conducted by [Files.com](http://Files.com) staff to ensure this resolution, and we have already taken steps internally to prevent this issue from recurring in the future. We greatly appreciate your patience and understanding as we resolved this issue. If you need additional assistance or continue to experience issues, please contact our Customer Support team.

Time: March 6, 2023, 6:03 p.m.

Status: Resolved

Update: All services have been restored and are operating normally. As of 9:57 AM Pacific Time, a fix has been implemented and we will continue to monitor the results. We will follow up with an Incident Report within ten (10) business days including the root cause and steps taken to address the root cause. If you need additional support, please do not hesitate to contact our Customer Support team by email or phone. Thanks for your support while we resolved this issue.

Time: March 6, 2023, 5:44 p.m.

Status: Investigating

Update: We are actively investigating failures on Box Remote Server Syncs and Mounts. We will provide additional details as they become available. Customers with urgent questions are encouraged to contact our Customer Support team by email. Thank you for your patience.