Time: Sept. 5, 2023, 5:11 p.m.

Status: Resolved

Update: We resolved this issue after identifying the cause. Initially, QUE went down due to a huge amount of read / write operations to the database. This overloaded both the API server and the database itself. We will be implementing updates in the future to mitigate operations where a huge number of read / write operations are invoked by end users. For now all systems are operating as expected. Please send us anything else you may notice to our normal support channels. Thank you for your patience with us and your continued business with Dataclay! Happy rendering!

Time: Sept. 5, 2023, 11:38 a.m.

Status: Investigating

Update: We recognize that Dataclay QUE service is down for our users. We are investigating now and will update status throughout the day here. We apologize for the inconvenience and thank you for your patience and continued business.

Time: Aug. 22, 2023, 6:45 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 22, 2023, 6:45 p.m.

Status: Monitoring

Update: We have not received any substantive updates from Google's support team since this issue occurred, but no further service interruptions have been noted. We were hoping there would be a more definitive answer and resolution, so that we could better mitigate the issue if it recurs. In the meantime, we are closing this incident.

Time: July 6, 2023, 12:54 a.m.

Status: Monitoring

Update: We are continuing to monitor the error rates for Bot + Google Sheets. In the last day or so, error rates have dropped dramatically, so your experience should be noticeably improved. We are still communicating with Google's support team to understand the source of the errors. While we have some more information, it's still not entirely clear why these errors started appearing regularly. We are reliant upon Google's engineering team to try to get better clarity. We will continue to update as more information becomes available. We are also working towards a new Templater release that will include the patched panel and fixes to mitigate these types of errors in the future.

Time: June 29, 2023, 5:16 p.m.

Status: Monitoring

Update: Google Sheets errors: we are continuing to monitor traffic and any new error reports from customers. So far, the Bot customers using the patched Templater panel appear to be maintaining connection to Sheets. This still doesn't resolve the underlying issue with Google's API infrastructure responding with 500 server errors, but those errors will abate and the patched Templater panel should be able to continue Bot operations. Google's support engineering team is investigating the cause, and we will continue to update as we have more information.

Time: June 23, 2023, 9:06 p.m.

Status: Identified

Update: Current status for the Bot + Google Sheets issue: we are making a patched Templater 3 panel available to affected customers. This does not resolve the 500 errors, but it allows Templater to refresh the authorization token once the errors clear. This is a short-term fix while we continue to work with Google to understand how to resolve the underlying 500 errors. We will likely be releasing patched versions of the full installers once we are certain that the errors are resolved. This fix should work for ALL versions of Templater: - Standard or floating - Windows or macOS - Mult-frame/multi-core (AE 2022 and newer) or Legacy (AE 2021 and earlier) The installation method for this patch is a bit more complex than the standard Templater installer since we wanted to get something out as quickly as possible. To install the patch, first download the archive from this link based on which type of license you have: Standard: https://assets.dataclay.com/templater-patches/Templater_3_14431-patch-release-nonfloat.zip Floating: https://assets.dataclay.com/templater-patches/Templater_3_14431-patch-release-float.zip Next, you'll want to navigate to the After Effects directory and go to `Support Files>Scripts>ScriptUI Panels`. Once there: - (Optional) Rename the `Templater 3.jsxbin` to `Templater 3.jsxbin.bak` - Copy the `Templater 3.jsxbin` from the ZIP archive directly to the `ScriptUI Panels` directory Once that's done, you should be able to launch Templater normally via the After Effects UI or CLI and enable Templater Bot. We will be keeping an eye on support channels over the weekend and into next week to ensure that the patch is successful. Subscribe to this incident to be alerted for any new updates. If you have any questions, please reach out to Dataclay support.

Time: June 23, 2023, 7:16 p.m.

Status: Investigating

Update: We are internally testing a patched Templater panel that we hope will work around the errors and prevent the Bot from disconnecting from Google Sheets. If we believe that this resolves the issue, we hope to make the patch available for users to download in the next few hours. We are simultaneously communicating with Google to try to understand why this occurring in the first place. We will continue to update this as we have more information.

Time: June 22, 2023, 4:04 p.m.

Status: Investigating

Update: We are continuing to work towards a resolution for the Google Sheets issue. At the moment, the temporary workarounds are still recommended for those who are experiencing 500 server errors and disconnections from the data source. While this appears to be an issue with Google's APIs, and not our software, we are exploring several ways to mitigate the effects. Please continue to follow updates through this channel and on our social media accounts.

Time: June 21, 2023, 7:46 p.m.

Status: Investigating

Update: We are investigating reports of an issue where Templater Bot users with Google Sheets data are encountering 500 server errors after the Bot has been running for a while. This issue only seems to affect Sheets + Bot. Temporary workarounds: - Close After Effects and reopen (error might recur at an undetermined interval) - Manually set row ranges for renders instead of "Enable Bot" - Export Sheets data to a local JSON file. With JSON, you can continue to use the Bot. Example installable Sheets exporter: https://workspace.google.com/marketplace/app/export_sheet_data/903838927001 Or, write a custom Google Apps Script: https://thenewstack.io/how-to-convert-google-spreadsheet-to-json-formatted-text/

Time: June 15, 2023, 7:51 p.m.

Status: Resolved

Update: We have rolled back the latest updates to further investigate the issues that cropped up after deploying to production. QUE is now operational, but will be updated in the near future with real time stats.

Time: June 15, 2023, 7:45 p.m.

Status: Investigating

Update: We have not yet identified the issue, but we have reverted to QUE built before the most recent status update. We will continue to update.

Time: June 15, 2023, 1:36 p.m.

Status: Investigating

Update: We are investigating degraded QUE performance due to recent deployment for real time stats.

Time: April 28, 2023, 1:29 a.m.

Status: Resolved

Update: We have deployed a fix to the QUE API. Filtering by a campaign for a job ID should no longer timeout. If you are continuing to experience any issues, please contact support.

Time: April 28, 2023, 12:54 a.m.

Status: Identified

Update: We have discovered the source of the issue and are going to be deploying a fix soon.

Time: April 28, 2023, 12:39 a.m.

Status: Investigating

Update: We are investigating an issue with the QUE API hanging when using the Job ID to filter for a specific job in a Campaign. Other filter properties and endpoints appear to be operational. This may affect users of our Zapier integration if they are using the "Filter Jobs (Advanced)" Zap action if they are filtering by Job ID, as well.

Time: Feb. 27, 2023, 8:41 p.m.

Status: Resolved

Update: QUE is now back up after restarting the dashboard and API environments. Thank you for your patience!

Time: Feb. 27, 2023, 8:20 p.m.

Status: Investigating

Update: The database servers that QUE runs on underwent maintenance and in doing so caused the QUE environment to go down. We are working towards restoring both the API and the QUE Dashboard. Thank you for your patience.