Time: May 30, 2023, 11:44 a.m.

Status: Postmortem

Update: Starting at 24-th of May at 10:27 CEST a subset of Templafy One custtomers started experiencing slow response times when accessing the application. The issue was caused by an unexpected situation in our offline component, that resulted in an excesive number of operations on our storage account, causing it to have a slower response. Upon identifying the issue, at 11:35 CEST, we promptly implemented short-term preemptive measures to stabilize the system and restore performance to acceptable levels. These measures, including immediate mitigation steps and intensified monitoring, proved effective in maintaining system stability and ensuring ongoing performance within our standards. In parallel, we have initiated a comprehensive review of our system to implement long-term changes aimed at enhancing efficiency and performance. Our team is diligently working on optimizing the codebase to identify and implement improvements that will ensure optimized performance in future scenarios.

Time: May 25, 2023, 9:39 a.m.

Status: Resolved

Update: The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

Time: May 24, 2023, 1:24 p.m.

Status: Monitoring

Update: The system is stable and operating within our performance standards. We will continue to closely monitor the situation to maintain stability.

Time: May 24, 2023, 9:35 a.m.

Status: Monitoring

Update: We have applied mitigation steps and performance is improving. We are continuing to monitor the system to ensure ongoing stability and performance. In parallel we are also continuing investigating the root cause so that we are able to prevent similar issues in the future. Details will be provided in the postmortem.

Time: May 24, 2023, 8:27 a.m.

Status: Identified

Update: We have identified an issue that affects a subset of customers. Because of the issue, customers are experiencing slow response times. Further updates will be posted here soon.

Time: May 31, 2023, 1:14 p.m.

Status: Postmortem

Update: Starting at 23-th of May at 11:00 CEST a subset of Templafy One custtomers started experiencing slow response times when accessing the application. The issue was caused by an unexpected situation in our offline component, which resulted in an excessive number of operations on our storage account, causing it to have a slower response. After identifying the issue we started to apply mitigation steps to stabilize the system and restore performance to acceptable levels. At 12:10 CEST we could already confirm that the system was working within the desired performance metrics. Following this the engineering team maintained continuous monitoring of the system to ensure that all operational metrics pertaining to performance stayed within optimal parameters.

Time: May 23, 2023, 11:40 a.m.

Status: Resolved

Update: The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

Time: May 23, 2023, 10:10 a.m.

Status: Monitoring

Update: The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

Time: May 23, 2023, 9 a.m.

Status: Investigating

Update: We are seeing a subset of customers are experiencing slow response times. The engineering team is working on identifying the issue.

Time: May 4, 2023, 12:30 p.m.

Status: Postmortem

Update: **Summary:** Early this week, we identified an issue with Templafy VSTO Add-ins in Microsoft Office for PC. A subset of users received a “Publisher cannot be verified” notification when starting Office desktop applications that had Templafy VSTO Add-ins installed. Our service was not down, and users who chose to install the Office Add-in from the unknown publisher could continue using our product via Office desktop. As an immediate response, we assembled a team from Engineering, Product, Support, and InfoSec to scope and address the issue. Templafy identified that the issue was only impacting customers using a particular subset of VSTO add-ins and versions. Subsequently, on Tuesday, Templafy implemented and then deployed a fix for the impacted customers, without requiring any further action from the clients, to ensure that the issue was resolved. **Issue overview:** On Sunday, 30th of April, we received support tickets indicating users were being prompted with a Microsoft Office Customization Installer notification, stating “Publisher cannot be verified.” Upon investigation, we determined that the issue was limited to Windows users who opened Microsoft Word, PowerPoint, Excel, or Outlook using Templafy One and Hive with the following VSTO Add-ins and their versions: * Library version 6.1.36 - 6.1.73 * Productivity version 5.0.701 - 7.1.205 * Email Signature version 6.1.36 - 6.1.65 Templafy Web Add-ins \(For MS Office on-prem, Online & Mac\) were not impacted. Customers on the latest stable version of Templafy Hive were also not impacted. **Root cause:** Upon thorough investigation, we determined that the root cause was that the signing certificate, used to sign the VSTO Add-ins, could not be verified by Microsoft Office anymore due to a change in the Time Stamp Authority requirements. The signing certificate we used was valid until the 28th of April. To ensure that Add-ins are valid after the certificate expires, we use a Time Stamp Authority \(TSA\) when signing add-ins. As long as the Time Stamp Authority is trusted, add-ins remain valid after the certificate expires. To enhance global security, on the 1st of June 2021, [the CA/B Forum](https://cabforum.org/wp-content/uploads/Baseline-Requirements-for-the-Issuance-and-Management-of-Code-Signing.v2.8.pdf#page=40) increased the required key length for Time Stamp Authority certificates to a minimum of 3072[.](https://bit.ly/3LAEoXW).) The Time Stamp Authority \(TSA\) we used when signing our add-ins was using a key length of 2048. Therefore, it was no longer trusted, resulting in the add-ins becoming invalid after the certificate expiration date. We had anticipated this problem and had switched to a new Time Stamp Authority \(TSA\) when signing executable files in March 2021, but we had not correctly switched to the new TSA when signing manifest files. Because of this, after the certificate expiration date on the 28th of April, users with impacted VSTO Add-in versions who opened an Office application received the "Publisher cannot be verified" notification. **Resolution:** To resolve the issue, we manually re-signed Library, Email Signature, and Productivity VSTO Add-ins using a new certificate with a valid Time Stamp Authority without changing any behavior of the software. We manually validated the re-signed VSTO Add-ins with both Templafy One and Hive customers. After confirmation, we re-signed all affected add-ins and programmatically updated Templafy Desktop packages for all affected Templafy One customers. Similarly, we programmatically updated all affected Hive customers to the re-signed add-ins. This process was completed by the end of Tuesday, 2nd of May. The packages with the re-signed VSTO Add-ins were automatically installed by Templafy Desktop on end users' PCs after midnight Tuesday \(local time\). Going forward, we will take extra precautions to ensure that our signed packages remain valid and adhere to the necessary key length standards in order to prevent similar problems in the future. **Timeline:** * Sunday, 30th of April: We identified the incident and received the first support tickets. * Monday, 1st of May: We began investigating and escalated to involve our Engineering, Product, Support, and InfoSec departments. We identified the scope, cause, and solution. At the end of Monday, we got confirmation from a few customers that the fix was working. * Tuesday, 2nd of May: We automatically re-signed all add-ins and updated all affected customers. * Wednesday, 3rd of May: Customers received the corrected VSTO Add-ins and the issue was officially resolved for all impacted customers. Lastly, we would like to thank you for your patience and understanding during this incident. We know that disruptions to your business operations are costly and frustrating, and as outlined above, we are committed to doing everything in our power to prevent similar incidents from occurring in the future. If you have any further questions or concerns, please do not hesitate to reach out to [[email protected]](mailto:[email protected]). We are always here to assist you and help you get the most out of our product. We value your business and appreciate your ongoing trust in our service.

Time: May 3, 2023, 4:24 p.m.

Status: Resolved

Update: The incident has been resolved, and further information will be provided in a postmortem. We apologize for the impact to affected customers.

Time: May 2, 2023, 9:22 p.m.

Status: Monitoring

Update: We have mitigated the incident for all affected customers. All affected customers have been updated to the re-signed versions of the add-ins. To deploy this update, the Templafy Desktop app will install the re-signed add-ins the next time it checks for updates, which will automatically happen after midnight local time. It can also be triggered manually by restarting the machine or through the Templafy Desktop application itself. No additional action will be needed from the admin or end-user. For any customers using server mode (e.g. Citrix), an update would have to be triggered to fetch the newest update. If the end-user has disabled the add-in in Office in the past 24 hours, they might need to manually activate it again. Read more here: https://support.templafy.com/hc/en-us/articles/115005420025 While this issue is resolved, we will not close this incident until we have confirmed updates have been deployed for all customers. Please expect a detailed post-mortem within the next 24 business hours. To report any further issues, please reach out to [email protected]. Thank you for your patience as we resolved this issue.

Time: May 2, 2023, 4:15 p.m.

Status: Identified

Update: We have updated the majority of affected customers on Templafy One to the re-signed versions of the add-ins. Templafy Desktop will automatically install the re-signed add-ins the next time it checks for updates. This will automatically happen after midnight (local time) or can be triggered manually by the end-user by restarting the machine or through the Templafy Desktop application. No additional action will be needed from either the admin or end-user side. In a few cases, we have not yet been able to update the add-ins. This is in two situations: If you are using Server Mode (e.g. Citrix) or have special characters in your existing package. If an end-user has disabled the add-in in Office in the past 24 hours, they might need to manually activate it again. Read more here: https://support.templafy.com/hc/en-us/articles/115005420025 With this update, we have mitigated the incident for the majority of the affected customers. We are continuing work to create an update that also applies to customers on Templafy Hive as well as customers using Server Mode add-ins. If you’re on Templafy One (and the two cases above don’t apply to you) and still experience any issues, please reach out to [email protected]

Time: May 2, 2023, 9:02 a.m.

Status: Identified

Update: We now have a full overview of affected versions and customers. If you are a Hive customer on an automatic release track (like most tenants), you are not affected. All affected versions of Library, Email Signature, ProductivityPlus, and Check have now been re-signed with the new certificate, and the re-signed add-in versions have been tested and confirmed working with multiple customers. For customers on Templafy Hive: New versions of add-ins have been uploaded to the admin center, and we are currently working on automatically updating all affected customers to the version containing the re-signed add-ins. The only change to the new add-in version will be the re-signing of add-in files. For customers on Templafy One: New versions of existing Desktop packages are being uploaded to the admin center. These new packages contain re-signed add-in files but are otherwise the same as the active desktop package. New versions of ProductivityPlus, Check, and Email Signature have been uploaded to the admin center and are available from the dropdowns with an x.1 version extension. Like the other add-ins, no other changes than re-signing have happened to these add-ins. Next Steps We are actively working on an automatic update of the add-in version in the admin center for both One and Hive customers. In this automated update, versions will be changed to the re-signed version of what is already active on the tenant. When this has been done, no additional action will be needed from either the admin or end-user side. We expect this to happen throughout today (European time). Templafy Desktop will automatically install the re-signed add-ins the next time it checks for updates. This will automatically happen after midnight (local time) or can be triggered manually by the end-user by restarting the machine or through the Templafy Desktop application. Customers using server mode (e.g. Citrix) For customers using server mode such as a Citrix environment, an update of Templafy Desktop will need to be triggered on the customer side after the changes above have been rolled out.

Time: May 1, 2023, 8:49 p.m.

Status: Identified

Update: Templafy has identified a solution for this issue and is in the process of coordinating automatic deployment to all customers to ensure service is returned without customer involvement. Details of the fix: - We have already re-signed all impacted Library add-ins and are in the process of re-signing all impacted e-mail signature and Productivity add-ins - This is being validated for both One and Hive customers Additional details for Hive customers: - All Hive customers on either the stable or insider tracks have not been impacted - After all impacted add-ins have been validated, we will deploy the fix for Hive customers immediately Additional details for One customers: - We are developing a method for programmatically updating all Templafy Desktop packages with re-signed add-ins - We will provide an update ASAP as to when this deployment can be expected

Time: May 1, 2023, 5:32 p.m.

Status: Identified

Update: The latest stable Templafy Hive add-in versions are not impacted by this issue. This issue impacts the following VSTO add-ins and their versions: Library add-in version 6.1.36 - 6.1.73 Productivity version 5.0.701 - 7.1.205 Email Signature version 6.1.36 - 6.1.65 Templafy Hive customers experiencing this issue can upgrade to the latest stable version in order to resolve it.

Time: May 1, 2023, 3:01 p.m.

Status: Identified

Update: Cause: In April 2021, Templafy updated the timestamp server that we used to sign our Templafy VSTO certificates due to the timestamp server being purchased by another company. The certificates are used by Templafy to ensure that our software has not been tampered with and can be installed safely and silently by end users. Due to this change in timestamp server, Templafy started using a timestamp server that was flagged invalid at a later stage, therefore signing VSTO packages with the invalid timestamp server. Templafy is still identifying all affected VSTO versions. Potential resolution: Templafy is in the process of signing all new VSTO packages with the correct timestamp server as well as working diligently to identify a programmatic / centralized solution to make sure all previous VSTO packages are signed with the correct timestamp server. Please continue to follow the status page for more updates as they become available. Thank you.

Time: May 1, 2023, 2:10 p.m.

Status: Identified

Update: The engineering team is continuing to work on the incident and we will provide more information once available.

Time: May 1, 2023, 1:12 p.m.

Status: Identified

Update: We are continuing to investigate the issue as well as scope the affected versions of Templafy add-ins. We will continue to provide status updates every hour.

Time: May 1, 2023, 11:58 a.m.

Status: Identified

Update: We have identified the cause of "Publisher cannot be verified" messages. The recent routine certificate changes performed by Templafy have exposed a configuration change that was made on our root certificate timestamp servers. This issue has been escalated to the highest degree and all technical teams are involved to resolve the issue as soon as possible.

Time: May 1, 2023, 10:03 a.m.

Status: Investigating

Update: We have identified an issue where customers receive a notification popup when starting an Office application where Templafy VSTO Add-ins are installed. This only affects Windows users. For more information about it and how to proceed please see our knowledge base article: https://support.templafy.com/hc/en-us/articles/10485903891485-Trusted-publisher-notification Currently, our engineering team is actively working on implementing a solution.

Time: April 21, 2023, 11:06 a.m.

Status: Postmortem

Update: On 19-th of April 2023 9:53 AM CEST, a software incident occurred during the rollout of a new certificate. The previous certificate was inadvertently removed from the Trusted Publisher store, causing a Certification prompt to be displayed for Add-Ins that still utilized the old certificate. To resolve the incident, the engineering team developed and rolled out a fix, which added both the old and new certificates to the Trusted Publisher store. The fix was deployed at 12:57 CEST. This prevented the Certification prompt from being displayed and restored normal operation. To avoid similar incidents in the future, the team is implementing several measures, such as improving the deployment process to ensure the proper configuration of Trusted Publisher store and conducting additional testing to catch potential issues before rollout. The incident caused inconvenience to users and required immediate action from the engineering team to restore normal operation. However, through a swift and effective response, the team was able to resolve the incident and put measures in place to prevent similar incidents in the future.

Time: April 19, 2023, 12:05 p.m.

Status: Resolved

Update: The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

Time: April 19, 2023, 10:57 a.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: April 19, 2023, 9:53 a.m.

Status: Identified

Update: We have identified an issue where customers receive a notification popup when starting an Office application where Templafy VSTO Add-ins are installed. This only affects Windows users. For more information about it and how to proceed please see our knowledge base article: https://support.templafy.com/hc/en-us/articles/10485903891485-Trusted-publisher-notification Currently, our engineering team is actively working on implementing a solution.

Time: April 12, 2023, 11:21 a.m.

Status: Postmortem

Update: Starting at 6th April 2023 11:50 AM CET, Templafy One customers started to get an exception when trying to upload slides and/or presentations. This behaviour was introduced by system misconfiguration. We noticed the issue at 11th April 8:23 PM CET from a customer report. The engineering team started working on this issue at 11th April 2023 9:00 PM CET and the issue was mitigated at 11th April 2023 10:15 PM CET.

Time: April 11, 2023, 8:15 p.m.

Status: Resolved

Update: The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.

Time: April 11, 2023, 7:52 p.m.

Status: Identified

Update: We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

Time: April 11, 2023, 6:37 p.m.

Status: Investigating

Update: We are currently investigating this issue.