Time: May 23, 2023, 6:41 p.m.

Status: Postmortem

Update: **Teem Detailed Root Cause Analysis – Severity 2 – May 9, 2023** **Unable to Access Teem via Outlook Plugin**   **Description:** On May 9, 2023, multiple customers began reporting the inability to access Teem through the Outlook Plugin. When trying to access Teem through the plugin, customers were presented with the following error, “"Unfortunately, we were unable to associate your external account with Teem. Contact your administrator for help."   **Type of Event:** Service Disruption   **Services\\Modules Impacted:** Production   **Timeline:** On May 9, 2023, at 9am, MST an initial report was submitted to customer support about the inability to access Teem through the Outlook Plugin. Around the 11 o’clock hour, additional customers began to report the same issue and all customers were notified that we are currently investigating the issue via Status Page. Around 4:01pm MST, the issue was identified and implemented by our engineering team and our Status Page was updated to a Monitoring Phase. Customers began to confirm the resolution and there were no additional reports to Teem Support. Our Status Page was updated from Monitoring to Resolved at 8:45pm, MST.   **Total Duration of Event:** 11hrs 45mins   **Root Cause Analysis:** The team identified an internal process that involved certificate updates.   **Preventative Action:** Internal teams are actively researching preventative analysis. However, if this issue happens in the short term, another internal process has been put in place to quickly resolve this issue.

Time: May 10, 2023, 2:45 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: May 9, 2023, 10:01 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: May 9, 2023, 10:01 p.m.

Status: Monitoring

Update: The issue with Users unable to sign into Teem via Outlook Plugin has been identified and a fix has been implemented. We are moving into a Monitoring phase for the next couple of hours. We will provide another update at 6pm MST. Thank you for your continued patience as we worked to resolved this issue.

Time: May 9, 2023, 9:46 p.m.

Status: Investigating

Update: Our engineering team continues to investigate this issue and are working on finding the root cause. We appreciate your patience.

Time: May 9, 2023, 7:33 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: May 9, 2023, 4:58 p.m.

Status: Investigating

Update: We are currently investigating an issue where users aren't able to access Teem through the Outlook Plugin. We are asking that you please use app.teem.com to access the application for now. We will provide another update at 1pm MST.

Time: March 10, 2023, 3:08 a.m.

Status: Resolved

Update: We apologize for the inconvenience, this is not an incident. If you're experiencing an issue with calendars, please reach out to our support team at [email protected]. We appreciate your patience.

Time: March 9, 2023, 10:29 p.m.

Status: Investigating

Update: We continue an ongoing investigation from our Service Disruption yesterday, March 9, 2023. Users that are trying to sync their calendars are seeing the following error intermittently. "The request failed. The remote server returned an error: (401) Unauthorized."

Time: March 9, 2023, 12:33 a.m.

Status: Resolved

Update: After monitoring, our support team has received confirmation from multiple customers, after verifying their Source Systems are no longer locked, forcing a re-sync on the calendars are working as designed with out error. If you have additional questions, please reach out to our support team @ [email protected]. A detailed RCA will be followed within 10 business days. We appreciate your patience during this service disruption.

Time: March 8, 2023, 8:20 p.m.

Status: Monitoring

Update: In efforts to resolve this issue, our team found the root cause to be related to a hotfix that was released on the evening of Monday, March 6, 2023. The hot fix was released to resolve an auto refresh issue that involved calendar credentials, thus causing 401 unauthorized errors. An internal decision was made to roll back the hot fix as there were continual retries and caused source systems (O365) to lock accounts. Some accounts are being unlocked based on source system timing and others may need to be unlocked manually. We are continuing to monitor this issue. Please let our support team know if you continue to experience these errors after you have unlocked your source system accounts.

Time: March 8, 2023, 5:58 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: March 8, 2023, 5:16 p.m.

Status: Identified

Update: Our engineering team has identified the issue and will be implementing a fix shortly. We appreciate your patience through this disruption.

Time: March 8, 2023, 4:39 p.m.

Status: Investigating

Update: We are currently investigating an issue with O365 Calendar Errors - Customers are receiving emails about their calendar sync being disabled. While on the Teem platform errors also appear on the resource calendar such as "The parameter count cannot be less than zero errors" or "Too many authorization attempts" and "401 Unauthorized".

Time: Feb. 27, 2023, 2:13 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 27, 2023, 1:37 p.m.

Status: Investigating

Update: We are currently investigating an issue with Teem. We will update you when we have more information.

Time: Oct. 24, 2022, 9:17 p.m.

Status: Postmortem

Update: **Teem Detailed Root Cause Analysis | Severity 1 & 2 | October 6 – 20, 2022** **Description:** _On October 6, 2022, at approximately 8:50am, customer support started to receive reports of issues throughout the Teem platform with calendars having syncing latency. When customers book events in Outlook or Google calendars, the event boards would not reflect the events for as 10 mins to 2 hours. This issue lasted intermittently till October 20, 2022._ **Type of Event:** _Service Disruption_   **Services\\Modules Impacted:** _Calendars_   **Remediation:** _Our engineering team has created new monitoring systems and have implemented new hardware to effectively identify these issues quickly going forward._ **Timeline:** _**Thursday, October 6, 2022**_ _8:50am MDT: Customers began reporting issues of calendars having some latency on syncing events to the event board and support escalates the issue to engineering._ _8:54am MDT: Our Engineering team begins investigation and are actively exploring calendar issues through 12pm MDT._ _1:31pm MDT: We’ve identified the issue to be with both Microsoft and Google_ _3:03pm MDT: Engineering has identified the issue and implemented a fix. Essentially, correcting the event queues by adding more resources to clear out the calendar requests quickly._ _6:05pm MDT: Event queues continue to process their way through._ _7:32pm MDT: Support received feedback from most of our customers saying that this issue has been resolved._ _7:35pm MDT: The status page was marked as resolved. However, the issue was downgraded to an S3 as support awaits more feedback from customers._ _**Friday, October 7, 2022**_ _9:27am MDT: Status page is updated with a note advising customers: On 10/06/2022, Teem customers using O365 & Google Calendars experienced disruption with Calendars not Syncing. Our engineering team has implemented a fix. You may experience a delay in resolution due to event jobs that continue to process due to the disruption. We will continue to monitor the resolution till the events have been cleared._ _10:37am MDT: Engineering confirms that the queue is still clearing. However, number of calendar events have not grown in the queue, which is a good sign. They continue to add more resources to help process all requests through the queue._ _**Wednesday, October 12, 2022**_ _2:52pm MDT: Customers begin to report that Microsoft and Google calendars are not syncing, and a new incident was reported on the status page making our customers aware._ _2:58pm MDT: Engineering acknowledged that they’re beginning to investigate and are exploring notification pushes for calendars._ _4:36pm MDT: Engineering has asserted logs and will review why some and not all calendars are updating._ _7:01pm MDT: Status page updated to as we continue to investigate the issue._ _11:00pm MDT: Engineering has a working theory on the issue and their team continues efforts to assert it is the root cause. Calendar watchers are being registered and an attempt to sync begins._ _11:05pm MDT: The status page is updated as we are still investigating the issue._ _3:01pm MDT: The status page is updated as we are still investigating the issue._ _**Thursday, October 13, 2022**_ _7:54am MDT: The status page is updated as we are still investigating the issue._ _12:21pm MDT: The engineering team has done initial testing on their working theory and the team begins to prep solution for bulk application._ _12:28pm MDT: The status page is updated as we are still investigating the issue._ _2:56pm MDT: The engineering team is positive that the issue has been identified and are running updates to verify solution._ _4:32pm MDT: The status page is updated from Investigation to Identified._ _6:28pm MDT: Engineering has implemented a fix and numbers for calendar requests begin to trend downwards. Support team begins to verify calendars with customers._ _7:29pm MDT: The status page is updated from Identified to Monitoring._ _8:40pm MDT: The status page is updated as we continue to monitor, and our support team continues to verify the solution with customers._ _**Friday, October 14, 2022**_ _6:16am MDT: Customers have verified that the issue has been resolved and the status page has been updated from Monitoring to Resolved._ _6:38am MDT: The status page has a note added to the incident - While this issue may appear to be resolved, support will continue monitoring all customers that have been affected by the Calendar Sync disruption. If you continue experiencing these issues. Please reach out to our support team at_ [[email protected]_](mailto:[email protected])_._ _**Monday, October 17, 2022**_ _3:41pm MDT: Customers being to report calendar syncing issues are intermittent. Some calendars are syncing and not all. The status page is updated, and all customers are notified that we are investigating this issue. Engineering also acknowledges the issue and begins investigating._ _4:07pm MDT: Support works with engineering to provide some examples of calendars from customers as investigation continues_ _7:05pm – 11:13pm MDT: The status page is updated, and customers are notified that new logs are being added to the system to help us better identify the issue. Investigation continues through the night and into the next morning._ _**Tuesday, October 18, 2022**_ _3:03am – 11:08am MDT: The team continues to investigate logs that were added to identify the issue and status page is updated as such._ _1:49pm MDT:  The engineering team is finding the queue for calendar requests are backed up and are working on distributing requests. The team is now adding more logs and are monitoring them to understand where the bottle neck is._ _8:37pm MDT: The status page is updated as our engineering team continues to monitor logs that have been implemented. We're also looking at current configurations to assist the latency in calendar events._ _11:00pm MDT: The engineering team has setup new servers for processing queues and updated queue processing configurations. The team begins to test, monitor, and fine tune configuration._ _**Wednesday, October 19, 2022**_ _12:30am MDT: The status page is updated as our engineering team continues to monitor logs that have been implemented. Queues have been restarted and have processed. The team has force synced 200 test calendars and they all processed within 2 minutes._ _9:05am MDT: The status page is updated as we are still investigating this issue._ _10:18am MDT: Additional monitoring has been put into place and queues begin processing as normal_ _1:46pm MDT: Support reaches out to customers to verify calendar issues and continue to monitor customer requests. The status page is also updated with information asking customers to reach out to support if they are still experiencing the issue._ _6:00pm MDT: The status page is updated to monitoring. Monitoring continues into the next day._ _**Thursday, October 20, 2022**_ _6:05pm MDT: Status page is updated from Monitoring to Resolved. After a day of monitoring, we are marking this issue as Resolved. Our team has implemented new logs to help us tell a better story for created events going forward. We've also reconfigured job managers which has been processing requests smoothly and quickly. We continue to implement new hardware to improve our services. Please reach out to our support team if you have any questions. We appreciate your patience as we work through this._  **Root Cause Analysis:** _Insufficient processing, logs, and monitoring systems for Calendar queues._ **Preventative Action:** _The engineers have reconfigured the queue processing system and sustained monitoring is suggestive of normal processing. The engineering team implemented a plan to reduce the chances of a recurrence of the queue processing problems. The plan includes provisioning of additional infrastructure to add a new layer of high availability. Adding this new high availability layer to the system will enable us to continue processing queue data even in the event of some parts of the system failing. It will also allow us to continue processing while provisioning additional server capacity and restarting queue processing subsystems if needed. The team has deployed this additional infrastructure and has begun testing and tuning the configuration. As part of the efforts to recover the system it was necessary to gain additional insights into our queue processing. There are new monitors available now that give insights into queue processing, we did not have previously. However, we still need more monitoring to confidently say the system is processing every message as expected and to understand our throughput patterns so we can watch/alert for anomalies that may indicate an active or imminent failure.  The engineering team is going to add additional monitoring and build tooling to enable Support to better diagnose message processing through the queues. This tooling will empower support to address more questions with confidence without having to involve engineering. The implementation of additional monitoring is in progress now and tooling for support will continue into the next sprint cycle for the team._

Time: Oct. 21, 2022, 12:05 a.m.

Status: Resolved

Update: After a day of monitoring we are marking this issue as Resolved. Our team has implemented new logs to help us tell a better story for created events going forward. We've also reconfigured job managers which has been processing requests smoothly and quickly. We continue to implement new hardware to improve our services. Please reach out to our support team if you have any questions. We appreciate your patience as we work through this.

Time: Oct. 20, 2022, midnight

Status: Monitoring

Update: Our team is continuing to monitor the issues at hand before we move into a resolved state. If you're still experiencing any issues please don't hesitate to reach out to our support team.

Time: Oct. 19, 2022, 8:19 p.m.

Status: Monitoring

Update: Since our team has implemented additional monitoring resources, we should start to see calendars beginning to sync and process normally. If you are still experiencing this issue please reach out to support ([email protected]) and provide details around the room(s) that you are trying to book. - Calendar name - Is your calendar event syncing? - If so, how long is it taking to sync? - Are your rooms syncing for some or all? Our monitoring continues and our next update will be provided 6:20pm MDT.

Time: Oct. 19, 2022, 3:05 p.m.

Status: Monitoring

Update: Our engineering team continues to monitor resources that have been put into place. Our next update will be provided at 1pm MDT.

Time: Oct. 19, 2022, 10:45 a.m.

Status: Monitoring

Update: Our engineering team has identified/implemented more resources such as logging, adding additional hardware and made some configuration updates. We continue to test and monitor the calendar queues through to resolution. We appreciate your continued patience as we work through this. Our next update will be posted at 8:30am MST.

Time: Oct. 19, 2022, 6:30 a.m.

Status: Investigating

Update: Our engineering team continues to monitor logs that have been implemented. We're also taking a look at current configurations to assist the latency in calendar events. We apologize for the inconvenience and appreciate your continued patience as we work through this. Our next update will be posted at 4:30am MST.

Time: Oct. 19, 2022, 2:37 a.m.

Status: Investigating

Update: Our engineering team continues to monitor logs that have been implemented. We're also taking a look at current configurations to assist the latency in calendar events. We apologize for the inconvenience and appreciate your continued patience as we work through this. Our next update will be posted at 12:30am MST.

Time: Oct. 18, 2022, 9:30 p.m.

Status: Investigating

Update: Our engineering team continues to investigate calendar queues and monitoring possible bottle necks that are causing this latency. We apologize for the inconvenience and appreciate your continued patience as we work through this. Our next update will be posted at 7:30pm MDT.

Time: Oct. 18, 2022, 5:08 p.m.

Status: Investigating

Update: Our team continues to monitor the latency in calendar events syncing. We appreciate your patience as we work through this. Our next update will be provided at 3:05pm MDT.

Time: Oct. 18, 2022, 1:16 p.m.

Status: Investigating

Update: Our team continues to monitor the latency in calendar events syncing. We appreciate your patience as we work through this. Our next update will be provided at 11:05am MDT.

Time: Oct. 18, 2022, 9:03 a.m.

Status: Investigating

Update: Our team continues to monitor the latency in calendar events syncing. We appreciate your patience as we work through this. Our next update will be provided at 7:05am MDT.

Time: Oct. 18, 2022, 5:15 a.m.

Status: Investigating

Update: Our engineering team continues to investigate and monitor event logs. Our next update will be provided at 3:05am MDT.

Time: Oct. 18, 2022, 1:05 a.m.

Status: Investigating

Update: Our engineering team continues to investigate this issue. We are adding logs to the system to help us better identify the issue. Our next update will be provided 11:05pm MDT.

Time: Oct. 17, 2022, 9:41 p.m.

Status: Investigating

Update: We are currently investing an issue with Teem Calendars, where events are not syncing and will provide an update for you at 7:41pm.