Time: Oct. 9, 2019, 10:33 p.m.

Status: Postmortem

Update: There were several issues that were addressed during the emergency downtime that were all contributing factors to the performance issue: * Indexes on several heavily used tables were bloated/corrupted leaving the primary database instance starved for processing resources. This was corrected along with adding additional capacity and alerting. * Various instance and network level errors indicated potential issues with the hardware hosting the virtual database instance. The system was force migrated to new hardware and processes cleared and restarted ensuring proper function and replication. Additionally, failover processes and triggers have been reviewed and updated to help avoid single node disruption of the wider system. * Several background and asynchronous tasks were better tuned and balanced to avoid resource over utilization.

Time: Oct. 3, 2019, 3:36 p.m.

Status: Resolved

Update: This incident has been resolved. The database performance has continued to stay at normal levels as a result of the back-end changes. Post mortem to be posted by Thursday, October 10th

Time: Oct. 2, 2019, 7:47 p.m.

Status: Monitoring

Update: We have continued monitoring, and the implemented back-end changes have resulted in database performance staying at normal levels. We will continue monitoring this evening and will provide another update by 10 AM MT/12 PM ET tomorrow.

Time: Oct. 1, 2019, 8:59 p.m.

Status: Monitoring

Update: The backlog of requests has returned to normal levels. We are continuing to monitor and will provide updates as further information is found.

Time: Oct. 1, 2019, 6:26 p.m.

Status: Monitoring

Update: At 8:30AM MT this morning there was a significant increase in sync requests to our systems. This backlog of requests is trending downward and it is expected that some calendars will be out of sync at this time while the systems catch up. Next Update by: 3PM MT/5PM ET

Time: Oct. 1, 2019, 3:38 p.m.

Status: Monitoring

Update: A code release was pushed last night to address the small subset of O365/Exchange calendars, as well as improve calendar syncing overall. We are monitoring the results and will update by 4PM MT/6PM ET

Time: Sept. 30, 2019, 8:24 p.m.

Status: Monitoring

Update: We are investigating reports of a subset of calendars not syncing for O365, Exchange, and Google calendars that appear to be related to this incident. Updates will be provided as they become available.

Time: Sept. 27, 2019, 10:52 p.m.

Status: Monitoring

Update: All systems are operational. Teem will leave this incident open and continue to monitor systems closely throughout the weekend to verify all global clients are fully functional.

Time: Sept. 27, 2019, 3:49 p.m.

Status: Monitoring

Update: All systems are operational and Teem will continue to closely monitor the platform throughout the day. During the maintenance window overnight the team shifted hosted hardware, restarted and performed maintenance on the primary database, including updates, reindexing, general clean up and reinitialization dependent services. Teem will be adding additional capacity to the system throughout the day. In addition we will continue to monitor our systems. As of this update, all systems are currently operational.

Time: Sept. 26, 2019, 10:54 p.m.

Status: Investigating

Update: We are continuing to investigate this issue as services have operated with degraded performance throughout the day. While a root cause has not yet been confirmed we have identified issues relating to our database cluster and will be scheduling an emergency maintenance window later this evening to address the issue.

Time: Sept. 26, 2019, 3:54 p.m.

Status: Investigating

Update: Teem is currently investigating a wide spread performance issue across the site.

Time: Oct. 9, 2019, 3:38 p.m.

Status: Postmortem

Update: Our investigation found two distinct issues: The first has to do with push notifications received from Office 365 that contain instructions that Teem should renew our subscription. These unique push notifications would trigger Teem to start a new subscription to changes on the calendar folder. However, it was found that Office 365 would repeatedly send these push notifications and that in certain cases this could create a race condition where the subscription saved to Office 365 was different than that saved in Teem. To remediate this, we have implemented a scheduling system for calendar jobs that lets Teem react after a short time to the first of these unique push notifications, and ignore any subsequent notifications. Monitoring has shown that this change is effective in renewing our calendar subscription and removing these race conditions. The second issue had to do with normal errors that occur when working with Office 365. When Teem received these errors, after trying multiple times over the course of a few minutes, we would stop attempting to sync the calendar. While most calendars naturally recovered because of a subsequent calendar change, a subset of calendars would not have a later change, and thus stay out of sync. Our investigation showed that some of these errors were temporary. Using the scheduling system described above, we now isolate temporary errors and schedule a retry for a later time. This helps ensure that we can sync in the events as expected.

Time: Oct. 3, 2019, 3:40 p.m.

Status: Resolved

Update: This incident has been resolved. We have continued monitoring the calendar-related update, and systems are working as expected. Post mortem to be posted by Thursday, October 10th.

Time: Oct. 2, 2019, 7:45 p.m.

Status: Monitoring

Update: We have continued monitoring the calendar-related update released September 19th, and O365 calendars continue working as expected, except due to the non-calendar specific database event reported last Friday. We will continue monitoring this evening and will provide another update by 10 AM MT/12 PM ET tomorrow.

Time: Sept. 28, 2019, 10:11 p.m.

Status: Monitoring

Update: We are continuing to monitor the process and will release update for any additional issues.

Time: Sept. 25, 2019, 7:47 p.m.

Status: Monitoring

Update: Teem is continuing to monitor the process and released update for any additional issues.

Time: Sept. 20, 2019, 3:47 p.m.

Status: Monitoring

Update: An update has been deployed to production. We are monitoring to see if this update resolves the calendar sync issues.

Time: Sept. 17, 2019, 3:35 p.m.

Status: Identified

Update: Engineering has identified the root cause, and are implementing a plan for resolution. We will provide another update as soon as the changes have been made.

Time: Sept. 16, 2019, 5:13 p.m.

Status: Investigating

Update: Engineering believes they have identified the root cause and they are currently verifying. We will provide another update as soon as the root cause is confirmed.

Time: Sept. 13, 2019, 9:03 p.m.

Status: Investigating

Update: Teem is continuing to investigate the root cause of this issue. Additional logging has been implemented to assist with this effort.

Time: Sept. 12, 2019, 12:25 a.m.

Status: Investigating

Update: Teem is continuing to investigate the root cause of the issue. In the interim: 1. Any O365 calendar found to be out of sync will automatically be re-synced every 8 hours 2. If you encounter a calendar which is out of sync, you can follow these instructions to manually initiate a re-sync: https://www.teem.com/support/eventboard-room-scheduling/re-syncing-calendars?from_search=36544921

Time: Sept. 11, 2019, 6:02 p.m.

Status: Investigating

Update: Teem is investigating a reported calendar sync issue affecting multiple O365 customers. Affected calendars may fail to sync with O365 until a manual re-sync is performed.

Time: Sept. 20, 2019, 9:33 p.m.

Status: Resolved

Update: Teem has worked with Google to identify the root cause behind the issue with Android For Work deployment. EventBoard deployments utilizing Android Enterprise (previously Android for Work) have been restored. Users are now able to provision new devices by entering afw#teem in the account field during device setup, or by downloading the app through the Google Play Store (https://play.google.com/store/apps/details?id=enderlabs.eventboardandroid.full)

Time: Sept. 11, 2019, 6:25 p.m.

Status: Identified

Update: Teem has identified a solution that we are testing. This solution will allow you to deploy EventBoard to your Android devices without Android Enterprise (Android for Work). As we finish our testing, we will update the status and our documentation on the new way of deploying EventBoard on Android.

Time: Sept. 6, 2019, 5:04 p.m.

Status: Investigating

Update: EventBoard deployments utilizing Android Enterprise (previously Android For Work) are currently unavailable. This only affects provisioning on Android devices. Updates to Android devices and new deployments on iOS and Crestron are not affected. Teem is currently working to provide short and long-term solutions for Android deployment.

Time: Aug. 29, 2019, 4:47 p.m.

Status: Resolved

Update: The fix has been deployed and verified and the issue has been successfully resolved. NOTE: Finder/Flightboard list mode is a long running process and for the changes to take effect clients will need to manually reload/refresh their instances or the system will reload automatically at 12pm device local time

Time: Aug. 29, 2019, 4:16 p.m.

Status: Monitoring

Update: A fix has been implemented and is currently being deployed. Once verified we will full close the incident. NOTE: Finder list mode is a long running process and for the changes to take effect clients will need to manually reload/refresh or the system will reload automatically at 12pm device local time

Time: Aug. 29, 2019, 3:49 p.m.

Status: Identified

Update: Teem has identified an issue in a recent release preventing Finder/Flightboard from displaying properly in list view. The problem has been diagnosed, and a fix will be deployed shortly.

Time: July 2, 2019, 3:32 p.m.

Status: Resolved

Update: Cloudflare has resolved their outstanding issue and we have verified all systems are operational.

Time: July 2, 2019, 2:57 p.m.

Status: Monitoring

Update: Cloudflare has provided additional context regarding this issue and they are monitoring their systems to verify this problem has been resolved. According to Cloudflare, their service has returned to baseline performance levels - so we expect our service response times to stabilize. Another update will be provided prior to 10 AM.

Time: July 2, 2019, 2:22 p.m.

Status: Identified

Update: Cloudflare has implemented a fix and they are monitoring to see if the problem is resolved.

Time: July 2, 2019, 2:16 p.m.

Status: Identified

Update: Our web proxy and CDN provider Cloudflare is currently having issues that are affecting our services. We will update when we have more information or these issues have been resolved.