Time: Oct. 15, 2019, 7:38 p.m.

Status: Resolved

Update: This incident has been resolved. All systems are healthy. If you have devices that are still on 2.10.5, please update to 2.10.6 as soon as possible. If you need any assistance, please open a Support ticket and we'll be happy to assist.

Time: Oct. 14, 2019, 10:25 p.m.

Status: Monitoring

Update: We are continuing to see improvements across all systems due to the movement off of iOS version 2.10.5 to 2.10.6. Response times from Teem's servers are now within a normal range. All devices (iOS, Android, and Crestron) using Eventboard are expected to be able to authenticate and/or activate. There may be periods of intermittent slowness, if that persists please open a Support ticket for further troubleshooting. We will continue to monitor and will update again tomorrow by 3 PM MT/5 PM ET.

Time: Oct. 14, 2019, 7:10 p.m.

Status: Monitoring

Update: With the release of iOS 2.10.6 we have seen the number of devices on 2.10.5 drop significantly, and it continues to do so. Due to the movement off of 2.10.5 to 2.10.6, we have seen a large drop in inbound reports of devices unable to authenticate and/or activate across all devices (iOS, Android, and Crestron), and we are currently following up on previous reports. Teem systems are trending back towards normal. There may still be intermittent slowness but it is anticipated that will continue to improve over the next few hours. Next update will be by 5 PM MT/7 PM ET

Time: Oct. 14, 2019, 4:18 p.m.

Status: Monitoring

Update: Eventboard iOS version 2.10.6 for Eventboard has been approved and has begun propagating, which can take several hours before it appears in the app store. If the iOS device has 'auto-update' enabled, the Eventboard app will update to 2.10.6 when it is available. Important to note: iOS Devices on Eventboard 2.10.5 should be updated to 2.10.6 as soon as it is available. For iOS devices on 2.10.4 or lower, updating to 2.10.6 is recommended for the additional features and fixes, but it isn't essential to fixing the connectivity/syncing issues. Additionally, Teem has implemented changes on the back end to block traffic from devices on iOS version 2.10.5 and to free up resources and improve performance on all other iOS versions and platforms (Android and Crestron). Next Update will be by 2PM MT/4PM ET

Time: Oct. 14, 2019, 1:59 p.m.

Status: Monitoring

Update: For all devices that are not on iOS 2.10.5, which includes all Crestron and Android devices, we are seeing many more devices that are now able to activate and authenticate. However, some devices are still appearing offline or having intermittent issues. For devices on iOS 2.10.5, they continue showing as offline. When 2.10.6 is released in the App Store and these devices update, they will begin coming back online. Next Update by 12PM MT/2PM ET

Time: Oct. 14, 2019, 7:59 a.m.

Status: Monitoring

Update: We have deployed back-end changes and are seeing the load on Teem’s servers decrease significantly. 

The expected behavior for all devices that are not on iOS 2.10.5, which includes all Crestron and Android devices, is that they can now activate and authenticate as normal. However, some intermittent issues may also be seen. For devices running iOS 2.10.5 the expected behavior is that there will be slowness in activating, and also some devices may continue reporting as offline. Until 2.10.6 is available in the App Store this behavior is expected to continue.

 Next Update: 8AM MT/10AM ET

Time: Oct. 14, 2019, 5:56 a.m.

Status: Identified

Update: iOS version 2.10.6 is not yet available in the app store. As such, we are continuing to work on a workaround that will rate limit traffic from iOS devices on 2.10.5. Next Update by 2AM MT/4 AM ET

Time: Oct. 14, 2019, 1:57 a.m.

Status: Identified

Update: Both Eventboard and LobbyConnect are continuing to have issues authenticating with Teem servers on all devices (Android, iOS, Crestron). Eventboard version 2.10.6 has yet to be released in the iOS app store. We are going to continue looking into back-end changes to our environment that will help until 2.10.6 is released. To re-state the issue, it is caused by Eventboard iOS version 2.10.5. However, all versions and devices are affected as it is causing increased load on Teem's servers. When 2.10.6 is released and devices on 2.10.5 begin upgrading to 2.10.6 the load on Teem's servers will decrease, and this will lead to normal load times. Next Update in 4 hours at Midnight MT/2 AM ET

Time: Oct. 13, 2019, 2:47 p.m.

Status: Identified

Update: Both Eventboard and LobbyConnect are continuing to have issues authenticating with Teem servers. The fix, which is included in version 2.10.6, is still pending release in the app store. We anticipate it will be available sometime today. When it is released, if your iOS device is set to auto-update apps then Eventboard will be updated automatically as soon as 2.10.6 is available in the app store. Next Update: Today by 8 PM MT/10 PM ET

Time: Oct. 13, 2019, 6:39 a.m.

Status: Identified

Update: Both Eventboard and LobbyConnect are continuing to have issues authenticating with Teem servers, this is due to the queuing up of requests. To be clear, this behavior will continue until Eventboard 2.10.6 on iOS is released in the app store and devices running 2.10.5 are updated with the new code. We are still awaiting confirmation that version 2.10.6 on iOS has been made available in the app store. Next Update: Tomorrow, Sunday, October 13th by 10 AM MT/12 PM ET

Time: Oct. 13, 2019, 4:37 a.m.

Status: Identified

Update: Eventboard 2.10.6 on iOS has been submitted to Apple and is pending approval. Next Update: Tomorrow, Sunday, October 13th by 10 AM MT/12 PM ET

Time: Oct. 13, 2019, 1:29 a.m.

Status: Identified

Update: We have identified the issue. Eventboard iOS version 2.10.5, the latest release, included a race condition which increased calls made to our servers. This additional load led to increased latency and device connectivity issues. We have developed a fix, and are in process of implementing this in iOS version 2.10.6. Only iOS devices on 2.10.5 will need to be updated. Note: Crestron and Android devices will not need to be updated Next Update: Tonight by 11:30 PM MT/1:30AM ET

Time: Oct. 13, 2019, midnight

Status: Investigating

Update: As part of the investigation, it was found app.teem.com was timing out, this has been addressed and app.teem.com is loading as normal. We are continuing to investigate the Eventboard & LobbyConnect device syncing issues. The next update will be by 8 PM MT/10 PM ET this evening.

Time: Oct. 12, 2019, 10:55 p.m.

Status: Investigating

Update: We have verified that calendar events are unaffected, and are syncing successfully. The current issue appears to be specific to devices. The next update will be by 8 PM MT/10 PM ET this evening.

Time: Oct. 12, 2019, 10:46 p.m.

Status: Investigating

Update: We are seeing connectivity issues with Eventboard and Lobbyconnect. We are currently investigating. Next update will be by 8 PM MT/10 PM ET this evening.

Time: Oct. 14, 2019, 9:04 p.m.

Status: Resolved

Update: This incident has been resolved. Booking future events in Eventboard successfully saves the event with no issues in 2.10.6. Note: this is unrelated to the ongoing Eventboard syncing issues.

Time: Oct. 11, 2019, 5:47 p.m.

Status: Monitoring

Update: We’ve deployed a fix that is now available in version 2.10.5 of Eventboard on iOS. After updating to 2.10.5 the Eventboard app will no longer close after saving an event. We will continue to monitor over the weekend. The next update will be Monday, October 14 by 3 pm MT/5 pm ET.

Time: Oct. 9, 2019, 3:25 p.m.

Status: Identified

Update: We’re currently investigating an issue on Eventboard: when booking a future meeting from the Eventboard device the moment the room is reserved the app will close and bring the user to the iPad home screen. The room reservation does save to the calendar and device. 

This behavior is only occurring on Eventboard v2.10.4. The ETA for release will likely be next week as it will require an iOS update. The next update will be Friday, October 11 by 3 pm MT/5 pm ET.

Time: Oct. 14, 2019, 9:06 p.m.

Status: Resolved

Update: This incident has been resolved. After updating to 2.10.6 the timer will not count up if a room is not checked into, it will instead release the room and show it as available. Note: this is unrelated to the ongoing Eventboard syncing issues.

Time: Oct. 11, 2019, 5:45 p.m.

Status: Monitoring

Update: We’ve deployed a fix that is now available in version 2.10.5 of Eventboard on iOS. After updating to 2.10.5 the timer will not count up if a room is not checked into, it will instead release the room and show it as available. We will continue to monitor over the weekend. The next update will be Monday, October 14 by 3 pm MT/5 pm ET.

Time: Oct. 10, 2019, 7:24 p.m.

Status: Identified

Update: We have found the cause and created a fix. After going through internal testing it will be submitted to Apple for review. The ETA for the iOS release with the fix is still sometime next week. The next update will be tomorrow, October 11 by 3 pm MT/5 pm ET.

Time: Oct. 9, 2019, 2:30 p.m.

Status: Identified

Update: We’re currently experiencing an issue on Eventboard where once an event check-in time expires, two things occur: 1. the device does not release the room 2. the counter continues counting up. This only affects iOS version 2.10.4. This does not affect Android. Our team is working to identify the root cause and implement a solution. The ETA for release will likely be next week as it will require an iOS update. The next update will be tomorrow, October 10 by 3 pm MT/5 pm ET.

Time: Oct. 9, 2019, 3:30 p.m.

Status: Resolved

Update: Adding rooms and desks continues to work normally. This issue will now be closed. We’re sorry for the inconvenience this caused.

Time: Oct. 8, 2019, 8:17 p.m.

Status: Monitoring

Update: A fix has been implemented, and rooms and desks are now able to be added normally. We will monitor through the night before closing the incident tomorrow. Next update by 11AM MT/1PM ET tomorrow.

Time: Oct. 8, 2019, 7:16 p.m.

Status: Investigating

Update: We’ve determined that adding a desk or room is not working. Do note that adding campuses, buildings and floors do work. We're working on fixing this now. We will provide another update again by 5 pm MT/7 pm ET.

Time: Sept. 26, 2019, 7:51 p.m.

Status: Postmortem

Update: The issue was caused by a memory limit that stopped our user sync jobs from creating additional cache objects in our system. This was difficult to pinpoint, as the reported errors in user sync were inconsistent and difficult to reproduce. Once we were able to reproduce and determine the root cause, we increased the limit of our cache which allowed the jobs to operate normally. We have added monitoring and alerting, which should prevent this issue from occurring in the future.

Time: Sept. 26, 2019, 7:51 p.m.

Status: Resolved

Update: Not all users were being synchronized to the Teem platform from external directories.