Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for Teem and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Teem.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
API | Active |
Authentication (SSO) | Active |
Datadog Events | Active |
EventBoard | Active |
Exchange Sync | Active |
Google Apps Calendar | Active |
LobbyConnect | Active |
Mandrill US East | Active |
Mandrill US West | Active |
Mobile Data | Active |
Phone System | Active |
Support Articles | Active |
Web Interface | Active |
View the latest incidents for Teem and check for official updates:
Description: After our fix we have confirmed that Calendar Sync functionality has returned to normal.
Status: Resolved
Impact: Major | Started At: April 22, 2021, 9:29 a.m.
Description: Earlier this month, we notified you of the unexpected technical challenges some customers experienced as a result of a major infrastructure upgrade for the Teem platform. We take performance and security very seriously, which is why we initiated the update to align with SOC 2 Type 2 data security standards. We appreciate your patience and understanding while we worked to resolve the temporary outages. Our first priority was to get you back up and running; our second is to give you some background on what happened. Here’s what we determined through our root cause analysis and how we’re preventing this moving forward. **Issue: O365 Sign-in Error 500** Cause: During a routine deployment, one of the third-party software libraries Teem SSO relies on was inadvertently upgraded to the latest major release of that library, which included noticeable changes. **Remediation**: · Reverted the library by specifying previous version with package management to avoid any unintentional upgrades **Issue: Post-Upgrade Device Connectivity** **Cause**: Teem’s core service had an interruption starting on Jan.9, 2021. During the interruption, when the EventBoard device made API calls to the service, the response status could be one of many error codes, including 401 Unauthorized. While this status code was in error, the device executed its designed security protocols and logged off. During a logout, EventBoard deletes all API tokens, downloaded themes, settings, and calendar data. It then reverts to a not-signed-in state and provides a 6-digit pin code for reactivation. Some customers were stuck on a “Authenticating with Teem …” screen. In these cases, after logging out EventBoard showed a message saying it could not communicate with Teem, and selecting “Retry” locked the app on that screen \(a secondary symptom of the core issue\). After logging out, the core service interruption would return an error instead of a pin code **Remediation**: · Deployed hotfix allowing devices to automatically activate and log in at pin code screen if they still exist in Teem database and are connected to only one Teem customer instance · Modified EventBoard app to increase fault tolerance on false 401s and to no longer get stuck at “Authenticating with Teem …” screen · Modified core service \(monolith\) so it doesn’t return 401s incorrectly **Issue:** [**App.Teem.com**](http://App.Teem.com) **Platform Outage** **Cause**: When deploying code on Jan. 14, 2021, an errant pip upgrade caused servers to not receive the deploying code and services to be stopped, interrupting all aspects of Teem. **Solutions**: · Updated deploy script to pin pip version · Cleared salt cache and confirmed correct deploy script on servers · Changed canary process to better detect downed servers · Ongoing: Updating underlying framework and all packages We apologize for any inconvenience and are continually working toward providing a more reliable experience for you. For additional information or to report an issue, please reach out to your Account Manager or visit [help.teem.com](http://help.teem.com) to contact our Customer Support team. Thank you
Status: Postmortem
Impact: Major | Started At: Jan. 15, 2021, 6:52 p.m.
Description: Earlier this month, we notified you of the unexpected technical challenges some customers experienced as a result of a major infrastructure upgrade for the Teem platform. We take performance and security very seriously, which is why we initiated the update to align with SOC 2 Type 2 data security standards. We appreciate your patience and understanding while we worked to resolve the temporary outages. Our first priority was to get you back up and running; our second is to give you some background on what happened. Here’s what we determined through our root cause analysis and how we’re preventing this moving forward. **Issue: O365 Sign-in Error 500** Cause: During a routine deployment, one of the third-party software libraries Teem SSO relies on was inadvertently upgraded to the latest major release of that library, which included noticeable changes. **Remediation**: · Reverted the library by specifying previous version with package management to avoid any unintentional upgrades **Issue: Post-Upgrade Device Connectivity** **Cause**: Teem’s core service had an interruption starting on Jan.9, 2021. During the interruption, when the EventBoard device made API calls to the service, the response status could be one of many error codes, including 401 Unauthorized. While this status code was in error, the device executed its designed security protocols and logged off. During a logout, EventBoard deletes all API tokens, downloaded themes, settings, and calendar data. It then reverts to a not-signed-in state and provides a 6-digit pin code for reactivation. Some customers were stuck on a “Authenticating with Teem …” screen. In these cases, after logging out EventBoard showed a message saying it could not communicate with Teem, and selecting “Retry” locked the app on that screen \(a secondary symptom of the core issue\). After logging out, the core service interruption would return an error instead of a pin code **Remediation**: · Deployed hotfix allowing devices to automatically activate and log in at pin code screen if they still exist in Teem database and are connected to only one Teem customer instance · Modified EventBoard app to increase fault tolerance on false 401s and to no longer get stuck at “Authenticating with Teem …” screen · Modified core service \(monolith\) so it doesn’t return 401s incorrectly **Issue:** [**App.Teem.com**](http://App.Teem.com) **Platform Outage** **Cause**: When deploying code on Jan. 14, 2021, an errant pip upgrade caused servers to not receive the deploying code and services to be stopped, interrupting all aspects of Teem. **Solutions**: · Updated deploy script to pin pip version · Cleared salt cache and confirmed correct deploy script on servers · Changed canary process to better detect downed servers · Ongoing: Updating underlying framework and all packages We apologize for any inconvenience and are continually working toward providing a more reliable experience for you. For additional information or to report an issue, please reach out to your Account Manager or visit [help.teem.com](http://help.teem.com) to contact our Customer Support team. Thank you
Status: Postmortem
Impact: Critical | Started At: Jan. 15, 2021, 2:43 a.m.
Description: Earlier this month, we notified you of the unexpected technical challenges some customers experienced as a result of a major infrastructure upgrade for the Teem platform. We take performance and security very seriously, which is why we initiated the update to align with SOC 2 Type 2 data security standards. We appreciate your patience and understanding while we worked to resolve the temporary outages. Our first priority was to get you back up and running; our second is to give you some background on what happened. Here’s what we determined through our root cause analysis and how we’re preventing this moving forward. **Issue: O365 Sign-in Error 500** Cause: During a routine deployment, one of the third-party software libraries Teem SSO relies on was inadvertently upgraded to the latest major release of that library, which included noticeable changes. **Remediation**: · Reverted the library by specifying previous version with package management to avoid any unintentional upgrades **Issue: Post-Upgrade Device Connectivity** **Cause**: Teem’s core service had an interruption starting on Jan.9, 2021. During the interruption, when the EventBoard device made API calls to the service, the response status could be one of many error codes, including 401 Unauthorized. While this status code was in error, the device executed its designed security protocols and logged off. During a logout, EventBoard deletes all API tokens, downloaded themes, settings, and calendar data. It then reverts to a not-signed-in state and provides a 6-digit pin code for reactivation. Some customers were stuck on a “Authenticating with Teem …” screen. In these cases, after logging out EventBoard showed a message saying it could not communicate with Teem, and selecting “Retry” locked the app on that screen \(a secondary symptom of the core issue\). After logging out, the core service interruption would return an error instead of a pin code **Remediation**: · Deployed hotfix allowing devices to automatically activate and log in at pin code screen if they still exist in Teem database and are connected to only one Teem customer instance · Modified EventBoard app to increase fault tolerance on false 401s and to no longer get stuck at “Authenticating with Teem …” screen · Modified core service \(monolith\) so it doesn’t return 401s incorrectly **Issue:** [**App.Teem.com**](http://App.Teem.com) **Platform Outage** **Cause**: When deploying code on Jan. 14, 2021, an errant pip upgrade caused servers to not receive the deploying code and services to be stopped, interrupting all aspects of Teem. **Solutions**: · Updated deploy script to pin pip version · Cleared salt cache and confirmed correct deploy script on servers · Changed canary process to better detect downed servers · Ongoing: Updating underlying framework and all packages We apologize for any inconvenience and are continually working toward providing a more reliable experience for you. For additional information or to report an issue, please reach out to your Account Manager or visit [help.teem.com](http://help.teem.com) to contact our Customer Support team. Thank you
Status: Postmortem
Impact: Minor | Started At: Jan. 12, 2021, 2:02 a.m.
Description: The issue was fixed and normal operations resumed at approximately 7:00pm MT on Sept. 8. Teem has been monitoring sync traffic since that time. After working with vendors to eliminate possible problems and bottlenecks, a subtle networking issue with the hosting platform itself was discovered. The issue left the subsystems that handle calendar synchronization for Office 365 and Exchange clients in a partially working state with most networking successfully transacting but many or our O365 clients heavily affected. Replacement VMs were created, tested and deployed to clear any underlying issues with the networking, hypervisor or hardware.
Status: Resolved
Impact: Major | Started At: Sept. 8, 2020, 1:39 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.