Time: March 25, 2024, 4:48 p.m.

Status: Postmortem

Update: **Teem by Eptura detailed Root Cause Analysis | 3/11/2024**  **S2 – O365 SSO Login Issues**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  **On Monday on Monday March 11th Internal Teem members and customers started experiencing issues with logging into the Teem application via O365. At around 10:15 PM a fire alarm is pulled by Jared Collins Teem Support Manager**    **Type of Event:**  Outage    **Services/Modules impacted:**  SSO/Logging    **Timeline:**` `**The timeline is posted in MST.**  We received issues of logging in at 5:25 PM but it was still logging perfectly fine after a few tries. We confirmed logging worked over the next few hours and kept a close eye on things. At 10:06 PM A customer reaches out that logging is having issues, internally we are now having issues as well. 10:08 PM a Jira is started to be created for our Engineering team to jump on the issue. 10:15 PM a Fire Alarm is pulled. 10:21 PM Status page is updated with the status of issue at hand. 10:23 PM Engineering confirms they are working on the issue. 11:54 PM Status page is updated once more for the evening. 8:00 AM status page is updated. 12:00 PM Status page is updated again that the issue is currently still at hand. 2:37 PM Engineering has found a fix and implemented said fix to environment. 3:05 PM customers confirming that the issue is resolved. 4 PM the status page is taken down.    **Total Duration of Event:**  16 Hours    **Root Cause:**   The cause of the issue at hand was an outdated token living on a server that is older. The token was re-established and the issue was then fixed.    **Remediation:**  We have put Fire Alarms in to notify the team of SSO expiration a month before it happens so we can update the token and ensure that SSO has no issues going forward.    **Preventative Action:**   Having a system in place to notify of token expiration.

Time: March 12, 2024, 10:07 p.m.

Status: Resolved

Update: The issue is now resolved. We appreciate your patience while we resolved this issue. We'll be posting the RCA by March 26th.

Time: March 12, 2024, 9:02 p.m.

Status: Monitoring

Update: Our Engineering team has found a fix and implemented it. We have had some confirmation the issue is resolved. We will be in monitoring for the next hour while awaiting further confirmations.

Time: March 12, 2024, 8:43 p.m.

Status: Identified

Update: The issue with O365 SSO has been identified and a fix is being implemented. We will post another update at 8:00pm CST or earlier when testing is complete.

Time: March 12, 2024, 6:04 p.m.

Status: Investigating

Update: Our Engineering team is continuing the investigating to determine the cause of the disruption. Next update will be posted at 5:00pm CST

Time: March 12, 2024, 2:01 p.m.

Status: Investigating

Update: We are continuing to investigate this issue we will update at 1:00 PM CST

Time: March 12, 2024, 6:26 a.m.

Status: Investigating

Update: We are continuing to investigate this issue we will update at 8:00 AM MST

Time: March 12, 2024, 4:21 a.m.

Status: Investigating

Update: We are currently having issues with what seems to be mainly O365 SSO. We are investigating this issue and will make an update when we have gathered more information.

Time: Feb. 23, 2024, 4:44 p.m.

Status: Postmortem

Update: **Teem by Eptura detailed Root Cause Analysis | 2/12/2024** **S1 – Inability to Access Teem**   We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.   **Description:** **On Monday February 12, 2024, around 7:02pm MST both internal and external customers experienced an inability to access Teem. At approximately 7:06pm MST, internal team members were alerted that Teem Login Page has failed a monitoring check. Internal teams immediately began to investigate the issue.**   **Type of Event:** Outage   **Services/Modules impacted:** All production.   **Timeline:** **The timeline is posted in MST.** **We received an alert at 7:02 PM. Our internal Engineering team was able to jump onto the issue. 7:16 PM. We continued investigating and jumped on a call with AWS engineering to get this issue resolved. 8:39 PM Our internal teams are still on a call with AWS engineering. 9:40 PM The call is continuing with AWS as our Engineering team and AWS engineering teams are working together to get database online. 3:02 AM Our Engineering team posted a status page as a Severity 1 letting our customers know the issues at hand 4:20 AM We are still investigating the issue. 5:59 AM Investigation continues 7:11 AM the issue has been resolved and we have moved Virtual Machines to allow hosting for our database. We then notified our customer base and put the status page into a monitoring state. 9:26 AM the issue has been confirmed resolved and the status page has now been updated to reflect.**     **Total Duration of Event:** 12 Hours   **Root Cause:** AWS hosting required an update to our Virtual Machine we are hosting our Database on. Our Engineering team wasn’t notified due to an email being sent to an old email address. We were notified of the outage right away due to failsafe's put in place. Investigation and attempts at resolution started immediately.   **Remediation:** We have moved to RDS for AWS and this should now no longer cause issues with downtime on server updates. We have also updated all emails and notification systems.   **Preventative Action:** Having the correct email in place as well as being on a hosting server allows for quick switchover without downtime.

Time: Feb. 13, 2024, 4:26 p.m.

Status: Resolved

Update: As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive post automatically.

Time: Feb. 13, 2024, 2:11 p.m.

Status: Monitoring

Update: A fix has been implemented. We are moving into the Monitoring Phase for the next 2 hours. 10:00am CST

Time: Feb. 13, 2024, 12:59 p.m.

Status: Investigating

Update: We are currently investigating an issue with TEEM. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 9 AM CST.

Time: Feb. 13, 2024, 11:20 a.m.

Status: Investigating

Update: We are currently investigating an issue with TEEM. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 7 AM CST.

Time: Feb. 13, 2024, 10:02 a.m.

Status: Investigating

Update: We are currently investigating an issue with Teem. We will update you when we have more information.

Time: Oct. 27, 2023, 7:50 p.m.

Status: Postmortem

Update: **Eptura Workplace Detailed Root Cause Analysis | Severity 2 | October 23, 2023**   **SSO Certificate Failed**  ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  On October 23, at approximately 9:30am EDT, support received multiple reports of calendars not syncing events due to an SSL error. ‌ **Type of Event:**  Performance Degradation    **Services\\Modules Impacted:**  Production: Calendars    **Timeline:** On October 23, at approximately 9:30am EDT, support received multiple reports of calendars not syncing events due to an SSL error. After a short investigation support was able to replicate this issue and all customers were notified of the disruption via Status Page at 9:33am EDT. 11:07am EDT internal teams have identified the issue and began working on the fix. At 11:33am the fix was implemented, and the status page was updated to a monitoring phase. As not additional reports were received, and customers confirmed the resolution the status page was updated to a resolved phase at 1:56pm.  ‌ **Total Duration of Event:**  2hrs 3mins    **Root Cause:**   Internal teams found a certificate that expired on the exchange adapter for Teem.    **Preventative Action:**    Internal teams continue to monitor all systems and have created internal processes to ensure alerts are set up to prevent this from happening in the future.

Time: Oct. 23, 2023, 5:56 p.m.

Status: Resolved

Update: As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.

Time: Oct. 23, 2023, 3:33 p.m.

Status: Monitoring

Update: A fix has been implemented. We are moving into the Monitoring Phase for the next 1 hour.

Time: Oct. 23, 2023, 3:07 p.m.

Status: Identified

Update: The issue with "Teem calendar error - SSL error "SSL: CERTIFICATE_VERIFY_FAILED]" has been identified and a fix is being implemented. We will post another update at 1:07pm MST or sooner.

Time: Oct. 23, 2023, 1:33 p.m.

Status: Investigating

Update: We are currently investigating a calendar issue with Teem. Some customers may see an error on their calendars - "SSL: CERTIFICATE_VERIFY_FAILED]"

Time: Sept. 12, 2023, 6:05 p.m.

Status: Postmortem

Update: **Teem Detailed Root Cause Analysis – Severity 2 – 08/29/2023**   **Description:** On August 29, 2023, at approximately 4:45am EDT, customer support started to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow.   **Type of Event:** Service Disruption   **Services\\Modules Impacted:** Production – Calendars   **Timeline:** On August 29, 2023, at approximately 4:45am EDT, customer support began to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow. At approximately 10:26am EDT, the engineering team identifies the issue and begins to implement the fix. At 10:36am EDT all customers are made aware that the fix has deployed to production and that we are moving into a monitoring phase. Customers begin to confirm the fix and at 11:01 EDT, the status page is marked as resolved. ‌ **Total Duration of Event:** **6hrs 16mins** ‌ **Root Cause:** The team continues to narrow down the root cause. However, the DevOps team identifies an internal configuration issue and begins to apply a manual fix.   **Preventative Action:** Our internal teams are working on enhancing the observability to bring awareness to the appropriate teams.

Time: Aug. 29, 2023, 3:01 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 29, 2023, 2:36 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Aug. 29, 2023, 2:26 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Aug. 29, 2023, 11:55 a.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Aug. 29, 2023, 8:45 a.m.

Status: Investigating

Update: We are currently investigating an issue with Teem. We will update you when we have more information.

Time: June 15, 2023, 4:52 p.m.

Status: Postmortem

Update: **Teem Detailed Root Cause Analysis – Severity 2 – June 5, 2023**  **Outlook and Google Events are not Syncing from Teem to Source Calendar**    **Description:**  _On June 5, 2023, at approximately 11:43am MST, customers support received additional reports of “Outlook and Google Calendar Events are not Syncing from Teem to Source Calendar”. An event created in Teem should sync back to the source calendar and vice versa._    **Type of Event:**  _Performance Degradation_    **Services\\Modules Impacted:**  _Production \(Reservation Module\)_     **Timeline:**` `  _On June 5, 2023, at approximately 11:43am MST, customer support received additional reports of “Outlook and Google Calendar events are not Syncing from Teem to Source Calendar”. The initial ticket was escalated to a S2 incident at 11:43am MST and posted to our status page alerting customers. Internal teams continue to investigate this issue through June 6, 2023. Our internal team identified the root cause of the issue at 9:10am MST and began work on the resolution. The status page was updated from investing to an identified phase. At 3:30pm MST a hotfix was implemented, and the status page was moved to a monitoring phase. At 4:27pm MST after the hotfix was implemented, internal teams acknowledged that calendar events are slowly being processed through the queue. The internal teams continue to monitor the queue through June 7, 2023. At 6:56am MST, all prior events created before the hotfix was implemented have been cleared. Internal teams continue to monitor new events that were created after the hotfix was released. As monitoring continued, customers reported that all calendar events are syncing at about 4:02pm MST. At 5:15pm MST, the incident was marked as resolved._     **Total Duration of Event:**  _2 days 5 hours 32 mins_    **Root Cause Analysis:**  _Engineering teams found this was related to a release pushed out May 19, 2023._     **Preventative Action:**   _While the engineering team fixed the regression bug. They found that there was an internal process during the release roll out that needed improvement with QA._

Time: June 15, 2023, 4:52 p.m.

Status: Resolved

Update: This incident was resolved, Wednesday June 7, 2023 (17:15 MDT). The RCA will follow shortly.

Time: June 7, 2023, 11:15 p.m.

Status: Monitoring

Update: As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.

Time: June 7, 2023, 7:24 p.m.

Status: Monitoring

Update: As we continue to monitor the calendar events queue, we are seeing newer events processing in a timely manner. If you are seeing a small delay in new events syncing to source calendar, using the "Calendar Refresh" functionality in Teem will assist for quicker processing of newer calendar events.

Time: June 7, 2023, 4:47 p.m.

Status: Monitoring

Update: Calendar events that were created before the fix was implemented have been successfully processed. As new calendar events are being created, our engineering and dev ops team continue to monitor the calendar events updated queue.

Time: June 7, 2023, 1:41 p.m.

Status: Monitoring

Update: Since the fix was implemented your external calendar events have been synced, we continue to monitor the calendar events update queue internally. You may experience a delay receiving new calendar events as the queue continues to process.

Time: June 6, 2023, 9:30 p.m.

Status: Monitoring

Update: A fix has been implemented. We are moving into the Monitoring Phase. If you reported a support ticket, our team will be reaching out shortly to confirm the resolution. Thank you for working with us during this disruption.

Time: June 6, 2023, 7:58 p.m.

Status: Identified

Update: We continue to work toward a resolution for the issue, S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar.

Time: June 6, 2023, 5:19 p.m.

Status: Identified

Update: We continue to work toward a resolution for the issue, S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar.

Time: June 6, 2023, 3:10 p.m.

Status: Identified

Update: The issue with S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar has been identified. The team is working on a fix and we will continue to keep you posted on the progress. Thank you and we appreciate your continued patience.

Time: June 6, 2023, 1:41 p.m.

Status: Investigating

Update: The team continues to work through this complex issue involving S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar. We are still in an investigating phase and will continue to provide updates throughout the day.

Time: June 6, 2023, 11:37 a.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 5, 2023, 11:22 p.m.

Status: Investigating

Update: The team continues to investigate this issue. We appreciate your patience as we work through this.

Time: June 5, 2023, 8:17 p.m.

Status: Investigating

Update: Our team continues to investigate "S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar". We appreciate your patience as we analyze this.

Time: June 5, 2023, 5:43 p.m.

Status: Investigating

Update: We are currently investigating this issue.