Time: June 14, 2024, 6:37 p.m.

Status: Resolved

Update: We appreciate your patience and are pleased to share that our team has identified the source of the Insight data disruption and is implementing a fix. We are responding directly to open tickets with details on the resolution for specific customer environments and will continue to monitor overall Insights performance. For further questions, please reach out to our dedicated support team. Thank you.

Time: June 13, 2024, 9:20 p.m.

Status: Investigating

Update: Our engineering team continues to investigate the inability pull data from Insights. We appreciate your patience as we work through this.

Time: June 13, 2024, 5:13 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 13, 2024, 12:55 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 13, 2024, 6:59 a.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 13, 2024, 2:05 a.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 12, 2024, 9:14 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: June 12, 2024, 5:11 p.m.

Status: Investigating

Update: We are currently investigating an issue with Insights not generating data. Our internal team is currently investigating to determine the cause of the disruption. The next update will be posted at 4:10pm CST.

Time: April 18, 2024, 2:57 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: April 18, 2024, 12:24 a.m.

Status: Investigating

Update: We are currently investigating an issue with Teem. We will update you when we have more information.

Time: May 9, 2024, 12:33 p.m.

Status: Postmortem

Update: **Teem Detailed Root Cause Analysis | 4.05.2024** **404 Error Site Wide Outage** ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. ‌ **Description:** On April 5th, 2024,  internal teams and customer support noticed issues happening with logging and being able to access sites. Reports of some Teem instances experiencing inability to login with a forward of a 404 error also were presented. This issue impacted a large scale of Teem Customers. This cause of this site outage was due to a migration to a new server that would allow easier rollbacks and data integrity, that unfortunately had a performance error and needed to be done early. Upon receiving notice of the site access issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue. **Type of Event:** Site Access **Remediation:** Upon immediate notice of the login issues, our dedicated Teem team was able to restart the migration process to successful completion, resolving the issue. **Root Cause Analysis:** There was a production outage caused by a database that we were hosting on. We needed to then move the data base to our standby server and set up a new one. We had planned the migration for a few weeks later and were preparing for it but the server encountered errors before the planned maintenance happened forcing us to start the migration early. We started the migration server backed up the DB and re-established connection with services. **Preventative Action:** The Teem team was able to start a migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure the best customer experience.

Time: April 18, 2024, 2:57 a.m.

Status: Resolved

Update: As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase.

Time: April 18, 2024, 1:27 a.m.

Status: Monitoring

Update: A fix has been implemented. We are moving into the Monitoring Phase for the next 30 minutes.

Time: April 18, 2024, 12:24 a.m.

Status: Investigating

Update: We are currently experiencing an issue with the Teem application resulting in a 404 when trying to access the site.

Time: May 20, 2024, 5:25 p.m.

Status: Postmortem

Update: **Teem by Eptura detailed Root Cause Analysis | April 11, 2024**  **S2 Google Calendar Service not Synchronizing**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  Customers using the Google Calendar service experienced events that were not automatically synced. During this time, a workaround was provided to force a manual sync, updating the calendars.     **Type of Event:**  Functionality Issue    **Services/Modules impacted:**  Production/ Google Calendar Service    **Timeline** \(Reported MST\)**:**` `  On the late afternoon of April 11th, 2024, at approximately 3:50pm, multiple customers reported an issue with their Google Calendar Service not automatically syncing calendar events. Customers were provided a temporary workaround to manually force sync their calendars. All customers were made aware of the Severity 2 incident via Teem Status Page. The investigation continued through April 19, 2024, when the CloudOps team identified the root cause of the issue. On April 22, 2024, at approximately 11:08am, all customers were notified via Status Page that the fix had been implemented and we moved into a monitoring phase. After continuous monitoring, no additional reports for Google Calendar Events and customers confirming that their Calendar events were syncing automatically, the Severity 2 incident was marked as resolved on April 29, 2024, at 10:23am.    **Total Duration of Event:**  17 days, 18 hours, 33 minutes    **Root Cause:**   We observed that the PgBouncer and PgBouncer\_ro services will not run simultaneously on job managers. Due to the startup script, it is unclear which of the two services is running, and it seems that the "last to start wins" scenario occurs. In an instance restart, a different service could "win" and cause further inconsistency. We have also discovered that three of our Job Managers are running outdated code.    **Remediation:**  These services shared a unix socket directory. By providing different unix socket directories, the services both would run simultaneously and eliminate the inconsistency. This eliminated significant errors on the jobmanagers    **Preventative Action:**   Our team is dedicated to continuously improving the Google Calendar Service by enhancing our current processes and implementing robust monitoring systems. We appreciate your patience and cooperation during this disruption.

Time: April 29, 2024, 4:23 p.m.

Status: Resolved

Update: We deeply appreciate your patience as our team worked diligently to resolve the recent calendar event synchronization timing issue. We are pleased to inform you that we have successfully identified the root cause of the problem and have implemented a fix to resolve it. Our team is committed to ensuring normal calendar performance and will continue to monitor this issue closely to ensure the best possible customer experience. For any further questions and concerns, please reach out to our dedicated support team. Thank you.

Time: April 22, 2024, 5:08 p.m.

Status: Monitoring

Update: We appreciate your patience as our team is working diligently to resolve the issue with unpredictable timing of calendar event synchronization. We have identified an issue that was causing these symptoms, and have developed and implemented a fix to resolve the issue. Our team is dedicated to ensuring normal calendar sync performance, and we will continue to monitor the issue to ensure full resolution and provide additional status updates if needed. Thank you.

Time: April 19, 2024, 5:52 p.m.

Status: Identified

Update: We appreciate your patience as our team is working diligently to resolve the issue with unpredictable timing of calendar event synchronization. We have identified a potential issue which could be causing these symptoms and are actively working to address it. Currently, we have observed that the PgBouncer and PgBouncer_ro services will not run simultaneously on job managers. Due to the startup script, it is unclear which of the two services is running, and it seems that the "last to start wins" scenario occurs. In the event of an instance restart, a different service could potentially "win" and cause further inconsistency. To resolve this, we have worked on a solution where these services now have separate unix socket directories. By providing different unix socket directories to both services, they can run simultaneously and eliminate the inconsistency. This eliminated significant errors on jobmanagers. Our team is dedicated to restoring normal calendar sync performance, and we will keep you updated with additional status updates as we continue to monitor and make progress towards a resolution.

Time: April 18, 2024, 8:34 p.m.

Status: Investigating

Update: Our team is currently working to resolve an issue that is impacting sync times for customers using Google calendars. We want to assure you that our team is fully committed to resolving this issue as swiftly as possible. We recognize the importance of timely event syncing, and we apologize for any delays you may be experiencing. Restoring normal calendar sync performance is our top priority, and we will keep you updated with additional status updates as we make progress towards a resolution. Thank you.

Time: April 18, 2024, 4:34 p.m.

Status: Investigating

Update: We are continuing to investigate this issue on priority. We apologize for the delay, next update will be shared at 3:30 PM CST.

Time: April 18, 2024, 12:38 p.m.

Status: Investigating

Update: As the fix implemented haven't resolved the issue completely, we have moved to the investigation phase. Our Engineering team is currently investigating the issue with Google Calendar Service to determine the cause of disruption. The next update will be posted at 11:30 AM CST.

Time: April 17, 2024, 12:42 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues for next 12 hours.

Time: April 16, 2024, 9:03 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues for next 12 hours.

Time: April 16, 2024, 3:56 p.m.

Status: Monitoring

Update: A fix had been identified and applied to optimize the performance of the Google Calendar Sync. We are moving into the Monitoring Phase for the next 4 hours and next update will be shared at 3 PM CST.

Time: April 16, 2024, 11:49 a.m.

Status: Investigating

Update: We are continuing to investigate this issue on priority. We apologize for the delay, next update will be shared at 11 AM CST.

Time: April 16, 2024, 8:32 a.m.

Status: Investigating

Update: As the fix implemented haven't resolved the issue completely, we have moved to the investigation phase. Our Engineering team is currently investigating the issue with Google Calendar Service to determine the cause of the disruption. The next update will be posted at 7 AM CST.

Time: April 15, 2024, 12:51 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues for next 12 hours.

Time: April 14, 2024, 7:47 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues for next 12 hours.

Time: April 13, 2024, 8:12 a.m.

Status: Monitoring

Update: A fix has been implemented. We are moving into the Monitoring Phase for the next 12 hours.

Time: April 13, 2024, 4:13 a.m.

Status: Identified

Update: We are continuing to investigate this issue on priority. We apologize for the delay, next update will be at 3 am CST

Time: April 12, 2024, 9:06 p.m.

Status: Identified

Update: As the previous fix implemented did not resolve the issue completely. We are continuing the investigation with Google Calendar Service and have determined the cause of the disruption and are working on a fix. The next update will be posted at 8 PM CST.

Time: April 12, 2024, 4:27 p.m.

Status: Investigating

Update: We are continuing to investigate this issue on priority. We will post another update at 4 PM CST.

Time: April 12, 2024, 1:22 p.m.

Status: Investigating

Update: As the fix implemented haven't resolved the issue completely, we have moved to the investigation phase. Our Engineering team is currently investigating the issue with Google Calendar Service to determine the cause of the disruption. The next update will be posted at 12 PM CST.

Time: April 12, 2024, 10:27 a.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues for next 4 hours.

Time: April 12, 2024, 5:58 a.m.

Status: Monitoring

Update: A fix has been implemented. We are moving into the Monitoring Phase for the next 4 hours.

Time: April 12, 2024, 1:56 a.m.

Status: Identified

Update: The issue with Google Calendar Service. has been identified and a fix is being implemented. We will post another update at 1am CST.

Time: April 12, 2024, 1:35 a.m.

Status: Investigating

Update: We are continuing to investigate this issue in regard to Google Calendar Sync. The next update will be posted at 11:45pm MDT.

Time: April 11, 2024, 9:46 p.m.

Status: Investigating

Update: We are currently investigating an issue with customers using Google Calendar Service. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 7:45pm MDT.

Time: April 17, 2024, 5:41 p.m.

Status: Postmortem

Update: **Teem Detailed Root Cause Analysis | 4.05.2024**  **404 Error Site Wide Outage**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  On April 5th, 2024,  internal teams and customer support noticed issues happening with logging and being able to access sites. Reports of some Teem instances experiencing inability to login with a forward of a 404 error also were presented. This issue impacted a large scale of Teem Customers.  This cause of this site outage was due to a migration to a new server that would allow easier rollbacks and data integrity, that unfortunately had a performance error and needed to be done early.  Upon receiving notice of the site access issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue.    **Type of Event:**  Site Access     **Remediation:**  Upon immediate notice of the login issues, our dedicated Teem team was able to restart the migration process to successful completion, resolving the issue.    **Timeline:**  ` `April 5  * \(4:14 AM\) – Internal teams were noticing issues with logging as well as site access  * \(4:21 AM\) – Customers also reporting the issue, at this point we have already notified the team  * \(4:24 AM\) – Investigation process has started with internal teams  * \(5:23 AM\) – Internal teams notify us of continued investigation but that the issue is close to being found  * \(6:43 AM\) – The issue has been found and a fix has been implemented with the rollback of the server to the new DB still needing around a few hours to finish.  * \(12:45 PM\) – The database has been moved and restored.    **Total Duration of Event:**  \(0 day/8 hours/30minutes\)     **Root Cause Analysis:**    There was a production outage caused by a database that we were hosting on. We needed to then move the data base to our standby server and set up a new one. We had planned the migration for a few weeks later and were preparing for it but the server encountered errors before the planned maintenance happened forcing us to start the migration early. We started the migration server backed up the DB and re-established connection with services.    **Preventative Action:**   The Teem team was able to start a migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure the best customer experience.

Time: April 5, 2024, 6:45 p.m.

Status: Resolved

Update: We have had confirmation from internal teams as well as customers of the issue being resolved. We appreciate everyone's patience as our engineers worked to resolve this.

Time: April 5, 2024, 5:30 p.m.

Status: Monitoring

Update: A fix has been implemented by the Engineering team. We'll be going into Monitoring for the next hour, an update will be given at 14:30 CST.

Time: April 5, 2024, 3:57 p.m.

Status: Identified

Update: We are continuing to work on the fix for this issue. We will post another update at 13:00 CST.

Time: April 5, 2024, 2:04 p.m.

Status: Identified

Update: The issue with TEEM has been identified and a fix is being implemented. We will post another update at 11:00 CST.

Time: April 5, 2024, 10:17 a.m.

Status: Investigating

Update: We are currently investigating an issue with Teem. We will update you when we have more information.