Is there an Totango outage?
Totango status: Systems Active
Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for Totango and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 4 minutes ago
Get notified about any outages, downtime or incidents for Totango and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Totango.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Assets | Active |
| Campaigns | Active |
| Data Collection Gateway | Active |
| Data Processing Pipelines | Active |
| Email to Touchpoints | Active |
| Marketing Website | Active |
| Mobile Application | Active |
| Salesforce Connector | Active |
| Tasks and SuccessPlays | Active |
| Totango Web Application | Active |
View the latest incidents for Totango and check for official updates:
Description: This incident has been resolved.
Status: Resolved
Impact: Major | Started At: Aug. 4, 2022, 9:14 a.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Major | Started At: Aug. 4, 2022, 9:14 a.m.
Description: # **Event Description** Users attempting to login, work with segments or view account profiles were latency, system timeouts and inability to log in. # **Findings & Timeline** _**Reported: August 1st, 2022 17:52 GMT**_ _At 17:52 GMT, Totango Engineering team reported a latency and Totango users began to experience issues with viewing segments, account profiles and occasional issues logging into the application_ _**Identified: August 1st, 2022 18:48 GMT**_ _At 18:48 GMT, Totango Engineering team implemented a short term fix to allow for normal operations to resume temporarily. However the root cause had not yet been completely resolved._ _At 19:42 GMT The system issues returned and users were no longer able to access the application._ _At 20:54 GMT The issue was isolated, allowing us to restore normal operations to the application._ _**Resolved: August 1st, 2022 21:24 GMT**_ _The root cause was fully resolved._ **Root Cause** _A sudden and large volume of account data was introduced on the shared index in ElasticSearch. This data load caused all services on the shared index to timeout all search requests, leading to the Data API thread pool to fill and eventually shutdown causing the system to become unresponsive._ # **Corrective Action** * _Re-shard the problematic index out of the shared index - **Done**_ * _Increased monitoring and alert systems when a service is approaching a high volume of documents on the shared index - Ongoing_
Status: Postmortem
Impact: Minor | Started At: Aug. 1, 2022, 5:52 p.m.
Description: # **Event Description** Users attempting to login, work with segments or view account profiles were latency, system timeouts and inability to log in. # **Findings & Timeline** _**Reported: August 1st, 2022 17:52 GMT**_ _At 17:52 GMT, Totango Engineering team reported a latency and Totango users began to experience issues with viewing segments, account profiles and occasional issues logging into the application_ _**Identified: August 1st, 2022 18:48 GMT**_ _At 18:48 GMT, Totango Engineering team implemented a short term fix to allow for normal operations to resume temporarily. However the root cause had not yet been completely resolved._ _At 19:42 GMT The system issues returned and users were no longer able to access the application._ _At 20:54 GMT The issue was isolated, allowing us to restore normal operations to the application._ _**Resolved: August 1st, 2022 21:24 GMT**_ _The root cause was fully resolved._ **Root Cause** _A sudden and large volume of account data was introduced on the shared index in ElasticSearch. This data load caused all services on the shared index to timeout all search requests, leading to the Data API thread pool to fill and eventually shutdown causing the system to become unresponsive._ # **Corrective Action** * _Re-shard the problematic index out of the shared index - **Done**_ * _Increased monitoring and alert systems when a service is approaching a high volume of documents on the shared index - Ongoing_
Status: Postmortem
Impact: Minor | Started At: Aug. 1, 2022, 5:52 p.m.
Description: **Event Description** _Users were not able to login and also logged-in users experienced system slow response time_ **Findings & Timeline** _**First Noticed: July 6th, 2022 at 13:50 UTC**_ _Our monitoring systems alerted us to an issue with application performance and failures to login. Investigation started by engineering teams._ _**Problem Area Identified: July 6th, 2022 at 14:00 UTC**_ _At this point, our team was able to identify the problem and began work on steps to mitigate and trace the root cause. There were a handful of users in the system appearing to overload our systems with numerous API calls._ _**July 6th, 2022 at 15:30 UTC**_ _Continuing efforts are made to mitigate the issue, though the root cause has not yet been identified._ _**July 6th, 2022 at 17:00 UTC**_ _The root cause analysis found the problematic code. A decision to revert a recent code deployment made_ _**July 6th, 2022 at 17:30 UTC**_ _The revert of the problematic code showed good signs and resolved the issue with the exception of impacting only 2 users._ _**July 6th, 2022 at 18:30 UTC**_ _Steps were taken to contact the two users and guide them through logging off the system to rectify the erroneous API calls._ **Root Cause** * _As part of fixing an issue, where contacts were being added to touchpoints without choosing them, we introduced a new code that in some cases caused an endless component render which fetched contacts from the DB. This code was only running for certain users where a specific setting for showing all hierarchy contacts was off._ **Preventive Action** _Fix the original bug that caused this issue. Make sure the call to fetch contacts from the DB is done only once - **Done**_ _Enforce rate limitations across all components - Not Started_ _User token isolation to better control rate limitations - Not Started_
Status: Postmortem
Impact: Minor | Started At: July 6, 2022, 2:22 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
