Last checked: 3 minutes ago
Get notified about any outages, downtime or incidents for Kustomer and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Kustomer.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
Component | Status |
---|---|
Regional Incident | Active |
Prod1 (US) | Active |
Analytics | Active |
API | Active |
Bulk Jobs | Active |
Channel - Chat | Active |
Channel - Email | Active |
Channel - Facebook | Active |
Channel - Instagram | Active |
Channel - SMS | Active |
Channel - Twitter | Active |
Channel - WhatsApp | Active |
CSAT | Active |
Events / Audit Log | Active |
Exports | Active |
Knowledge base | Active |
Kustomer Voice | Active |
Notifications | Active |
Registration | Active |
Search | Active |
Tracking | Active |
Web Client | Active |
Web/Email/Form Hooks | Active |
Workflow | Active |
Prod2 (EU) | Active |
Analytics | Active |
API | Active |
Bulk Jobs | Active |
Channel - Chat | Active |
Channel - Email | Active |
Channel - Facebook | Active |
Channel - Instagram | Active |
Channel - SMS | Active |
Channel - Twitter | Active |
Channel - WhatsApp | Active |
CSAT | Active |
Events / Audit Log | Active |
Exports | Active |
Knowledge base | Active |
Kustomer Voice | Active |
Notifications | Active |
Registration | Active |
Search | Active |
Tracking | Active |
Web Client | Active |
Web/Email/Form Hooks | Active |
Workflow | Active |
Third Party | Active |
OpenAI | Active |
PubNub | Active |
View the latest incidents for Kustomer and check for official updates:
Description: Postmark has resolved an event affecting Pod 1 & Pod 2 where users may have seen delays in emails not being sent or received. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at [email protected] if you have additional questions or concerns.
Status: Resolved
Impact: Minor | Started At: July 28, 2022, 6:27 p.m.
Description: Postmark has resolved an event affecting Pod 1 & Pod 2 where users may have seen delays in emails not being sent or received. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at [email protected] if you have additional questions or concerns.
Status: Resolved
Impact: Minor | Started At: July 28, 2022, 6:27 p.m.
Description: # **Summary** On 7/18/22 between 11:50am ET and 4:43pm ET, users of Kustomer platform in prod1 experienced high latency. The system was operational during this time but responses from the system were slower than usual. # **Root Cause** There was a spike in latency in the API service starting at 11:50am ET. This was traced back to extremely high load in a caching system that ramped up CPU usage very quickly. The caching system sustained 100% CPU usage which resulted in a backlog of items waiting to be processed. This backlog manifested in the latency experienced by users in this time period. # **Timeline** 07/18 11:50am - API latency detected; caching system metrics within acceptable range. 07/18 2:31pm - Reports of customers experiencing latency and slowness; caching system metrics now showing 100% CPU usage. 07/18 2:51pm - Engineer on call begins investigation. 07/18 3:00pm - Initiated failover in caching system to rule out system/hardware problems. Metrics showed improvement only for a few minutes before regression. 07/18 3:29pm - Provision a new caching system optimized for high CPU load. 07/18 4:18pm - All services switched to utilize the newly provisioned caching system; latency subsided; CPU usage back in normal range. 07/18 4:43pm - Incident fully resolved. ## **Lessons/Improvements** * **\[IN PROGRESS\]** Optimize usage of the caching system * **\[IN PROGRESS\]** Implement cache sharding to increase the horizontal scale automatically
Status: Postmortem
Impact: Minor | Started At: July 18, 2022, 6:51 p.m.
Description: # **Summary** On 7/18/22 between 11:50am ET and 4:43pm ET, users of Kustomer platform in prod1 experienced high latency. The system was operational during this time but responses from the system were slower than usual. # **Root Cause** There was a spike in latency in the API service starting at 11:50am ET. This was traced back to extremely high load in a caching system that ramped up CPU usage very quickly. The caching system sustained 100% CPU usage which resulted in a backlog of items waiting to be processed. This backlog manifested in the latency experienced by users in this time period. # **Timeline** 07/18 11:50am - API latency detected; caching system metrics within acceptable range. 07/18 2:31pm - Reports of customers experiencing latency and slowness; caching system metrics now showing 100% CPU usage. 07/18 2:51pm - Engineer on call begins investigation. 07/18 3:00pm - Initiated failover in caching system to rule out system/hardware problems. Metrics showed improvement only for a few minutes before regression. 07/18 3:29pm - Provision a new caching system optimized for high CPU load. 07/18 4:18pm - All services switched to utilize the newly provisioned caching system; latency subsided; CPU usage back in normal range. 07/18 4:43pm - Incident fully resolved. ## **Lessons/Improvements** * **\[IN PROGRESS\]** Optimize usage of the caching system * **\[IN PROGRESS\]** Implement cache sharding to increase the horizontal scale automatically
Status: Postmortem
Impact: Minor | Started At: July 18, 2022, 6:51 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.