Is there an Ultimate AI outage?
Ultimate AI status: Systems Active
Last checked: 9 minutes ago
Get notified about any outages, downtime or incidents for Ultimate AI and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 9 minutes ago
Get notified about any outages, downtime or incidents for Ultimate AI and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Ultimate AI.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Analytics | Active |
| Backend Integrations | Active |
| Core Services | Active |
| Dashboard | Active |
| Public API | Active |
| Web Chat Widget | Active |
| Chat integrations | Active |
| Freshchat | Active |
| Freshdesk Automation | Active |
| Giosg Automation | Active |
| Intercom Automation | Active |
| LiveChat.com CRM Integration | Active |
| Salesforce | Active |
| Sunshine | Active |
| Zendesk Chat | Active |
| Zendesk Support Automation | Active |
| Thirdparty Providers | Active |
| LiveChat Agent apps | Active |
| LiveChat API | Active |
| LiveChat Chat widget | Active |
| LiveChat Integrations | Active |
| LiveChat Subprocessors' service | Active |
| Salesforce EU37 | Active |
| Salesforce EU40 | Active |
| Salesforce UM7 | Active |
| Zendesk Sunshine Conversations Sunshine Conversations Core API | Active |
| Zendesk Sunshine Conversations Web SDK | Active |
View the latest incidents for Ultimate AI and check for official updates:
Description: On 08. October 2024 our Redis cache for Sunshine Automation reached its memory limit at around 00:30 CEST. Redis, acting as a cache to store frequently accessed data, experienced memory overuse due to improper key expiration, which prevented the bots integrated with Sunshine Automation from processing messages effectively. Our alert systems failed to notify the on-call team, delaying the start of the investigation until around 07:00 CEST. After the investigation, the team increased the Redis memory, resolving the incident by 08:00 CEST. Message processing returned to normal and the system is now stable. As a follow-up we will address the Redis key expiration issue and improve our alert system for Redis memory usage to ensure faster response times in the future.
Status: Postmortem
Impact: Major | Started At: Oct. 8, 2024, 5:27 a.m.
Description: On October 1, 2024, One of our bots experienced a significant spike in chat messages. The incident began at approximately 20:50, leading to operational challenges that peaked around 21:25. As chat volumes increased, our system encountered rate limiting from the Sunshine Conversations API, which was not handled gracefully. This resulted in CPU overutilization across multiple pods, degrading performance. Oncall team, after being notified, updated the system’s maximum resources, ultimately stabilising the platform.
Status: Postmortem
Impact: Critical | Started At: Oct. 1, 2024, 8:20 p.m.
Description: Due to a high and unexpected load paired with a recent change that made our queued events be less performant, since last night at around 00:30 CEST some asynchronous operations and triggers are suffering a delay to be consumed by our workers that could cause them to take up to 5 hours to execute, the mentioned events and triggers are the following ones: - Inactivity timers and their related actions - Chat ended events and their related actions - Feedback reply triggers - Delayed ticket messages We rolled back the changes and the queue is getting back to a healthy state, our estimation is that it will be fully consumed by 21:00 CEST today, from that point onwards the delays should disappear.
Status: Resolved
Impact: Major | Started At: Sept. 5, 2024, 10:30 p.m.
Description: Live chat status page has been updated with All impacted systems reported operational.
Status: Resolved
Impact: Major | Started At: Sept. 2, 2024, 1:36 p.m.
Description: Around 14:23 UTC on 2024/09/02, our system began predicting non-existent intents for Zero Shot \(ZS\) bots, which inadvertently triggered actions like tags appearing on tickets. The issue arose after we released a new `inference` version to production. Upon investigation, we identified that the problem was caused by an error in our cache logic within the IntentsFetcher class. Specifically, there was a mistake in how the cache key was generated to determine if the same bot was requesting the data. This error only impacted Zero Shot bots due to a recent modification in the ZS pipeline. This has not impacted the end-users directly.Because this bug did not generate any service errors, it was challenging to detect immediately. However, once identified, we quickly mitigated the impact by rolling back the affected service to a previous version, which has now resolved the issue.
Status: Postmortem
Impact: Major | Started At: Sept. 2, 2024, 10:49 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
