Is there an Ultimate AI outage?
Ultimate AI status: Systems Active
Last checked: 9 minutes ago
Get notified about any outages, downtime or incidents for Ultimate AI and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 9 minutes ago
Get notified about any outages, downtime or incidents for Ultimate AI and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Ultimate AI.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| Analytics | Active |
| Backend Integrations | Active |
| Core Services | Active |
| Dashboard | Active |
| Public API | Active |
| Web Chat Widget | Active |
| Chat integrations | Active |
| Freshchat | Active |
| Freshdesk Automation | Active |
| Giosg Automation | Active |
| Intercom Automation | Active |
| LiveChat.com CRM Integration | Active |
| Salesforce | Active |
| Sunshine | Active |
| Zendesk Chat | Active |
| Zendesk Support Automation | Active |
| Thirdparty Providers | Active |
| LiveChat Agent apps | Active |
| LiveChat API | Active |
| LiveChat Chat widget | Active |
| LiveChat Integrations | Active |
| LiveChat Subprocessors' service | Active |
| Salesforce EU37 | Active |
| Salesforce EU40 | Active |
| Salesforce UM7 | Active |
| Zendesk Sunshine Conversations Sunshine Conversations Core API | Active |
| Zendesk Sunshine Conversations Web SDK | Active |
View the latest incidents for Ultimate AI and check for official updates:
Description: One of our customers experienced a malicious conversation. This exposed an unforeseen inefficiency in our database layer. A temporary fix was deployed within 90 minutes of incident discovery. The fix limited the speed of consuming events, in effect limiting the load on the database it caused. This stabilized the production systems. A long-term solution introduced a hard limit on conversation length. According to our analytics, no production bots should be affected by this limitation.
Status: Postmortem
Impact: Critical | Started At: Jan. 31, 2024, 3:42 p.m.
Description: Around 10:00 AM UTC we have noticed an increase errors in our internal error reporting tool, and a quick investigation revealed that one of our database clusters, managed by an external provider, cannot handle the current load. This affected our dashboard as well as incoming chats. An investigation revealed that our database was automatically scaled down to a less performant instance based on the previous 24 hours. We reacted by manually scaling it up to the previous cluster size using external provider’s console, however due to a malfunction on the external provider side, this process did not complete as it usually should. At 10:40 AM UTC we contacted the external provider’s support team, and they force-applied our change from their end. At 11:57 AM UTC our dashboard became operational. At 12:12 PM UTC chats on all CRMs became operations. As first measure to avoid a similar future situation, we now enforce a higher minimum cluster size to avoid excessive automatic down scaling. Furthermore, we are in contact with the provider’s support to investigate why our changes were not applied.
Status: Postmortem
Impact: Critical | Started At: Nov. 27, 2023, 10:13 a.m.
Description: Around 19:32:13 UTC on 09.11.2023, an `integration-middleware` service started giving the following error on the impacted clients. * ``` :\"Error: Request failed with status code 403\\n ``` This incident started post-upgrade of one of the K8s cluster's nodePools where the impacted service was operating, We checked the release notes of k8s to see if anything was introduced which might have caused the service to throw these exceptions. * After a thorough investigation we concluded that 3 newly added NAT IP were not added to the allowed list of impacted clients which led to this incident. After an upgrade of our K8s cluster, the impacted service was migrated to a node, which was using the new NAT IPs for the egress traffic. Adding the new IPs to affected clients allow-list mitigated the issue.
Status: Postmortem
Impact: None | Started At: Nov. 10, 2023, 8:15 a.m.
Description: Our team has resolved both issues and our analytics should be fully operational again. We will follow up with a post-mortem soon.
Status: Resolved
Impact: Minor | Started At: Oct. 30, 2023, 10:05 a.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Major | Started At: Oct. 26, 2023, 9:15 a.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
