Is there an Avochato outage?
Avochato status: Systems Active
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Avochato and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 6 minutes ago
Get notified about any outages, downtime or incidents for Avochato and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Avochato.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| API | Active |
| avochato.com | Active |
| Mobile | Active |
View the latest incidents for Avochato and check for official updates:
Description: ## What Happened The morning after a data migration to improve inbox search performance and add new search functionality, a series of complex queries caused our search service to enter a bad state and became throttled during the impact period. This appeared to be due to the size of our Elasticsearch shards exceeding a critical limit in terms of size. We exceeded the size threshold due to additional columns that were added to support new indices, multiplied by the size of the production dataset. As a result, search results \(including the default inbox experience, contacts list, etc\) timed out until we could completely reboot the search infrastructure. ### Impact Customer Data \(including conversations, messages, tags, etc\) was not lost and messages continued to deliver as intended during the period. Most functionality was available including account and user management, though the experience for searching through the inbox was severely degraded. Customers who were online trying to use the app or API during the impact period would not have easily been able to lookup conversations or contacts. It was difficult to create and populate broadcasts during the period. Users were still able to navigate to conversations directly from links and from notifications. ## Resolution Engineers used our metrics to identify the issue was specifically related to our search infrastructure. The team scaled up additional instances of our Elasticsearch infrastructure to try and solve the problem. The team then rebuilt the Elasticsearch cluster as they were not able to "reboot" it, and began routing traffic to the new cluster. This resolved the issue for customers on a rolling basis, as some connections hit the new instance while other data routed to bad shards. The rebuild process was clocked at taking about 1 hour and 15 minutes to complete syncing all data to build new Elasticsearch indices, so unfortunately some customers were impacted during this entire period. We have since moved from using 8 Elasticsearch shards to 16, and reindexed the dataset, which cut the total size of each shard in half. ## Additional Notes In a separate issue that occurred around the same time period, the [www.avochato.com](http://www.avochato.com) domain was flagged automatically by Avast Antivirus' anti-phishing browser extension. We believe this was a false positive, but it caused Avast users who had the extension installed to be unable to view Avochato. Users who whitelisted Avochato in their extension were able to continue to log in, and the team worked quickly to submit an appeal. Avast has since then removed us from their phishing blacklist. Avochato poses no known phishing threat to its users, but we encourage users who suspect phishing attack vectors to submit their reports to [www.avochato.com/bugbounty](http://www.avochato.com/bugbounty) Thanks again for your patience while we resolved this issue, and for being an Avochato customer, Christopher, CTO & CISO
Status: Postmortem
Impact: Minor | Started At: Oct. 23, 2021, 3:05 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Oct. 6, 2021, 6:07 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Sept. 28, 2021, 4:12 p.m.
Description: This incident has been resolved.
Status: Resolved
Impact: Minor | Started At: Sept. 17, 2021, 3 p.m.
Description: ## What Happened Inboxes were receiving stale data that was sometimes 10-15 minutes out of date. This meant that even when an inbox was updated to reflect an incoming call or new message, it would rapidly be replaced by stale data, which made it really hard to find new or active conversations. Messages and calls were being received in real-time, but the Avochato inbox was not being properly updated to reflect that, and if you missed a notification or refreshed your inbox, you would not always get the latest list of active conversations \(though you could still view those conversations and respond to them if you still had a link to them from a notification\). This was exacerbated by ever-increasing daily platform usage such as syncing contacts and new conversations, which slowed our services in inboxes. It was particularly challenging for inboxes with many open conversations or conversations with very long message histories. This year, we analyzed and foresaw these growth challenges specifically with indexing, and have been making architectural changes to avoid these types of slowdowns. However, our growth rate outpaced our estimates of shipping the new version of our search indexing. During this time, no data was lost, though it was in some cases difficult to respond to incoming calls as well as handle live chats, and certain automation like closing conversations or marking conversations as addressed would appear to the end user as incorrect status in the app. ## Resolution First, the engineering team implemented a patch to the adapter handling indexing platform-wide, but that did not make a significant enough impact to eliminate the symptoms of this incident. After analyzing the results, we decided to edit a specific index to exclude certain expensive queries, namely full-text searching of incoming messages for all new records. Historical messages in conversations will still be searchable for the time being. We are actively rearchitecting how we index messages within conversations \(among other things\) so that they are easily fully searchable from the inbox and our API, as well as increasing the speed and accuracy of fetching conversations in an inbox on mobile and desktop devices. We thank you for your patience and for choosing Avochato as your business communications platform of choice, Christopher Neale CTO & CISO
Status: Postmortem
Impact: Minor | Started At: Aug. 25, 2021, 6 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
