Is there an Avochato outage?
Avochato status: Systems Active
Last checked: 5 minutes ago
Get notified about any outages, downtime or incidents for Avochato and 1800+ other cloud vendors. Monitor 10 companies, for free.
Last checked: 5 minutes ago
Get notified about any outages, downtime or incidents for Avochato and 1800+ other cloud vendors. Monitor 10 companies, for free.
Outage and incident data over the last 30 days for Avochato.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage. It's completely free and takes less than 2 minutes!
Sign Up NowOutlogger tracks the status of these components for Xero:
| Component | Status |
|---|---|
| API | Active |
| avochato.com | Active |
| Mobile | Active |
View the latest incidents for Avochato and check for official updates:
Description: ## What Happened One of our third-party software vendors suffered a platform-wide outage which prevented us from leveraging their APIs to deliver SMS and Voice functionality. During the impact period, we were not able to leverage their APIs for sending messages and managing Avochato Numbers, as well as routing calls. Attempts to send messages were queued and retried until we could successfully deliver the messages. Customers sending traffic through our other SMS and Voice partners were not impacted. ## Resolution The vendor performed their procedures for triaging and resolving the incident, and successfully returned to full operational capability.
Status: Postmortem
Impact: Critical | Started At: Feb. 26, 2021, 1:30 p.m.
Description: ## What happened Actions taken by our cloud hosting provider as part of an investigation into Avochato infrastructure unexpectedly caused applications and browsers to be temporarily unable to reach Avochato from the web or the Avochato mobile apps. ## Resolution Our ops team worked quickly to immediately rebuild our application cluster and get up and running, but some devices with cached DNS may have experienced 500 errors and the inability to take or make phone calls while DNS rolled out. We are working with our provider to make sure this misunderstanding does not happen in the future. We apologize for any inconvenience and thank you for choosing Avochato.
Status: Postmortem
Impact: None | Started At: Feb. 16, 2021, 10:30 a.m.
Description: ## What Happened Platform automation handling live notifications for messages led to excessive queueing of updates to critical tables in our write database, which in turn led to longer turnaround times for live updates inside the inbox. After identifying the root cause, Engineering deployed a fix \(which succeeded in stemming the root cause\), but in the meantime, we over-scaled our workers to adjust to their load which caused connection issues for non-workers. This led to an escalation in page load times while our congested databases could no longer serve our applications in a timely manner. Our cloud operations automatically scaled to handle the increased pressure from the root cause while we resolved the issue, but over-scaled improperly, leading to increased load times when accessing the app. Load times additionally spiked as we swapped the databases then returned to normal. During this period, one specific database seemingly had failure despite no usage. This led to a second period of increased load times after a brief reprieve. This database was safely replaced and servers were were routed to a replacement, and load times returned to normal. ## Impact Initial delays in receiving live inbox updates followed by high page load times when viewing the app. Delays in updating conversations and contacts. Due to delays in receiving live updates, some messages that appeared to be double-sent manually were delivered exactly once as intended. ## Resolution The team has already deployed updates to prevent the root cause from occurring. We have made adjustments to prevent the second issue including adjusting the maximum connections to the write database as well as safely replacing the faulty database that appeared to failover during this period. Our engineering team will audit configurations that led to the lack of cloud resources when auto-scaling.
Status: Postmortem
Impact: Minor | Started At: Feb. 12, 2021, 7:43 p.m.
Description: A software update relating to the broadcast scheduling mechanism introduced a regression in our system, which caused broadcasts to stay stuck in an initial “sending” state. This has been resolved with a subsequent software update. This issue was not caught by automated or QA testing, and we have updated our automated testing and quality assurance controls to prevent a similar issue from happening in the future. Affected broadcasts eventually retried and delivered their message to their audience after a delay.
Status: Postmortem
Impact: None | Started At: Dec. 15, 2020, 3:39 p.m.
Description: ## What Happened A sudden escalation in CPU in our production database cloud database provider led to a period of longer than normal load times. While our platform was under high load for Cyber Monday, our investigation leads us to believe this was a hardware-related failure and our operations team responded accordingly. Additionally, the period of slow response times led to our Slack application automatically turning off as per Slack’s platform policy. Engineers manually turned the app back on numerous times, but unfortunately some messages sent via Slack threads were unable to be delivered during the windows when the Slack app was disabled. ## Resolution Data was not lost as we were able to fail over to a read replica database. We have upgraded database hardware to move away from potentially degraded hardware and performed routine system maintenance.
Status: Postmortem
Impact: Minor | Started At: Nov. 30, 2020, 4 p.m.
Join OutLogger to be notified when any of your vendors or the components you use experience an outage or down time. Join for free - no credit card required.
