Time: Aug. 19, 2022, 4:13 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 18, 2022, 11:57 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Aug. 18, 2022, 11:11 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: Aug. 18, 2022, 10:35 p.m.

Status: Investigating

Update: We are continuing to investigate the cause of a delay in displaying new messages in the inbox. Note that messages continue to be delivered and be received. Thank you for your patience.

Time: Aug. 18, 2022, 10:15 p.m.

Status: Investigating

Update: We're investigating an issue with up to 15 minute delays in displaying messages and conversations in the inbox.

Time: Aug. 10, 2022, 4:43 p.m.

Status: Postmortem

Update: ## What Happened Due to continued growth of activity on the platform, a fast-growing and critical database table exceeded the 32 bit memory requirement for inserting new primary keys, preventing additional tags from being created. The inability to create new rows in this table impacted a number of features including adding contacts to broadcasts, adding tags to contacts, and more. Until the column type could be upgraded from Integer to BigInteger, those features were impacted and other business logic within the Avochato application was subsequently impacted. Unfortunately, the upgrade process took significant time due to the size of the table itself, leading to a protracted incident window for the impacted feature set. In order to resolve the issue, the engineering team placed the Avochato platform in maintenance mode during 4:15-5pm in order to attempt to resolve the issue, which directly impacted the ability to read conversations in the inbox during the maintenance window. A second process was applied overnight and into the following business day to fully migrate the database and resolve the issue without losing data until the new primary key column using the BigInteger type was populated, reindexed, and could be swapped for the old column. ## Resolution The team performed an in-place migration to change the column type from Integer to BigInteger. This took quite a bit of time to perform on a table with over 2 billion rows, but once the migration was complete, platform functionality immediately returned to normal. Meanwhile, the design of the Avochato platform allowed failure attempts for things like tagging, broadcasting, and contact uploading to be queued for retry while the platform was impacted. Those tasks were then completed successfully in the order they were received as soon as the incident was resolved. Moving forward, we are auditing our legacy database tables to ensure that primary or foreign keys across our data warehouse will scale decades into the future as we continue to experience growth on the platform. We know how critical Avochato is for communicating with your teams, and appreciate your patience during this period.

Time: Aug. 10, 2022, 4:36 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: Aug. 9, 2022, 7:47 p.m.

Status: Monitoring

Update: Broadcasting, tagging, and related Avochato functionality has been restored. We are continuing to monitor our systems post-maintenance. Over the next 24 hour period, our system will work through the backlog of business logic relating to contact uploads, tags, and broadcasts.

Time: Aug. 9, 2022, 7:16 p.m.

Status: Identified

Update: Our maintenance has successfully resolved the primary issue causing tags and broadcasts. Our system is continuing to work through the backlog of tasks since the start of the incident. Attempts at uploading contacts to broadcasts and manually tagging contacts since the start of the incident will slowly re-attempt. We apologize for the inconvenience and will continue updating as we perform a rolling restart of the system.

Time: Aug. 9, 2022, 6:55 p.m.

Status: Identified

Update: Inbox functionality has been restored while we proceed to the next phase of maintenance. Thank you for your patience. We are continuing to work on resolving issues with Broadcasts and Tags.

Time: Aug. 9, 2022, 6:49 p.m.

Status: Identified

Update: In order to complete the required maintenance, inboxes will temporarily stop serving conversations. We are trying to minimize this window as much as possible and apologize for the inconvenience.

Time: Aug. 9, 2022, 5:01 p.m.

Status: Identified

Update: We are continuing to work on resolving issues tagging contacts and broadcasting. We appreciate your patience.

Time: Aug. 9, 2022, 3:02 p.m.

Status: Identified

Update: Resolution is still in progress, but some app functionality including broadcasting and tagging are still impacted. We appreciate your patience as we work to resolve this issue.

Time: Aug. 8, 2022, 11:34 p.m.

Status: Identified

Update: In order to expedite resolving issues with Broadcasts and Tag creation, we will be performing emergency database maintenance. Inboxes may not be accessible at this time while we complete maintenance. We appreciate your patience.

Time: Aug. 8, 2022, 11:14 p.m.

Status: Identified

Update: We're currently experiencing issues tagging objects in our database and are working on a solution. Broadcasts and campaigns are disabled and will not function in the meantime until we find a resolution.

Time: Aug. 4, 2022, 4:58 p.m.

Status: Postmortem

Update: ## What Happened An update to how we handle state did not properly handle sessions generated in older versions of the app for previously logged in users. This caused the login screen of the iOS app to get in a state where it could not log an older or invalid session out and also could not proceed to the login screen for specific users upgrading from older versions of Avochato. ## Resolution We’ve patched the specific iOS component that handles state when upgrading versions and it should do a better job of cleaning up after older versions of Avochato whenever the app is upgraded or uninstalled and reinstalled. No further action is necessary at this time.

Time: Aug. 4, 2022, 4:57 p.m.

Status: Resolved

Update: This issue is resolved in our most recent version of Avochato for iOS, 2.19, which is live in the AppStore.

Time: Aug. 3, 2022, 7:57 p.m.

Status: Monitoring

Update: Version 2.18 of Avochato for iOS is now available. We are monitoring progress for users who were unable to log in using the latest version. Older versions of the iOS app, Avochato for Android, and our desktop and mobile web experiences have been unaffected during this period.

Time: Aug. 2, 2022, 9:01 p.m.

Status: Investigating

Update: We are investigating an issue causing some users to get stuck at the loading screen on our most recent version of the Avochato for iOS app.

Time: May 26, 2022, 8:26 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: May 26, 2022, 8:16 a.m.

Status: Identified

Update: The maintenance is done, but we will continue to monitor and make sure the infrastructure is stable.

Time: May 26, 2022, 8:16 a.m.

Status: Identified

Update: Unfortunately our infrastructure hosting provider is doing emergency maintenance on our databases. Servers should be back to normal shortly.

Time: Dec. 15, 2021, 5:07 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Dec. 15, 2021, 4:11 p.m.

Status: Identified

Update: We have received communication from our cloud services provider of a partial outage within their data center. Recovery is in progress. Access to the Avochato mobile and desktop app may be intermittent depending on your location.

Time: Dec. 15, 2021, 4:04 p.m.

Status: Investigating

Update: We're seeing latency and slow requests in our environment and investigating.