Time: March 13, 2019, 6:28 a.m.

Status: Resolved

Update: According to Google (and our metrics) everything should be back to normal. Thank you for bearing with us. Emails that were sent during the outage, and which you didn't cancel or retry yourself, have now been retried and have sent successfully. If you have further questions, please get in touch [email protected].

Time: March 13, 2019, 5:55 a.m.

Status: Monitoring

Update: It looks like the error rates have returned to normal. We're going to keep an eye on the situation and ensure your emails are sent correctly. We'll resolve this completely when we hear good news from Google.

Time: March 13, 2019, 2:41 a.m.

Status: Identified

Update: We've been in touch with the Gmail team, and they are working on the issue. It is affecting users of mail.google.com too, so they expect to have a resolution quickly. We'll keep this incident open until we hear back from them that the issue is resolved.

Time: March 13, 2019, 2:07 a.m.

Status: Investigating

Update: We are investigating an issue with Gmail attachments. This may cause sends to fail when you reply or forward an email that contains an attachment, and will mean that you cannot download attachments from Gmail. We've escalated this to the Gmail support team, as it seems to be an outage with that API endpoint.

Time: March 2, 2019, 8:08 a.m.

Status: Resolved

Update: This maintenance was completed successfully. We're monitoring things closely, but everything appears to be back to normal. Please reach out to me if you have any questions: [email protected] Conrad

Time: March 2, 2019, 7:23 a.m.

Status: Monitoring

Update: We are upgrading our database configuration after the outage yesterday. We estimate that this will require ~10 minutes of downtime starting at 11:50 pm PST (our lowest traffic point of the week). During this window, emails will receive as normal but they will not send. Any emails that you do send will be safely stored within Superhuman, and will deliver just as soon as connectivity is restored. Please reach out to me if you have any questions: [email protected] Conrad CTO

Time: March 1, 2019, 12:59 a.m.

Status: Postmortem

Update: Hello all, Yesterday, Superhuman was down for nearly two hours due to a failure with our database. I am deeply sorry for this. We know that email is mission critical, and that this much downtime is unacceptable. During the downtime, emails could not be sent and it was not possible to log into Superhuman. If you were already logged in, you could still receive emails. If you were not logged in, then it was not possible to log in. The failure was due to two simultaneous issues: 1\. Our database was running low on disk space. 2\. One of the availability zones that our database runs in was unable to provision more disk space. For our database, we use Google Cloud SQL in High Availability mode. We also use the built-in feature to “automatically increase disk space”. We failed to realize two important things about this setup: 1\. The automatic disk space increase is very conservative. Based on current load, it would only allocate enough space for a few additional hours at peak traffic. 2\. Increasing disk space is an operation that requires both availability zones to be active. We spoke with Google Cloud Support who explained all off this in detail, and then we took the decision to temporarily disable high availability so that we could resize the primary database. This is the timeline of events: 09:40. The auto-scaler detected we had less than 25GB of free space and started to increase capacity, but this failed. 12:03. Our database ran out of disk space. 12:03-12:11. We tried to manually increase disk space and failover to another zone, but these both failed. 12:11. We opened a ticket with Google Cloud Support. 12:59. We were on the phone with Google, who provided a detailed explanation of the issue. 13:30. We disabled high availability on our database, and resized it in the working zone. 13:34. Our database was back up again. 13:34-13:50. Clients began to reconnect and send email. 13:59. Normal operations resume, though our database is not high availability for the time being. As a result of this incident, we are going to make several changes: 1\. Tonight, we are going to re-enable high availability on our database. This will cause ~10m of downtime, but we will do it when we have our lowest traffic: 11:50 pm PST. 2\. We have built our own database auto-scaler that will trigger much before the built-in auto-scaler. 3\. We have added alerting on database disk-utilization metrics so that we can pre-empt any similar failures. 4\. We will fix the client so that if the backend is unexpectedly down, it will not log you out so that you can continue to read and process email. 5\. We are going to practice failing over to our secondary read replica. This will be helpful if we are ever again in a situation where both our primary-replica pairs are not functioning. Again, I am truly sorry that this happened. These steps will ensure that we do not have a similar incident in the future. If you have any questions, please just ask: [[email protected]](mailto:[email protected]). Conrad CTO

Time: Feb. 28, 2019, 10:09 p.m.

Status: Resolved

Update: The issue is fully resolved, and all emails that were delayed have now sent. Please don't hesitate to reach out to [email protected] if you have any questions.

Time: Feb. 28, 2019, 9:39 p.m.

Status: Monitoring

Update: Our database is now back up, and all systems are functioning normally. All emails that were sent during the outage should go out shortly. We will now conduct a thorough investigation into what happened, and we will ensure it cannot happen again.

Time: Feb. 28, 2019, 8:32 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Feb. 28, 2019, 8:08 p.m.

Status: Investigating

Update: We're seeing elevated errors with our database. Investigating…

Time: Feb. 8, 2019, 4:22 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: Feb. 8, 2019, 5:45 a.m.

Status: Investigating

Update: This turned out to be a mistake on our end, which is now resolved. Access should be restored everywhere over the next few minutes

Time: Feb. 8, 2019, 5:19 a.m.

Status: Investigating

Update: We are currently investigating an issue where Superhuman is inaccessible from some locales, we're working with our infrastructure provider to debug.

Time: Dec. 12, 2018, 8:47 p.m.

Status: Resolved

Update: We’ve now fixed the underlying issue, and notified all users who’s emails did not send.

Time: Dec. 12, 2018, 4:46 a.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: Dec. 12, 2018, 2:23 a.m.

Status: Investigating

Update: Message sending and signing in should now be working normally. We're still investigating the underlying root causes internally.

Time: Dec. 11, 2018, 11:20 p.m.

Status: Investigating

Update: We're investigating an issue that causes sending and signing in to Superhuman to slow, and sometimes fail. Your emails are safely stored on your device, and will send when the issue is resolved.