Time: April 21, 2020, 6:51 p.m.

Status: Postmortem

Update: Sorry about that, everyone! We’re in the middle of upgrading our email sending providers and adding DMARC support, and unfortunately the drastic usage spike we’ve been experiencing lately caused us to unexpectedly run up against a daily email sending cap with our new email provider \(Amazon\). That’s why emails were suddenly being delayed, and when we managed to work with Amazon to get our usage limit raised, a bug in our retry sending code \(that’s supposed to retry sending email in case a provider goes down, sporadic network failure, etc\) resulted in duplicate emails being sent to many users. This bug was a tricky one because it only manifested when we bumped up against the usage cap AND had production levels of email flowing through the system - and unfortunately that means we didn’t catch it during testing. We’ve temporarily fallen back to sending through our old email provider, and after we purge duplicates from the new queue and get any messages that should be sent flushed out, we’ll confirm the retry logic bug is fixed and carefully begin transitioning to Amazon again, after working with them to raise our sending limit yet again to give us lots of headroom this time. Thanks for your patience as we continue to deal with abnormally high usage and still try to push the product forward at the same time!

Time: April 21, 2020, 6:45 p.m.

Status: Resolved

Update: This incident has been resolved... there are a handful of backed up emails we'll be sending throughout the day, but all new email sent during the last several hours looks good.

Time: April 21, 2020, 4:26 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: April 21, 2020, 3:41 p.m.

Status: Identified

Update: The issue has been identified and a fix is being implemented.

Time: April 21, 2020, 3:40 p.m.

Status: Investigating

Update: Due to increased usage of Populi's online learning features right now, we hit a limit with our email provider that led to messages being marked as "rejected" so we retried sending them.... however, when they raised our limit the retires are flooding in and sending multiple copies of each message... sorry for the hassle! We're working with them to halt the flow of repeat email as soon as possible....

Time: April 8, 2020, 12:33 a.m.

Status: Resolved

Update: This incident has been resolved.

Time: April 8, 2020, 12:01 a.m.

Status: Identified

Update: We are continuing to work on a fix for this issue.

Time: April 7, 2020, 11:57 p.m.

Status: Identified

Update: We pushed an update to a low-level encryption library that isn't playing nicely with API token validation... we're reverting and should be back to normal in a few minutes. Sorry for the hassle!

Time: Jan. 24, 2020, 12:28 a.m.

Status: Resolved

Update: Sorry folks - a network administrator was making a configuration change to a firewall and made a mistake. Populi was down for approximately 2 minutes until they were able to revert the change - thanks for your patience!

Time: Oct. 22, 2019, 11:47 p.m.

Status: Resolved

Update: We're not seeing the errors from S3 anymore, so marking this resolved.

Time: Oct. 22, 2019, 7:01 p.m.

Status: Identified

Update: Populi uses Amazon S3 to store files, and it looks like a high percentage of our requests to S3 attempting to store files are failing right now.

Time: June 1, 2019, 6:46 p.m.

Status: Resolved

Update: This incident has been resolved.

Time: June 1, 2019, 6:24 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: June 1, 2019, 5:56 p.m.

Status: Identified

Update: We've received reports that recently changed people aren't being reflected properly in search results. We've identified the issue and are implementing a fix now.