Populi Status: Check if Populi down or having an outage.

Component	Status
API	Active
Email	Active
Main Application	Active
Media Encoding	Active
Search	Active
SMS	Active

Internet routing issues

Description: Everything continues to look good, so we're closing this incident.

Status: Resolved

Impact: Major | Started At: July 22, 2024, 11:53 p.m.

Updates:

Time: July 23, 2024, 3:18 a.m.

Status: Resolved

Update: Everything continues to look good, so we're closing this incident.
Time: July 23, 2024, 1:12 a.m.

Status: Monitoring

Update: Traffic from all internet providers looks good but we will continue to monitor for a while before resolving this incident... thanks for your patience!
Time: July 23, 2024, 12:41 a.m.

Status: Monitoring

Update: The issue appears to have been with an Internet connection into Populi's primary datacenter - failover has occurred and we're seeing usage statistics return to normal. We'll continue to monitor....
Time: July 22, 2024, 11:53 p.m.

Status: Investigating

Update: Populi is currently unavailable from several larger network providers such as Comcast... this appears to be a widespread Internet problem and our datacenter is fine (just unreachable for many users). We are continuing to investigate and will post updates here...

Server crash this morning

Description: At 9:35am Pacific this morning, a critical caching server crashed and Populi was unavailable to the majority of our users for about 6 minutes as we failed over to another server. Sorry about that!

Status: Resolved

Impact: Major | Started At: June 20, 2024, 4:30 p.m.

Updates:

Time: June 20, 2024, 5:58 p.m.

Status: Resolved

Update: At 9:35am Pacific this morning, a critical caching server crashed and Populi was unavailable to the majority of our users for about 6 minutes as we failed over to another server. Sorry about that!

Populi unavailable for a large subset of users

Description: From roughly 4:20am - 4:50am Pacific time a large subset of Populi users were unable to access the service. This was because a weekly database consistency check started running at roughly 3am and eventually locked up a critical database server, leading to widespread system failure. We eventually managed to restart the database and restore service, and we'll be taking a close look at that check today to prevent it causing issues in the future!

Status: Resolved

Impact: None | Started At: July 3, 2023, 11:30 a.m.

Updates:

Time: July 3, 2023, 3:32 p.m.

Status: Resolved

Update: From roughly 4:20am - 4:50am Pacific time a large subset of Populi users were unable to access the service. This was because a weekly database consistency check started running at roughly 3am and eventually locked up a critical database server, leading to widespread system failure. We eventually managed to restart the database and restore service, and we'll be taking a close look at that check today to prevent it causing issues in the future!

Populi logins aren't working for some users

Description: This incident has been resolved - it was a timezone bug and we'll do our best to make sure it doesn't happen next year!

Status: Resolved

Impact: Major | Started At: March 12, 2023, 2:56 a.m.

Updates:

Time: March 12, 2023, 3:33 a.m.

Status: Resolved

Update: This incident has been resolved - it was a timezone bug and we'll do our best to make sure it doesn't happen next year!
Time: March 12, 2023, 3:14 a.m.

Status: Monitoring

Update: Daylight savings time is tricky and apparently broke logins for a bit - everything appears to be back to normal but we're keeping an eye on things...
Time: March 12, 2023, 2:56 a.m.

Status: Investigating

Update: We are currently investigating this issue.

Networking issues

Description: From roughly 11:30am to 12:30pm, some requests to Populi experienced delayed network connections or timed out - our apologies for the inconvenience! We've reconfigured some things and raised load-balancing limits to allow more traffic during peak times.

Status: Resolved

Impact: None | Started At: Oct. 17, 2022, 6:30 p.m.

Updates:

Time: Oct. 17, 2022, 10:54 p.m.

Status: Resolved

Update: From roughly 11:30am to 12:30pm, some requests to Populi experienced delayed network connections or timed out - our apologies for the inconvenience! We've reconfigured some things and raised load-balancing limits to allow more traffic during peak times.