Time: Nov. 11, 2023, 3:13 a.m.

Status: Resolved

Update: This issue has been resolved with the software release earlier today that addressed the underlying behavior on the new UI post details page when video content blocks were contained on the page. All components marked fully operational.

Time: Nov. 9, 2023, 11:07 p.m.

Status: Identified

Update: The root cause of this behavior has been identified and a software fix has been identified and is in progress.

Time: Nov. 9, 2023, 6:08 p.m.

Status: Investigating

Update: We are continuing to investigate this issue.

Time: Nov. 9, 2023, 5:19 p.m.

Status: Investigating

Update: We are currently investigating multiple reports of a display behavior following the recent UI update this week. We are working on a hot fix that will fully address the issue and also have a mitigation measure that we can put in place for individual communities experiencing the problem. For Customers observing this behavior, please open a support ticket at https://support.firstup.io and reference Known Issue ID CSUP-3534. Additional background information is also available on the known issue article at https://support.firstup.io/hc/en-us/articles/18926725571223 .

Time: Feb. 29, 2024, 7:36 p.m.

Status: Postmortem

Update: The root cause of this service disruption has been identified to be the same as the one posted in another incident’s postmortem. Please follow this [link](https://status.firstup.io/incidents/4kpmpkmvkrz0) for additional details.

Time: Jan. 3, 2024, 2:55 a.m.

Status: Resolved

Update: Marking this incident resolved and all components fully operational. We have not seen any further recurrences of this issue and all services have remained stable since.

Time: Nov. 9, 2023, 6:07 p.m.

Status: Monitoring

Update: We are continuing to monitor for any further issues.

Time: Nov. 7, 2023, 8:49 p.m.

Status: Monitoring

Update: All services have been restored and functioning as expected. We will be placing this incident under monitoring for now.

Time: Nov. 7, 2023, 8:35 p.m.

Status: Investigating

Update: We are still investigating this issue. Another update in 30 minutes.

Time: Nov. 7, 2023, 8:06 p.m.

Status: Investigating

Update: We are still investigating this issue. Another update in 30 minutes.

Time: Nov. 7, 2023, 7:35 p.m.

Status: Investigating

Update: We are seeing a recurrence of this issue, and are currently looking into it. Another update in 30 minutes.

Time: Nov. 2, 2023, 4:25 p.m.

Status: Monitoring

Update: All services have been restored and functioning as expected. We will be placing this incident under monitoring for now.

Time: Nov. 2, 2023, 4 p.m.

Status: Investigating

Update: We are currently investigating reports of general platform performance degradation, which may include general access to Studio and Employee experience. We will provide an update within 30 minutes.

Time: Nov. 9, 2023, 5:25 p.m.

Status: Postmortem

Update: On October 18th, 2023, a code change was released to address an issue where a campaign could not be edited because an author was not assigned to the campaign during its creation. This release caused this incident where user generated content auto-selected the default-community-author as the author of the content, instead of the actual user the created the content. We identified that a feature flag that selected the content creator as the author of the content was turned off within the released code, resulting in author selection defaulting to the default-community-author. The feature flag was turned on at 6:20PM ET to mitigate this incident, and subsequent user generated content used the correct content author. We also identified that 108 pieces of content were affected by this incident and ran a task to update them with the correct authors. To prevent this incident from happening again, we have included feature flag testing and verification in our code pre-release QA processes.

Time: Oct. 23, 2023, 5:53 p.m.

Status: Resolved

Update: All systems have remained operational and functioning as expected. Also, any posts that were affected have been re-authored with their correct authors. This service degradation will now be marked as resolved.

Time: Oct. 18, 2023, 10:27 p.m.

Status: Monitoring

Update: The fix for this service degradation has been deployed in the production environment, and normal operations are be restored. We will be monitoring the systems involved for now.

Time: Oct. 18, 2023, 9:38 p.m.

Status: Identified

Update: The fix for this service degradation has been verified in our test environment, and we are currently working on deploying it in the production environment. We will provide another update once that is completed.

Time: Oct. 18, 2023, 8:24 p.m.

Status: Identified

Update: We are still testing a potential fix for this service degradation in out test environment. Another update within the hour.

Time: Oct. 18, 2023, 7:20 p.m.

Status: Identified

Update: We are still testing a potential fix for this service degradation in out test environment. Another update within the hour.

Time: Oct. 18, 2023, 6:15 p.m.

Status: Identified

Update: A potential root cause for this service degradation has been identified, and we are working on a fix in our test environment. Another update will be provided within the hour.

Time: Oct. 18, 2023, 5:11 p.m.

Status: Investigating

Update: We continue investigating this service degradation. Another update within the hour.

Time: Oct. 18, 2023, 3:45 p.m.

Status: Investigating

Update: We are currently investigating reports where user generated content are defaulting to community author. We will provide an update within the hour.

Time: Dec. 7, 2023, 6:53 p.m.

Status: Postmortem

Update: On Monday October 9th 2023, a large rake task was ran on the platform that took too long to process. This caused a backup in the user import queue, resulting in user imports being stuck in a processing state while waiting for the queue to drain. While investigating the issue, the rake task completed running, and the queue was able to accept and process user import requests as normal. We have since made changes to the batch sizes allowed in the queue to prevent large tasks from overwhelming the queue, as well as implemented policies where rake tasks would be run outside of normal business hours to prevent a recurrence of this service disruption.

Time: Oct. 10, 2023, 7:29 p.m.

Status: Resolved

Update: All systems have remained active and fully functional, and bulk user uploads have been processing as expected. This service degradation will now be marked as resolved. We thank you for your patience while we worked to restore this service.

Time: Oct. 9, 2023, 10:42 p.m.

Status: Monitoring

Update: This service degradation has now been mitigated, and bulk user uploads are now processing as expected. We will be placing this incident under monitoring for now, just to make sure everything remains fully functional.

Time: Oct. 9, 2023, 10:30 p.m.

Status: Identified

Update: We continue to work towards a solution to the identified potential cause of this service degradation. Another update will be provided within the hour.

Time: Oct. 9, 2023, 9:30 p.m.

Status: Identified

Update: We have identified a potential cause for this service degradation, and are working towards a solution. Another update will be provided within the hour.

Time: Oct. 9, 2023, 9:06 p.m.

Status: Investigating

Update: We are aware of, and currently investigating reports where bulk user imports are stuck in processing. We will provide another update within the hour.

Time: Aug. 28, 2023, 7:47 p.m.

Status: Postmortem

Update: On August 1st 2023, at 8:00 PM PDT, a software enhancement was released as part of the implementation of Campaign Guaranteed Delivery to ensure timely communication delivery within specified timeframes. Prior to this release, and during our QA testing of the software enhancement, a use case where a campaign’s intended audience was considerably large was not tested. Given that large campaign audiences take some time to process, Override campaigns that had a large audience were blocked if the audience “processing end-time” conflicted with the actual “set sending time” for the campaign. ‌ On August 2nd 2023, at 9:50 AM PDT, the software enhancement was rolled back to mitigate this issue, and the affected campaigns were replanned to resend their communications immediately. ‌ To prevent a recurrence of this issue, we have added pre-release testing with large campaigns and audiences as part of out QA process.

Time: Aug. 2, 2023, 9:29 p.m.

Status: Resolved

Update: The affected component has remained stable and all campaign email services are fully functional. This service disruption is now resolved.

Time: Aug. 2, 2023, 4:51 p.m.

Status: Monitoring

Update: We have just finished rescheduling email communications for immediate delivery to any users that have not already received or viewed the content. Users should start seeing emails delivered if the affected campaign(s) have not been cancelled.   Please note that new campaigns are not be affected by this service disruption moving forward. We will be placing this service disruption under monitoring for now.

Time: Aug. 2, 2023, 3:51 p.m.

Status: Identified

Update: We are still working on resending any campaign emails that did not send out. If you believe your campaign may have been affected by this service disruption, and do not wish for its email communication to be resent, please notify us as soon as possible (if you haven’t done so already). Another update will be provided in 30 minutes.

Time: Aug. 2, 2023, 3:13 p.m.

Status: Identified

Update: We have identified a potential root cause for this service disruption, and have determined that only campaigns that were published after 12:00AM ET today are in the scope of this service disruption. We are currently working on resending any campaign emails that did not send out. If you believe your campaign falls within the above scope, and do not wish for its emails to be resent, please notify us as soon as possible (if you haven’t done so already). Another update will be provided in 30 minutes.

Time: Aug. 2, 2023, 2:53 p.m.

Status: Investigating

Update: We are aware of and currently investigating reports where campaigns that were published today to send email notifications immediately have not yet delivered the emails to their audiences. We will provide another update in 30 minutes.