Time: June 16, 2023, 9:39 p.m.

Status: Postmortem

Update: On 06/14/2023 starting at 8:00 AM PT to 3:15 PM PT, our Firstup alert monitoring system promptly notified our engineering team about campaigns queuing in the system. Our engineering team identified a spike in our campaign engine planner, caused by a data task aimed at improving content rendering, inadvertently led to delays across the platform. To minimize future impact, safeguards and policies have been implemented, including running these tasks during off-peak hours. Regular monitoring, testing, and documentation will be conducted to prevent similar issues and ensure optimal platform performance.

Time: June 15, 2023, 10:04 p.m.

Status: Resolved

Update: The service disruption to Connect has been resolved. Thank you for your patience whilst we carried out our investigation. Please contact Support at support.firstup.io should you encounter any further issues.

Time: June 14, 2023, 11:29 p.m.

Status: Monitoring

Update: We have identified the root cause and have taken steps to prevent the issue from reoccurring. We will continue to monitor for the next 24 hours. Next update will be 6/15/2023 at 5 PM PT

Time: June 14, 2023, 10:12 p.m.

Status: Monitoring

Update: We show that the campaigns processing queue has caught up and all campaigns should have been delivered. We continue to work on identifying the root cause for this service disruption, and will provide additional information as more information is made available. We will be monitoring this service in the time being.

Time: June 14, 2023, 9:47 p.m.

Status: Identified

Update: As we continue investigating the root cause of this incident, our engineers were able to remove some Database locks believed to have contributed to the campaigns delivery delays as a mitigation step. We show that the campaigns processing queue has started to catch up, and some impacted customers may have seen older campaigns delivered already. However, we do not have an ETA yet as to when all published campaigns will finish processing. We will provide you with another update within the next 1 hour.

Time: June 14, 2023, 8:31 p.m.

Status: Investigating

Update: We continue to investigate the delays in campaign deliveries. The scope of the delays still remain as affecting both Scheduled and Immediate campaigns. Another update will be provide at or before 2:30 PM PT.

Time: June 14, 2023, 7:37 p.m.

Status: Investigating

Update: We continue to investigate the delay and working to identify the root cause. Scheduled and Immediate campaigns are affected. Next Update: 1:30 PM PST

Time: June 14, 2023, 6:50 p.m.

Status: Investigating

Update: We continue to investigate the delay and working to identify the root cause.

Time: June 14, 2023, 5:57 p.m.

Status: Identified

Update: We have identified the delays are still occurring for Campaigns. We are continuing to work to resolve the delays.

Time: June 14, 2023, 5 p.m.

Status: Monitoring

Update: We have identified campaigns scheduled between 8:00 AM PST to 9:00 AM PST did not get delivered. Engineering is working to backfill the delayed Campaigns to be delivered. All services are now operational and we will continue to monitor.

Time: June 16, 2023, 8:14 p.m.

Status: Postmortem

Update: Our upstream vendor - AWS - has provided the following root cause analysis to the event on their platform that caused multiple dependent services to become degraded or unavailable on Firstup platform on Tuesday, June 13th 2023.  More specifically, the Firstup platform leverages the AWS Lambda service to perform routine functions that include:  * Scanning attachments for viruses * Finding program logins using community codes * Calculating user activity metrics * Processing user sync files Additionally, we observed that the AWS Mediaconvert service was impaired during this time which impacted Firstup Customers’ ability to upload video files. This vendor specific RCA is being included for reference purposes only and is included inline below: _“Between 11:49 AM PDT and 3:37 PM PDT, we experienced increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers \(including through API Gateway\) and indirectly through the use of other AWS services. Additionally, customers may have experienced authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also have experienced issues when attempting to initiate a Call or Chat to AWS Support. As of 2:47 PM PDT, the issue initiating calls and chats to AWS Support was resolved. By 1:41 PM PDT, the underlying issue with the subsystem responsible for AWS Lambda was resolved. At that time, we began processing the backlog of asynchronous Lambda invocations that accumulated during the event, including invocations from other AWS services. As of 3:37 PM PDT, the backlog was fully processed. The issue has been resolved and all AWS Services are operating normally.”_ Please see [here](https://health.aws.amazon.com/health/status) for more information.

Time: June 13, 2023, 11:46 p.m.

Status: Resolved

Update: Following confirmation from our upstream vendor that their issue has been resolved, we can also confirm that all dependent Firstup services are up and running as expected. This incident will now be marked as resolved.

Time: June 13, 2023, 9:23 p.m.

Status: Monitoring

Update: Our vendor has stated all services have recovered. Firstup is starting to see services recover and we will continue monitor.

Time: June 13, 2023, 9:04 p.m.

Status: Identified

Update: Our third party vendor continues to work to resolve the issue. Next Update will be 4:00 PM MST

Time: June 13, 2023, 8:12 p.m.

Status: Identified

Update: Due to an issue with our upstream vendor our Firstup customer may experience platform service degradation on multiple services including but not limited to video processing and reporting. Next update will be 3:00 PM MST

Time: June 9, 2023, 10:37 p.m.

Status: Postmortem

Update: On 6/1/2023 at 10:55 AM PST, Firstup platform had an increase in commenting on posts which put stress on our database. Due to a sudden influx of a high volume of comments a safety mechanism was automatically triggered which limited additional comments from being able to post. This mechanism was put in place from platform incident on 2/2/2023 to reduce extra extraneous load on the database caching service in order to prevent impact from spreading to other \(non-commenting related\) micro services. We have implemented a performance enhancement by removing a bottleneck in the process of users submitting comments.

Time: June 2, 2023, 1:11 p.m.

Status: Resolved

Update: Our extensive monitoring has indicated that all affected systems have remained stable and fully functional. This incident will now be resolved, and a root cause analysis will be provided once a postmortem has been completed.

Time: June 1, 2023, 7:30 p.m.

Status: Monitoring

Update: A fix has been implemented and we are monitoring the results.

Time: June 1, 2023, 6:39 p.m.

Status: Identified

Update: We have identified the issue that is causing this service degradation with unable to comment, and are currently working on a fix for it. Another update will be provided within the next 1 hour.

Time: June 1, 2023, 6:06 p.m.

Status: Investigating

Update: We are currently investigating reports of unable to post comments on Microapps and Content

Time: June 9, 2023, 7:11 p.m.

Status: Postmortem

Update: On May 31st 2023, at 10:31 AM Pacific Time, a configuration change intended to add flexibility in video display scaling resulted in prior videos to not be playable for Safari and iOS users due to a change in the naming convention of the stored video assets. On June 1st 2023, at 1:21 PM Pacific Time, a fix was developed and deployed, and prior videos were backfilled by 2:10 PM Pacific Time to utilize the correct naming convention, allowing all videos to resume playing as expected. Our QA testing has been revamped to include testing video assets and their use cases in the platform before any configuration change deployment, in an effort to prevent this scenario from happening again.

Time: June 2, 2023, 8:54 p.m.

Status: Resolved

Update: Our monitoring has indicated that mobile videos have been rendering as expected after the deployed fix. This incident will now be placed in a resolved status.

Time: June 1, 2023, 9:04 p.m.

Status: Monitoring

Update: The developed fix for this issues has been vetted and deployed in the production environment, and mobile videos should now be rendering as expected. We will be placing this service under monitoring to ensure stability.

Time: June 1, 2023, 7:56 p.m.

Status: Identified

Update: A fix for this issue has been developed, and is currently undergoing QA testing and final approval. Another update will be provided within 1 hour.

Time: June 1, 2023, 7:10 p.m.

Status: Identified

Update: We continue to work on a fix for this issue, and will provide another update within the next 1 hour.

Time: June 1, 2023, 6:07 p.m.

Status: Identified

Update: We continue to work on a fix for this issue, and will provide another update within the next 1 hour.

Time: June 1, 2023, 5:07 p.m.

Status: Identified

Update: We have identified a potential configuration issue that is causing this service degradation, and are currently working on a fix for it. Another update will be provided within the next 1 hour.

Time: June 1, 2023, 4:32 p.m.

Status: Investigating

Update: We are currently investigating reports where videos in mobile devices are not rendering as expected, with and error message “The media could not be loaded, either because the server or network failed, or because the format is not supported.” We will provide you with an update within the next 1 hour.

Time: June 30, 2023, 7:52 p.m.

Status: Postmortem

Update: On May 22nd 2023, one of our caching servers received an unexpected amount of load. This caused the depending services to be impacted, cascading to other services as well. As mitigation step, we added a circuit breaker in order to limit the scope of impact on dependent services, as well as increased the memory in our caching server to promptly restore services. As a long term preventative measure, we have increased the time-to-live for cashed data, which reduces the frequency of calls to the caching server, hence reducing the load on the server.

Time: June 26, 2023, 6:35 p.m.

Status: Resolved

Update: All services affected by this service degradation have remained available and fully functional. This is now considered resolved.

Time: May 22, 2023, 5:54 p.m.

Status: Monitoring

Update: All database connections have been restored, and all services are now available and accessible. We will be placing this incident under monitoring

Time: May 22, 2023, 5:31 p.m.

Status: Identified

Update: As we continue to work on restoring all database connections, some services such as Web Experience are back up and running. Another update will be provided within 30 minutes.

Time: May 22, 2023, 5:01 p.m.

Status: Identified

Update: A database connection issue has been identified as the cause of the multiple services being degraded. We are working to mitigate this issue. Another update will be provided within 30 minutes.

Time: May 22, 2023, 4:35 p.m.

Status: Investigating

Update: We are investigating reports of multiple services being degraded. These may include - among others - Microapps and Web Experience. We will provide an update within 30 minutes.