Time: June 12, 2023, 7:42 p.m.

Status: Postmortem

Update: On Tuesday May 9th 2023, a configuration change was made in our Database \(DB\) environment to increase DB performance. On Friday May 12th 2023, the above change surfaced an issue in our Bulk Upload system where queries would run for too long, resulting in DB locking. As a mitigation step, these locks were cleared allowing Bulk Upload jobs to continue processing successfully. The root cause was later identified to be a missing DB index, and on Monday May 22nd 2023, the missing DB index was added to resolve the DB locking issue.

Time: June 2, 2023, 2:45 p.m.

Status: Resolved

Update: After extensive monitoring of the services that were affected by this service disruption, they all have remained available and operational. This incident is now being placed in a resolved status.

Time: May 22, 2023, 2:58 p.m.

Status: Monitoring

Update: The proposed hot fix for this issue has been confirmed and approved, and has now been deployed in the production environment. For any failed bulk import jobs, the files can now be re-uploaded for processing. We will be placing this issue under monitoring for now.

Time: May 22, 2023, 2:40 p.m.

Status: Identified

Update: We have identified a potential root cause for this issue, and a hot fix for it is now being tested in a staging environment. Once confirmed and approved, it will be deployed in the production environment. We will provide you with another update within 1 to 2 hours.

Time: May 22, 2023, 1:35 p.m.

Status: Investigating

Update: We continue to investigate this recurrence, and will provide another update within 1 to 2 hours.

Time: May 22, 2023, 11:08 a.m.

Status: Investigating

Update: We are investigating a recurrence of the issue affecting user imports. At this time, bulk user imports are failing with an 'Internal Server Error' for customers with communities in our US Data Center. We will provide an update within 1 hour.

Time: May 12, 2023, 4:30 p.m.

Status: Monitoring

Update: User Bulk Import service is still running as expected. We will place this service under monitoring, and provide an update if and when new information is made available.

Time: May 12, 2023, 2:01 p.m.

Status: Identified

Update: This issue has now been mitigated, and user bulk imports are now processing as expected. We are working on backfilling any previously failed jobs. An update will be provided once this is completed.

Time: May 12, 2023, 1:20 p.m.

Status: Investigating

Update: We are aware of, and are currently investigating an issue where bulk user upload jobs are failing with an internal server error in our US datacenter. We will provide an update within 1 hour.

Time: May 5, 2023, 4:55 p.m.

Status: Resolved

Update: The issue causing service disruption to Campaigns has been resolved. Thank you for your patience whilst we carried out our investigation. Please contact Support at support.firstup.io should you encounter any further issues.

Time: May 4, 2023, 6:03 p.m.

Status: Monitoring

Update: We have applied a fix for the issue affecting Campaigns and are currently monitoring. We will continue to monitor through 5/5/2023 11:00 AM PST. Next Update will be 11:30 AM PST 5/5/2023

Time: May 4, 2023, 5:10 p.m.

Status: Monitoring

Update: We will be deploying a hot fix at 10:30 AM PST to resolve this issue. Any scheduled campaigns between 10:30 AM PST to 11:00 AM PST may be delayed.

Time: May 4, 2023, 3:15 p.m.

Status: Monitoring

Update: This issue has been identified and engineering is working on a fix. Currently, delivery of campaigns are working as expected. We may experience deliver issues until a fix is implemented. Engineering is monitor for further errors and will resolve has they occur. Customer's can resend the campaigns as the service is back up.

Time: May 3, 2023, 6:38 p.m.

Status: Postmortem

Update: On April 17th 2023,  starting at 1:00 AM ET, campaign email delivery stopped sending emails from the campaigns because the process that was responsible for sending out emails was running out of memory. This happened because of a specific campaign that was scheduled at that time. To prevent this from happening in the future, we have increased the memory available, set up alerts to notify us when memory usage approaches a certain limit, and decreased the overall amount of memory being used. We apologize for any inconvenience this may have caused and are committed to ensuring a seamless experience for our customers.

Time: April 25, 2023, 6:18 p.m.

Status: Resolved

Update: Our extensive monitoring has indicated that all affected systems have remained stable and fully functional. This incident will now be resolved, and a root cause analysis will be provided once a postmortem has been completed.

Time: April 17, 2023, 1:47 p.m.

Status: Monitoring

Update: A fix for this service disruption has been implemented, and campaign emails have began sending out again. Please allow a few moments for the messages queue to catch up with the backlog. We will be placing the affected systems in active monitoring to ensure stability.

Time: April 17, 2023, 1:30 p.m.

Status: Identified

Update: We have identified a potential root cause for this service disruption, and are working on a fix for it. We will provide another update in 30 minutes.

Time: April 17, 2023, 1 p.m.

Status: Investigating

Update: We are investigating a service disruption affecting Campaigns. Campaigns (both those scheduled and set to send immediately) may not have sent any communication out to users. Our next update will be in 30 minutes.

Time: April 22, 2023, 5:16 p.m.

Status: Postmortem

Update: ## **Summary:** When Microapps refreshed content after 10:07 AM PT on Thursday, April 13th , a "content not found" error was displayed unexpectedly.  Total duration of the platform incident was 26 minutes, but impact could have been extended beyond then in certain instances due to localized content caching behavior.  The problem was the result of an unexpected side effect stemming from an unrelated infrastructure change that was not properly scheduled or fully tested before being pushed to the production network. ## **Impact:**  All Microapps showed the same “content not found” error if content was refreshed within the affected time window.  This would have been similar irrespective of how the Microapps are deployed \(embedded, digital signage, etc.\).  ## **Root Cause:**  An infrastructure change to global load balancing rules resulted in all Microapp network traffic being redirected to an alternate Firstup service endpoint not configured to handle Microapp requests.  As a result, a fallback HTML page was returned instead which visually rendered the ‘no content found’ error, and was subsequently cached by many local browsers. ## **Mitigation:** The errant global load balancer configuration was reverted immediately after a flood of alerts starting coming in at 10:10 AM PT.  This change ensured new Microapp requests that came in would be serviced by the appropriate network endpoint.  Cloud infrastructure network caches were subsequently purged and work was coordinated with individual Customers on techniques to purge local cache entries containing the fallback HTML page.  In extreme circumstances, the Microapps were recreated and deployed to a new URL that had not been previously cached \(such as in the case with some embedded uses within Sharepoint\). ## **Recurrence Prevention:** Two primary actions have been taken to prevent both the root cause, and reduce the duration of impact. First, infrastructure change management policy and process will be followed going forward for any updates to the global load balancer and will include full non-production testing and verification, and a planned maintenance window in non-emergency situations for future updates. Second, is a code change to Microapps that will append a version hash to the end of the response object, making it easier to bypass cached content during software updates or other potential unexpected events that result in unexpected content being cached at the client.  This change is being thoroughly tested now, and will be released during a normal software change window given the low risk of recurrence, but high long term value.

Time: April 22, 2023, 5:16 p.m.

Status: Resolved

Update: Remaining Microapps in scope of this incident are all confirmed to be displaying expected content feeds. Marking all components as fully operational and resolving this incident.

Time: April 14, 2023, 5:37 p.m.

Status: Monitoring

Update: A fix for this service disruption has now been deployed in the production environment. A browser refresh or clicking on the “Try Again” link in the error message is required to clear this issue. We will monitor the affected systems for performance and stability.

Time: April 14, 2023, 5:06 p.m.

Status: Identified

Update: We have identified a potential root cause for this service disruption, and are working on vetting a fix for it. Another update will be provided within 1 hour.

Time: April 14, 2023, 4:55 p.m.

Status: Investigating

Update: We continue to investigate this service disruption. Another update will be provided within the next 1 hour.

Time: April 14, 2023, 3:58 p.m.

Status: Investigating

Update: We are currently investigating reports of Microapps for Sharepoint and digital signage showing "No content found”. An update will be provided within the next 1 hour.

Time: April 13, 2023, 6:39 p.m.

Status: Resolved

Update: We have applied a fix for the issue affecting Microapps and have forced cache clearing from the server side. This should resolve "content not found" error. If users are still receiving "content not found", carry out the following steps: 1. Reload the page (if the issue still persist step two needs to be carried out). 2. Clear cache locally on the browser will resolve the issue. This incident is now resolved. Please contact Support at support.firstup.io should you encounter any further issues.

Time: April 13, 2023, 6:06 p.m.

Status: Identified

Update: We have identified a root cause for the issue affecting Microapps and a fix has been implemented. Some customer's may still see content not found and we are working to fix the cache to resolve the issue. Customer's may also attempt to clear cache from the browser itself.