Time: April 24, 2023, 7:03 p.m.

Status: Postmortem

Update: On Tuesday April 4th 2023, at 3:25 PM ET, a core database table became overloaded and slowed down or stalled user bulk upload and related activities in the application.  At 10:49 PM ET, an offending query to the database table was identified and altered to restore normal operation.   On Thursday April 6th 2023, a database maintenance was performed to move the affected database to its own cluster, as part of a wholistic fix to this issue.

Time: April 7, 2023, 5:41 a.m.

Status: Resolved

Update: The Development team has applied a fix for Bulk user upload. The platform is now fully operational. Thank you for your patience whilst we investigated this issue. Please contact Support at support.firstup.io if you encounter any further issues.

Time: April 6, 2023, 7:10 p.m.

Status: Monitoring

Update: While bulk upload jobs continue to process as expected, we will be performing an emergency database maintenance tonight as part of a wholistic fix to this issue. Please see https://status.firstup.io/incidents/2z56hkznzsdm for more details.

Time: April 5, 2023, 3:21 a.m.

Status: Monitoring

Update: We have implemented a fix to mitigate this platform degradation, and bulk upload jobs are now being processed as expected. Please allow a few hours for the jobs’ queue to catch up. We will be monitoring the platform’s performance.

Time: April 5, 2023, 1:10 a.m.

Status: Identified

Update: We continue to work on a fix for this issue. Another update will be provided within 1 hour.

Time: April 5, 2023, 12:07 a.m.

Status: Identified

Update: A potential root cause for this issue has been identified, and we are working on a fix for it. Another update will be provided within 1 hour.

Time: April 4, 2023, 11:28 p.m.

Status: Investigating

Update: We are aware of a platform slowdown that has manifested itself as bulk user upload jobs being stuck in processing. We are currently investigating this incident, and will provide a status update within the next 1 hour.

Time: April 20, 2023, 8:02 p.m.

Status: Postmortem

Update: On March 22nd 2023, at 2:25 PM Pacific Time, a misconfigured message queue modification was performed in the EU environment that resulted in campaign jobs pointing to the wrong message queue, and consequently failing to initialize or process as expected. On March 23rd 2023, at 7:30 AM Pacific Time, the erroneous configuration was corrected and campaigns resumed processing as expected. Firstup has taken the following steps to aid in faster campaign processing and delivery issues detection, as well as prevent future recurrences of this incident: * Implemented a canary campaign in the EU environment. * Include a full non-production testing and verification of any message queue configuration changes.

Time: March 23, 2023, 7:09 p.m.

Status: Resolved

Update: Our monitoring has indicated that campaign emails continue to be sent and delivered as expected. This service disruption is now resolved.

Time: March 23, 2023, 3:05 p.m.

Status: Monitoring

Update: A fix has been implemented for the service disruption, and campaigns are now sending and delivering as expected. We will monitor this service closely for the next few hours before declaring this service disruption as resolved.

Time: March 23, 2023, 2:09 p.m.

Status: Identified

Update: We are still working on implementing a fix for this issue, and will provide another update as soon as more information is made available.

Time: March 23, 2023, 12:17 p.m.

Status: Identified

Update: We have identified a possible root cause for the issue affecting campaign deliveries for EU communities. We will provide further updates once we have applied a fix.

Time: March 23, 2023, 12:10 p.m.

Status: Investigating

Update: We are continuing to investigate the service disruption affecting campaigns for EU communities. Our next update will be in 30 minutes.

Time: March 23, 2023, 11:23 a.m.

Status: Investigating

Update: We are investigating a service disruption affecting campaign deliveries to email and push notification channels - for communities hosted on our EU infrastructure. Customers on our EU infrastructure may see that campaigns show as completed in Studio, but that emails and push notifications have not been sent. Note, this does not affect customer communities hosted on our US infrastructure. Our next update will be in 30 minutes.

Time: March 21, 2023, 7:22 p.m.

Status: Postmortem

Update: On March 20th 2023, at 12:40 PM Pacific Time, an upgrade to a core html rendering library was performed, which resulted in New Studio content with Polls showing a spinner and were not functional. This upgrade was rolled back on March 21st 2023, at 10:27 AM Pacific Time to fix the issue, and all affected content with Polls were re-rendered to regain full functionality. Firstup engineering will be enabling a global Deployment Analysis monitor to alert on change in performance or new errors, to aid in faster incident detection and mitigation after a system upgrade.

Time: March 21, 2023, 6:24 p.m.

Status: Resolved

Update: All previously affected Polls content have now been re-rendered. This service disruption is now resolved.

Time: March 21, 2023, 5:25 p.m.

Status: Monitoring

Update: This service disruption has been mitigated, and polls are now available. Engineering is currently working on re-rendering any previously affected polls content. Another update within 1 hour.

Time: March 21, 2023, 5:03 p.m.

Status: Identified

Update: We have identified a potential root cause for this issue, and are working to mitigate the service disruption. Another update in 30 minutes.

Time: March 21, 2023, 4:59 p.m.

Status: Investigating

Update: We are currently investigating reports of Polls not loading as expected in Employee Experience. We will provide an update in 30 minutes.

Time: March 20, 2023, 9:15 p.m.

Status: Resolved

Update: This incident has been resolved. We identified an issue with our vendor causing the reports not to load. The vendor has resolved the issue at 1:25 PM PST. Firstup has confirmed reports are working as expected.

Time: March 20, 2023, 7:06 p.m.

Status: Investigating

Update: As we continue to investigating the intermittent service disruption where Reports are slow to load, or not loading at all, we show that concurrency scaling is working as expected and helping with the Reports load, and most Reports are now loading. Another update in 1 hour.

Time: March 20, 2023, 5:52 p.m.

Status: Investigating

Update: We continue to investigating the intermittent service disruption where Reports are slow to load, or not loading at all. Another update in 1 hour.

Time: March 20, 2023, 4:44 p.m.

Status: Investigating

Update: We continue to investigating a service disruption where Reports are slow to load, or not loading at all. Another update in 30 minutes.

Time: March 20, 2023, 4:44 p.m.

Status: Investigating

Update: We are continuing to investigate the issue. Reports may continue to load slowly, or may not load at all. We will provide an update in 30 minutes.

Time: March 20, 2023, 4:07 p.m.

Status: Investigating

Update: We are aware of and currently investigating a service disruption where Reports are slow to load, or not loading at all. We will provide an update in 30 minutes.

Time: June 16, 2023, 12:17 a.m.

Status: Resolved

Update: Marking this issue as resolved. This incident was opened initially as risk-based, and then evolved in to two separate service disruptions that were subsequently mitigated. We do not feel there is any near-term risk of recurrence now having re-established contacts with the social media platform, good progress towards moving to the updated API, and a developer subscription under Twitter's new service plans.

Time: May 3, 2023, 12:20 a.m.

Status: Identified

Update: Twitter feeds should now all be refreshed, and the incident has once again been mitigated. Work continues for the long-term fix on the new version of Twitter's API and service tier, but we believe any recurrent disruption of service between now and then to be highly unlikely. Moving incident back to monitoring state and marking all components fully operational.

Time: May 1, 2023, 6:14 p.m.

Status: Identified

Update: We are aware of Twitter feeds not refreshing/updating as expected since 4/25. This is due to a change from Twitter and we are working to resolve it. Currently, we do not have an ETA but we will keep you up to date as Firstup makes progress.

Time: April 7, 2023, 9:44 p.m.

Status: Monitoring

Update: Impact has been mitigated, and all connected Twitter feeds have been successfully refreshed. There remains a high probability of a recurrence between now and end of next week, when we have fully implemented and tested Twitter integration with their new API. In the interim, we will continue to monitor for any disruptions and do what we can to minimize any service impact to the Firstup platform.

Time: April 7, 2023, 6:59 p.m.

Status: Investigating

Update: Moving this incident out of a risk-only designation. We have observed unexpected failures with Twitter API calls. We are currently investigating the reason behind the errors, but all content pulled in from Twitter sources for content feeds will fail in the interim.

Time: April 4, 2023, 8:24 p.m.

Status: Identified

Update: Twitter has announced new access tiers to their platform API as well as a stated requirement for all developers to port any applications to the new version 2 of the API within 30 days or less. We have spun up a developer work stream to make these updates so that connected Twitter accounts can remain a viable a source for content feeds. Firstup remains committed to providing the best functionality possible that the new Twitter API allows.

Time: Feb. 14, 2023, 9:07 p.m.

Status: Investigating

Update: Twitter have sent a further update out that this change has been delayed for a few more days - no definite date has been provided as yet. Our Engineering Team will be reviewing all options from Twitter when fully known.

Time: Feb. 9, 2023, 3:06 a.m.

Status: Investigating

Update: Twitter has moved this change window to Feb 13, 2023 so we do not expect any potential disruption of service tomorrow Feb 9, 2023. We will keep this incident open for communication updates however as the situation progresses.

Time: Feb. 8, 2023, 10:07 p.m.

Status: Investigating

Update: We are initiating a platform incident for risk, based on the contents of a public tweet (https://twitter.com/TwitterDev/status/1621026986784337922) from the official Twitter Dev account, purporting that access to the Twitter API will require a "paid basic tier" as of February 9th. We have not received any further communication through alternate channels or via Twitter's Developer Team detailing exactly when/how this will be enforced and what it will ultimately look like. Our Engineering Team remains at the ready to respond as necessary to make sensible adjustments that may be required to retain current interoperability between Twitter and the Firstup platform. Further information will be posted in this platform-specific status page as it becomes available. If access to the Twitter API should be unexpectedly severed, that could impact content feeds configured to pull Twitter content in to Programs, but native sharing of content out to Twitter should continue to function as it does not utilize the Twitter Developer API.