Time: June 26, 2023, 5:08 p.m.

Status: Resolved

Update: What a day. AWS having issues like this, where it actually makes mainstream news, is unbelievably rare. Having said that, it's given us some things to think about. We can't recall an outage with such an impact since 2012. Following this, it's given us a +100 to look into multi-region availability. Having said that, it looks like our data collector remained available throughout. Early on in the problem, we saw a large queue backlog, but it was quickly handled. And our current visitors continued to work throughout the day. We are currently working through millions of pageviews in our backlog which need to be aggregated into your dashboard stats. The good news is that it seems like we weren't impacted severely. One final note we'd like to make regarding the pageview backlog. We've had numerous backlogs with pageviews since our inception. We're pleased to share that we are almost ready to move to Elasticsearch, where we'll see no more backlogs and real-time data reporting. This will be a huge win for us and our customers, and we can't wait to have it completed. Update Interestingly, since we identified the attack, our pageview collector seems to have been running just fine. Looking at our database, we can see a large backlog of data, and the issue seems to be that pageviews aren't aggregating to your dashboard. Once AWS fixes the problems they're having, the aggregation will resume. One thing we want to make clear, this has nothing to do with the recent DDoS attacks we were receiving, this is related to a large scale AWS problem. These regional outages are incredibly rare and, unfortunately, we are in a position where we simply have to wait. As we say, we seem to only have been mildly impacted. Identified Our service provider is currently having issues and we are looking into it.

Time: June 26, 2023, 5:09 p.m.

Status: Resolved

Update: We've been under DDoS attacks for a while now, on and off. Today we had our first win. We engaged our 24x7 DDoS team, they identified patterns in the attack and were able to completely eliminate the attack within a few hours. Because of previous protection measures put in place, Fathom was up & down, rather than going offline for a solid chunk of time. This is a big step forward for us in response to these attacks. More info: https://usefathom.com/blog/ddos-attack

Time: June 26, 2023, 5:10 p.m.

Status: Resolved

Update: The attack on 14th November lasted around 3 hours, and our data collector was up & down, but we didn't want to update this page until we'd finished implementing our new layer of defence. As we've said in previous updates, we've been the target of a highly motivated, malicious attack that is looking to damage Fathom. We've made the following changes: - We now have a 24x7 DDoS attack mitigation team. This a highly specialized team who, in the event of a huge DDoS attack against Fathom, will help us absorb it. - We have developed a new threat checking system within our application. It's invisible, and works to block perceived attacks that get past our firewall. - We are going to be building new spam protections using machine learning over the next few months. - All analytics software gets hit with spam. It's not fun for us or for the handful of our customers who were targeted. However, we are taking this attack as a learning opportunity, and we are now developing an advanced spam protection system that will look to a) prevent spam where possible and b) clean up spam gracefully. As we've said before, we truly appreciate your patience during these attacks and we're sorry that they're happening. Identified We are under attack again. We have actually been under attack for the past week, on and off, and this is a targeted, motivated attack on our service. We have mitigated some spam, which is sent every single hour, but occasionally we are flooded with hundreds of thousands of concurrent requests. There is not a lot we can do when under this kind of attack but we are working on solutions and will be bringing in a 24/7 DDoS protection service very soon.

Time: June 26, 2023, 5:12 p.m.

Status: Resolved

Update: The attack ended November 7th PT. As we stated before, only 0.01% of customers were hit with spam, but all customers were affected. The attack was intermittent and typically ran in periods of 3-4 hours at random times of the day, not 24/7. Our system throttled incoming pageviews in defense, so we didn't collect 100% of page views sent to us. We apologize for the inconvenience here. We can assure you that we did everything within our power to absorb this malicious, unprovoked attack on Fathom. Moving forward, here is how we are responding to the challenge of spam & Layer 7 DDoS attacks: We are working with our service providers to see what we can do about this in the future. We are now building our own spam detection system that will be complete this week. We already rolled out Version 1 of our spam protection system during the attack and mitigated many millions of additional spam pageviews. We knew that this attack would only lead to Fathom becoming a better analytics product. And that's exactly what is happening right now. And finally, thank you so much to everyone who has shown us support. Every single person affected has been so understanding during this challenging time and we just want to say thank you. We love building this software for you, and we appreciate all of your patience. Posted at 7:34pm PT Update The attack is back. We have mitigated the majority of the spam but the dashboard it still having intermittent availability issues. At this moment in time, we're continuing to work on anti-spam measures, introducing new technology in response to the attack. Update We haven't updated this until now because we weren't certain that the attack was over. It seems to have stopped today. Less than 0.01% of customers were affected by the spam attack, but we still saw backlogs. We've had plans to improve the way we process pageviews (to prevent backlogs & improve aggregation speed) for a while now. Following this attack, we've now prioritized these tasks, and will be rolling out the upgrades as soon as they are ready. Update We mitigated a large chunk of the attack by using pattern matching. Unfortunately, the attack was done via a botnet, and a lot of the traffic looked legitimate, so there was no kind of blocking we could do. The attack has now stopped. For those who have been targeted as part of this spam attack, please email us, and we will clear the spam on your account. One of the big problems we had with this attack was that the floods of traffic lead to issues with the Current Visitors box and the speed of data aggregation. We know that stopping spam is impossible, and all analytics companies are subject to it, but we want to be able to absorb spam traffic without backlogs. We are now working on a self-serve spam removal tool and rebuilding our aggregation system to ensure backlogs don't occur in the future. Update The attack has continued into today. The backlog is still happening and some customers are being hit with large amounts of referral spam. We are still actively working on this issue. Thanks so much again for your patience with this matter. Identified Fathom is being targeted by malicious attack, sending tons of referral traffic our way. All customers will experience delays in seeing new traffic show up on their dashboards (but that data is still being collected). We appreciate your patience. Update We have mitigated the attack, and are working through our backlog. As this has happened before, we are working on additional protections and backlog defences. If you have any referral spam on your dashboard, let us know and we'll purge it for you. Thanks again for your patience.

Time: June 26, 2023, 5:13 p.m.

Status: Resolved

Update: Resolved Everything is back to normal. We were encountering rate limiting from a service that the dashboard relied on (AWS Parameter Store). We have increased the throughput and things are now working as expected. Identified We received reports that the Fathom dashboard wasn't loading.