Time: June 26, 2023, 5:04 p.m.

Status: Resolved

Update: Today’s attack was unique because it was completely unintentional. There was a problem with a customer’s site, because they had programmed an infinite loop on their event tracking code. So, what would happen is that a visitor would load their page, and then an event would fire itself at a constantly-high rate until the page was closed. (Making things worse: the page played a popular and very fantastic song that’s 3:08 long, so the page was left open for quite a while by most people.) Now, we’ve hardened our security a lot since we were first DDoS’ed last year, and our firewall routinely blocks similar attacks every week. However, the issue with this incident is that our security was focused on page collection, not event collection. As of now, we've put additional security in front of event collection to prevent this from happening again. Fathom did not go offline, but it did create a backlog. Once we isolated and blocked the offending customer’s event (and had them remove the code from their site), our backlog cleared in less than five minutes. How will this be avoided in the future? We’re migrating to a new database (finished March 12, 2021) that can easily handle things like this, and it will process backlogs like the above much faster. We’ve now added security checks to event as well. If a similar event happened in the future, our software would automatically block offenders (even if their music tastes are quite acceptable). Let us know if you have any questions. We’re always just an email away. Identified There’s currently a backlog in our queue due to a targeted attack from a motivated party. All stats are still being collecting, but will take longer to show up on your dashboard. We’re working with our 24x7 DDoS AWS team to resolve this currently. We are working on blocking more attacks like these and a building tool to mitigate them, and it’s almost ready for us to put into action (but not quite yet).

Time: June 26, 2023, 4:50 p.m.

Status: Resolved

Update: Parts of us-east-1 appeared to have fallen offline for around 7 minutes. During this time, our services were only partially available (we use multiple clouds to process traffic). Sorry for the inconvenience here, we run across multiple availability zones and this one was out of our control.

Time: June 26, 2023, 4:51 p.m.

Status: Resolved

Update: Two customers using our old custom domain system (deprecated last year) ran into issues with SSL certificates being revoked by LetsEncrypt, our old certificate provider. We looked into it and it was widespread, affecting everything who hasn't moved to the new custom domain infrastructure. This had a huge impact across the web, and affected 2.7 million websites, not just Fathom customers using the old infrastructure. As soon as we were made aware of this issue, we started working with Caddy founder, Matt Holt, to come up with a solution and re-issue the certificates. There may have been a slight drop in traffic for anyone using our old custom domain solution, and we apologize for that. This was out of our hands completely and we moved as swiftly as possible. On a related note, nobody should be using our old custom domain solution. We have a brand new, globally distributed, rapid-fast custom domain solution that we launched last year. We will likely maintain the old infrastructure for the rest of 2022, but we really do advise that you move ASAP.

Time: June 26, 2023, 4:51 p.m.

Status: Resolved

Update: We performed some upgrades to our default Fathom script (cdn.usefathom.com) on the 11th October 2021. We moved the ingest endpoint to a new CDN, with added security and global availability (to improve performance worldwide). For 1-2 hours during this move, whilst DNS was propagating, we had some incorrect configuration on our new CDN which meant that not all pageviews were being tracked. So for customers who aren't using custom domains, you'll see that your pageviews will have dropped during that time. Unfortunately, we have no way of "back-filling" missing pageviews, as we don't keep any kind of access logs. Again, customers who are using custom domains were not affected. However, folks using our default Fathom script will notice a slight drop during that period. The reason this issue wasn't caught sooner is because we monitor for downtime, not incorrect configurations, and the response wasn't technically broken. And then the reason why some pageviews were collected and some weren't was down to the fact that global DNS propogation takes time, meaning some of your website visitors were hitting our old infrastructure, whilst some will have hit the new infrastructure. Following this, we're going to be implementing changes around testing. We'll now be monitoring end to end, ensuring that the pageview is collected and that it appears in our database. Clearly monitoring for uptime alone isn't enough, and we need full, 24/7, minute by minute checks for the full end-to-end process. When testing manually, we had assumed the DNS had propogated, but it was still hitting the old servers. We apologize to everyone affected here.

Time: June 26, 2023, 5 p.m.

Status: Resolved

Update: Today, we tried out some new rules that were meant to reduce the strictness of our firewall. Unfortunately, the rule change meant huge amounts of spam was able to pass through, and we had to make a snap decision to go with a nuclear option, which was to clear the queue backlog (around 6 million pageviews), as 99% of traffic coming in was spam. Because of this, some sites lost up to 1.5 hours of traffic data. If you weren't hit with spam, you'll see a tiny blip on your stats for today (between 6PM - 8PM PT). If you were hit with spam, you'll see a huge spike. If you were hit with the spam, please reach out us. This isn't our first time dealing with spam, and we've invested a lot of time into spam protection. We'll be rolling out an additional firewall in Version 3 and, of course, we've reverted the rule change. We sincerely apologize for the inconvenience here. We have been under DDoS attacks since November 2020, and the series of events here were unfortunate.