The art of blocking referrer spam

Lately I’ve noticed a steady increase in the amount of referrer spam I’m getting, so I decided to see if there was a simple way to trap and ban these bots. The typical approach is usually to maintain a blacklist of domain names and deny them using mod_rewrite rules. The downside to this approach is the amount of time and effort that goes into maintaining your blacklist.

This last flood made 500 requests and lasted eight seconds before my firewall could block them all. The attack consisted of 186 unique IP addresses and 49 unique spam referrals. However, this batch of spambots were not that hard to single out as they were all using the same user-agent string.

Yep, every one of those brutes were screaming “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)” out loud. It’s not a user-agent string I expect to get any legitimate traffic from (Netcraft being the sole exception), so I’ll feed them to my blocklist en masse.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(mozilla\/4.0).* [NC]
RewriteCond %{HTTP_REFERER} !www\.netcraft\.com [NC] 
RewriteRule ^(.*)$ - [L,R=403]
</IfModule>

For those interested, you’ll find the offending IP’s and domains at my github repository.