Blocking DNS query spam with fail2ban

$Id: dns-fail2ban.html,v 1.8 2020/06/24 20:52:20 madhatta Exp $

I recently noticed that named was eating up about a whole CPU on teaparty.net. That's not normal, so I looked at my query logs, and found very large numbers — as in tens of millions — of lines like

13-Jun-2020 21:14:54.865 queries: client @0x7f1894023a20 173.189.48.140#80 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (178.18.123.147)

That particular query comes (or at least purports to come) from a dynamic IP address at a local ISP in Little Rock, Arkansas, but they came from all over. During a 24-hour period I saw about 25 million queries from several hundred addresses. My server isn't configured to recursively-resolve for anyone except itself, so I wasn't trying to answer them, but simply noting their arrival was tying up quite a lot of resources. It was annoying.

The queries came from machines which don't seem to be anything to do with pizzaseo.com's own DNS hosting infrastructure, so I'm not convinced that pizzaseo are responsible, whoever they are. I'm not even convinced the queries come from the machines whose source IPs are on them, since it'd be trivial to forge these requests. I concluded that complaining to pizzaseo, whoever they are, would be ineffective, and complaining to the ISPs associated with the source addresses would be extremely time-consuming.

So I fixed the problem with fail2ban. If you don't have named query logging turned on, you'll need to turn it on, with something like this in named.conf:

logging {
        channel querylog{
                file "/var/log/named/named-querylog" versions 5 size 100m;
                severity info;
                print-category yes;
                print-time     yes;
        };
        category queries { querylog; };
};
You may need to create the directory /var/log/named, and give permission to the user named to write therein.

Then configure the following "active jail" (a fail2ban term of art for an association between a logfile, a regexp to search for, and an action to take on finding matches at more than a specified rate, for the specified regexp, in the specified logfile) in /etc/fail2ban/jail.local (other pathnames may apply to other distros):

[named-pizzaseo]
enabled  = true
filter   = pizzaseo
action   = iptables[name=named, port=53, protocol=udp, blocktype=DROP]
logpath  = /var/log/named/named-querylog
maxretry = 100
findtime = 60
bantime  = 1800
ignoreip = 127.0.0.1/8 ::1

Note the blocktype=DROP. By default, fail2ban sends rejections, specifically icmp-port-unreachable. But we don't want to blitz anyone who's sending these lookups with a torrent of ICMP packets; they're either not sending the lookups at all, or they don't know they're sending them. In neither case will bombarding them with ICMP packets help.

The above jail references a filter called "pizzaseo" which (oddly enough) doesn't ship with CentOS 7's fail2ban, so you'll need to configure it by creating the file /etc/fail2ban/filter.d/pizzaseo.conf with the following contents:

[INCLUDES]
before = common.conf

[Definition]
failregex = .*queries: client @0x.* <HOST>#.*\(pizzaseo.com\): query: pizzaseo.com
ignoreregex =
When you restart fail2ban, a shiny new iptables chain should spring into life as soon as the first attack happens, which is defined (in the active jail) as 100 matching log lines from a single IP address, in a sixty-second window. A match results in a half-hour ban on all inbound DNS queries from that address via UDP. The resulting chain should look something like this, at the bottom of your iptables -L -n -v output:
Chain f2b-named (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 174K   10M DROP       all  --  *      *       74.69.50.233         0.0.0.0/0           
62076 3600K DROP       all  --  *      *       188.212.100.185      0.0.0.0/0           
 1980  148K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

And Bob's your uncle. The queries are now not reaching named, which means they don't have to be logged or dealt with, and you can watch the packet counts on those rules click up and up, counting problems you're not having.

Is it all fixed now?

Once things had been quiet for a few days, I noticed they no longer were. Reading the named query logs revealed tens of millions of lines like
15-Jun-2020 14:56:26.272 queries: client @0x732a45bd4dc0 87.174.90.116#53670 (.): query: . IN ANY +E(0) (178.18.123.147)

These are queries for the root domain, which nobody should be sending me, and certainly not at the rate these were arriving. They weren't as expensive to deal with as the pizzaseo.com lookups, but they were still a pain. So I included them in the same solution, by amending the regexp line in pizzaseo.conf to read

failregex = .*queries: client @0x.* <HOST>#.*\(pizzaseo.com\): query: pizzaseo.com      
            .*queries: client @0x.* <HOST>#.*\(.\): query: . IN ANY 
which matches both the original DNS queries, and the newly-annoying lot, and bans the querying parties just the same.

A note on fail2ban and firewalld

I use iptables to manipulate my firewall rules; it was good enough for Van Jacobson (joke), it's good enough for me (not a joke). C7 comes out of the box with a fuzzy daemon called firewalld, which I routinely disable. When I first tried to get this working, fail2ban would log a match, say it was blocking the traffic, then not block it.

This turned out to be because the fail2ban binary comes in the package fail2ban-server. Other packages are available in the repo, including fail2ban, a sort of meta-package which in turn pulls in a number of other fail2ban-related packages, including fail2ban-firewalld. That last adds a file which redirects what should be iptables-based blocks into firewalld-based blocks, which of course weren't doing anything on my system. Removing the fail2ban-firewalld package fixed the problem.

I am indebted to Chris Herdt, for writing about this in his blog.

Back to Technotes index
Back to main page