$Id: dns-fail2ban.html,v 1.12 2021/07/22 15:14:57 madhatta Exp $
I recently noticed that named was eating up about a whole CPU on teaparty.net. That's not normal, so I looked at my query logs, and found very large numbers of lines like
13-Jun-2020 21:14:54.865 queries: client @0x7f1894023a20 173.189.48.140#80 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (178.18.123.147)
That particular query comes (or at least purports to come) from a dynamic IP address at a local ISP in Little Rock, Arkansas, but they came from all over. During a 24-hour period I saw about 25 million queries from several hundred addresses. My server isn't configured to recursively-resolve for anyone except itself, so I wasn't trying to answer them, but simply noting their arrival was tying up quite a lot of resources. It was annoying.
The queries came from machines which don't seem to be anything to do with pizzaseo.com's own DNS hosting infrastructure, so I wasn't convinced that pizzaseo are responsible, whoever they are. I'm not even convinced the queries come from the machines whose source IPs are on them, since it'd be trivial to forge these requests. I concluded that complaining to pizzaseo, whoever they are, would be ineffective, and complaining to the ISPs associated with the source addresses would be extremely time-consuming.
Edit: my assumptions above have been given some support by an email from the company which used to be Pizza SEO, who have confirmed they know nothing about this except what they've gleaned from the many (angry) emails they've received.
Anyway, I fixed the problem with fail2ban. If you don't have named query logging turned on, you'll need to turn it on, with something like this in named.conf:
logging {
channel querylog{
file "/var/log/named/named-querylog" versions 5 size 100m;
severity info;
print-category yes;
print-time yes;
};
category queries { querylog; };
};
You may need to create the directory /var/log/named, and give permission to the user
named to write therein.
Then configure the following "active jail" (a fail2ban term of art for a live association between a logfile, a regexp to search the logfile for, and an action to take on finding regexp matches at more than a specified rate) in /etc/fail2ban/jail.local (other pathnames may apply to other distros):
[named-pizzaseo] enabled = true filter = pizzaseo action = iptables[name=named, port=53, protocol=udp, blocktype=DROP] logpath = /var/log/named/named-querylog maxretry = 100 findtime = 60 bantime = 1800 ignoreip = 127.0.0.1/8 ::1
Note the blocktype=DROP. By default, fail2ban sends rejections, specifically icmp-port-unreachable. But we don't want to blitz anyone who's sending these lookups with a torrent of ICMP packets; they're either not sending the lookups at all, or they don't know they're sending them. In neither case will bombarding them with ICMP packets help.
The above jail references a filter called "pizzaseo" which (oddly enough) doesn't ship with CentOS 7's fail2ban, so you'll need to configure it by creating the file /etc/fail2ban/filter.d/pizzaseo.conf with the following contents:
[INCLUDES] before = common.conf [Definition] failregex = .*queries: client @0x.* <HOST>#.*\(pizzaseo.com\): query: pizzaseo.com ignoreregex =When you restart fail2ban, a shiny new iptables chain should spring into life as soon as the first attack happens, which is defined (in the active jail) as 100 matching log lines from a single IP address, in a sixty-second window. A match results in a half-hour ban on all inbound DNS queries from that address via UDP. The resulting chain should look something like this, at the bottom of your iptables -L -n -v output:
Chain f2b-named (1 references) pkts bytes target prot opt in out source destination 174K 10M DROP all -- * * 74.69.50.233 0.0.0.0/0 62076 3600K DROP all -- * * 188.212.100.185 0.0.0.0/0 1980 148K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0If the chain is busy, and changing a lot in near-real time because you get lots of dodgy lookups, Sea Monkey kindly points out that the watch command can be used to keep a continuous eye on the state of the chain, eg with sudo watch -n 2 iptables -L f2b-named -n -v. Note that until fail2ban instantiates the chain at the time of the first detected attack, watch will display an error message from iptables. Some distros have a fairly unedifying error message; don't get confused.
Either way, Bob's your uncle. The queries are now not reaching named, which means they don't have to be logged or dealt with, and you can watch the packet counts on those rules click up and up, counting problems you're not having.
15-Jun-2020 14:56:26.272 queries: client @0x732a45bd4dc0 87.174.90.116#53670 (.): query: . IN ANY +E(0) (178.18.123.147)
These are queries for the root domain, which nobody should be sending me, and certainly not at the rate these were arriving. They weren't as expensive to deal with as the pizzaseo.com lookups, but they were still a pain. So I included them in the same solution, by amending the regexp line in pizzaseo.conf to read
failregex = .*queries: client @0x.* <HOST>#.*\(pizzaseo.com\): query: pizzaseo.com
.*queries: client @0x.* <HOST>#.*\(.\): query: . IN ANY
which matches both the original DNS queries, and the newly-annoying lot, and bans the querying
parties just the same.
This turned out to be because the fail2ban binary comes in the package fail2ban-server. Other packages are available in the repo, including fail2ban, a sort of meta-package which in turn pulls in a number of other fail2ban-related packages, including fail2ban-firewalld. That last adds a file which redirects what should be iptables-based blocks into firewalld-based blocks, which of course weren't doing anything on my system. Removing the fail2ban-firewalld package fixed the problem.
I am indebted to Chris Herdt, for writing about this in his blog.