PF DNS query attack: PF unable to block IP addresses from table

micski

micski

A BIND DNS on FreeBSD is under attack from hundreds of hosts, that is sending DNS queries non-stop. Every query is about an unknown domain pizzaseo.com. The DNS has access control lists, that limit recursive queries and cache to known users only, but obviously this attack are able to break through that. Even with recursive funtion set to none (disabled), the attack continues. Millions of queries has been filling the log. Here is an example from the query log.

Code: 
29-Apr-2021 03:02:43.865 queries: info: client @0x804552d60 45.170.252.5#80 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (1.2.3.4)
29-Apr-2021 03:02:43.871 queries: info: client @0x804552d60 45.170.252.5#80 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (1.2.3.4)
29-Apr-2021 03:02:43.883 queries: info: client @0x804552d60 45.170.252.5#80 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (1.2.3.4)
29-Apr-2021 03:02:43.910 queries: info: client @0x804552d60 95.156.213.134#27017 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (1.2.3.4)
29-Apr-2021 03:02:43.977 queries: info: client @0x804552d60 45.170.252.5#80 (pizzaseo.com): query: pizzaseo.com IN RRSIG + (1.2.3.4)

The response rate limit (RRL) option in BIND DNS does not work either on this attack. It just bursts right through. As surprising as that is, I would assume, that PF is better at taking care of this.

I made a script, that catches the IP addresses, that makes these queries, and creates a sorted unique list for PF to read into a table. The script works, the list is created and the table is loaded. Here is an output from PF and the table.

Code: 
# pfctl -t ban -T show
   212.32.207.227
   212.174.190.160
   213.127.79.252
   213.152.121.229
   213.163.87.151

I use a PF rule, that should block traffic from the IP addresses in this table. This rule is documented in the OpenBSD handbook and I use the same rule in other blacklists and those work. See the rule below.

Code: 
block in quick on $wan from <ban> to any

When monitoring the counters, I see, that the rule does get evaluated and does get packets. See check below.

Code: 
pfctl -v -s rules
block drop in quick on vtnet0 from <ban> to any
  [ Evaluations: 6384      Packets: 6067      Bytes: 351886      States: 0     ]
  [ Inserted: uid 0 pid 67018 State Creations: 0     ]

But, as you see from the BIND DNS query log, the DNS query just bursts right through and continues to race through. IP addresses, that are listed in the query log, are also listed in the loaded PF table.

How is this possible?
 
Do you have any pass quick rules above it? Can you zero out (pfctl -z) the stats, wait for a bit (when the unwanted traffic is the most prevalent), and then see what rules are actually ticking up?
 
There are not any pass quick rules above the block rule. I zeroed out the counters after your recommendation here, waited for a bit, and then I checked, that the block rule is getting lots of hits, while the UDP DNS rule is getting almost no hits, but the DNS query log are racing with IP addresses, that are present in the PF table. I did a fresh test.

So PF thinks, it blocks this attack, but the packets actually goes right through to DNS.
 
If you add logging to the block rule, do you see those IPs (in the table) definitely matching the block rule?

Do you need to flush anything to make sure the tables/rules are actually being used? I'm a bit rusty so might be talking nonsense - I vaguely remember that rules won't match existing connections without a flush or something like that.

Grasping at straws - if you explicitly block tcp/udp and the appropriate ports e.g. 53 does it make any difference? It really shouldn't because it looks like the default is any protocol and any port.

Good luck.
 
I had the same thought, but flushing after adding an IP address to a table is not mention in the OpenBSD handbook - and neither mentioned in any of PF guides, I have seen. The problem with flushing is, that it also kills other existing SSH connections and similar. It's a good point though. Maybe someone in here knows, if it is possible to flush the IP addresses in the table only.
 
Can you go upstream to a provider and ask them to block the traffic - obviously you want to get it working on pf but if they can stem the flow a bit that buys you some time.
 
micski said:
I had the same thought, but flushing after adding an IP address to a table is not mention in the OpenBSD handbook - and neither mentioned in any of PF guides, I have seen. The problem with flushing is, that it also kills other existing SSH connections and similar. It's a good point though. Maybe someone in here knows, if it is possible to flush the IP addresses in the table only.
Click to expand...
The manual has a section on flushing: https://www.freebsd.org/cgi/man.cgi?query=pfctl&sektion=8 but yes, the last time I did flushing I threw the baby out with the bathwater and caused some unhappiness so definitely worth waiting for wiser folk to help on this.
 
The flush rule option seems to only apply to rate limiting rules and the kill option has no effect. It confirms, that it killed 0 states and 0 sources. I would assume, that UDP is stateless, which makes it harder to kill, if it was kept running in some internal allow cache.
 
Check your state table. It's likely that enough traffic is being sent to keep the UDP state alive, and as long as the state exists traffic will be allowed to pass, even if there's a block rule. Or to rephrase that: existing states keep matching the rule that created them, it's not re-evaluated against the new rules.

Look for '-k' in the pfctl man page.
 
I only have laptops I can access locally so I don't allow remote access and my ruleset is set to block. I don't use flush or tables but maybe you can find some use in it. I've posted it before so it's no big deal:

Code: 
### Macro name for external interface
ext_if = "em0"
netbios_tcp = "{ 22, 23, 25, 80, 110, 111, 123, 512, 513, 514, 515, 6000, 6010 }"
netbios_udp = "{ 123, 512, 513, 514, 515, 5353, 6000, 6010 }"

### Reassemble fragmented packets
scrub in on $ext_if all fragment reassemble

### Default deny everything
block log all

### Pass loopback
set skip on lo0

### Block spooks
antispoof for lo0
antispoof for $ext_if inet
block in from no-route to any
block in from urpf-failed to any
block in quick on $ext_if from any to 255.255.255.255
block in quick log on $ext_if from { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 } to any

### Block all IPv6
block in quick inet6 all
block out quick inet6 all

### Block to and from port 0
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

### Block specific ports
block in quick log on $ext_if proto tcp from any to any port $netbios_tcp
block in quick log on $ext_if proto udp from any to any port $netbios_udp

### Keep and modulate state of outbound tcp, udp and icmp traffic
pass out on $ext_if proto { tcp, udp, icmp } from any to any modulate state

This is it at work:

Code: 
root@bakemono:/ # pfctl -s all
FILTER RULES:
scrub in on em0 all fragment reassemble
block drop log all
block drop in on ! lo0 inet from 127.0.0.0/8 to any
block drop in on ! em0 inet from 192.168.1.0/24 to any
block drop in inet from 192.168.1.34 to any
block drop in on ! lo0 inet6 from ::1 to any
block drop in from no-route to any
block drop in from urpf-failed to any
block drop in quick on em0 inet from any to 255.255.255.255
block drop in log quick on em0 inet from 10.0.0.0/8 to any
block drop in log quick on em0 inet from 172.16.0.0/12 to any
block drop in log quick on em0 inet from 192.168.0.0/16 to any
block drop in log quick on em0 inet from 255.255.255.255 to any
block drop in quick inet6 all
block drop out quick inet6 all
block drop quick proto tcp from any port = 0 to any
block drop quick proto tcp from any to any port = 0
block drop quick proto udp from any port = 0 to any
block drop quick proto udp from any to any port = 0
block drop in log quick on em0 proto tcp from any to any port = ssh
block drop in log quick on em0 proto tcp from any to any port = telnet
block drop in log quick on em0 proto tcp from any to any port = smtp
block drop in log quick on em0 proto tcp from any to any port = http
block drop in log quick on em0 proto tcp from any to any port = pop3
block drop in log quick on em0 proto tcp from any to any port = sunrpc
block drop in log quick on em0 proto tcp from any to any port = ntp
block drop in log quick on em0 proto tcp from any to any port = exec
block drop in log quick on em0 proto tcp from any to any port = login
block drop in log quick on em0 proto tcp from any to any port = shell
block drop in log quick on em0 proto tcp from any to any port = printer
block drop in log quick on em0 proto tcp from any to any port = x11
block drop in log quick on em0 proto tcp from any to any port = x11-ssh
block drop in log quick on em0 proto udp from any to any port = ntp
block drop in log quick on em0 proto udp from any to any port = biff
block drop in log quick on em0 proto udp from any to any port = who
block drop in log quick on em0 proto udp from any to any port = syslog
block drop in log quick on em0 proto udp from any to any port = printer
block drop in log quick on em0 proto udp from any to any port = mdns
block drop in log quick on em0 proto udp from any to any port = x11
block drop in log quick on em0 proto udp from any to any port = x11-ssh
pass out on em0 proto tcp all flags S/SA modulate state
pass out on em0 proto udp all keep state
pass out on em0 proto icmp all keep state

STATES:
all tcp 192.168.1.34:23705 -> 34.210.39.83:443       ESTABLISHED:ESTABLISHED
all tcp 192.168.1.34:58876 -> 204.109.59.195:443       TIME_WAIT:TIME_WAIT
all udp 192.168.1.34:123 -> 208.67.72.43:123       MULTIPLE:SINGLE

INFO:
Status: Enabled for 22 days 03:06:51          Debug: Urgent

State Table                          Total             Rate
  current entries                        3              
  searches                         4787888            2.5/s
  inserts                            58432            0.0/s
  removals                           58429            0.0/s
Counters
  match                             135151            0.1/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start            60000 states
adaptive.end             120000 states
src.track                     0s

LIMITS:
states        hard limit   100000
src-nodes     hard limit    10000
frags         hard limit     5000
table-entries hard limit   200000

OS FINGERPRINTS:
762 fingerprints loaded
root@bakemono:/ #
 
You'd have to show us your full ruleset before anyone can tackle this problem, otherwise it'll be a guessing game, thus a waste of time.
 
micski said:
A BIND DNS on FreeBSD is under attack from hundreds of hosts, that is sending DNS queries non-stop. Every query is about an unknown domain pizzaseo.com. The DNS has access control lists, that limit recursive queries and cache to known users only, but obviously this attack are able to break through that.
Click to expand...

If you only restricted recursive queries to given ACLs, then the host will still accept queries from everyone and thus replies with an error to bogus queries like those you see.
You also have to set 'allow-query' and restrict it to your networks.

The response rate limit (RRL) option in BIND DNS does not work either on this attack. It just bursts right through. As surprising as that is, I would assume, that PF is better at taking care of this.
Click to expand...

How have you configured rate-limiting?
If you are *really* and *absolutely* sure you need an open resolver, and you *really* have to use this single server also as a resolver for your own network, you usually use views to separate your prefixes from the rest of the world and define rate-limits for these views.
But usually it is far easier and safer to use a dedicated resolver for your networks and a separate one for the rest of the world - but unless you *have* to serve you own domains from your own nameservers (there *have* to be at least 2!), there is usually no need to run a world-accessible, open resolver.

As for the rate-limiting: this usually works perfectly fine - please show us your named.conf.options (or the 'options' block in your named.conf) and check/show us the logfiles (bind is usually very talkative) - otherwise as ShelLuser already stated this whole thread will be nothing but a guessing game.



edit:
regarding the strategy to use PF and extracting IPs from logfiles: usually those attacks are carried out over thousands of unique IPs which individually often only make a single request, so blocking them won't stop the attack.
What *does* mitigate a lot of such attacks is using the fact that those botnets consist 99% of windows boxes and thus blocking by OS fingerprint - an authorative NS for a domain should never get any requests directly from clients, so blocking windows systems entirely on port 53 can be a very effective strategy that doesn't interfere with normal operations.
 
#15: If you have two DNS servers, that are authoritative for domains, then I can not see, how you would not have them "open to the world" by listening and responding to queries about the domains. When someone tries to open a website, then the top level domain will reference the registered authoritative DNS servers and that someone will now ask these servers through a DNS server. If you block DNS queries, or simply kill BIND, then this will not work and that someone will not be able to open the website.
 
#12: I tested the -k option, but for some reason this had no effect The attack continued from the same IP addresses. I suspect, that this optioin does not work for UDP packets. I also tried to create a rule, that could identify these UDP packets, block them and clear their states, but the examples, I could find, did not work for UDP. If anyone are able to produce such a rule, I would be very interesting in seeing one.
 
So it seems, that the only way, that DNS attacks can be blocked by PF is through a rule, that is able to identify the rate of UDP packets from same IP addresse, then add it to a table, then clear the state for this IP address and then finally drop it. Is this actually possible? I have seen similar rules for TCP packets, but this is UDP, which might be different. If PF is not able to do this, I can not imagine, that BIND would be able to, and thus one would have to accept, that these DNS attacks runs in the background.
 
micski said:
So it seems, that the only way, that DNS attacks can be blocked by PF is through a rule, that is able to identify the rate of UDP packets from same IP addresse, then add it to a table, then clear the state for this IP address and then finally drop it.
Click to expand...
Can you back up this claim with any actual factual examples? Because I have a hard time believing this since it would also imply that PF couldn't protect you against DoS or DDoS and I know better than that.

I'm still convinced that if a table in PF gets ignored then there's a reason for this, in my opinion another than acclaimed possible bugs.
 
ShelLuser said:
Can you back up this claim with any actual factual examples? Because I have a hard time believing this since it would also imply that PF couldn't protect you against DoS or DDoS and I know better than that.

I'm still convinced that if a table in PF gets ignored then there's a reason for this, in my opinion another than acclaimed possible bugs.
Click to expand...
I use pf tables all the time. I'm sure that's not the problem. He still hasn't shown us his ruleset.
ShelLuser said:
You'd have to show us your full ruleset before anyone can tackle this problem, otherwise it'll be a guessing game, thus a waste of time.
Click to expand...
 
micski said:
#12: I tested the -k option, but for some reason this had no effect The attack continued from the same IP addresses. I suspect, that this optioin does not work for UDP packets. I also tried to create a rule, that could identify these UDP packets, block them and clear their states, but the examples, I could find, did not work for UDP. If anyone are able to produce such a rule, I would be very interesting in seeing one.
Click to expand...
Have you looked at the state table?

As others have pointed out: it's *very* hard to help you if you don't supply the information we ask for. Blindly guessing at what the problem is with incomplete information is not very efficient.
 
Firewall is not the answer to the DNS amplification. You will reach the PF table limit records in no time blocking all spoofed IPs. You need to fix your DNS design first by separating your SOA server from the normal DNS caching server and then enforce ACL to the range of the IPs which are allowed to query your DNS server. If you don't have the hardware resources to do this then use a DNS hosting service with DDoS protection like Cloudflare for your authoritative DNS servers.
 
#19: There is a huge difference from TCP to UDP attacks. However, my claim assumed, that if we had to use PF, then it might not be possible to block DNS attacks with PF alone. It just does not offer the needed UDP features. I know, that other methods are possible.
 
#22: Do we know for sure, that the IPs are spoofed and not just hacked botnet controlled servers? I understand, that normal DNS servers and SOA DNS servers should be separated. Why is that though? Does BIND break down in its internal ACL system, if authoratative zones exist?
 
You must log in or register to reply here.
Back
Top