Solved IPFW stops bhyve guest from getting IP address from DHCP server

R

rawthey

I'm trying to setup some bhyve guests for the first time on my 13.0-RELEASE system but the guests were unable to obtain IP addresses from the DHCP server on my router unless I disabled ipfw. It appears that ipfw was blocking the responses from the DHCP server.

Google searches didn't come up with any solutions for me although one article relating to a somewhat different problem suggested changing sysctl net.link.bridge.ipfw. As an experiment I tried setting this to 1 and found that with ipfw running DHCP worked for the FreeBSD and Windows guests I'd created. But I know nothing about the significance of this sysctl setting and I'm wary of blindly changing things like this in case I'm introducing problems elsewhere.

So did I do the right thing or should I have set up some additional ipfw rules ?

Code: 
curlew:/root# uname -a 
FreeBSD curlew 13.0-RELEASE-p11 FreeBSD 13.0-RELEASE-p11 #0: Tue Apr  5 18:54:35 UTC 2022     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

curlew:/root# ifconfig 
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> 
        ether 40:8d:5c:84:3d:74 
        inet 192.168.1.13 netmask 0xffffff00 broadcast 192.168.1.255 
        inet6 fe80::428d:5cff:fe84:3d74%re0 prefixlen 64 scopeid 0x1 
        inet6 2a02:8010:6418:0:428d:5cff:fe84:3d74 prefixlen 64 autoconf 
        inet6 2a02:8010:6418:0:3c54:1293:7650:d114 prefixlen 64 autoconf temporary 
        media: Ethernet autoselect (1000baseT <full-duplex>) 
        status: active 
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> 
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        groups: lo 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> 
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 
        ether 0a:cc:49:c4:20:81 
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> 
                ifmaxaddr 0 port 4 priority 128 path cost 2000000 
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> 
                ifmaxaddr 0 port 1 priority 128 path cost 20000 
        groups: bridge vm-switch viid-4c918@ 
        nd6 options=9<PERFORMNUD,IFDISABLED> 
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 
        description: vmnet-freebsd131-0-public 
        options=80000<LINKSTATE> 
        ether 58:9c:fc:10:e4:65 
        groups: tap vm-port 
        media: Ethernet autoselect 
        status: active 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> 
        Opened by PID 5631

curlew:/root# ipfw list 
00100 allow ip from any to any via lo0 
00200 deny ip from any to 127.0.0.0/8 
00300 deny ip from 127.0.0.0/8 to any 
00400 deny ip from any to ::1 
00500 deny ip from ::1 to any 
00600 allow ipv6-icmp from :: to ff02::/16 
00700 allow ipv6-icmp from fe80::/10 to fe80::/10 
00800 allow ipv6-icmp from fe80::/10 to ff02::/16 
00900 allow ipv6-icmp from any to any icmp6types 1 
01000 allow ipv6-icmp from any to any icmp6types 2,135,136 
01100 check-state :default 
01200 allow tcp from me to any established 
01300 allow tcp from me to any setup keep-state :default 
01400 allow udp from me to any keep-state :default 
01500 allow icmp from me to any keep-state :default 
01600 allow ipv6-icmp from me to any keep-state :default 
01700 allow udp from 0.0.0.0 68 to 255.255.255.255 67 out 
01800 allow udp from any 67 to me 68 in 
01900 allow udp from any 67 to 255.255.255.255 68 in 
02000 allow udp from fe80::/10 to me 546 in 
02100 allow icmp from any to any icmptypes 8 
02200 allow ipv6-icmp from any to any icmp6types 128,129 
02300 allow icmp from any to any icmptypes 3,4,11 
02400 allow ipv6-icmp from any to any icmp6types 3 
02500 allow tcp from 192.168.1.0/24{1-199} to me 22 keep-state :default 
02600 allow tcp from 192.168.1.0/24{1-199} to me 80 keep-state :default 
02700 allow tcp from 192.168.1.0/24{1-199} to me 81 keep-state :default 
02800 allow tcp from 192.168.1.0/24{1-199} to me 137 keep-state :default 
02900 allow tcp from 192.168.1.0/24{1-199} to me 138 keep-state :default 
03000 allow tcp from 192.168.1.0/24{1-199} to me 139 keep-state :default 
03100 allow tcp from 192.168.1.0/24{1-199} to me 445 keep-state :default 
03200 allow tcp from 192.168.1.0/24{1-199} to me 143 keep-state :default 
03300 allow tcp from 192.168.1.0/24{1-199} to me 443 keep-state :default 
03400 allow udp from 192.168.1.0/24{1-199} to me 81 keep-state :default 
03500 allow udp from 192.168.1.0/24{1-199} to me 137 keep-state :default 
03600 allow udp from 192.168.1.0/24{1-199} to me 138 keep-state :default 
03700 allow udp from 192.168.1.0/24{1-199} to me 139 keep-state :default 
03800 allow udp from 192.168.1.0/24{1-199} to me 445 keep-state :default 
03900 allow udp from 192.168.1.0/24{1-199} to 192.168.1.255 81,137,138,445 in 
65000 count ip from any to any 
65100 deny ip from any to 255.255.255.255 
65200 deny ip from any to 224.0.0.0/24 in 
65300 deny udp from any to any 520 in 
65400 deny tcp from any 80,443 to any 1024-65535 in 
65500 deny log logamount 500 ip from any to any 
65535 deny ip from any to any
 
Not related to the issue but plan your upgrade to 13.1. 13.0-RELEASE will be end-of-life at the end of this month.
 
rawthey said:
Thanks that fixed it.
Click to expand...
Looks like I spoke too soon.
After adding the rule allow udp from any to any 67 out keep-state everything was working fine until midway through this morning when DHCP requests suddenly started failing.
Wireshark showed outgoing DHCP discovery packets from 0.0.0.0 to 255.255.255.255 but no responses when ipfw was running as shown in dhcp-blocked.pdf but when I stopped ipfw responses from the DHCP server started coming through as in dhcp-working.pdf
 

Attachments

rawthey said:
Looks like I spoke too soon.
After adding the rule allow udp from any to any 67 out keep-state everything was working fine until midway through this morning when DHCP requests suddenly started failing.
Wireshark showed outgoing DHCP discovery packets from 0.0.0.0 to 255.255.255.255 but no responses when ipfw was running as shown in dhcp-blocked.pdf but when I stopped ipfw responses from the DHCP server started coming through as in dhcp-working.pdf
Click to expand...

I've been trying to follow this, recognising the 'workstation' ruleset plus rules presumably for your bhyve guest hosts?, but I'm confused about the addressing.

Given I know nothing of bhyve nor TAP specifically ...

Your ifconfig shows ether:
40:8d:5c:84:3d:74 with ip4 as
192.168.1.13, but neither appear on the wireshark report, nor do the other ethernet addresses (vm_public and tap0). Is that IP by DHCP?

So who/where is 192.168.1.214, with ether addr
FreeBSDF_07:9b:ec (58:9c:fc:07:9b:ec)?

There's no apparent NAT, so are you routing or forwarding for 192.168.1.0/24{1-199}?

Is 192.168.1.1 your router / uplink, or just DHCP server?

Where did you add that new rule?

So in a nutshell, who exactly is 'me' to ipfw?

# ipfw -ted show may help locate which rule is blocking what, and when.
 
rawthey said:
Google searches didn't come up with any solutions for me although one article relating to a somewhat different problem suggested changing sysctl net.link.bridge.ipfw. As an experiment I tried setting this to 1 and found that with ipfw running DHCP worked for the FreeBSD and Windows guests I'd created. But I know nothing about the significance of this sysctl setting and I'm wary of blindly changing things like this in case I'm introducing problems elsewhere.
Click to expand...

Good waryness ... that's to run ipfw in bridge (layer 2) mode, ie on ethernet packets, either instead of or as well as layer 3.

See ipfw(8)
/bridge|layer

for the shorter story, however the manual teaches far better than google - or the handbook - about all aspects of ipfw.

That search's first hit is a simple but very useful ASCII diagram about packet flows through ipfw, and the sysctls that manage it.
 
smithi said:
I've been trying to follow this, recognising the 'workstation' ruleset plus rules presumably for your bhyve guest hosts?, but I'm confused about the addressing.
Click to expand...
40:8d:5c:84:3d:74 with ip4 192.168.1.13 is my PC, the bhyve host and is 'me' in ipfw.

58:9c:fc:07:9b:ec is ihe bhyve guest with IP 192.168.1.214 assigned by the DHCP server in my broadband router at 192.168.1.1

The guest connects through tap0 which is bridged through vm-public to my network port re0

Code: 
curlew:/root# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 40:8d:5c:84:3d:74
        inet 192.168.1.13 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::428d:5cff:fe84:3d74%re0 prefixlen 64 scopeid 0x1
        inet6 2a02:8010:6418:0:428d:5cff:fe84:3d74 prefixlen 64 autoconf
        inet6 2a02:8010:6418:0:3c54:1293:7650:d114 prefixlen 64 autoconf temporary
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0a:cc:49:c4:20:81
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-freebsd131-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:e4:65
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 5631

Yes I'm using the ipfw 'workstation' ruleset and added the rule allow udp from any to any 67 out keep-state towards the end of the workstation section just before the 'Allow all connections from trusted IPs' section.

smithi said:
Good waryness ... that's to run ipfw in bridge (layer 2) mode, ie on ethernet packets, either instead of or as well as layer 3.

See ipfw(8)
/bridge|layer

for the shorter story, however the manual teaches far better than google - or the handbook - about all aspects of ipfw.

That search's first hit is a simple but very useful ASCII diagram about packet flows through ipfw, and the sysctls that manage it.
Click to expand...

So it looks like my original uneducated guess to have net.link.bridge.ipfw=1 is the right approach in this situation. With my very limited knowledge of packet filtering I assume because ipfw was filtering at the IP level the packets for the bhyve guest were being blocked because the destination address 192.168.1.214 didn't match that of the host machine 192.168.1.13. Having set this sysctl everything seems to be working fine without needing any additional ipfw rules .
 
You don't need net.link.bridge.ipfw=1

ipfw add 1650 allow udp from any to any 67 keep-state
ipfw add 1660 allow udp from any to any 68 keep-state
 
VladiBG said:
You don't need net.link.bridge.ipfw=1
Click to expand...

I'm not so sure, but it's not clear (to me) whether traffic from/to 'not me' is to be handled by this ipfw, or will guest hosts have their own firewalls? (re next msg)

VladiBG said:
ipfw add 1650 allow udp from any to any 67 keep-state
ipfw add 1660 allow udp from any to any 68 keep-state
Click to expand...

Ok, but from the tcpdump keep-state won't, since while src and dst ports (67 & 68) match, src & dst addrs do not - ie that pair would work without keep-state, and extra state table entries will be ignored, I think. Small beer ...
 
VladiBG said:
You don't need net.link.bridge.ipfw=1

ipfw add 1650 allow udp from any to any 67 keep-state
ipfw add 1660 allow udp from any to any 68 keep-state
Click to expand...
That fixed it for DNS but I wasn't able to browse any websites until I added rules to open up ports 53, 80 & 443 and I expect I'll need to open up more ports as the need arises, e.g. for ftp.

But everything seems to work OK without any extra rules if I have net.link.bridge.ipfw=1 which looks like the simplest option for me. But is this an acceptable solution or am I creating the possibilities of undesirable side effects?
 
rawthey said:
40:8d:5c:84:3d:74 with ip4 192.168.1.13 is my PC, the bhyve host and is 'me' in ipfw.

58:9c:fc:07:9b:ec is ihe bhyve guest with IP 192.168.1.214 assigned by the DHCP server in my broadband router at 192.168.1.1

The guest connects through tap0 which is bridged through vm-public to my network port re0

Code: 
curlew:/root# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 40:8d:5c:84:3d:74
        inet 192.168.1.13 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::428d:5cff:fe84:3d74%re0 prefixlen 64 scopeid 0x1
        inet6 2a02:8010:6418:0:428d:5cff:fe84:3d74 prefixlen 64 autoconf
        inet6 2a02:8010:6418:0:3c54:1293:7650:d114 prefixlen 64 autoconf temporary
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 0a:cc:49:c4:20:81
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-freebsd131-0-public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:e4:65
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 5631
Click to expand...

Thanks for all that. I'm an old dog, but might still learn a new trick or two.

Reason for delayed response is I've been poring through old backups looking for setup details for a filtering bridge (with dummynet shaping) we setup c. 2003 between a govt. supplied satellite-down ISDN-up link for small rural towns with no internet access, and up to a dozen mostly windows boxes at a community centre.

Still hunting ...

I suggest looking at
https://docs.freebsd.org/en/articles/filtering-bridges/
to see what guided me back then, when bridge.ko preceded if_bridge, the later sysctls are similar but different and more numerous, ARP issues don't apply, and other things - but it does, I think, describe some issues you might encounter with bridging in general.

Handbook Section 32.6 looks pretty good on a quick browse.

rawthey said:
Yes I'm using the ipfw 'workstation' ruleset and added the rule allow udp from any to any 67 out keep-state towards the end of the workstation section just before the 'Allow all connections from trusted IPs' section.
Click to expand...

I suggest copying rc.firewall to eg /etc/rc.myfirewall, specifying that (full path) in rc.conf, and playing at will.

I'd certainly consider adding the rules after that diagram in the PACKET FLOW section of ipfw(8), initially at least with logging, to clearly see which bridged packets show up at layer2 (if net.link.bridge.ipfw=1), and then again at layer3 (if net.link.ether.ipfw=1). We used that or similar for distinct sections for bridged and local / routed traffic, and to select packets needing shaping on then very limited bandwidth.

rawthey said:
So it looks like my original uneducated guess to have net.link.bridge.ipfw=1 is the right approach in this situation. With my very limited knowledge of packet filtering I assume because ipfw was filtering at the IP level the packets for the bhyve guest were being blocked because the destination address 192.168.1.214 didn't match that of the host machine 192.168.1.13.
Click to expand...

That too :)

rawthey said:
Having set this sysctl everything seems to be working fine without needing any additional ipfw rules .
Click to expand...

Yes, but beware the dragons ...
 
smithi said:
I'd certainly consider adding the rules after that diagram in the PACKET FLOW section of ipfw(8), initially at least with logging, to clearly see which bridged packets show up at layer2 (if net.link.bridge.ipfw=1), and then again at layer3 (if net.link.ether.ipfw=1). We used that or similar for distinct sections for bridged and local / routed traffic, and to select packets needing shaping on then very limited bandwidth.
Click to expand...
if_bridge(4)

net.link.ether.ipfw control Layer2 processing (ether_input, ether_output) for IPFW by default it's off (net.link.ether.ipfw=0)
net.link.bridge.ipfw control layer2 (bridged) filtering for the bridge interface but if you enable it using net.link.bridge.ipfw=1 you need also to enable net.link.ether.ipfw otherwise IPFW will not filter any traffic on the bridge and will also stop packet filtering for the bridge (net.link.bridge.pfil_bridge), IP filtering for the bridge (net.link.bridge.pfil_onlyip) and member interface filtering for the bridge (net.link.bridge.pfil_member) so it will disable the entire IPFW for the traffic which is going to the bridge.

So if you don't want to have firewall for the bridge interface members just set the following:
net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0

If you want to use IPFW and filter the TCP/IP traffic which is coming from the bridge then don't enable net.link.bridge.ipfw and leave net.link.bridge.pfil_member=1 ; net.link.bridge.pfil_bridge=1 and net.link.bridge.pfil_onlyip=1 this will cause IPFW to inspect the traffic for bridge (which is default on). Then you need create the IPFW rules and allow the traffic that you want.

Untitled Diagram.drawio.png


For example to allow DHCP from 192.168.1.1 to bhyve guest VM (random IP)
allow udp from any to any 67
allow udp from 192.168.1.1 to any 68
Click to expand...

To allow ICMP traffic from any to 192.168.1.0/24
allow icmp from any to 192.168.1.0/24
Click to expand...

to allow DNS traffic from 192.168.1.0/24 to any
allow tcp from 192.168.1.0/24 to any 53 setup keep-state
allow udp from 192.168.1.0/24 to any 53 keep-state
Click to expand...

to allow http/https traffic from 192.168.0/24 to any
allow tcp from 192.168.0/24 to any 80 setup keep-state
allow tcp from 192.168.0/24 to any 443 setup keep-state
Click to expand...

and so on...

I'm not so sure, but it's not clear (to me) whether traffic from/to 'not me' is to be handled by this ipfw, or will guest hosts have their own firewalls? (re next msg)
Click to expand...
in IPFW "me" keyword refers to the IP addresses on all interfaces in this case this is 127.0.0.1 lo0 an 192.168.1.13 re0
The traffic from the router 192.168.1.1 to the bhyve guest 192.168.1.214 is not handled by the "me" keyword so you need to allow this traffic to pass in the bridge member interface "re0" in order to reach the bhyve guest.
 
smithi said:
I'd certainly consider adding the rules after that diagram in the PACKET FLOW section of ipfw(8), initially at least with logging, to clearly see which bridged packets show up at layer2 (if net.link.bridge.ipfw=1), and then again at layer3 (if net.link.ether.ipfw=1).
Click to expand...

I stand by recommending using those rules to split and count (& log?) the layer2 (aka bridged) traffic, especially to check the right combination of sysctls are active - while taking on board your vastly superior knowledge of if_bridge and so the pfil settings. 2003 may as well have been the Dark Ages...

smithi said:
Yes, but beware the dragons ...
Click to expand...

I'll stand by that too <&^}=

VladiBG said:
if_bridge(4)

net.link.ether.ipfw control Layer2 processing (ether_input, ether_output) for IPFW by default it's off (net.link.ether.ipfw=0)
net.link.bridge.ipfw control layer2 (bridged) filtering for the bridge interface but if you enable it using net.link.bridge.ipfw=1 you need also to enable net.link.ether.ipfw otherwise IPFW will not filter any traffic on the bridge and will also stop packet filtering for the bridge (net.link.bridge.pfil_bridge), IP filtering for the bridge (net.link.bridge.pfil_onlyip) and member interface filtering for the bridge (net.link.bridge.pfil_member) so it will disable the entire IPFW for the traffic which is going to the bridge.

So if you don't want to have firewall for the bridge interface members just set the following:
net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0

If you want to use IPFW and filter the TCP/IP traffic which is coming from the bridge then don't enable net.link.bridge.ipfw and leave net.link.bridge.pfil_member=1 ; net.link.bridge.pfil_bridge=1 and net.link.bridge.pfil_onlyip=1 this will cause IPFW to inspect the traffic for bridge (which is default on).
Click to expand...

Thanks for all that detail. I'm again rereading
if_bridge(4) and ipfw(8) in conjunction, given that the latter isn't up to mentioning pfil(9) at all.

But with the second scenario - filtering the bridged IP traffic - you'd still need to have net.link.ether.ipfw=1, wouldn't you?

VladiBG said:
Then you need create the IPFW rules and allow the traffic that you want.
Click to expand...

I'll leave the rest, and your very helpful diagram, for rawthey to digest.

cheers
 
smithi said:
I'll leave the rest, and your very helpful diagram, for rawthey to digest.
Click to expand...
Thanks smithi and VladiBG. This thread has certainly given me much to work on but with my limited skills in this area some of it has been hard to follow. Anyway after a few dead ends and U-turns I think I've got it sorted.

The short answer is that leaving net.link.bridge.ipfw=0 and setting net.link.bridge.pfil_bridge=0 and net.link.bridge.pfil_member=0 gives the result I need but I'll provide details of how I got there for anyone who is interested or cares to point out errors in my reasoning.

First some background. My main PC, running FreeBSD, on my home LAN connects to a Fritzbox 7530 router which provides NAT plus some basic firewalling. Also connected to the LAN are a Linux PC, a Windows PC, a Linux laptop and an Android tablet. I'm reasonably happy to rely on the router to protect my PC from external threats, in fact I've run this PC without any onboard firewall for several years without any problem but extra protection is no bad thing. I don't consider my the other computers to pose much threat but there is a slight risk from any potentially compromised visitors' devices like smartphones or WiFi connected devices in the home like my internet radio, Google Chromcast TV dongle or HP inkjet printer so a while ago I decided to try to learn a bit about ipfw to improve the security of my FreeBSD PC.
All my regularly connected PC's have fixed IP addresses and the router provides a small range of DHCP addresses starting at 192.168.1.200 for other devices and visitors. This way I can open things up to allow my known devices to access my PC for things like SMB shares, my local web server and ssh but keep tighter restrictions for the less trusted devices using DHCP. I'll generally rely on DHCP for bhyve guests, largely because if I'm setting up a new guest with an OS I'm not familiar with I don't want to be messing around with configuring it's network settings at the installation stage, also it's probably best to regard an unfamiliar new operating system as 'untrusted'. I've attached a copy of the firewall rules to this post, it's effectively using the 'workstation' option with an extra rule to enable the Windows PC to discover my SMB shares.

With the default sysctl settings the client failed to acquire a DHCP lease, the response from the request being blocked by the rule 65500 deny log logamount 500 ip from any to any

I temporarily added 2 extra rules to enable the client to obtain an IP address at boot time.
Code: 
${fwcmd} add 50000 allow log logamount 500 udp from any to any 67
${fwcmd} add allow log logamount 500 udp from 192.168.1.1 to any 68

/var/log/security showed the rule was working.
Code: 
Aug 11 12:48:26 curlew kernel: ipfw: 50000 Accept UDP 0.0.0.0:68 255.255.255.255:67 in via tap0
Aug 11 12:48:26 curlew kernel: ipfw: 50000 Accept UDP 0.0.0.0:68 255.255.255.255:67 in via vm-public
Aug 11 12:48:26 curlew kernel: ipfw: 50000 Accept UDP 0.0.0.0:68 255.255.255.255:67 in via tap0
Aug 11 12:48:26 curlew kernel: ipfw: 50100 Accept UDP 192.168.1.1:67 192.168.1.219:68 in via re0
Aug 11 12:48:26 curlew kernel: ipfw: 50100 Accept UDP 192.168.1.1:67 192.168.1.219:68 in via vm-public
Aug 11 12:48:26 curlew kernel: ipfw: 50100 Accept UDP 192.168.1.1:67 192.168.1.219:68 out via vm-public
Aug 11 12:48:26 curlew kernel: ipfw: 50100 Accept UDP 192.168.1.1:67 192.168.1.219:68 out via tap0

The client then had a fully configured network interface but all other traffic for the client was blocked by the 65500 rule.

smithi said:
I'd certainly consider adding the rules after that diagram in the PACKET FLOW section of ipfw(8), initially at least with logging, to clearly see which bridged packets show up at layer2 (if net.link.bridge.ipfw=1), and then again at layer3 (if net.link.ether.ipfw=1).
Click to expand...
I added the following rules:
Code: 
ipfw add 40000 pass log logamount 500 udp from any to any 53 layer2 in
ipfw add 40100 pass log logamount 500 udp from any 53 to any layer2 in
ipfw add 40200 pass log logamount 500 udp from any to any 53 not layer2 in
ipfw add 40300 pass log logamount 500 udp from any 53 to any not layer2 in
ipfw add 40400 pass log logamount 500 udp from any to any 53 not layer2 out
ipfw add 40500 pass log logamount 500 udp from any 53 to any not layer2 out
ipfw add 40600 pass log logamount 500 udp from any to any 53 layer2 out
ipfw add 40700 pass log logamount 500 udp from any 53 to any layer2 out

and I was then able to get DNS info with drill freebsd.org
Code: 
Aug 11 14:04:41 curlew kernel: ipfw: 40200 Accept UDP 192.168.1.219:61468 192.168.1.1:53 in via tap0
Aug 11 14:04:41 curlew kernel: ipfw: 40200 Accept UDP 192.168.1.219:61468 192.168.1.1:53 in via vm-public
Aug 11 14:04:41 curlew kernel: ipfw: 40400 Accept UDP 192.168.1.219:61468 192.168.1.1:53 out via vm-public
Aug 11 14:04:41 curlew kernel: ipfw: 40400 Accept UDP 192.168.1.219:61468 192.168.1.1:53 out via re0
Aug 11 14:04:41 curlew kernel: ipfw: 40300 Accept UDP 192.168.1.1:53 192.168.1.219:61468 in via re0
Aug 11 14:04:41 curlew kernel: ipfw: 40300 Accept UDP 192.168.1.1:53 192.168.1.219:61468 in via vm-public
Aug 11 14:04:41 curlew kernel: ipfw: 40500 Accept UDP 192.168.1.1:53 192.168.1.219:61468 out via vm-public
Aug 11 14:04:41 curlew kernel: ipfw: 40500 Accept UDP 192.168.1.1:53 192.168.1.219:61468 out via tap0
These are all 'not layer2' rules so this traffic was being filtered at level 3 at each stage through the bridge between tap0 and re0 and back.
All other external traffic for the guest was still blocked by the 65500 rule.

Then I set net.link.ether.ipfw=1 and repeated drill freebsd.org with the same result as above, i.e. the 'not layer2' rules were applied and all other outgoing traffic from the guest was still blocked by the 65500 rule.

For the next step I reset net.link.ether.ipfw to zero and set net.link.bridge.ipfw=1. This resulted in all outgoing traffic from the guest being enabled but none of it was logged, not even for ports 53, 67 and 68 for which I'd provided logging rules. But more on this later in this post where I'd learnt a bit more.

VladiBG said:
net.link.ether.ipfw control Layer2 processing (ether_input, ether_output) for IPFW by default it's off (net.link.ether.ipfw=0)
net.link.bridge.ipfw control layer2 (bridged) filtering for the bridge interface but if you enable it using net.link.bridge.ipfw=1 you need also to enable net.link.ether.ipfw otherwise IPFW will not filter any traffic on the bridge and will also stop packet filtering for the bridge (net.link.bridge.pfil_bridge), IP filtering for the bridge (net.link.bridge.pfil_onlyip) and member interface filtering for the bridge (net.link.bridge.pfil_member) so it will disable the entire IPFW for the traffic which is going to the bridge.

So if you don't want to have firewall for the bridge interface members just set the following:
net.link.bridge.pfil_member=0
net.link.bridge.pfil_bridge=0
Click to expand...
This sounds like exactly what I need but first I experimented keeping net.link.bridge.ipfw=0 and setting net.link.bridge.pfil_bridge=0 and running drill freebsd.org. The log file showed that ipfw was still filtering on re0 and tap0 but ignoring traffic on the bridge vm-public

Code: 
Aug 11 15:37:50 curlew kernel: ipfw: 40200 Accept UDP 192.168.1.219:24935 192.168.1.1:53 in via tap0
Aug 11 15:37:50 curlew kernel: ipfw: 40400 Accept UDP 192.168.1.219:24935 192.168.1.1:53 out via re0
Aug 11 15:37:50 curlew kernel: ipfw: 40300 Accept UDP 192.168.1.1:53 192.168.1.219:24935 in via re0
Aug 11 15:37:50 curlew kernel: ipfw: 40500 Accept UDP 192.168.1.1:53 192.168.1.219:24935 out via tap0

But other external traffic was still being blocked by rule 65500 on re0

Then I reset net.link.bridge.pfil_bridge=1 (it's default value) and set net.link.bridge.pfil_member=0 and ran drill freebsd.org. The log file showed that ipfw was only filtering on via vm-public and not on either of the members of the bridge tap0 and re0

Code: 
Aug 11 15:45:37 curlew kernel: ipfw: 40200 Accept UDP 192.168.1.219:37923 192.168.1.1:53 in via vm-public
Aug 11 15:45:37 curlew kernel: ipfw: 40400 Accept UDP 192.168.1.219:37923 192.168.1.1:53 out via vm-public
Aug 11 15:45:37 curlew kernel: ipfw: 40300 Accept UDP 192.168.1.1:53 192.168.1.219:37923 in via vm-public
Aug 11 15:45:37 curlew kernel: ipfw: 40500 Accept UDP 192.168.1.1:53 192.168.1.219:37923 out via vm-public

And other incoming traffic was still being blocked by rule 65500 on vm-public.

Finally I set both net.link.bridge.pfil_bridge=0 and net.link.bridge.pfil_member=0 and the guest could communicate fully with the network without anything being logged in /var/log/security. I also added some temporary extra logging rules so I could confirm that traffic between the host and the network was still being protected as expected

I've added net.link.bridge.pfil_member=0 and net.link.bridge.pfil_bridge=0 to /etc/sysctl.conf and removed all my temporary additions to rc.firewall and everything seems to be working fine.

And I think I now understand why net.link.bridge.ipfw=1 resulted in passing all traffic for the guest without anything showing in the log file.

From if_bridge(4)
Code: 
net.link.bridge.ipfw    Set to 1 to enable layer2 filtering with
                        ipfirewall(4), set to    0 to disable it.  This
                        needs    to be enabled for dummynet(4) support.
                        When ipfw is enabled,    pfil_bridge and
                        pfil_member will be disabled so that IPFW is
                        not run twice; these can be re-enabled if
                        desired.
So by setting net.link.bridge.ipfw=1 I had unknowingly cleared net.link.bridge.pfil_bridge and net.link.bridge.pfil_member both to zero producing the desired effect of allowing traffic through from the guest but unnecessarily(?) applying filtering to the ethernet layer instead of (or perhaps as well as) the IP layer
 

Attachments

You must log in or register to reply here.
Back
Top