pf nat not working

K

kevingnet

Ok, I've spend quite a bit of time, several days lately, and several weeks before.

My setup, Arris cable modem, FreeBSD system, with two NICs
em0 = external to cable modem
em1 = unmanaged switch

The broken way I can access the internet now, is by having a cable connect to a port on my Arris modem, basically bypassing the non working nat setup I'm trying to do.
rc.conf:
Code: 
zfs_enable="YES"
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
nfscbd_enable="YES"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_client="YES"
rpcbind_enable="YES"
mountd_flags="-r"

defaultrouter="192.168.0.1"
ifconfig_em0="inet 192.168.0.4 netmask 255.255.255.0"
ifconfig_em0_ipv6="inet6 accept_rtadv"
ifconfig_em1="inet 192.168.1.4 netmask 255.255.255.0"
ifconfig_em1_ipv6="inet6 accept_rtadv"

pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
gateway_enable="YES"
ipv6_gateway_enable="YES"

/etc/sysctl.conf
Code: 
net.inet.ip.forwarding=1      # (default 0)
#net.inet.ip.fastforwarding=1  # (default 0)
#net.inet6.ip6.forwarding=1    # (default 0)
Code: 
ifconfig 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
    ether 00:25:90:68:8e:04
    inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255 
    inet6 fe80::225:90ff:fe68:8e04%em0 prefixlen 64 scopeid 0x1 
    inet6 2601:647:4501:db30:225:90ff:fe68:8e04 prefixlen 64 autoconf 
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
    ether 00:25:90:68:8e:05
    inet 192.168.1.4 netmask 0xffffff00 broadcast 192.168.1.255 
    inet6 fe80::225:90ff:fe68:8e05%em1 prefixlen 64 scopeid 0x2 
    inet6 2601:647:4501:db30:225:90ff:fe68:8e05 prefixlen 64 autoconf 
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether e0:46:9a:2c:d5:a1
    inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255 
    inet6 fe80::e246:9aff:fe2c:d5a1%re0 prefixlen 64 scopeid 0x3 
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (none)
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo 
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
Two versions of pf.conf
CURRENT: (desperate attempt to do nat)
Code: 
ext_if="em0"
int_if="em1"

nat on $ext_if from $int_if to any -> ($ext_if)
pass in all
PREVIOUS: (no go)
Code: 
ext_if="em0"
int_if="em1"
#int_if="em1"

localnet = $int_if:network

nat on $ext_if from $localnet to any -> ($ext_if)
#block in all
pass from { lo0, $localnet } to any keep state

client_out = "{ domain, auth, nntp, http, https }"
pass inet proto tcp from $localnet to any port $client_out flags S/SA keep state

udp_services = "{ domain, ntp }"
pass quick inet proto { tcp, udp } to any port $udp_services keep state
To state the obvious, I'm not a networking expert. I copied these files from somewhere else, they might be old. I sort of understand a little, some of the concepts I guess. I've also read the manual and some man pages. For the life of me, I can't get this thing to work. Is there something wrong in the configs, or might it be some ghost in my computer?
 
Remove the entries from sysctl.conf, these are already set by the gateway_enable and ipv6_gateway_enable in rc.conf.

If possible I would suggest putting the modem in "bridge-mode". At the moment both your modem/router and the FreeBSD host is doing NAT. This can be made to work but doing NAT twice is not ideal.
 
I tried what you said, however nat still doesn't happen. What you suggested makes sense, I was thinking about doing that but I was concerned about wireless. Would wireless also be natted, that would be ideal. I'd rather have freebsd do all/most network management.
 
I've bridged my cable modem/router so I get the external internet IP address directly on the interface of my FreeBSD firewall. But the bridge mode does mean I cannot use the wireless or the telephony (SIP) of the modem/router any more. I don't use the telephone at all and for wireless I have a separate access-point so for me this is not a problem.

Are you able to configure the modem/router? If you can I would forgo the NAT on the FreeBSD host and use regular routing. But this requires adding a static route to the modem/router.
 
Tried and set it up in bridge mode and all internet was gone, I had to reset the modem. I'm back online though.

Still, I'm only using freebsd as zfs file server, which is awesome. However, I've never really been able to configure it as a nat router like I had with openbsd. I might have to go back to openbsd, this is really ridiculous, it has been about a year and still no nat. The modem has a good firewall though, but damn. I want my firewall back.

Anyway, thanks for all your support.

I'll try a little more this weekend.
 
I also have another wireless modem that I can connect to my third nic, and might try what you suggest, in fact that would be ideal. Could you send your config files to see how you did it or compare?

The weird thing though is that when I set it up in bridge mode, all access was gone. Might this have to do with the fact that I use 192.168.0.4 for my external nic connected to the modem and not 192.168.0.1 like it's usually the case? I even tried that before modem reset to no avail. But I just don't know.

Btw, my internet is superfast after the reset.
 
kevingnet said:
Tried and set it up in bridge mode and all internet was gone, I had to reset the modem.
Click to expand...
After putting the modem in bridge mode, enable DHCP on the em0 interface, you should receive your internet IP address. With my modem I get a 10.0.0.x address first and after the modem is done initializing I eventually get the internet address.

Basic pf.conf:
Code: 
ext_if="em0"
int_if="em1"

set skip on lo0

nat on $ext_if inet from any to any -> ($ext_if)
(the default rules allow all traffic)
 
Thanks for the info and quick reply.

What do you mean by: "enable DHCP on the em0 interface" ?
Is this done in the modem? or, do I install dhcpd in freebsd? Sorry for the basic question...

Btw, for my internal net, I use all static IPs, DHCP is only used for wireless, but at the moment, I'm leaving wireless out of the equation...
 
I think I know what you meant. By enabling DHCP on em0, it will get the *public* ip from my service provider. Is this correct? I'll try that in a bit.
 
kevingnet said:
What do you mean by: "enable DHCP on the em0 interface" ?
Is this done in the modem? or, do I install dhcpd in freebsd? Sorry for the basic question...
Click to expand...
Change in rc.conf:
Code: 
ifconfig_em0="DHCP"

kevingnet said:
Btw, for my internal net, I use all static IPs, DHCP is only used for wireless, but at the moment, I'm leaving wireless out of the equation...
Click to expand...
That's fine, just leave the settings on em1 as they are now.

kevingnet said:
By enabling DHCP on em0, it will get the *public* ip from my service provider. Is this correct?
Click to expand...
Correct.
 
I tried it and again my modem went to take a nap. I'm starting to suspect something wrong with my hardware or service. I think it should have worked. I mean, you and many others have similar configurations and they seem to work.

There's one thing I may have done wrong though... that was that I disabled DHCP when I set the modem to bridge mode, because that's the internal DHCP. The WAN DHCP I left intact, so that I was able to see that in fact, em0 did get an ip address from my ISP.

I was getting:
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
    ether 00:25:90:68:8e:04
    inet6 fe80::225:90ff:fe68:8e04%em0 prefixlen 64 scopeid 0x1 
    inet 24.23.137.23 netmask 0xfffffc00 broadcast 255.255.255.255 
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
As of now, I had to reset the modem to get me back on line. I'm at the end of the line.
 
The em0 output looks good. That's indeed what's supposed to happen. Some providers however will give you an odd gateway though, that may be the reason why it's not working. When you get the correct IP address (24.x.x.x) check your routing table, specifically look for the default gateway. If you don't get a gateway try this on the command line: route add default -iface em0
 
Thanks so much for your help... I'll try that... I was without internet for a few hours for some reason.... I'll give it another shot...
 
Tried it and I have access to the internet from the freebsd server on em0, I presume. However, my workstation connected to a switch and then to em1 still has no access to the internet.. I added the command you sent and it said it was already added

route add default -iface em0
add net default: gateway em0 fib 0: route already in table

So, I'm not sure why it's not doing nat. I'm using this pf.conf
ext_if="em0"
int_if="em1"

set skip on lo0

nat on $ext_if inet from any to any -> ($ext_if)

Another weird thing is that in bridged mode, I don't have any access to the modem configuration....
 
The internet is incredibly slow also, from the server under this configuration...

I just switched em1 to re0, just in case the card was bad... still same symptoms...
 
I had to put everything back the way it was and reset the modem again.... I have to seriously considering decommissioning freebsd as a router, openbsd used to work fine... anyway thanks for your help... at this point I don't know what else to try...

nat just doesn't happen... I actually now disabled pf altogether since it never actually worked, I'm only using the one in the modem. Probably what the xfinity people wanted all along...
 
I just found out that some modems have an RG PassThrough mode that when enabled, does not allow NAT for some reason... my previous modem had that setting, but the new POS I use now, does not... I love xfinity..

I just found an advanced page, but it may be only for an older model, it also said that when enabled it will require the ISP to issue a new IP for every client... big chance of that happening...I can see why the feature is no longer there, since they only provide you with one IP address, that makes sense. Which brings me back to the question of how exactly do I do NAT with freebsd, it shouldn't depend on what modem I use, I think/hope.. I think it should only have to do with network configuration.

Another thing I tried is that in my sysctl.conf file I had quite a few network optimization settings, which I disabled (I tried with both on and off) as a test these last few times to see if it had any effect. The only effect I saw was the internet was slower, but no NAT.

I'll try that later...
 
Here is a long shot, of what I suspect might be happening...
When the modem does NAT, and I try to do NAT with freebsd, (double NAT) packets going out of freebsd into the modem, have this one IP address, the one in the NIC, when the reply packets for that address return, the modem is only so dumb to not know where they should go, other than the originating ip, the one in the freebsd, ext_if/em0, and that's where they end, somehow something gets lost in the translation and therefore the packets do not reach the origin destination, for example my linux box, they just stop at the freebsd box..
I'll figure out a way (tcpdump, maybe) to monitor the packets and see what the hell is going on...
 
Ok, that's it, this is too much bull, I'm buying a small computer with two NICs to put openbsd in it and that should take care of this issue, this is really what I wanted all along, I'd rather use openbsd for firewall, I'll keep my cool freebsd server for everything else, in fact, the original plan was to run openbsd in bhyve using a pass through nic and use it that way, but, it's becoming evident that that's not going to fly...at some point the ROI isn't going to be there, I value my time, I mean these approximately over 500 hours I've put into this over one year...
 
I've been able to verify, I think/hope, that my hardware might be ok, by setting up a bridge. When I setup a bridge I can see that I'm able to access the internet, using only one ethernet cable from my ext_if only to the modem, and my int_if to my switch. Without the bridge and when using this thing called pf/nat, it does not. I have to connect a second cable from the switch to the modem, which is cheating, there's no nat going on there. Which is how I've been running my server for way over a year.

I used tcpdump to see the packets and with the bridge I can see them going from em0 to re0, with and without pf enabled.

Still, I suspect there might be something in my configuration or some kind of bug in pf or the device driver, or somewhere else.

Is anyone else experiencing this or I'm all alone in this?
 
I'm very close to pulling the trigger (don't worry, it's not what you might think) on one of those diy router thingies. Debating weather to get a Soekris, EdgeRouterX, or Ubiquity, Alli is also really cool but I got fast internet and it wouldn't cut it, because it only goes to 100mbps, the other 3 can do 1gbps.
 
I modified pf.conf and I'm now able to ping external internet addresses such as yahoo's 98.138.253.109
However, I could not ping yahoo.com, from the internal network,
I'm able to use the internet and ping from the freebsd machine, the external network.

I also replaced the switch, just in case. So, I'm using a different nic and new switch.

This sort of tells me what I suspected, something wrong with my config.

Could someone post a simple config that shows NAT.

Here's what I have, I think a pretty common setup:

1) Cable modem, wan port, connected to em0, on my freebsd pf router
2) Switch connected to re0, on my freebsd pf router
3) Debian Jessie linux computer, my workstation, connected to the switch, with static ip 192.168.1.21

pf.conf:
ext_if="em0"
int_if="re0"

set skip on lo0

localnet = $int_if:network

nat on $ext_if from $localnet to any -> ($ext_if)
#block all
pass from { lo0, $localnet } to any keep state

client_out = "{ nfs, ssh, domain, pop3, auth, nntp, http, \
https, cvspserver, 2628, 5999, 8000, 8080 }"

pass inet proto tcp from $localnet to any port $client_out flags S/SA keep state
#pass in inet proto tcp to $ext_if port ssh
pass in inet proto tcp to $int_if port ssh

udp_services = "{ domain, ntp }"
pass quick inet proto { tcp, udp } to any port $udp_services keep state

This isn't the pf.conf I'll be using, I copied it, and modified it a bit...
 
I'm a software developer, and I'm starting to get a really morbid curiosity as to why the packets aren't transferred between the two NICs. I know that eventually I'll get to the root of it. The reason for my questions were as ever, time is limited and I thought that someone might know about this. In view that I believe that most people seem to have a working version of pf under freebsd, I'll have to assume that it works. Also, apparently, my hardware seems ok, and I think that because when I set it in bridged mode it works, besides pf operates as software so it's unlikely that I have a hardware issue. So that leaves us with the remaining which is the configuration. Which I posted above. Actually, it might also be the device driver, or perhaps although doubtful even the kernel itself.

So... any takers?

I think I'm still going to get another system for OpenBSD though, always sort of liked it and I guess I'm a masochist of sorts. But, damn it, why can't I get this thing to work in FreeBSD?
 
I'll try something like this:
interface_1="rl0" # STATIC IP
interface_2="rl1" # DHCP

nat_1="{ 192.168.0.2, 192.168.0.3 }"
nat_2="{ 10.0.0.2, 10.0.0.3 }"

nat on $interface_1 from $nat_1 to any -> 192.168.0.1
nat on $interface_2 from $nat_2 to any -> 12.34.56.78 # ip leased from dhcp

pass all
 
Found this: (I'll try when I go back home)
interface_2="rl1"
lan_2="rl4"
router_2="12.34.56.78"

pass in on $lan_2 route-to ( $interface_2 $router_2 ) from $nat_2
 
You must log in or register to reply here.
Back
Top