Solved Can I do routing with FreeBSD using a PC? What are the experiences?

[inf3rno]

I don't like the router my ISP gave to me, so I was thinking on putting it to modem mode and buying an Asus router I could use with OpenWRT. Meanwhile I realized that I have a home server that runs 24/7 so maybe I could turn it into a router if I buy an ethernet card with multiple ports. I found that pfSense does exactly this, but it uses some sort of customized kernel and I would rather call it a distribution of FreeBSD. I don't like the uncertainty that comes with a separate distribution, so I'd like to stick with FreeBSD. Is there a way to solve this with installing something from ports? What is the experience, is it worth it, or is it better to have a separate router and stick with server applications only? There are a few drawbacks I can think of, for example having a dedicated circuitry for the task can be a lot faster or better to keep it simple, because a crashing driver or application can kill the router too. These might not be big issues for me.

 

[Jose]

I use Openbsd for this purpose at home, but I've used Freebsd for gateways in large-scale production environments. You shouldn't need anything that's not in base.

This chapter in the Handbook probably has all the information you need:

Chapter 33. Firewalls

FreeBSD has three firewalls built into the base system: PF, IPFW, and IPFILTER. This chapter covers how to define packet filtering rules, the differences between the firewalls built into FreeBSD and how to use them

www.freebsd.org

 

[zirias@]

First of all, FreeBSD sure has better "routing" capabilities than your typical consumer hardware targeted at private home LANs. But for giving you a good answer, you should tell what exactly you don't like about your current router, and what features you want to use

I do have a server at home that also acts as my router, and one thing to consider is: If your server is down, your connection to the internet is down. It happened to me when I made a mistake with the bootcode, and I had to download stuff with my mobile phone to write a USB stick allowing me to repair my mess.

The second thing to consider is: You typically want your router to do firewalling. But you also want to protect your server. Therefore, the best solution is a dedicated box between all your internal devices (including your server) and the internet. My solution to this problem is virtualization: My router/firewall is a bhyve vm, which gets all NICs (my server board has two of them) exclusively via PCI pass-through. This is not the same level of protection as a dedicated box, but it's IMHO the second-best thing. Of course, there's quite some configuration to do, setting up bridges, etc.

About your idea of having many "ports", I think you mean actually several independent interfaces? There might be an alternative: I bought a manageable switch supporting VLAN and link aggregation. That way, I just have a trunked connection using both NICs to my switch, and VLANs separate the "way outside" (to the modem) from the internal network, the management network and the guest network. I think this is a more flexible approach, and smaller manageable switches can be pretty affordable.

 

[Lamia]

The one problem I foresee you having is a PPPoE app/pkg not working. If you don't need it for Internet connection, then FreeBSD might be the right OS/firmware. That pkg (rppoe, I think) with its port is NOT regularly maintained. PfSense does a good job. And I can of some good alternatives too. But be prepared for weeks, not days, of troubleshooting DHCP, routing, DNS, PF and many more.

 

[zirias@]

The one problem I foresee you having is a PPPoE app/pkg not working. If you don't need it for Internet connection, then FreeBSD might be the right OS/firmware. That pkg (rppoe, I think) with its port is NOT regularly maintained.

ppp(8) supports PPPoE, you don't need any packages. I never used any and never had problems.

 

[PMc]

Practically every such dedicated "plastic" router has our Berkeley networking stack as it's core component. The main thing you don't get when using FreeBSD is all the bugs and security holes they put in when porting (and the dependency on their bugfixes).
What you also don't get is their customization for the specific use-case of a soho router. So you'll have the fun of doing the configuration on your own, and learn from your mistakes...

There is a couple of architectural consequences to consider, as Zirias already pointed out. You may also want to have some of the usual things for a production environment, e.g. working and tested backups, a replacement power-supply on the shelf, and another test system where to install new versions first. Basically the thinking-sports of "what do I do if XXX fails?"
But then, if you keep the plastic router and just put it to PPPoE direct-thru, then you can always change that back in an emergency and hook some laptop onto it, so the risk isn't big.

Another thing to consider is: if the server is the router, then if it gets some hiccup, you will by no means be able to dial in remotely to restart it. Whereas with a separate router there are some means to implement that.

 

[ralphbsz]

Yes, it can be done. With a cheapo Intel CPU running at a few GHz, you can max out 10gigE networks, without exorbitant CPU load. One of the questions is, as Zirias and Lamia pointed out, that your computer has to support the hardware interface and protocol of the upstream connection. PPPoE is pretty easy, where it gets hard is if the router also has the modem in it and you need to connect physically to DSL lines or stuff like that. Yes, you can get cards and adapters for those, but that gets a bit esoteric.

The question is, as PMc said: Do you want to be responsible for all the setup, mistakes, configuration, maintenance, and inconvenience? If you've never set up a firewall/packet filter, and don't understand how IP and multiple networks interact, this will be a voyage of discovery, with probably days you'll spend without a working internet connection. What's more interesting today is: For lack of physical (paper) books, you'll probably want to set this up and debug it using online documentation. How are you going to read that online documentation to fix your router, when the router is broken?

By the way, if you think I'm trying to discourage you: No, this is exactly what I do at home; my home server is also the firewall / router, connected to the DSL modem. But I've been slowly setting this up over the last 16 or 18 years, with lots of trial and error, learning from mistakes, and redundancy and testing.

 

[mark_j]

Yes, it can be done. With a cheapo Intel CPU running at a few GHz, you can max out 10gigE networks, without exorbitant CPU load. One of the questions is, as Zirias and Lamia pointed out, that your computer has to support the hardware interface and protocol of the upstream connection. PPPoE is pretty easy, where it gets hard is if the router also has the modem in it and you need to connect physically to DSL lines or stuff like that. Yes, you can get cards and adapters for those, but that gets a bit esoteric.

I believe he said he was going to bridge his router to his freebsd-based router, so that takes care of the physical connection.

Setting up really isn't that hard. Either way, he can always reset his router's bridge mode.
My only 'issue' is has he considered WiFi? Purchasing the correct card is important as far as FreeBSD support of the full a/b/c/n range of the 802.11 protocol.

 

[ralphbsz]

My only 'issue' is has he considered WiFi? Purchasing the correct card is important as far as FreeBSD support of the full a/b/c/n range of the 802.11 protocol.

I tried using first an OpenBSD and then a FreeBSD machine as a WiFi access point. It kind of sort of works. But it had so many small recurring problems that eventually I just threw a really good access point at it, and now I'm much happier. That was about 5 years ago, it might have gotten better.

 

[msplsh]

Power draw is an issue with using a standard PC versus a dedicated appliance.
Maintenance and security updates, of course. Intrusion could also be a much larger deal.
Need to know how to configure dhcpd and a firewall.

 

[mark_j]

If power is a problem then an ARM based system would be a good solution. One I could think of off-hand is the Raspberry Pi 4. Alas, it's unusable on FreeBSD at present. You can easily run a USB to RJ45 adapter(s) as your network connectors and as ralphbsz said, plug in an access point device for WiFi (assuming support of said device in FreeBSD ARM).

There's also plenty of ARM devices that have multiple network interfaces and so on. Getting them running might not be worth the cost of time vs the cost of power consumption; especially on FreeBSD - you might have more luck with Linux in that regard.

 

[Lamia]

If power is a problem then an ARM based system would be a good solution. One I could think of off-hand is the Raspberry Pi 4. Alas, it's unusable on FreeBSD at present. You can easily run a USB to RJ45 adapter(s) as your network connectors and as ralphbsz said, plug in an access point device for WiFi (assuming support of said device in FreeBSD ARM).

There's also plenty of ARM devices that have multiple network interfaces and so on. Getting them running might not be worth the cost of time vs the cost of power consumption; especially on FreeBSD - you might have more luck with Linux in that regard.

You will get some good ARM/NIC devices running PfSense on eBay at reasonable prices. msplsh reiterated what I earlier mentioned - to what extent is their patience to gain the technical know-how even if somethings never work during the expedition?

 

[inf3rno]

First of all, FreeBSD sure has better "routing" capabilities than your typical consumer hardware targeted at private home LANs. But for giving you a good answer, you should tell what exactly you don't like about your current router, and what features you want to use

I do have a server at home that also acts as my router, and one thing to consider is: If your server is down, your connection to the internet is down. It happened to me when I made a mistake with the bootcode, and I had to download stuff with my mobile phone to write a USB stick allowing me to repair my mess.

The second thing to consider is: You typically want your router to do firewalling. But you also want to protect your server. Therefore, the best solution is a dedicated box between all your internal devices (including your server) and the internet. My solution to this problem is virtualization: My router/firewall is a bhyve vm, which gets all NICs (my server board has two of them) exclusively via PCI pass-through. This is not the same level of protection as a dedicated box, but it's IMHO the second-best thing. Of course, there's quite some configuration to do, setting up bridges, etc.

About your idea of having many "ports", I think you mean actually several independent interfaces? There might be an alternative: I bought a manageable switch supporting VLAN and link aggregation. That way, I just have a trunked connection using both NICs to my switch, and VLANs separate the "way outside" (to the modem) from the internal network, the management network and the guest network. I think this is a more flexible approach, and smaller manageable switches can be pretty affordable.

I have something with Intel Puma chipset. It gives a really bad quality internet connection, but that is not fixable even if I put it in modem mode afaik. Changing ISP is not an option currently and they don't allow any custom modem to their network. Their modem+router gives only very basic functions and in the long run I'll need a lot more than that. I want to have a WireGuard server on the router to access my home network remotely. I'd like to add a guest network too. I'd like to add wifi access points in the house or at least add a strong wifi antenna. I would add a faster connection between my developer PC and the server, which would require buying two additional ethernet cards, one for the PC one for the server. The alternative would be buying the exact same cards plus adding a switch to the network that can handle link aggregation for 1 Gbps or maybe later 10 Gbps. Obviously I would rather spare the price of that switch. There are many other advantages like solid cooling and CPU power, it is possible to upgrade the hardware and software components. If I need more RAM, I can easily add it. If there is a security fix I can install it and I don't have to rely on a router manufacturer to publish it, which many times does not happen. The software would be open source. I would be able to program the firewall, add a guest network, add a VPN server, media server or whatever I want, and so on... I think the disadvantages are bigger complexity, somewhat less reliability, probably somewhat more power consumption. What I am worried about is the data transfer speed and the cost in computing power. I don't want to run my server on 100% or not even on 50% constantly because of routing. A 10% would be acceptable. Another thing that when I run a lot more stuff on the server the routing might got less CPU time and it can slow down the network. I think this can be solved by giving a dedicated CPU core to the routing software, but I am not sure. I guess dedicated routers have special hardware that makes them a lot faster for routing than using software solutions, but maybe I am wrong. I don't know much about the differences, still I think it is worth a try, because it would give a lot more freedom.

I don't get why I'd need to move the firewall to a virtual machine. Should I expect that it will be hacked and attackers take control of that VM?

 

[inf3rno]

Practically every such dedicated "plastic" router has our Berkeley networking stack as it's core component. The main thing you don't get when using FreeBSD is all the bugs and security holes they put in when porting (and the dependency on their bugfixes).
What you also don't get is their customization for the specific use-case of a soho router. So you'll have the fun of doing the configuration on your own, and learn from your mistakes...

There is a couple of architectural consequences to consider, as Zirias already pointed out. You may also want to have some of the usual things for a production environment, e.g. working and tested backups, a replacement power-supply on the shelf, and another test system where to install new versions first. Basically the thinking-sports of "what do I do if XXX fails?"
But then, if you keep the plastic router and just put it to PPPoE direct-thru, then you can always change that back in an emergency and hook some laptop onto it, so the risk isn't big.

Another thing to consider is: if the server is the router, then if it gets some hiccup, you will by no means be able to dial in remotely to restart it. Whereas with a separate router there are some means to implement that.

Thanks! Actually it is a modem + router the ISP gave and since they don't allow any custom modem to their system I have to keep it at least as a modem. So there is no big risk, I just put it into router mode instead of modem mode if the server fails somehow.

 

[inf3rno]

Yes, it can be done. With a cheapo Intel CPU running at a few GHz, you can max out 10gigE networks, without exorbitant CPU load. One of the questions is, as Zirias and Lamia pointed out, that your computer has to support the hardware interface and protocol of the upstream connection. PPPoE is pretty easy, where it gets hard is if the router also has the modem in it and you need to connect physically to DSL lines or stuff like that. Yes, you can get cards and adapters for those, but that gets a bit esoteric.

The question is, as PMc said: Do you want to be responsible for all the setup, mistakes, configuration, maintenance, and inconvenience? If you've never set up a firewall/packet filter, and don't understand how IP and multiple networks interact, this will be a voyage of discovery, with probably days you'll spend without a working internet connection. What's more interesting today is: For lack of physical (paper) books, you'll probably want to set this up and debug it using online documentation. How are you going to read that online documentation to fix your router, when the router is broken?

By the way, if you think I'm trying to discourage you: No, this is exactly what I do at home; my home server is also the firewall / router, connected to the DSL modem. But I've been slowly setting this up over the last 16 or 18 years, with lots of trial and error, learning from mistakes, and redundancy and testing.

It is not DSL. I must keep the cable modem part of their device, I don't have a choice in that. I guess the routing part is a lot less esoteric. I have experience in web development and programming, and I have some basic knowledge in security and networking. I am pretty sure I'll learn new things, but I would widen my knowledge in the topic without this project too, so it is not a big deal...

 

[inf3rno]

I tried using first an OpenBSD and then a FreeBSD machine as a WiFi access point. It kind of sort of works. But it had so many small recurring problems that eventually I just threw a really good access point at it, and now I'm much happier. That was about 5 years ago, it might have gotten better.

I am not sure yet about it. I think I'll need one AP downstairs, because the signal is very weak and I doubt any wifi card can go through the thick concrete between the two floors. I have a cheap n-wifi card to do wifi upstairs, but that should cover only the room where the server is. So in theory it will be fine with 1 AP and with that card or if not, then I would rather buy one more AP than a more expensive card. I don't really need ac wifi, n is good enough. Most of the family members just browse and check facebook. There is no heavy download through wifi or such things. So even cheap wifi routers second hand would suffice for now.

 

[inf3rno]

Power draw is an issue with using a standard PC versus a dedicated appliance.
Maintenance and security updates, of course. Intrusion could also be a much larger deal.
Need to know how to configure dhcpd and a firewall.

I have this CPU: https://ark.intel.com/content/www/u...n-processor-e3-1230-v5-8m-cache-3-40-ghz.html
I calculated and if it goes with the 80W TDP they write, so with 100% CPU usage, then I spend around $10 on power. So in theory it is not an issue. The server would run 24/7 without the routing too, and I am not sure, but somebody wrote that routing does not use a lot of CPU time, so the difference is even less, probably negligible.

 

[inf3rno]

Thanks for the answers! So the conclusion that it is doable. I'd better start learning how.

 

[zirias@]

I don't get why I'd need to move the firewall to a virtual machine. Should I expect that it will be hacked and attackers take control of that VM?

That's a possibility, although I don't think it is very likely. If you have a separate firewall box between your whole internal network and the internet, that's ideal. If not, and you want to use a server as a firewall, you should at least consider doing this on a virtual machine. The reason is that otherwise, all network packets have to traverse all the network-handling code in your server (also in the kernel). You don't know which bugs "malicious" packets will trigger, it could also be bugs in code executed before the packet filtering, or even in the packet filtering itself. That's why a firewall should be a separate box. If you can't do that, a virtual machine is the second preferred solution. With PCI-passtrhough, the host still "gets" incoming network packets, but it won't try to decode the slightest bit of them, as even the device driver for the NIC sits in the virtual machine.

 

[msplsh]

I calculated and if it goes with the 80W TDP they write, so with 100% CPU usage, then I spend around $10 on power

That's also just the CPU, not the rest of the system (drives, board, PSU loss). Even if you still wanted custom, a SBC like an APU is max 12W. Compare this to a dedicated unit like a UniFi USG at max 7 watts. 10x the power consumption. Then map that out over a year. ?‍ That would finance buying a dedicated device. Up to you. Just wanted to point out that it's not nothing.

 

[Jose]

That's a possibility, although I don't think it is very likely. If you have a separate firewall box between your whole internal network and the internet, that's ideal. If not, and you want to use a server as a firewall, you should at least consider doing this on a virtual machine. The reason is that otherwise, all network packets have to traverse all the network-handling code in your server (also in the kernel). You don't know which bugs "malicious" packets will trigger, it could also be bugs in code executed before the packet filtering, or even in the packet filtering itself. That's why a firewall should be a separate box. If you can't do that, a virtual machine is the second preferred solution. With PCI-passtrhough, the host still "gets" incoming network packets, but it won't try to decode the slightest bit of them, as even the device driver for the NIC sits in the virtual machine.

You're trading potential bugs in the networking stack for potential bugs in the virtualization stack. Given that the BSD networking stack is legendary, I'll pick it over the virtualization stack every day and twice on Sundays.

I agree that a discrete machine is the best option, and that's indeed what I do. I built one around this board:

Amazon.com: ASRock Motherboard & CPU Combo Motherboards J3455B-ITX: Computers & Accessories

Buy ASRock Motherboard & CPU Combo Motherboards J3455B-ITX: Motherboards - Amazon.com ✓ FREE DELIVERY possible on eligible purchases

www.amazon.com

But in retrospect wished I had thought of using a PC Engines product.

It's a real bummer that you're saddled with that Puma chipset. We had one at work once upon a time, and we literally had to run out and buy an Arris/Motorola modem 'cause the Puma crapware kept freezing and dropping everyone off the Internet.

Intel Puma 6 modem chipset bug present in Puma 7 as well - Shaw | DSLReports Forums

Forum discussion: A bug in the Intel Puma 6 chipset that causes high latency spikes has been plaguing cable modem users for some time, including Shaw - it's present in the Cisco and Hitron modems that Shaw supplies to customers. You can test for it here:

www.dslreports.com

 

[zirias@]

You're trading potential bugs in the networking stack for potential bugs in the virtualization stack. Given that the BSD networking stack is legendary, I'll pick it over the virtualization stack every day and twice on Sundays.

IMHO, that doesn't make much sense because PCI passthrough is simple in comparison to networking, the host OS only does some "generic wiring" and the device driver for the NIC runs in the virtualized context already. You can't have better isolation short of the "best" solution of a dedicated firewall machine. But, of course, everyone is entitled to follow his own judgement

 

[Phishfry]

Just wanted to point out that it's not nothing.

I agree on the APU2 being an excellent platform for a router. I use them myself.
Another cost of doing business with a UniFi or other routers is system fixes/updates.
When a vulnerability is found FreeBSD quickly patches and you can update your firewall router.
The same cannot be said for purchased routers. You are dependent on them releasing a fixed firmware.
Worse they might just EOL perfectly working hardware for their enrichment.

Also routers use almost no CPU cycles so TDP is a useless metric. My APU2 Wireless AP / router uses 0.0% CPU

last pid:  7341;  load averages:  0.03,  0.02,  0.00                up 2+03:41:12  18:56:31
12 processes:  1 running, 11 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 4392K Active, 36M Inact, 155M Wired, 36M Buf, 1642M Free
Swap:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
7341 root          1  20    0  7836K  3200K CPU2    2   0:00   0.05% top
7334 root          1  20    0 12944K  7708K select  0   0:00   0.01% sshd
  834 root          1  20    0 10300K  5776K select  2   0:02   0.00% hostapd
  801 root          1  20    0  6384K  2376K nanslp  0   0:02   0.00% cron
  686 root          1  20    0  6332K  2408K select  1   0:01   0.00% syslogd
  701 nobody        1  20    0  9472K  3804K select  0   0:01   0.00% dnsmasq
  528 root          1  20    0  9176K  5000K select  2   0:00   0.00% devd
  474 root          1  52    0  6464K  2312K select  0   0:00   0.00% dhclient
  527 _dhcp         1  20    0  6464K  2348K select  2   0:00   0.00% dhclient
7337 root          1  20    0  7296K  3864K pause   2   0:00   0.00% csh
  798 root          1  20    0 12652K  7000K select  0   0:00   0.00% sshd
  854 root          1  52    0  6328K  2048K ttyin   2   0:00   0.00% getty

 

[Jose]

Or they may push a firmware "upgrade" when you least expect it:

eero Software Release Notes

eero software updates are released on a rolling basis, so your eero may not be updated immediately after a new version is released. New versions will be pushed to all customers’ networks (both 1st ...

support.eero.com

I guess it's one way to add excitement to your life.

 

[Phishfry]

Oh yea and I have increased my wpa_supplicant rekeying frequency so wireshark needs a very very fast computer to break WPA2.
Another bonus of homebrew.