Multiple nics and setfib

Understudy

Understudy

Hi All,

I am looking to setup two physical nics with two different ip addresses and two different gateways. The basics:

Code: 
 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 19:31:38 UTC 2015  
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386

I am looking to make one nic part of the private ip range 192.168.1.0/24 with a gateway of 192.168.1.255. I am looking to make the second nic part of a public ip range 77.77.77.0/24 with a gateway of 77.77.77.77 . As I am attempting to understand I need to setup setfib for this to work properly. My understanding is this has become part of the kernel default after version 10.

So I am looking to place a line in /boot/loader.conf.

Code: 
## Loader Config ##
net.fibs=2

Then modify my /etc/rc.conf

Code: 
## RC.CONF ##
static_routes="fibdefault"
route_fibdefault="default 192.168.1.255 -fib 1
route_fibdefault="default 77.77.77.77 -fib 2
ifconfig em0="DHCP"
ifconfig em1="77.77.77.76 netmask 255.255.255.240"

How does this look? Thank you in advance.
 
Code: 
route_fibdefault="default 192.168.1.255 -fib 1
route_fibdefault="default 77.77.77.77 -fib 2
You definitely can't do that for a start. /etc/rc.conf is just a list of variables, so route_fibdefault is set to the first value, then you overwrite it with a different value in the second line.

First off, do you actually need multiple gateways. Is it a problem if all access to non local addresses (anything that isn't 192.168.1.x) goes via 77.77.77.77? If not, the easiest solution is just to set a static 192 address on em0, and give a default gateway of 77.77.77.77.
Code: 
defaultrouter="77.77.77.77"
ifconfig_em0="192.168.1.x/24"
ifconfig_em1="77.77.77.76/28"
That's what I'd do unless there was a really specific reason why this machine couldn't just use the 77 address to access the Internet.

If you do want multiple routing tables, you should be able to use the default options and just manually set the default route for the second routing table.
Code: 
ifconfig_em0="DHCP"
ifconfig_em1="77.77.77.76 netmask 255.255.255.240"
Hopefully DHCP should set your main default gateway. It's then just a case of adding the gateway for the second routing table
Code: 
# setfib 1 route add default 77.77.77.77
There may be a way of doing this via rc.conf although 60 seconds on Google didn't find anything obvious. Most results seem to suggest putting that command in /etc/rc.local. If it complains about the route already existing, which it shouldn't really, you can use change instead of add, or delete the route first:
Code: 
setfib 1 route delete default
Bear in mind I haven't tested any of this, or ever used multiple routing tables.

Also, all applications will use the original routing table by default, and go via 192.168.1.255*. You'll have to actually run applications using setfib 1 for them to use the second routing table.

*Also 2, If 192.168.1.255 is the end of your range, you shouldn't be using it on a device. Assuming you're using the most common network size (subnet mask 255.255.255.0), the range would be 192.168.1.0-192.168.1.255, and 255 is the broadcast address.

Edit: I've just read the route man page a bit more and seen the fib stuff. So, the following *should* be feasible
Code: 
ifconfig_em0="DHCP"
ifconfig_em1="77.77.77.76 netmask 255.255.255.240"
static_routes="gw2"
route_gw2="default 77.77.77.77 -fib 1"
 
Thank you for your reply.

I can set a static 192.168.0.X address if I need to. I just had not planned on it.
I am not sure the the 192.168.0.X address would be accessible through the 77.77.77.77 gateway. I am doing this in coordination with an AT&T modem that hands off to me both private and public IPs. The issue being that they both run through separate gateways even if it is the same modem.


usdmatt said:
Code: 
defaultrouter="77.77.77.77"
ifconfig_em0="192.168.1.x/24"
ifconfig_em1="77.77.77.76/28"
That's what I'd do unless there was a really specific reason why this machine couldn't just use the 77 address to access the Internet.
Click to expand...

The 77.77.77.76 ip is used to access the internet.

I would like the office where the servers are located to be able to access the server with SSH from the internal network especially when there is a problem with the dns or other on the internet that prevents us from accessing the public IP.

usdmatt said:
*Also 2, If 192.168.1.255 is the end of your range, you shouldn't be using it on a device. Assuming you're using the most common network size (subnet mask 255.255.255.0), the range would be 192.168.1.0-192.168.1.255, and 255 is the broadcast address.
Click to expand...

I was just using it as an example the ISP 192.168.1.254 for the broadcast. My apologies for the confusion.

usdmatt said:
Edit: I've just read the route man page a bit more and seen the fib stuff. So, the following *should* be feasible
Code: 
ifconfig_em0="DHCP"
ifconfig_em1="77.77.77.76 netmask 255.255.255.240"
static_routes="gw2"
route_gw2="default 77.77.77.77 -fib 1"
Click to expand...

Would this go in the /etc/rc.conf?
 
Trying a couple of different things here but still not getting the desire result.
The idea being that emo is the default which would be fib 0 so you don't have to create a setfib for that.


/etc/rc.conf
Code: 
ifconfig_em0="DHCP"
static_routes="fibdefault"
route_fibdefault="default 77.77.77.77 -fib 1"
ifconfig em1="inet 77.77.77.76 netmask 255.255.255.240"

does not appear to work

# sysctl net.fibs
Code: 
net.fibs: 2

# setfib 1 netstat -rn
Code: 
Routing tables (fib: 1)
Internet:
Destination  Gateway  Flags  Netif Expire
127.0.0.1  link#3  UH  lo0
192.168.1.0/24  link#1  U  em0
Internet6:
Destination  Gateway  Flags  Netif
Expire
::/96  ::1  UGRS  lo0
::1  link#3  UH  lo0
::ffff:0.0.0.0/96  ::1  UGRS  lo0
fe80::/10  ::1  UGRS  lo0
fe80::%lo0/64  link#3  U  lo0
ff02::/16  ::1  UGRS  lo0

# ifconfig
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b8
  inet 192.168.1.143 netmask 0xffffff00 broadcast 192.168.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active

em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b9
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
 
Last edited by a moderator:
Is it even possible to put into NIC into fib 1? As I understand it, the interfaces are static across all fibs, you just have different routing tables.

I would like the office where the servers are located to be able to access the server with SSH from the internal network especially when there is a problem with the dns or other on the internet that prevents us from accessing the public IP.
Click to expand...

You realise that the machine is perfectly accessible via its private IP from any client that has a 192.168.1.x address, even if its default gateway is via a different interface?
 
usdmatt said:
You realise that the machine is perfectly accessible via its private IP from any client that has a 192.168.1.x address, even if its default gateway is via a different interface?
Click to expand...

It is an either or situation. I can either have it as a private ip and accessible via ssh from within the office or I can have a public ip and it is accessible from both as long as the DNS or something else doesn't flub up. I have had it where I cannot access it via the public ip because of outside issues. I would like to be able to at least have a type of solution on that.
 
Some work I have been doing:

Current /etc/rc.conf

Code: 
hostname="postal.brendhanhorne.com"
ifconfig_em0="DHCP"
default_router="192.168.1.255"
ifconfig_em1="up"
static_routes="fibnet fibgate"
route_fibnet="-net 77.77.77.76/28 -interface em1 -fib 1"
route_fibgate="default 77.77.77.77 -fib 1"
sshd_enable="YES"
moused_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
# SSH
sshd="YES"

Here are the results of the ifconfig
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b8
  inet 192.168.1.143 netmask 0xffffff00 broadcast 192.168.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:87:32:b9
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

And then I end up with a numbers that are off with a netstat command.
it say 77.77.77.74. It shouldn't. I can't ping it either.

# setfib 1 netstat -rn
Code: 
Routing tables (fib: 1)

Internet:
Destination  Gateway  Flags  Netif Expire
default  77.77.77.74  UGS  em1
77.77.77.74/28  00:30:48:87:32:b9  US  em1

Internet6:
Destination  Gateway  Flags  Netif Expire
fe80::%lo0/64  link#3  U  lo0

So basically nothing has changed. And I am getting information from a netstat command that only helps to confuse.
 
Last edited by a moderator:
It is an either or situation. I can either have it as a private ip and accessible via ssh from within the office or I can have a public ip and it is accessible from both as long as the DNS or something else doesn't flub up. I have had it where I cannot access it via the public ip because of outside issues. I would like to be able to at least have a type of solution on that.
Click to expand...

It isn't an either or situation though. If you have a public IP and gateway on the server it's accessible from the Internet. If you then add a private interface on your LAN, it's accessible from the LAN using the 192 address, even if the public interface is completely down; It might just take a few seconds longer to connect because the DNS lookup will fail.

I don't think a small ssh delay is worth the hassle of trying to engineer two gateways (something that is nowhere near as common as people seem to think), which will probably make no difference because the dns resolver is not going to somehow know which routing table it needs to use to access a dns server; It will just use the routing table the application (i.e ssh) was set to use when it was started.
 
usdmatt said:
Is it even possible to put into NIC into fib 1? As I understand it, the interfaces are static across all fibs, you just have different routing tables.
Click to expand...
Code: 
 # ifconfig vmnet0
vmnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:bc:37:00:00
        inet 10.255.255.3 netmask 0xfffffffe broadcast 255.255.255.255
        nd6 options=8<IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
        fib: 3
See man ifconfig:
fib fib_number
Specify interface FIB. A FIB fib_number is assigned to all
frames or packets received on that interface. The FIB is not
inherited, e.g., vlans or other sub-interfaces will use the
default FIB (0) irrespective of the parent interface's FIB. The
kernel needs to be tuned to support more than the default FIB
using the ROUTETABLES kernel configuration option, or the
net.fibs tunable.
Click to expand...
 
Small examples:
Code: 
ifconfig_vtnet0="10.0.12.1/24 fib 2"
ifconfig_vtnet1="10.0.13.1/24 fib 3"
ifconfig_vtnet2="10.0.14.1/24 fib 4"
static_routes="ISP1 ISP2 ISP3"
route_ISP1="-fib 2 default 10.0.12.2"
route_ISP2="-fib 3 default 10.0.13.3"
route_ISP3="-fib 4 default 10.0.14.4"
 
Understudy said:
Hi All,

I am looking to setup two physical nics with two different ip addresses and two different gateways. The basics:

Code: 
10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 19:31:38 UTC 2015 
root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386

I am looking to make one nic part of the private ip range 192.168.1.0/24 with a gateway of 192.168.1.255.
Click to expand...

I've never worked with fib before but I can tell you that you are trying to set the gateway to the broadcast address of that subnet. .254 is the last usable address in that range.
 
magikman said:
I've never worked with fib before but I can tell you that you are trying to set the gateway to the broadcast address of that subnet. .254 is the last usable address in that range.
Click to expand...
Yes, that was me having a brain fart. My apologies for the confusion.
 
You must log in or register to reply here.
Back
Top