propper routing and gateway configuration between 2 lan IPs

[DemoNIck]

There is a network topology which cannot be changed/altered as following:

[FILESERVER]---192.168.254.1---->|======|
[LANPC1]--------192.168.254.x---->| SWITCH |<--192.168.254.254--[MODEM]--->INTERNET
[LANPC2]--------192.168.254.x---->|======|

The FILESERVER (FreeBSD 11.1-RELEASE) is running along several other services, a local authoritative, validating, recursive, caching DNS (preinstalled unbound server) and a local authoritative DHCP server (ports/isc-dhcp43-server)

The MODEM, which in fact is a modem/router provided by my ISP (Speedport W 724V) has a custom firmware and many configuration options are locked (i.e. I cannot configure it in bridge mode and connect to the internet via pppoe from the FILESERVER). So the only option that I can think of is to use the MODEM to connect to the internet via it's own interface and route the traffic to my SWITCH via it's lan port (192.168.254.254). NAT is enabled on the MODEM. No other service is running on the MODEM.

What I want, is to route all traffic (LANPC1, LANPC2 etc), through the FILESERVER. The FILESERVER, when needed, will route/nat the lan traffic to the internet through the MODEM and vice versa (traffic from the internet will be routed/nat through the MODEM to the FILESERVER).

Right now, what I do (which is wrong) is this:

1. On the FILESERVER I have configured as default router the MODEM:
/etc/rc.conf: defaultrouter="192.168.254.254"

2. On the FILESERVER I have configured the DHCPD server to announce to the LAN dhcp clients as default gateway the FILESERVER:
/usr/local/etc/dhcpd.conf

subnet 192.168.254.0 netmask 255.255.255.0 {
  range 192.168.254.240 192.168.254.253;
  option routers 192.168.254.1;
}

The DHCP server is working as expected. LAN PCs get the correct IP(192.168.254.2x), gateway (192.168.254.1) and DNS (192.168.254.1)
The LAN PCs and the FILESERVER can access the Internet.
But the routing is of course wrong.

On the FILESERVER:

root@fileserver:~ # netstat -nr -4
Routing tables
Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.254.254    UGS       lagg0
127.0.0.1          link#3             UH          lo0
192.168.254.0/24   link#4             U         lagg0
192.168.254.1      link#4             UHS         lo0

root@fileserver:~ # traceroute www.in.gr
traceroute: Warning: www.in.gr has multiple addresses; using 213.133.127.247
traceroute to www.in.gr (213.133.127.247), 64 hops max, 40 byte packets
 1  modem (192.168.254.254)  0.629 ms  0.554 ms  0.515 ms
 2  80.106.108.247 (80.106.108.247)  7.565 ms  7.167 ms  7.602 ms
 3  79.128.228.193 (79.128.228.193)  16.898 ms  13.975 ms  14.083 ms
 4  79.128.228.241 (79.128.228.241)  14.767 ms
    79.128.228.61 (79.128.228.61)  14.480 ms  14.605 ms
 5  inet-athe7609k2.backbone.otenet.net (79.128.251.102)  26.855 ms  14.574 ms  13.686 ms
 6  athe7609k2-inet.backbone.otenet.net (79.128.251.101)  13.273 ms  13.584 ms  13.062 ms
 7  athe-crsa-nyma-crsa-2.backbone.otenet.net (79.128.224.37)  13.514 ms  14.961 ms  13.918 ms
.
.
.

if I try to traceroute a LANPC from the FILESERVER

root@fileserver:~ # traceroute pclan1.local.lan
traceroute to pclan1.local.lan (192.168.254.20), 64 hops max, 40 byte packets
 1  * * *
 2  * * *
.
.
.

which is expected, as the FILESERVER route all traffic through the MODEM.

From a LANPC:

C:\Users\user>tracert www.in.gr
Tracing route to www.in.gr [213.133.127.245]
over a maximum of 30 hops:
  1    <1 ms    <1 ms    <1 ms  fileserver.local.lan [192.168.254.1]
  2    <1 ms    <1 ms    <1 ms  modem.local.lan [192.168.254.254]
  3     8 ms     8 ms     7 ms  80.106.108.247
  4    15 ms    14 ms    15 ms  79.128.228.193
.
.
.

It makes sense.

If I traceroute from pclan1 to another pclan2 I get:

C:\Users\user>tracert pclan2.local.lan
Tracing route to pclan2.local.lan [192.168.254.21]
over a maximum of 30 hops:
  1    <1 ms    <1 ms    <1 ms  pclan2.local.lan [192.168.254.21]
Trace complete.

shouldn't the first hop be the FILESERVER?

Could you please propose the correct way the routing should be configured?
If, on the FILESERVER, I change the /etc/rc.conf: defaultrouter="192.168.254.254"
to defaultrouter="192.168.254.1", how could I "redirect" all traffic from 192.168.254.1 -> 192.168.254.254 and vise versa (from 192.168.254.254 -> 192.168.254.1)? Using static routes and/or through the pf?

Thank you in advance for your time.

Right now the FILESERVER configuration is:

/etc/rc.conf

# NETWORKING
hostname="fileserver.local.lan"
ifconfig_em0="up"
ifconfig_em1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto loadbalance laggport em0 laggport em1 192.168.254.1/24"
defaultrouter="192.168.254.254"
gateway_enable="YES"
local_unbound_enable="YES"
dhcpd_enable="YES"
dhcpd_ifaces="lagg0"
pf_enable="YES"
pf_flags=""
pf_rules="/usr/local/etc/pf.conf"

/etc/resolv.conf

domain local.lan
search local.lan
nameserver 127.0.0.1
nameserver 192.168.254.1
options edns0

/usr/local/etc/dhcpd.conf

option domain-name "local.lan";
option domain-name-servers 192.168.254.1;
default-lease-time 600;
max-lease-time 7200;
authoritative;
log-facility local7;

subnet 192.168.254.0 netmask 255.255.255.0 {
  range 192.168.254.240 192.168.254.253;
  option routers 192.168.254.1;
}

host PCLAN1 {
    hardware ethernet xx:xx:xx:xx:xx:xx;
    fixed-address 192.168.254.20;
}
host PCLAN2 {
    hardware ethernet xx:xx:xx:xx:xx:xx;
    fixed-address 192.168.254.21;
}

and for the local DNS the following configuration files
/var/unbound/unbound.conf

server:
    username: unbound
    directory: /var/unbound
    chroot: /var/unbound
    pidfile: /var/run/local_unbound.pid
    root-hints: "/var/unbound/root.hints"
    auto-trust-anchor-file: /var/unbound/root.key
    verbosity: 1
    interface: 0.0.0.0
    port: 53
    do-ip4: yes
    do-ip6: no
    do-udp: yes
    do-tcp: yes
    access-control: 192.168.254.0/24 allow
    hide-identity: yes
    hide-version: yes
    harden-glue: yes
    harden-dnssec-stripped: yes
    use-caps-for-id: yes
    cache-min-ttl: 3600
    cache-max-ttl: 86400
    prefetch: yes
    num-threads: 4
    msg-cache-slabs: 8
    rrset-cache-slabs: 8
    infra-cache-slabs: 8
    key-cache-slabs: 8
    rrset-cache-size: 256m
    msg-cache-size: 128m
    so-rcvbuf: 1m
   #  private-address: 192.168.254.0/24
    private-domain: "local.lan"
    unwanted-reply-threshold: 10000
    do-not-query-localhost: no
     val-clean-additional: yes

    local-zone: "local.lan." static
    local-data: "fileserver.local.lan. IN A 192.168.254.1"
    local-data: "PCLAN1.local.lan. IN A 192.168.254.20"
    local-data: "PCLAN2.local.lan. IN A 192.168.254.21"
    local-data-ptr: "192.168.254.1 fileserver.local.lan"
    local-data-ptr: "192.168.254.20 PCLAN1.local.lan"
    local-data-ptr: "192.168.254.21 PCLAN2.local.lan"

include: /var/unbound/forward.conf
include: /var/unbound/lan-zones.conf
include: /var/unbound/control.conf
include: /var/unbound/conf.d/*.conf

and
/var/unbound/forward.conf

forward-zone:
        name: .
        forward-addr: 195.170.2.2
        forward-addr: 195.170.0.1

finally the pf firewall is accepting anything for testing purposes
/usr/local/etc/pf.conf

pass in all keep state
pass out all keep state

 

[SirDice]

A plain (layer 2) switch has no IP address. Besides that, hosts in the same subnet can access each other directly and don't need a route/gateway. A route or gateway is only needed to access IP addresses outside of the subnet. So set the default gateway for all hosts to the modem's IP address.