Solved Routing between bridged interfaces

erley

erley

Hello,
I'm having strange problem with routing between bridged network interfaces.
Code: 
/internet/
    | ae0
+---------+                                 +--------+
|         |---wlan0 - - - - - - - - - - - - | Host2  |
|  Host1  |                +--------+       +--------+
|         |---bge0 - - - - | Host3  |
+---------+                +--------+
Host2 (192.168.0.2) and Host3 (192.168.0.3) cannot communicate between them, but can connect to Host1.
Bridge is assembled from wifi(wlan0) and ethernet(bge0) cards and has several IPs (used for jails):
Code: 
>ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:2c:67:a8:27:00
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet 192.168.0.18 netmask 0xffffffff broadcast 192.168.0.18
        inet 192.168.0.16 netmask 0xffffffff broadcast 192.168.0.16
        inet 192.168.0.15 netmask 0xffffffff broadcast 192.168.0.15
        inet 192.168.0.14 netmask 0xffffffff broadcast 192.168.0.14
        inet 192.168.0.17 netmask 0xffffffff broadcast 192.168.0.17
        id 00:10:18:5a:7b:d6 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:10:18:5a:7b:d6 priority 32768 ifcost 0 port 0
        member: bge0 flags=1e7<LEARNING,DISCOVER,STP,EDGE,AUTOEDGE,PTP,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000 proto rstp
                role designated state forwarding
        member: wlan0 flags=167<LEARNING,DISCOVER,STP,EDGE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 370370 proto rstp
                role designated state forwarding
        groups: bridge
Extract from /etc/rc.conf:

Code: 
routed_enable="YES"
gateway_enable="YES"

ifconfig_bge0="mtu 1500 link0 up"

wlans_ath0="wlan0"
create_args_wlan0="wlanmode hostap"
ifconfig_wlan0="up ssid Erley_WiFi mode 11g mtu 1500 txpower 50"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm wlan0 addm bge0 up"

ifconfig_bridge0_alias0="inet 192.168.0.1 netmask 255.255.255.0"
ifconfig_bridge0_alias1="inet 192.168.0.14 netmask 255.255.255.255"
ifconfig_bridge0_alias1="inet 192.168.0.15 netmask 255.255.255.255"
ifconfig_bridge0_alias2="inet 192.168.0.16 netmask 255.255.255.255"
ifconfig_bridge0_alias3="inet 192.168.0.17 netmask 255.255.255.255"
ifconfig_bridge0_alias4="inet 192.168.0.18 netmask 255.255.255.255"
Firewall is disabled and allows any traffic (just to simplify things at the moment).
Routing tables (X.Y.Z is public IP address):
Code: 
>sudo netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            X.Y.Z.254     UGS         ae0
X.Y.Z.0/24    link#2             U           ae0
X.Y.Z.67      link#2             UHS         lo0
127.0.0.1          link#3             UH          lo0
192.168.0.0/24     192.168.0.1        U       bridge0
192.168.0.1        link#5             UHS         lo0
192.168.0.14       192.168.0.14       UH          lo0
192.168.0.14/32    link#5             U       bridge0
192.168.0.15       192.168.0.15       UH          lo0
192.168.0.15/32    link#5             U       bridge0
192.168.0.16       192.168.0.16       UH          lo0
192.168.0.16/32    link#5             U       bridge0
192.168.0.17       link#5             UHS         lo0
192.168.0.17/32    link#5             U       bridge0
192.168.0.18       192.168.0.18       UH          lo0
192.168.0.18/32    link#5             U       bridge0
Am I missing something? Any help is welcome :)
 
This is of course the case:
Code: 
>ifconfig wlan0
wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:1e:58:48:11:11
    groups: wlan
    ssid Erley_WiFi channel 11 (2462 MHz 11g) bssid 00:1e:58:48:11:11
    regdomain ETSI indoor ecm authmode WPA privacy MIXED deftxkey 2
    TKIP 2:128-bit TKIP 3:128-bit txpower 20 scanvalid 60 protmode CTS wme
    burst dtimperiod 1 -dfs
    media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
    status: running
>ifconfig bge0
bge0: flags=9943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,LINK0,MULTICAST> metric 0 mtu 1500
    options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
    ether 00:10:18:5a:7b:d6
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
 
erley said:
I'm having strange problem with routing between bridged network interfaces.
Click to expand...
Bridges are a layer 2 connection, routing happens on layer 3. There is no routing between bridge members, they're all on the same broadcast domain.
 
SirDice said:
Bridges are a layer 2 connection, routing happens on layer 3. There is no routing between bridge members, they're all on the same broadcast domain.
Click to expand...
You are right, both interfaces are in the same broadcast domain.
My problem is that hosts connected to different bridged interfaces do not see each other, but can see Host1.
I would appreciate if someone can help me to debug what goes wrong...
 
Today I tried to simplify my network topology a bit more to eliminate possible causes.
I added 3rd ethernet network interface to bridge0:
Code: 
> ifconfig re0
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 00:13:f7:54:eb:f4
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
> ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:2c:67:a8:27:00
    inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
    inet 192.168.0.18 netmask 0xffffffff broadcast 192.168.0.18
    inet 192.168.0.16 netmask 0xffffffff broadcast 192.168.0.16
    inet 192.168.0.15 netmask 0xffffffff broadcast 192.168.0.15
    inet 192.168.0.14 netmask 0xffffffff broadcast 192.168.0.14
    inet 192.168.0.17 netmask 0xffffffff broadcast 192.168.0.17
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 2000000
    member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
    member: wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 370370
    groups: bridge
And connected it to Host2 (as shown on schema in my first message) directly with crosswire cable.
It eliminates possible misconfiguration of my wlan0 interface working in hostap mode.

Again the same problem, even being connected directly to Host1, both Host2&3 cannot ping each other, but can perfectly connect to Host1 itself. Something is still wrong...

PS
I even tried to remove wlan0 from bridge0:
Code: 
> ifconfig bridge0 deletem wlan0
with the same result.
 
Hi, just a quick thought.
Can Host1 ping Host2 and Host3?
Isn't there any possibility that Host2 and Host3 block incoming packets?
 
Hello genneko,
genneko said:
Hi, just a quick thought.
Can Host1 ping Host2 and Host3?
Isn't there any possibility that Host2 and Host3 block incoming packets?
Click to expand...
Yes, Host1 <--> Host2 and Host1 <--> Host3 pings pass ok in both directions.
There is no firewall at all on Host2 and Host3.
The firewall on Host1 was disabled (default to allow all) during tests.
All hosts have IPs in 192.168.0.0/24.
Very strange indeed...
 
Ah, I see. Thank you for your clarification.
Did you capture packets on the member interfaces (re0 and bge0) and bridge0 while running ping from Host2 to Host3?
tcpdump -i re0
tcpdump -i bge0
tcpdump -i bridge0

If the bridge actually doesn't forward packets, Host2 cannot get an ARP reply from Host3 and you would see on re0 that Host2 indefinitely transmits ARP requests. Is that the case?
 
I use bridged interfaces in a case more or less similar and I have no problem.
I wonder why you use link0 keyword on bge0? What is the effect? Have you tried without?

Also can you try without ip alias for bridge0?
 
Many thanks genneko! You pointed me to the right direction.
genneko said:
Did you capture packets on the member interfaces (re0 and bge0) and bridge0 while running ping from Host2 to Host3?
tcpdump -i re0
tcpdump -i bge0
tcpdump -i bridge0

If the bridge actually doesn't forward packets, Host2 cannot get an ARP reply from Host3 and you would see on re0 that Host2 indefinitely transmits ARP requests. Is that the case?
Click to expand...
You are right, it's time to use real tools :)
I think we get closer to solution, today I did this scenario:
  • cleared ARP cache on all hosts with "arp -d -a",
  • started 2 tcpdumps on Host1 on bge0 and re0,
  • started pinging from Host2 to Host3.
I can see ARP request/reply and ICMP requests without replies on re0 (link between Host1 and Host2):
Code: 
# tcpdump -i re0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:36:52.394247 IP Host2.60065 > Host1.domain: 55004+ A? Host3. (30)
23:36:52.395803 IP Host2 > Host3: ICMP echo request, id 27532, seq 1, length 64
23:36:53.413053 IP Host2 > Host3: ICMP echo request, id 27532, seq 2, length 64
23:36:54.437141 IP Host2 > Host3: ICMP echo request, id 27532, seq 3, length 64
23:36:55.461061 IP Host2 > Host3: ICMP echo request, id 27532, seq 4, length 64
23:36:56.484981 IP Host2 > Host3: ICMP echo request, id 27532, seq 5, length 64
23:36:57.509210 IP Host2 > Host3: ICMP echo request, id 27532, seq 6, length 64
23:36:57.605068 ARP, Request who-has Host3 tell Host2, length 46
23:36:57.605162 ARP, Reply Host3 is-at 74:d4:35:80:c7:ac (oui Unknown), length 46
23:36:58.533115 IP Host2 > Host3: ICMP echo request, id 27532, seq 7, length 64
23:36:59.557044 IP Host2 > Host3: ICMP echo request, id 27532, seq 8, length 64
23:37:00.580972 IP Host2 > Host3: ICMP echo request, id 27532, seq 9, length 64
23:37:01.605201 IP Host2 > Host3: ICMP echo request, id 27532, seq 10, length 64
23:37:02.629257 IP Host2 > Host3: ICMP echo request, id 27532, seq 11, length 64
On bge0 (link between Host1 and Host3) I can see ARP request and reply, but no ICMP packets coming:
Code: 
# tcpdump -i bge0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:36:57.605081 ARP, Request who-has Host3 tell Host2, length 46
23:36:57.605154 ARP, Reply Host3 is-at 74:d4:35:80:c7:ac (oui Unknown), length 46
I returned to my IPFW on Host1 as something looks wrong with it. Rules were simplified to this ("ipfw zero" done before the tests):
Code: 
# ipfw list
00100   4676    445652 allow ip from any to any via lo0
00200  54842  29145407 allow ip from any to any via bridge0
00300    745    152046 deny log logamount 5 ip from any to any
65535      0         0 allow ip from any to any
My understanding was that rules 100 and 200 would allow everything on loopback and bridged interfaces.
There is nothing related to ICMP in /var/log/security.
To test if I can eliminate completely any cause from firewall, I removed rules 100,200,300 leaving "allow ip from any to any" only.
And in this case pinging worked as it should!
Now I'm puzzled about what rules should I put to allow any traffic on my local network?
Obviously I cannot keep running with open firewall and have to customize what to permit on the outer network interface.
 
Emrion said:
I use bridged interfaces in a case more or less similar and I have no problem.
I wonder why you use link0 keyword on bge0? What is the effect? Have you tried without?

Also can you try without ip alias for bridge0?
Click to expand...
Thank you for your notes!
I stopped jails and removed aliases until the cause of connectivity problem is resolved.
As for "link0" I don't remember why I put it, have just removed, but it doesn't change anything.

BTW, could you point me what IPFW rules you use to allow all local traffic on your bridged interfaces?
Of course if you use IPFW, as I'd like to stay with it at the moment.
 
Well, things seem to be clearer now.
I added few new rules to IPFW to have this:
Code: 
> ipfw -a list
00100  20223   2090826 allow ip from any to any via lo0
00110 563151 452912380 allow ip from any to any via bridge0
00120     18      1512 allow log icmp from any to any icmptypes 8 keep-state :default
00130     50     11890 allow icmp from any to any icmptypes 3
00140      0         0 allow icmp from any to any icmptypes 11
01000    579     26517 deny log logamount 5 ip from any to any
65535 917675 746417087 allow ip from any to any
and now pings pass well between Host2 and Host3.
IPFW rule 120 logged this:
Code: 
> cat /var/log/security 
Jan 26 01:06:35 Host1 kernel: ipfw: 120 Accept ICMP:8.0 192.168.0.3 192.168.0.2 out via re0
Jan 26 01:06:35 Host1 kernel: ipfw: 120 Accept ICMP:0.0 192.168.0.2 192.168.0.3 in via re0
Jan 26 01:06:35 Host1 syslogd: last message repeated 1 times
Jan 26 01:06:35 Host1 kernel: ipfw: 120 Accept ICMP:0.0 192.168.0.2 192.168.0.3 out via bge0
Jan 26 01:06:36 Host1 kernel: ipfw: 120 Accept ICMP:8.0 192.168.0.3 192.168.0.2 in via bge0
Jan 26 01:06:36 Host1 syslogd: last message repeated 1 times
In other words, ICMP-related rules must precise underlying physical interface name in "via" option.
I don't know if it can be allowed once for all the bridge0 children. Do I miss something?
Anyway, the solution is quite clear now.
 
Hi erley,
So glad to hear you got closer to the solution.

Altough I know very little about IPFW-specific things (I'm using PF),
if_bridge(4)'s PACKET FILTERING section gave me some clue.

What I noted are the following sentences.
When filtering is enabled, bridged packets will
pass through the filter inbound on the originating interface, on the
bridge interface and outbound on the appropriate interfaces.
...
The packets destined to the bridging host will be seen by the filter on
the interface with the MAC address equal to the packet's destination MAC.
Click to expand...

With those in mind, I guess the behavior in question is related to whether a packet is destined to the bridge's MAC address (Host2/3 to Host1) or not (bridged between Host2 and Host3), rather than whether it's ICMP or not.

From the man page's description, I expect that a packet from Host2 to Host3 goes through the packet filter three times (re0, bridge0 and bge0), while a packet from Host2 to Host1 is processed only once on bridge0 (because it has the destination IP of the packet). I think it matches the observed behavior.

And the manual also says that filtering on a bridge and/or its members can be disabled by sysctl variables. If you don't need any filtering on the bridge, you can try disabling it.
 
Hello genneko,
The if_bridge(4) is the page a read last night :)
We were looking at the same thing.

My sysctl's I had during my previous tests were:
Code: 
> sysctl net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 1
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 1
net.link.bridge.pfil_onlyip: 1
Which seems to be default (I did not touch it in my /etc/sysctl.conf).
Today I tried different combinations, but no luck - IPFW always filters on bridge0 underlying network interfaces.
The rule
Code: 
allow ip from any to any via bridge0
is not enough to allow all traffic on all the bridged network interfaces.
So I finally added lines for each physical interface:
Code: 
allow ip from any to any via wlan0
allow ip from any to any via bge0
I removed re0 as it was a temporary thing for tests only.
My current sysctl's for bridge are:
Code: 
> sysctl net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 1
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0
All the traffic between bridged interfaces is now working ok.
I think there should be a way to avoid specifying all the bridged interfaces in IPFW rules, but still didn't figure out what sysctl values I have to put.
For example, what is the difference between net.link.bridge.pfil_member and net.link.bridge.pfil_local_phys ?
I learnt something, but I still have to read more docs. I'd like to understand what happens in depth.
 
I got the same question when I first read the manual, and I also learned a bit more.
To me, it seems that those variables are used to determine where the filtering occurs to what types of packets as follows.

Code: 
    (1)          (2)        (3)         (4)
     in           in       out         out
 ---> [wlan0] ---> [bridge0] ---> [bge0] --->

pfil_bridge: When enabled, bridged packets are filtered on the bridge interface at (2) and (3)
pfil_member: When enabled, bridged packets are filtered on the member intefaces at (1) and (4)
pfil_local_phys: This is not for bridged packets. When enabled, packets destined for the bridging host's MAC address (sent to the host or routed through the host) are filtered on the inbound member interface at (1), in addition to the receiving bridge interface at (2)

If my understanding is correct, you might be able to allow traffic between wlan0 and bge0 by simply disabling packet filtering on bridged packets. I think this can be achieved by changing the following sysctl's from their defaults.

Code: 
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0

And with the above setting, you would no longer need those IPFW rules you added yesterday.

Code: 
allow ip from any to any via wlan0
allow ip from any to any via bge0

This (disabling bridged packet filtering) shouldn't stop filtering on layer3 packets - packets to/from bridge0's IP address (Host2/3 <-> Host1) and packets routed between bridge0 and other IP interfaces (ae0 in your example). So the following rule would be still needed.

Code: 
allow ip from any to any via bridge0
 
I'm morbidly curious . I'm thrilled that FreeBSD works well as a bridge; it's networking flexibility and advancements in networking has never ceased to amaze me which is why I adore it. FreeBSD has always been able to replace a Cisco firewall or router with its supported network interfaces and open source routing protocols.

At the risk of hijacking this thread and possibly being modded, I have to ask this because I've been majorly shaking my head reading this thread as it majorly goes against my network design philosophy. If this has been addressed in another thread please link me in that direction.

Why are you bridging your wired and WiFi LANs through FreeBSD?
Why not put them on separate VLANs, route between them and call it a day?

Wired is obviously more secure than WiFi as you have L1 security; a hacker needs to obtain physical access to your wired network. Everyone has access to the airwaves which is screaming to be a separate VLAN for security reasons. You implement additional security access policies and protocols on the WiFi VLAN(s) and possibly have a bastion host or tar pit to deal with intruders.
 
Hi LVLouisCyphre,
LVLouisCyphre said:
Why are you bridging your wired and WiFi LANs through FreeBSD?
Why not put them on separate VLANs, route between them and call it a day?
Wired is obviously more secure than WiFi as you have L1 security; a hacker needs to obtain physical access to your wired network. Everyone has access to the airwaves which is screaming to be a separate VLAN for security reasons. You implement additional security access policies and protocols on the WiFi VLAN(s) and possibly have a bastion host or tar pit to deal with intruders.
Click to expand...
This is a simple SOHO network and I think similar configurations are used widely around (so this thread would help someone :) ).
In this particular case I'm not looking for a 100% bullet-proof network, but I'm pretty sure it already gives me very good security and seamless connectivity for all local network clients.
Here in this post I temporary simplified config for the sake of simplicity and to get closer to the most optimal and simple solution.

As yourself, what I really like in FreeBSD and why I use it - it can be configured to perform the way you want.
I like the fact that I control how by BSD box behaves and with FreeBSD I learnt many new things I was curious about.
 
Hello genneko,
genneko said:
I got the same question when I first read the manual, and I also learned a bit more.
To me, it seems that those variables are used to determine where the filtering occurs to what types of packets as follows.

Code: 
    (1)          (2)        (3)         (4)
     in           in       out         out
---> [wlan0] ---> [bridge0] ---> [bge0] --->

pfil_bridge: When enabled, bridged packets are filtered on the bridge interface at (2) and (3)
pfil_member: When enabled, bridged packets are filtered on the member intefaces at (1) and (4)
pfil_local_phys: This is not for bridged packets. When enabled, packets destined for the bridging host's MAC address (sent to the host or routed through the host) are filtered on the inbound member interface at (1), in addition to the receiving bridge interface at (2)

If my understanding is correct, you might be able to allow traffic between wlan0 and bge0 by simply disabling packet filtering on bridged packets. I think this can be achieved by changing the following sysctl's from their defaults.

Code: 
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0

And with the above setting, you would no longer need those IPFW rules you added yesterday.

Code: 
allow ip from any to any via wlan0
allow ip from any to any via bge0

This (disabling bridged packet filtering) shouldn't stop filtering on layer3 packets - packets to/from bridge0's IP address (Host2/3 <-> Host1) and packets routed between bridge0 and other IP interfaces (ae0 in your example). So the following rule would be still needed.

Code: 
allow ip from any to any via bridge0
Click to expand...
Yes, you are totally right!
After reading man if_bridge carefully several times, I set sysctl's as you suggested, now it looks this way:
Code: 
# sysctl net.link.bridge
net.link.bridge.ipfw: 1
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0

# ipfw list
00100 allow ip from any to any via lo0
00110 allow ip from any to any via bridge0
01000 deny log logamount 5 ip from any to any
and bingo! all works now - all packets go through bridge between interfaces and all the hosts can ping each other and connect.
To resume, it was important to set proper sysctl's and not rely on its default values (which were more restrictive and it is good btw).
The problem is solved now.
Thank you all who helped me chasing for the solution, with special thanks to genneko!
 
erley said:
This is a simple SOHO network and I think similar configurations are used widely around (so this thread would help someone :) ).
Click to expand...
What's simple to me may not be simple to others. I've run BGP, EIGRP and OSPF on my network for giggles; granted I could have ran a small to medium ISP out of the loft in a previous home. I've seen many problems caused with bridging (L2 loops) and violation of the 5-4-3 ethernet rule which is probably why I have an aversion to them.
 
You must log in or register to reply here.
Back
Top