jails Host can ping but not connect to VNET jail

DaLynX

DaLynX

I have two jails:
- 104 is a shared IP jail (classic) with an alias on the host's em0
- 115 is a vnet jail on vnet0.192

There is a bridge0 bridge with em0 and vnet0.192 as members

I cannot have the two jails talk together.
More precisely, 115 -> 104 works, but not 104 -> 115.
I did tests with netcat and tcpdump on the host's bridge0.
I can see SYN packets from 104 getting sent repeatedly but no response.
If I initiate from 115, I cannot see the initiation packet (??) but I can see the SYN+ACK coming in repeatedly from 104 too.

Any idea what's wrong and how to fix it?
 
Okay so I enabled bpf in the jail to get tcpdump running there. And now I am testing from the host directly. Still doesn't work.

Host
Code: 
ifconfig                                 em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 46:15:3c:47:74:9a
        inet 192.168.1.112 netmask 0xffffff00 broadcast 192.168.1.255           inet 192.168.1.104 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.106 netmask 0xffffff00 broadcast 192.168.1.255           inet 192.168.1.101 netmask 0xffffffff broadcast 192.168.1.101
        inet 192.168.1.107 netmask 0xffffff00 broadcast 192.168.1.255           media: Ethernet autoselect (10Gbase-T <full-duplex>)                    status: active                                                          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>            lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384                                                                           options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000                                       inet6 ::1 prefixlen 128                                                 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2                              groups: lo                                                              nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                       bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0                                                               ether 58:9c:fc:10:8c:37                                                 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15             maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200                root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0                member: vnet0.193 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                 ifmaxaddr 0 port 6 priority 128 path cost 2000                  member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>                       ifmaxaddr 0 port 1 priority 128 path cost 2000                  groups: bridge                                                          nd6 options=9<PERFORMNUD,IFDISABLED>                            vnet0.193: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500                                                     description: associated with jail: yarr as nic: epair0b
        options=8<VLAN_MTU>                                                     ether 02:ff:60:de:42:a8
        hwaddr 02:b3:9f:39:c0:0a                                                groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                     status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail
Code: 
ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:60:de:42:a9
        hwaddr 02:b3:9f:39:c0:0b
        inet 192.168.1.115 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::ff:60ff:fede:42a9%epair0b prefixlen 64 scopeid 0x7
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Test commands

- jail: nc -lvvv 12345
- host: nc 192.168.1.115 12345 -vvv

Network capture

From the jail side, tcpdump -n:
Code: 
tcpdump -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on epair0b, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:48:06.720297 ARP, Request who-has 192.168.1.115 tell 192.168.1.112, length 28
14:48:06.720311 ARP, Reply 192.168.1.115 is-at 02:ff:60:de:42:a9, length 28
14:48:06.720320 IP 192.168.1.112.58389 > 192.168.1.115.12345: Flags [S], seq 2197706155, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 2896660391 ecr 0], length 0
14:48:07.742712 IP 192.168.1.112.58389 > 192.168.1.115.12345: Flags [S], seq 2197706155, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 2896661416 ecr 0], length 0
14:48:09.962116 IP 192.168.1.112.58389 > 192.168.1.115.12345: Flags [S], seq 2197706155, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 2896663635 ecr 0], length 0
14:48:12.608419 ARP, Request who-has 192.168.1.115 tell 192.168.1.1, length 28
14:48:12.608435 ARP, Reply 192.168.1.115 is-at 02:ff:60:de:42:a9, length 28
14:48:14.183938 IP 192.168.1.112.58389 > 192.168.1.115.12345: Flags [S], seq 2197706155, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 2896667857 ecr 0], length 0

SYN packets come in, but they are ignored by the jail. Netcat remains blank.

Interestingly enough this jail has absolutely no problems connecting to the Internet.
 
I found this post: https://forums.freebsd.org/threads/vnet-jail-communication-with-the-host-system.86530/post-582439

I tried moving the host IP from em0 to bridge0 but it makes no difference.

But I think I found something: I did a new tcpdump capture with -vvv and it states some checksums are incorrect.

Code: 
16:21:20.404307 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)                                                          192.168.1.104.32394 > 192.168.1.115.12345: Flags [S], cksum 0x845a (incorrect -> 0x664d), seq 1814202172, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 3070624862 ecr 0], length 0
16:21:21.408912 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)                                                          192.168.1.104.32394 > 192.168.1.115.12345: Flags [S], cksum 0x845a (incorrect -> 0x625e), seq 1814202172, win 65535, options [mss 1460,nop,wscale 7,sackOK,TS val 3070625869 ecr 0], length 0

I tried disabling rxcsum and txcsum on em0 but it makes no difference.

I cannot disable vlanhwcsum. ifconfig returns without error but the option remains active.

I checked if the destination mac address was correct and it does match the ether value from ifconfig inside the jail.
 
Code: 
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS     bridge0
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#3             U       bridge0
192.168.1.101      link#2             UH          lo0
192.168.1.104      link#2             UHS         lo0
192.168.1.106      link#2             UHS         lo0
192.168.1.112      link#2             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#2                        URS         lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 link#2                        URS         lo0
fe80::%lo0/10                     link#2                        URS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         link#2                        URS         lo0

But in the meantime I tried to turn it off and on again and guess what, it works!

I have no idea happened. But I can see that TCP checksums are now displayed as correct in the network capture.
 
You must log in or register to reply here.
Back
Top