Solved expose VNET jail to WAN

N

nomih

Hello,

I have a webserver configured in a VNET jail.

The jail has its own IP 192.168.1.11 and I can access it in my LAN network without issue.

However when I'm outside of my network, I can't reach the webserver. I got either timed out or "connection reset by peer" errors.

I have the forward port rule setup in my router configuration with the jail ip targeted but it does not work. It's a GUI but the rule says:

Code: 
protocol: tcp
source ip: all
destination ip: 192.168.1.11 (the jail)
port WAN: 80
port LAN: 80
 
I'd check the following and based on findings determine next steps.

- routing of the jail
- firewall of the jail
- make sure your browser doesn't want to default to https

- use curl from outside to do request / use tcpdump on your jail host to make sure you actually see the traffic
 
- routing of the jail
Click to expand...
I followed the handbook

docs.freebsd.org

Chapter 17. Jails and Containers

Jails improve on the concept of the traditional chroot environment in several ways
docs.freebsd.org docs.freebsd.org

And this is what it looks like

my host /etc/rc.conf
Code: 
hostname="pouiksbrain"
ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
jail_enable="YES"
jail_parallel_start="YES"
defaultrouter="192.168.1.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.1.253/24 addm re0 up"
ifconfig_re0="up"

my /etc/jail.conf.d/blog.conf included in /etc/jail.conf
Code: 
blog {
     exec.consolelog = "/var/log/jail_console.${name}.log";

     allow.raw_sockets;
     exec.clean;
     mount.devfs;

     host.hostname = "${name}";
     path = "/usr/local/jails/containers/${name}";

     # VNET/VIMAGE
     vnet;
     vnet.interface = "${epair}b";

     # NETWORKS/INTERFACES
     $id = "11";
     $ip = "192.168.1.${id}/24";
     $gateway = "192.168.1.1";
     $bridge = "bridge0";
     $epair = "epair${id}";

     # ADD TO bridge INTERFACE
     exec.prestart  = "/sbin/ifconfig ${epair} create up";
     exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
     exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
     exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
     exec.start    += "/sbin/route add default ${gateway}";
     exec.start    += "/bin/sh /etc/rc";
     exec.stop      = "/bin/sh /etc/rc.shutdown";
     exec.poststop  = "/sbin/ifconfig ${bridge} deletem ${epair}a";
     exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}

The output of ifconfig on the host
Code: 
re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 9c:6b:00:2f:0a:03
        inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::9e6b:ff:fe2f:a03%re0 prefixlen 64 scopeid 0x1
        inet6 2a01:e0a:895:90b0:9e6b:ff:fe2f:a03 prefixlen 64 autoconf pltime 86400 vltime 86400
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:ff:f8
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair11a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
epair11a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: jail:blog
        options=8<VLAN_MTU>
        ether 02:e9:e6:0e:03:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

And in the jail
Code: 
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair11b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e9:e6:0e:03:0b
        inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

- firewall of the jail
Click to expand...

I didn't configured any. Is there some default configuration enabled?

- make sure your browser doesn't want to default to https
Click to expand...

Yes I made sure of that. Also I tested with a port scanner tool and the 80 port appears "timed out".

- use curl from outside to do request / use tcpdump on your jail host to make sure you actually see the traffic
Click to expand...

No traffic on the host from the outside and neither from the inside
 
nomih said:
$gateway = "192.168.1.1";
Click to expand...
In the jail, the gateway can't be its own ip address. I think it should be the one of your router/internet box.
It's generally useless to give an ip address to a bridge.

Edit: indeed, there is confusion between the different informations Op gives. Let him/her correct the thing.
 
Gateway looks correct. IP address should be removed from the bridge but doesn't do any harm. The biggest issue is defining ifconfig_re0 twice in rc.conf.

Code: 
ifconfig_re0="DHCP"
{...}
ifconfig_re0="up"

You need to be aware settings in rc.conf are variables, not commands. The second ifconfig_re0 overrules the first.

Code: 
#!/bin/sh

FOO="BAR"
FOO="NOTBAR"

echo $FOO

nomih said:
However when I'm outside of my network, I can't reach the webserver.
Click to expand...
You will need to configure port forwarding on your ISP router/modem (ports 80 for HTTP, 443 for HTTPS), nothing on your LAN is accessible from the internet because your local network uses a private range network and that traffic is NAT'ed.
 
sysctl net.inet.ip.forwarding?

edit:
also, what kind of uplink is this? cellular networks often use RFC1918 IP ranges and are therefore not routable from the outside, same goes for DSlite. Also some ISPs will simply block a lot of incoming ports (which violates net neutrality amongst other laws if you are in the EU)
 
I got it! Thanks to you for pointing

In the jail, the gateway can't be its own ip address. I think it should be the one of your router/internet box.
Click to expand...

The gateway ( 192.168.1.1) was not correct. The correct is 192.168.1.254.

I corrected /etc/jail.conf.d/blog.conf with $gateway = "192.168.1.254", restarted the jail and... voilà!
 
SirDice said:
way looks correct. IP address should be removed from the bridge but doesn't do any harm. The biggest issue is defining ifconfig_re0 twice in rc.conf.

ifconfig_re0="DHCP" {...} ifconfig_re0="up"
Click to expand...
Thanks for pointing that out. I didn't even notice.
 
You must log in or register to reply here.
Back
Top