Why can't my jail install a port or ftp in?

nonTechnical · Jan 7, 2016

Hiya,

I've just installed sysutils/ezjail, built my first jail, and can telnet out of it - so the network appears to work.

However, I can't install ports - they fail to download via FTP - and FTP connections don't work:

 # ftp ftp://ftp.freebsd.org/

Trying 96.47.72.72:21 ...

ftp: Can't connect to `96.47.72.72:21': Operation timed out

Trying 2610:1c1:1:606c::15:0:21 ...

ftp: Can't create socket for connection to `2610:1c1:1:606c::15:0:21': Protocol not supported

ftp: Can't connect to `ftp.freebsd.org:ftp'

ftp: Can't connect or login to host `ftp.freebsd.org:ftp'

Is this a built-in security measure of jails, or have I not yet configured my jail appropriately, or both?

Many thanks

SirDice · Jan 8, 2016

No, it's not a security measure. It's probably something that isn't configured correctly. How is the jail set up?

nonTechnical · Jan 9, 2016

SirDice said:
No, it's not a security measure. It's probably something that isn't configured correctly. How is the jail set up?

Thanks SirDice -

I followed the instructions from this site:

https://www.kirkg.us/posts/how-to-configure-a-freebsd-jail-on-a-digital-ocean-droplet/

which included setting up pf rules in the host. I've turned them on/off and still can't install ports.

Copying from the instructions, which show the same result on my server:

Code:

freebsd@hostname:~ % sudo pfctl -nvf /etc/pf.conf
ext_if = "vtnet0"
int_if = "lo1"
jail_net = "lo1:network"
nat on vtnet0 inet from 172.16.1.0/24 to any -> (vtnet0) round-robin

From the host:

Code:

$ ifconfig

vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 04:01:95:b8:23:01
   inet6 fe80::601:95ff:feb8:2301%vtnet0 prefixlen 64 tentative scopeid 0x1
   inet xxx.xxx.xxx.xxx netmask 0xffffff00 broadcast xxx.xxx.xxx.255
   inet 10.12.0.5 netmask 0xffff0000 broadcast 10.12.255.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet 10Gbase-T <full-duplex>
   status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 04:01:95:b8:23:02
   inet xx.xxx.xxx.239 netmask 0xffff0000 broadcast xx.xxx.255.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet 10Gbase-T <full-duplex>
   status: active
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet 172.16.1.1 netmask 0xffffff00
   inet 172.16.1.2 netmask 0xffffffff
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I've added 172.16.1.2 for a second jail I was to create. Tbh, I normally use a 192.168.x.x address range, so I'm a bit out of my depth using the address range from the instructions.

The jail has the same resolve.conf as the host -

Code:

nameserver 8.8.8.8
nameserver 8.8.4.4

From the host:

Code:

$ netstat -rn
Routing tables

Internet:
Destination  Gateway  Flags  Netif Expire
default  xxx.xxx.xxx.x  UGS  vtnet0
10.12.0.0/16  link#1  U  vtnet0
10.12.0.5  link#1  UHS  lo0
xx.xxx.0.0/16  link#2  U  vtnet1
xx.xxx.xxx.239  link#2  UHS  lo0
xxx.xxx.xxx.0/24  link#1  U  vtnet0
xxx.xxx.xxx.xxx  link#1  UHS  lo0
127.0.0.1  link#4  UH  lo0
172.16.1.1  link#5  UH  lo1
172.16.1.2  link#5  UH  lo1

Internet6:
Destination  Gateway  Flags  Netif Expire
::/96  ::1  UGRS  lo0
::1  link#4  UH  lo0
::ffff:0.0.0.0/96  ::1  UGRS  lo0
fe80::/10  ::1  UGRS  lo0
fe80::%vtnet0/64  link#1  U  vtnet0
fe80::601:95ff:feb8:2301%vtnet0  link#1  UHS  lo0
fe80::%lo0/64  link#4  U  lo0
fe80::1%lo0  link#4  UHS  lo0
ff01::%vtnet0/32  fe80::601:95ff:feb8:2301%vtnet0 U  vtnet0
ff01::%lo0/32  ::1  U  lo0
ff02::/16  ::1  UGRS  lo0
ff02::%vtnet0/32  fe80::601:95ff:feb8:2301%vtnet0 U  vtnet0
ff02::%lo0/32  ::1  U  lo0

From the host when trying to install a port in the jail:

Code:

$ netstat
Active Internet connections
Proto Recv-Q Send-Q Local Address  Foreign Address  (state)
tcp4  0  0 172.16.1.1.38427  pkg0.nyi.freebsd.http  SYN_SENT
tcp4  0  0 xxx.xxx.xxx.xxx.xxxxx  210-246-46-151.d.52876 ESTABLISHED
tcp4  0  36 xxx.xxx.xxx.xxx.xxxxx  210-246-46-151.d.52742 ESTABLISHED
udp4  0  0 172.16.1.1.syslog  *.*
udp4  0  0 localhost.ntp  *.*

Here's the output from trying to install nginx in the jail:

Code:

root@reverseproxy:/usr/ports/www/nginx # make install clean
===> Building/installing dialog4ports as it is required for the config dialog
===>  Cleaning for dialog4ports-0.1.5_2
===> Skipping 'config' as NO_DIALOG is defined
===>  License BSD2CLAUSE accepted by the user
===>  dialog4ports-0.1.5_2 depends on file: /usr/local/sbin/pkg - not found
===> Skipping 'config' as NO_DIALOG is defined
===>  License BSD2CLAUSE accepted by the user
=> pkg-1.6.2.tar.xz doesn't seem to exist in /var/ports/distfiles/.
=> Attempting to fetch http://files.etoilebsd.net/pkg/pkg-1.6.2.tar.xz
fetch: http://files.etoilebsd.net/pkg/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.us-east.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.eu.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.eu.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Operation timed out
=> Attempting to fetch http://distcache.us-west.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://mirror.shatow.net/freebsd/pkg/pkg-1.6.2.tar.xz
fetch: http://mirror.shatow.net/freebsd/pkg/pkg-1.6.2.tar.xz: Forbidden
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/pkg-1.6.2.tar.xz
fetch: http://distcache.FreeBSD.org/ports-distfiles/pkg-1.6.2.tar.xz: Protocol not supported
=> Couldn't fetch it - please try to retrieve this
=> port manually into /var/ports/distfiles/ and try again.
*** Error code 1

Stop.
make[5]: stopped in /basejail/usr/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make[4]: stopped in /basejail/usr/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make[3]: stopped in /basejail/usr/ports/ports-mgmt/dialog4ports
*** Error code 1

Stop.
make[2]: stopped in /basejail/usr/ports/ports-mgmt/dialog4ports
===> Options unchanged


===>  License BSD2CLAUSE accepted by the user
===>  nginx-1.8.0_3,2 depends on file: /usr/local/sbin/pkg - not found
===>  License BSD2CLAUSE accepted by the user
=> pkg-1.6.2.tar.xz doesn't seem to exist in /var/ports/distfiles/.
=> Attempting to fetch http://files.etoilebsd.net/pkg/pkg-1.6.2.tar.xz
fetch: http://files.etoilebsd.net/pkg/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.us-east.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.us-east.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.eu.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.eu.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://distcache.us-west.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz
fetch: http://distcache.us-west.FreeBSD.org/local-distfiles/portmgr/pkg-1.6.2.tar.xz: Protocol not supported
=> Attempting to fetch http://mirror.shatow.net/freebsd/pkg/pkg-1.6.2.tar.xz
fetch: http://mirror.shatow.net/freebsd/pkg/pkg-1.6.2.tar.xz: Forbidden
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/pkg-1.6.2.tar.xz
fetch: http://distcache.FreeBSD.org/ports-distfiles/pkg-1.6.2.tar.xz: Protocol not supported
=> Couldn't fetch it - please try to retrieve this
=> port manually into /var/ports/distfiles/ and try again.
*** Error code 1

Stop.
make[2]: stopped in /basejail/usr/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make[1]: stopped in /basejail/usr/ports/www/nginx
*** Error code 1

Stop.
make: stopped in /basejail/usr/ports/www/nginx

I guess the 'protocol not supported' is referring to ftp but I have no idea why it wouldn't be.

nonTechnical · Jan 10, 2016

I've just found this and am wondering if it solves the problem?

https://forums.freebsd.org/threads/pf-nat-and-jail.20742/

If so, could someone help me to adapt it to my pf.conf (above).

Thanks

ab2k · Jan 10, 2016

Hi,

Seems there is problem with NAT or network misconfiguration.
(IPv4 timing out on connection, ipv6 can connect to host but dies somewhere on negotiation).

Can you telnet 2610:1c1:1:606c::15:0 21 from jail ?

Please post /etc/rc.conf and output of jls command.

nonTechnical · Jan 11, 2016

Thanks ab2k,

Here's the host's /etc/rc.conf

Code:

hostname="reverseproxy"

# Ezjail Network Setup
cloned_interfaces="lo1"
ifconfig_lo1="inet 172.16.1.1 netmask 255.255.255.0"


#Disable logging from remote hosts and close that port,
#but still allow logging of localhost.
syslogd_flags="-ss"

update_motd="NO"


pf_enable="YES"
#pf_rules="/etc/pf.conf"
pflog_enable="YES"
pflog_logfile="/var/log/pf.log"


# Get date and time
# -------------------------------------------
# Disable ntpd to use OpenNTPD as more compatible for jails.
ntpd_enable="NO"
openntpd_enable="YES"
openntpd_flags="-sv"



#Enable Connections
# -------------------------------------------
sshd_enable="YES"


# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
# -------------------------------------------
dumpdev="NO"


# Restrict Sendmail to Localhost Only
# -------------------------------------------
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"


# Enable EzJail
# -------------------------------------------
ezjail_enable="YES"

At this time, I don't have any /etc/rc.conf in the jail - just using what sysutils/ezjail sets up using its global jail config.

Tbh, I'm a bit confused about how to configure rc.conf for jails, as I've tried to use one in the jail - tested setting a different hostname - but the sysutils/ezjail config file in the hosts /usr/jails/host file overrides it.

That IPv6 telnet address fails in the jail:

Code:

# telnet 2610:1c1:1:606c::15:0 21
Trying 2610:1c1:1:606c::15:0...
telnet: socket: Protocol not supported

Code:

# jls
  JID  IP Address  Hostname  Path
  1  172.16.1.1  reverseproxy  /usr/jails/reverseproxy

nonTechnical · Jan 11, 2016

I've also found this post that suggests the problem can be resolved by setting an environment variable to:

Code:

FTP_PASSIVE_MODE

http://serverfault.com/questions/166339/pf-firewall-issues-with-ftp-inside-freebsd-jail

However I've tested adding that line to the suggested files and fetch is still failing with:

Protocol not supported

SirDice · Jan 11, 2016

The host is missing:

Code:

gateway_enable="YES"

So there's never any routing being done between the jail and the host.

nonTechnical · Jan 11, 2016

SirDice said:
The host is missing:

Code:

gateway_enable="YES"

So there's never any routing being done between the jail and the host.

Damn, just tried and rebooted, and the jail still is complaining that fetch/ftp is an unsupported protocol when trying to install a port.

Is there any way I can check to see where ftp in could be disabled?

SirDice · Jan 11, 2016

I'm wondering why there are connections being made (tried) on IPv6 when there's no IPv6 configured.

And your NAT rule should be on vtnet1, not vtnet0, as that's your external interface.

nonTechnical · Jan 11, 2016

Here's the host's NICs again:

Code:

% ifconfig
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 04:01:99:30:45:01
   inet6 fe80::601:99ff:fe30:4501%vtnet0 prefixlen 64 tentative scopeid 0x1
   inet xxx.xxx.xxx.xxx netmask 0xfffff000 broadcast 159.203.207.255
   inet 10.12.0.6 netmask 0xffff0000 broadcast 10.12.255.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet 10Gbase-T <full-duplex>
   status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 04:01:99:30:45:02
   inet 10.134.98.71 netmask 0xffff0000 broadcast 10.134.255.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet 10Gbase-T <full-duplex>
   status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet 172.16.1.1 netmask 0xffffff00
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160

This may be different from my original post as I set up another server to reinstall everything and try to fix the same issue, with identical settings except for the external IP address, but it's having the same issue as the first.

I thought vtnet1 on each was the internal interface I set up during the droplet's creation - for droplet to droplet networking, and vtnet0 was the external interface?

SirDice · Jan 12, 2016

nonTechnical said:
I thought vtnet1 on each was the internal interface I set up during the droplet's creation - for droplet to droplet networking, and vtnet0 was the external interface?

You're correct. It wasn't clear which was which from your original post.

kpa · Jan 13, 2016

This line in vtnet0 configuration is suspicious:

Code:

inet6 fe80::601:99ff:fe30:4501%vtnet0 prefixlen 64 tentative scopeid 0x1

It says tentative which means the interface is halfway configured for IPv6 and I believe that's the reason why IPv6 connection is tried. Could you post your full /etc/rc.conf, what you posted above has nothing about the interface configurations.

nonTechnical · Jan 23, 2016

Thanks, that's my full /etc/rc.conf (from the host).

What am I missing in the way of interface config?

I followed this tutorial:

https://www.kirkg.us/posts/how-to-configure-a-freebsd-jail-on-a-digital-ocean-droplet/

ab2k · Jan 25, 2016

Hi,

I don't see ifconfig(8) parameters for vtnet* interfaces in your /etc/rc.conf. Please post your ifconfig(8) things from /etc/rc.conf (from main machine, not a jail) - mask away the IP's - i just want to see how you set it (IPv4/IPv6). Maybe you using DHCP to set it up for You?

You have 2 interfaces vtnet0, vtnet1 - you must set the addreses by yourself. And only those that provider giving You!

Why can't my jail install a port or ftp in?

nonTechnical

SirDice

Administrator

nonTechnical

nonTechnical

ab2k

nonTechnical

nonTechnical

SirDice

Administrator

nonTechnical

SirDice

Administrator

nonTechnical

SirDice

Administrator

kpa

nonTechnical

ab2k