Solved Jail outgoing internet's IPv6 packet 50% fail problem.

epopen

epopen

Hi All.

My server's jail need outgoing internet connection for Let's encrypt OCSP Staple, but connect problem IPv6 only.
Detail

ifconfig

Code: 
wan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8000a<TXCSUM,VLAN_MTU,LINKSTATE>
        ether 00:1e:68:c4:e1:9e
        inet6 2001:b011:a480:592f:21e:68ff:fec4:e19e prefixlen 64
        inet6 fe80::21e:68ff:fec4:e19e%wan0 prefixlen 64 scopeid 0x1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.0.0.254 netmask 0xffffff00
        inet 10.0.0.2 netmask 0xffffffff
        inet 10.0.0.1 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x5
        inet6 fd00::ffff:ffff:fffe prefixlen 96
        inet6 fd00::ffff:a00:2 prefixlen 128
        inet6 fd00::ffff:a00:1 prefixlen 128
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1460
        inet6 fe80::547:4b0e:4df6:1f7c%ng0 prefixlen 64 scopeid 0x6
        inet 122.117.86.253 --> 168.95.98.254 netmask 0xffffffff
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Here:
  1. ng0: Virtual interface created by net/mpd5 (VDSL dial-up PPPoE).
  2. lo1: Virtual interface clone from lo0, attach all of jail.
  3. wan0: "Physical interface to VDSL modem", attached IPv6 address by net/dhcp6 and got from above interface 'ng0'.
  4. 10.0.0.2 / fd00::ffff:a00:2: HTTP server Jail's address.
  5. 10.0.0.1 / fd00::ffff:a00:1: DNS server Jail's address.

/etc/pf.conf

Code: 
ext_if          = "ng0"
ext_inet6_if    = "wan0"

set limit               { states 10000, frags 5000 }
set loginterface        $ext_if
set loginterface        $ext_inet6_if
set skip on             lo0
set skip on             lo1
set optimization        aggressive
set block-policy        drop
set state-policy        if-bound
set require-order       yes
scrub in all fragment reassemble

nat log on $ext_if inet6 from fd00::ffff:a00:2 to any -> ($ext_inet6_if)
nat log on $ext_if inet from 10.0.0.2 to any -> ($ext_if)

rdr on $ext_if inet6 proto { udp, tcp } from any to ($ext_inet6_if) port domain -> fd00::ffff:a00:1
rdr on $ext_if inet proto { udp, tcp } from any to ($ext_if) port domain -> 10.0.0.1
rdr on $ext_if inet6 proto { tcp } from any to ($ext_inet6_if) port http https -> fd00::ffff:a00:2
rdr on $ext_if inet proto { tcp } from any to ($ext_if) port http https -> 10.0.0.2

block all
pass quick proto pfsync

== UDP ==
pass out quick on $ext_if inet6 proto udp all
pass out quick on $ext_if inet proto udp all
pass in quick on $ext_if inet6 proto udp from any to fd00::ffff:a00:1 port domain
pass in quick on $ext_if inet proto udp from any to 10.0.0.1 port domain

== TCP ==
pass out quick on $ext_if inet6 proto tcp all
pass out quick on $ext_if inet proto tcp all

pass in quick on $ext_if inet6 proto tcp from any to "fd00::ffff:a00:1 fd00::ffff:a00:2" port http https domain
pass in quick on $ext_if inet proto tcp from any to "10.0.0.1 10.0.0.2" port http https domain


/etc/jail.conf

Code: 
allow.nomount;
allow.noraw_sockets;
allow.noset_hostname;
allow.nosysvipc;
exec.clean;
exec.jail_user          = "root";
exec.start              += "/bin/sh /etc/rc";
exec.stop               = "/bin/sh /etc/rc.shutdown";
exec.system_user        = "root";
host.hostname           = "epopen.com";
interface               = "lo1";
mount.devfs;
path                    = "/usr/jail/${name}";
persist;                                       

domain {
    ip4.addr            = "10.0.0.1";
    ip6.addr            = "fd00::ffff:a00:1";
}

http {
    ip4.addr            = "10.0.0.2";
    ip6.addr            = "fd00::ffff:a00:2";
}

Short interval time test result as below, NG/OK interval (Total test time (10 times): less 1 minutes)
Code: 
ist test: NG.
# curl http://\[2404:6800:4008:c01::6a\]
curl: (7) Failed to connect to 2404:6800:4008:c01::6a port 80: Operation timed out

2nd test: OK.
# curl http://\[2404:6800:4008:c01::6a\]
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/</code> was not found on this server.  <ins>That’s all we know.</ins>

3rd test: NG.
4nd test: OK.
5th test: NG.
.....

But tested NG and next test NG still if both long interval time.
I assume queue issue, but haven't in PF rule.

IPv4 100% OK as below.
Code: 
# curl http://108.177.125.147
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

I tried to turn on forward but problem still.
Code: 
# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1

Connect pass 50%, so assume outgoing rule path completed.
Can help me hint/troublehoot/debug if possible?
Thanks all very much.
 
Last edited:
Hi All.

I found root cause, it is NAT's address rotate (round-robin)
Check PF's NAT status by pfctl -sn as below shown
Code: 
nat on ng0 inet6 from fd00::ffff:a00:2 to  any -> (wan0) round-robin
And interface wan0 attach TWO IPv6 address, Local fe80:... and Global 2001:...
Connect fail when use local address.

Solution: add :network:0 into wan0 interface -> wan0:network:0
Let NAT use global address only.
Either :network or :0 work file, combind (in the case) work fine also.

But one question exist.
pf.conf(5) wrote:
Code: 
:network  = Translates to the network(s) attached to the interface.
:0  = Do not include interface aliases.
Question:
1.Is not Local fe80:... attached address?
"Attached" meant external source like ISP given?
2.Is Local fe80:... aliases?

Thanks all :D
 
On the subject of ':0' the latest version of the man page says:

Code: 
           Host names may also have the :0 option appended to restrict the name
           resolution to the first of each v4 and non-link-local v6 address
           found.

The fe80 address is a link-local address.
 
Kristof Provost said:
On the subject of ':0' the latest version of the man page says:

Code: 
           Host names may also have the :0 option appended to restrict the name
           resolution to the first of each v4 and non-link-local v6 address
           found.

The fe80 address is a link-local address.
Click to expand...
Hi Kristof Provost

I got it :D
New case below ( ipv6_privacy="YES" added in /etc/rc.conf)
Code: 
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1460
        inet6 fe80::21ce:dd50:1703:cd73%ng0 prefixlen 64 scopeid 0x6 
        inet6 2001:b011:a480:47e2:21ce:dd50:1703:cd73 prefixlen 64 autoconf 
        inet6 2001:b011:a480:47e2:1d0:82a3:51e4:ff7 prefixlen 64 deprecated autoconf temporary 
        inet6 2001:b011:a480:47e2:104d:a54e:c923:8a29 prefixlen 64 autoconf temporary 
        inet 122.117.86.253 --> 168.95.98.254 netmask 0xffffffff 
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

And ping test result below
Code: 
# ping6 ipv6.google.com
PING6(56=40+8+8 bytes) 2001:b011:a480:47e2:104d:a54e:c923:8a29 --> 2404:6800:4012:1::200e
16 bytes from 2404:6800:4012:1::200e, icmp_seq=0 hlim=53 time=14.528 ms

NAT take non-deprecated temporary address only, correctly.

Thanks you very much :D
 
You must log in or register to reply here.
Back
Top