jails BastilleBSD Jail with pf NAT - IPv6 Requests not routed to jail

[marschro]

Hi all,

The Problem
My http(s) server is not responding to IPv6 requests from curl -v -L -6 my devpunx.com
Curl logs:

* Trying 2a03:b0c0:3:d0::1012:6001:80...
* Immediate connect fail for 2a03:b0c0:3:d0::1012:6001: Connection refused
* Closing connection 0

The Setup

• Host: FreeBSD 13.0-RELEASE
• BastilleBSD
• 2 jails: haproxy and nginx (IPv4 only, IPv6 not configured)
• Host has one IPv4 and one IPv6 Address
• Network Firewall (digitalocean) lets pass all IPv4 and IPv6 on :80 and :443
• Host uses NAT to the jails
• Minimalistic pf config on Host

  ext_if = "vtnet0"
  set block-policy return
  scrub in on $ext_if all fragment reassemble
  set skip on lo
  
  table <jails> persist
  nat on $ext_if from <jails> to any -> ($ext_if:0)
  rdr-anchor "rdr/*"
  
  block all
  pass proto ipv6-icmp all
  pass out quick keep state
  antispoof for $ext_if inet
  antispoof for $ext_if inet6
  
  pass in inet proto tcp from any to any port { 22 } flags S/SA modulate state

• haproxy jail has rdr set for
  • host:80 => haproxy_jail:80
  • host:443 => haproxy_jail:80
• haproxy balances traffic to nginx jail
• Setup is working perfectly on IPv4

So my understanding, what should happen is:

1. IPv6 Request hits the Network-Firewall and is passing through (I also enabled icmp)
2. IPv6 Request hits the Hosts pf, as all incoming :80 and :443 are redirected to the haproxy jail, I would assume, the rdr of the jail also forwards IPv6 as it does with IPv4 addresses and thus, the IPv6 request hits haproxy (?)
3. haproxy binds all :80 v4v6 addresses and routes them to the "internal" nginx server via its local 192.168.0.0/16 IPv4 address
4. ... and should resolve....

So basically my goal is, that an ipv6 request to the domain is resolved, but it is not.
There are now a bunch of questions to me:

1. What did I wrong or missed in this setup
2. Does this setup even make sense ?
   1. I cite this entry: https://forums.freebsd.org/threads/...ocal-address-usage-for-nat.67531/#post-400886
      "The whole idea of IPv6 is to be able to connect everything without having to resort to silly constructs like NAT."
   2. But I like the idea to have the jails communicate internally in the private network and are somehow in a separate space (also I only have only one public IPv6 address - maybe more, but I am happy that I understood IPv4 partially :/)
3. How can I debug where the IPv6 request is blocked?

Any hint or experience appreciated

Regards,
Martin

 

[SirDice]

I would assume, the rdr of the jail also forwards IPv6 as it does with IPv4 addresses and thus, the IPv6 request hits haproxy (?)

You assumed incorrectly. It only redirects the IPv4 traffic. Besides that, your Haproxy jail only has IPv4, you cannot redirect IPv6 to IPv4. Redirection only works on IPv6 to IPv6 and IPv4 to IPv4.

haproxy binds all :80 v4v6 addresses

Your HAproxy jail only has IPv4, so it has nothing to bind on with IPv6.

haproxy binds all :80 v4v6 addresses and routes them to the "internal" nginx server via its local 192.168.0.0/16 IPv4 address

This can work but requires the haproxy jail to have an IPv6 address and have Haproxy actually listen on it (and the firewall configured to either redirect or at least pass the incoming IPv6 traffic). Then you could proxy to an IPv4 'backend' host.

The host has nothing running on port 80 on the IPv6 address, so you get a "connection refused", in other words, the host responds with a RST to the incoming SYN packet.

 

[marschro]

You assumed incorrectly. It only redirects the IPv4 traffic. Besides that, your Haproxy jail only has IPv4, you cannot redirect IPv6 to IPv4. Redirection only works on IPv6 to IPv6 and IPv4 to IPv4.


Your HAproxy jail only has IPv4, so it has nothing to bind on with IPv6.


This can work but requires the haproxy jail to have an IPv6 address and have Haproxy actually listen on it (and the firewall configured to either redirect or at least pass the incoming IPv6 traffic). Then you could proxy to an IPv4 'backend' host.

The host has nothing running on port 80 on the IPv6 address, so you get a "connection refused", in other words, the host responds with a RST to the incoming SYN packet.

Okay, thank you SirDice - that clarified a lot for me.
Especially the fact, that NAT is not possible from IPv6 to IPv4 explains why this can never works.

What I would have to do is:

1. Enable jails for IPv6 - or at least the haproxy jail
2. Enable nat for inet and inet6 - sth like that:
   1. nat pass on $ext_if inet proto { tcp udp icmp } from $jails_ip4 to any -> ($ext_if) nat pass on $ext_if inet6 proto { tcp udp icmp6 } from $jails_ip6 to any -> ($ext_if)
3. and redirect manually and separately the inet and inet6 traffic to the jail - which makes the awesome convenience BastilleBSD brings up, by adding aliases to the loopback interface and writing the rdr-anchors, obsolete - what I don't want...

This whole IPv6 with Jails is a rabbit hole I don't want to go down.
I thought one could just rdr from an IPv6 to an IPv4... but in an article I once read "This whole IPv6 is an internet besides the internet" - and that is somehow right.
I also read this conversation which drove me away from the Idea, that my jails will be reachable via IPv6.

An alternative approach could be the following:

• Not putting the haproxy into a jail, but run it on the host. Then the proxy takes over the the role of dispatching the requests to the correct jails. haproxy can receive all kind of address families in its frontend and can pass them to the IPv4 backends (pure assumption - never tried it, but from the documentation this should work)
• The loss would be the convenience, that comes with jails, jail-templates etc. A crucial part of the stack then would not reside in a jail and I especially want this part to be automated as complete as possible...

I think I will just ignore the existence of IPv6 a bit more ....

 

[SirDice]

Not putting the haproxy into a jail, but run it on the host. Then the proxy takes over the the role of dispatching the requests to the correct jails. haproxy can receive all kind of address families in its frontend and can pass them to the IPv4 backends (pure assumption - never tried it, but from the documentation this should work)

That's exactly how I set it up on my server, HAProxy runs on the host and proxies to the various backend jails.

 

[marschro]

That's exactly how I set it up on my server, HAProxy runs on the host and proxies to the various backend jails.

Did the same - and its perfect... and I learned a lot about FreeBSD, Firewalls, IPV6 the last few days... Thanks SirDice for your great support.