jails After 14.1->14.2 upgrade, vnet=new ng_bridge network from/to jail<->host doesn't work anymore

[thepaya]

But everything else does work.

For example, the first DNS server in the jail's /etc/resolv.conf is the host itself, which doesn't work, the second one (1.1.1.1) works just fine. Connecting to the jail from the local network also works fine, but not from the host itself.
Nothing in the /etc/{jail,pf,rc}.conf configuration files changed with the upgrade from 14.1 to 14.2 (neither in the host nor the jail itself).
This got me really puzzled, normally I can find an error message or something to search for in the sources to understand what's happening but here I have no references at all
I've seen the mentions about strengthening of the sysctl jail tunables in the 14.2 release notes but none seem like they'd affect the network (at least directly), relevant /etc/sysctl.conf content included just in case.
Even looking at tcpdumps inside/outside of the jail or on the host's epairXa/b interfaces, everything seems to be working fine all packets reach all interfaces w/o problem. It's like "just the programs" listening on those ports are exclusively ignoring things to/from the jail's vnet stack.
See below a demo using port 22/ssh (to/from the jail), but also happens with DNS 53/udp or postgres 5432/tcp.

First some version info. From the jail:

freebsd-version -k ; freebsd-version -ru ; uname -bmvKU
freebsd-version: unable to locate kernel
14.2-RELEASE
14.2-RELEASE
FreeBSD 14.2-RELEASE releng/14.2-n269506-c8918d6c7412 GENERIC amd64 1402000 1402000 881d8d7f1313038d6c104b9d978cdb7ce2ed50a3

From the host:

# freebsd-version -kru ; uname -bmvKU
14.2-RELEASE
14.2-RELEASE
14.2-RELEASE
FreeBSD 14.2-RELEASE releng/14.2-n269506-c8918d6c7412 GENERIC amd64 1402000 1402000 881d8d7f1313038d6c104b9d978cdb7ce2ed50a3

The setup jail.conf:

vjail {
    path = "/tank/jails/${name}";
    host.hostname = "${name}";
    mount.fstab = "/etc/${name}.fstab";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail/${name}-console.log";

    allow.mount;
    allow.chflags;
    allow.raw_sockets;
    allow.mount.tmpfs;
    allow.mount.devfs;
    allow.mount.procfs;
    allow.mount.linprocfs;
    allow.mount.nullfs;
    allow.mount.linsysfs;
    allow.mlock;
    allow.sysvipc = "1";
    devfs_ruleset = "10";
    enforce_statfs = 1;
    exec.clean;
    persist;
    vnet = new;
    vnet.interface = "${epair}b";
    $id = "3";
    $ip = "192.168.0.${id}/24";
    $gateway = "192.168.0.254";
    $bridge = "bridge:";
    $epair = "epair${id}";

    # ng_bridge:
    exec.prestart = "/sbin/ifconfig ${epair}a destroy ||:";
    exec.prestart += "/sbin/ifconfig ${epair} create up";
    exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name} mtu 9000";
    exec.prestart += "/usr/sbin/ngctl connect ${epair}a: ${bridge} lower link${id}";
    exec.prestart += "/usr/sbin/ngctl msg ${epair}a: setpromisc 1";
    exec.prestart += "/usr/sbin/ngctl msg ${epair}a: setautosrc 0";
    exec.poststop = "/usr/sbin/ngctl shutdown ${epair}a:";
    exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}

Relevant jail's rc.conf:

hostname="vjail"
sendmail_enable="NONE"
rpcbind_enable="NO"
syslogd_flags="-ss"
ifconfig_epair3b="192.168.0.3/24 up mtu 9000"
defaultrouter="192.168.0.254"
sshd_enable="YES"

The host's pf.conf is too big/complicated to fit in here (and it was working fine in 14.1), plus it's set to skip on both { epair3a, epair3b } interfaces, and it should not interfere with packets going through an ng_bridge(4) anyway (right?).
That said, and to rule out pf, the only matching states that include the jail's ip address with incoming traffic are generated by this simple rule (which allows all outgoing LAN traffic from the host):

# pfctl -ss -vv | grep -A2 '192.168.0.3:22 '
No ALTQ support in kernel
ALTQ related functions disabled
all tcp 192.168.0.254:14548 -> 192.168.0.3:22       SYN_SENT:CLOSED
   [3226960649 + 16777216]  [0 + 4294901760]
   age 00:00:06, expires in 00:00:13, 3:0 pkts, 180:0 bytes, rule 28, random-id
# pfctl -sr -vv | grep '@28 '
No ALTQ support in kernel
ALTQ related functions disabled
@28 pass out quick on lagg0 inet from (lagg0:1) to <lan:2> flags S/SA keep state (no-sync)

For outgoing traffic this is the only rule getting matched (which also allows all incoming LAN traffic into the host:

# pfctl -ss -vv | grep -A2 '192.168.0.254:22.*192.168.0.3'
No ALTQ support in kernel
ALTQ related functions disabled
all tcp 192.168.0.254:22 <- 192.168.0.3:40217       SYN_SENT:ESTABLISHED
   [3035593433 + 256] wscale 9  [542541728 + 256] wscale 9
   age 00:00:01, expires in 00:00:15, 2:3 pkts, 120:180 bytes, rule 30, reassemble-tcp
# pfctl -sr -vv | grep -A2 '@30 '
No ALTQ support in kernel
ALTQ related functions disabled
@30 pass in quick on lagg0 inet from <lan:2> to (lagg0:1) flags S/SA keep state (no-sync)
  [ Evaluations: 67028     Packets: 19703     Bytes: 1913403     States: 401   ]
  [ Inserted: uid 0 pid 52771 State Creations: 9292  ]

Finally, this is the piece that creates the ng_bridge(4) with a /etc/start_if.lagg0 script that sets everything up on the host:

#!/bin/sh

ngctl shutdown bridge: ||:
ifconfig lagg0 destroy ||:

ifconfig ix0 -lro mtu 9000 up
ifconfig ix1 -lro mtu 9000 up
ifconfig ix3 up txcsum rxcsum -lro
ifconfig lagg0 create laggproto lacp laggport ix0 laggport ix1 up lagghash l3,l4
ifconfig lagg0 txcsum rxcsum -lro mtu 9000
ifconfig lagg0 192.168.0.254/24 up

ngctl mkpeer lagg0: bridge lower link0
ngctl name lagg0:lower bridge
ngctl connect lagg0: bridge: upper link1
ngctl connect ix3: bridge: lower link2

The security. section of the host's /etc/sysctl.conf:

# grep '^security' /etc/sysctl.conf
security.jail.allow_raw_sockets=1
security.jail.sysvipc_allowed=1
security.bsd.unprivileged_read_msgbuf=0
security.jail.mount_allowed=1
security.jail.mount_nullfs_allowed=0
security.jail.mount_linsysfs_allowed=1
security.jail.mount_linprocfs_allowed=1
security.jail.mount_fdescfs_allowed=1
security.jail.mount_tmpfs_allowed=1
security.jail.mount_procfs_allowed=1
security.jail.mount_zfs_allowed=0
security.jail.mount_devfs_allowed=1

Sidebar: I've had this ng_bridge(4) setup working since FreeBSD 11 (or so, it's been a while), mainly because if_bridge(4) didn't work with interfaces that had different MTU sizes (e.g. wifi), while ng_bridge has been rock solid for years.

Any help or hints are appreciated.

 

[thepaya]

An update: magically solved by changing the startup order. Pushed the jail to launch after sshd and pf (and thus, a bunch of other things) and now everything works.
Nothing special about it, I just specifically wanted to have ssh access to monitor the jail starting up right after a reboot.
Not quite solved (did a quick revert of the order change and again: same issue). But at least found a harmless workaround.