Solved Routing traffic to jails - PF restart required after reboot

E

emtib8

I'm running ngircd and musicpd, each inside its own jail, on FreeBSD 11.1, and have configured PF so that the jails have internet access and I can connect to both of them.
The problem is that if the machine is rebooted, I have to ssh into it and reload pf before I can connect to the irc server or to musicpd.
I'd like to make it so that there's no human intervention required after a reboot.

NOTE: the host only responds to ping before restarting pf.
What am I missing?
 
Probably, pf applying rules before jail networking established. Like, adding aliases, creating taps and so on. Move network stuff to rc.conf or hook pf reload in your jail management utility.
 
abishai said:
Probably, pf applying rules before jail networking established. Like, adding aliases, creating taps and so on. Move network stuff to rc.conf or hook pf reload in your jail management utility.
Click to expand...

I don't have a jail management utility.
Which parts would I have to move to /etc/rc.conf and what should they look like?

I tried adding the following to rc.conf and commenting out the equivalents in jail.conf, but it didn't work:
Code: 
jail_list="mpd irc"

jail_mpd_hostname="mpd.local"
jail_mpd_interface="lo1"
jail_mpd_ip="10.255.255.16"

jail_irc_hostname="irc.local"
jail_irc_interface="lo1"
jail_irc_ip="10.255.255.17"

Apologies for the possibly dumb question, but I've only just recently started using FreeBSD and I'm learning as I go.

This is my /etc/jail.conf:
Code: 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

irc {
    interface = "lo1";
    host.hostname = "irc.local";
    path = "/home/j/irc";
    ip4.addr = "10.255.255.17" ;
    mount.fstab = "/home/j/fstab/irc.fstab";
}

mpd {
    interface = "lo1";
    host.hostname = "mpd.local";
    path = "/home/j/mpd";
    ip4.addr = "10.255.255.16";
    mount.fstab = "/home/j/fstab/mpd.fstab";
}

And /etc/rc.conf:
Code: 
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="hs-0"

ifconfig_em0="DHCP"
cloned_interfaces="lo1"
ifconfig_lo1_alias1="inet 10.255.255.17 netmask 255.255.255.255" # IRC
ifconfig_lo1_alias0="inet 10.255.255.16 netmask 255.255.255.255" # MPD

pf_rules="/etc/pf.conf"
pflog_enable="yes"
pflog_logfile="/var/log/pflog"
pf_enable="yes"

sshd_enable="YES"
powerd_enable="YES"

dumpdev="AUTO"

jail_enable="YES"
 
Your config should work.
Here is mine and pf applies rules correctly. The only difference is that I'm assigning primary address and aliases. Maybe this is the matter?
Code: 
ifconfig_em0="inet 10.0.1.2 netmask 255.255.255.0 up"
ifconfig_em0_alias0="inet 10.0.1.3 netmask 255.255.255.255" # zoneminder
ifconfig_em0_alias2="inet 10.0.1.5 netmask 255.255.255.255" # proxy
ifconfig_em0_alias3="inet 10.0.1.6 netmask 255.255.255.255" # database

Code: 
# Defaults
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
host.hostname = "$name.xxxxxx.yyy";
allow.nomount;

# Dynamic wildcard parameter:
# Base the path off the jail name.
path = "/usr/local/jails/$name";

proxy {
interface = "em0";
ip4.addr = "10.0.1.5";
}

database {
interface = "em0";
ip4.addr = "10.0.1.6";
allow.sysvipc;
}

zm {
ip4.addr = "em0|10.0.1.3/32";
ip4.addr += "em1|192.168.1.2/32";
mount.fstab = "/etc/fstab.$name";
}
 
Perhaps I'm missing something in pf.conf?

pf.conf:
Code: 
ext_if="em0"
ext_addr="192.168.1.3"

int_if="lo1"
int_net_0="10.255.255.17"    # irc
int_net_1="10.255.255.16"    # mpd

ssh_port_0="4122"
irc_port_0="6697"
mpd_port_0="6608"
mpd_httpd_port_0="8008"

set loginterface $ext_if

nat on $ext_if from $int_net_0 to any -> ($ext_if:0)
nat on $ext_if from $int_net_1 to any -> ($ext_if:0)
rdr pass on $ext_if inet proto tcp from any to $ext_if port $irc_port_0 -> $int_net_0 port $irc_port_0
rdr pass on $ext_if inet proto tcp from any to $ext_if port $mpd_port_0 -> $int_net_1 port $mpd_port_0
rdr pass on $ext_if inet proto tcp from any to $ext_if port $mpd_httpd_port_0 -> $int_net_1 port $mpd_httpd_port_0

block in log all
pass in on $ext_if proto tcp from any to $ext_if port $ssh_port_0 keep state

pass out log all
 
Nothing obvious. But, you told that host pings. Why it pings if config has no exceptions for icmp ? Can you check that pf rules are loaded at all after reboot ? If not, you definitely can check console for extended information.
 
pfctl -sa shows nothing under filter rules after a reboot.
The rules only load when I reload pf.
 
I connected a monitor and caught a glimpse: could not parse host specification.
The rules aren't being loaded.
Is there any way I could delay pf until the jails are loaded, or load a second set of rules afterwards?
 
This is not related to jails.
pf should tell line number as well.
I'm starting to believe that record
Code: 
cloned_interfaces="lo1"
ifconfig_lo1_alias1="inet 10.255.255.17 netmask 255.255.255.255" # IRC
ifconfig_lo1_alias0="inet 10.255.255.16 netmask 255.255.255.255" # MPD
is incorrect. Try to set network specification and then alias. (See my rc.conf) Then, find the bogus line number.
 
Alias specification is OK, but the alias1 vs alias0 -- that I'm not sure of. Try using alias2 instead of alias0.
 
emtib8 said:
I'm running ngircd and musicpd, each inside its own jail, on FreeBSD 11.1, and have configured PF so that the jails have internet access and I can connect to both of them.
The problem is that if the machine is rebooted, I have to ssh into it and reload pf before I can connect to the irc server or to musicpd.

What am I missing?
Click to expand...
I suspect that there might be some adversaries attacking the ports being used by the services. I would personally want to take some pre-cautions/precautions, when running an IRC protocol.
emtib8 said:
[BGCOLOR=#dee3e7]pfctl -sa[/BGCOLOR] shows nothing under filter rules after a reboot.
The rules only load when I reload pf.
Click to expand...
How about - #pfctl -vnf - before & after rebooting ?
Could you try run and access other services on the host since you said ping works?
 
abishai said:
Alias specification without network specification is OK ?
Click to expand...
He does specify the network, though be it /32 one. All my jails have configuration such as:

Code: 
$ grep lo252 /etc/rc.conf
cloned_interfaces="lo252 lo666"
ifconfig_lo252_alias1="inet 192.168.252.2 netmask 255.255.255.255"
ifconfig_lo252_alias2="inet 192.168.252.3 netmask 255.255.255.255"
ifconfig_lo252_alias3="inet 192.168.252.4 netmask 255.255.255.255"
ifconfig_lo252_alias4="inet 192.168.252.6 netmask 255.255.255.255"

And I have no problem. I'm not sure if you can skip the alias #, or shuffle them in the configuration. I think it's worth trying. Right now I have nowhere to test it, but I hinted it as a suggestion ..
 
_martin said:
And I have no problem. I'm not sure if you can skip the alias #, or shuffle them in the configuration. I think it's worth trying. Right now I have nowhere to test it, but I hinted it as a suggestion ..
Click to expand...
The below should work. I also use alias0....
Code: 
ifconfig_lo252_alias0="inet ........
 
Found a spare minute or two to spawn FreeBSD to check this -- yes, it does work, you can change the order of the alias #.

I'd focus on that error message OP saw on screen -- start with who (which script) is logging that error message.
 
Sorry for the long silence. Had a busy week.
_martin said:
Found a spare minute or two to spawn FreeBSD to check this -- yes, it does work, you can change the order of the alias #.

I'd focus on that error message OP saw on screen -- start with who (which script) is logging that error message.
Click to expand...
pfctl is logging the error message.
The errors are on the lines on which I have int_net_0 and int_net_1
 
The error doesn't show up in pflog; just onscreen when the machine boots.
Code: 
could not parse host specification
 
I went through those configs and I can't find anything that would cause the parse error during boot. I loaded them to my VM and all was ok. Are these config files you're sharing complete config files or just excerpts ? Do you have the line numbers where the error occurred ?

What is the state of the firewall after fresh boot - what does pfctl -sr say ? Are all interfaces/IPs as they should be after boot?

I would start by process of elimination. Try to disable the jail_enable in rc.conf, check if pf loads properly. Try to have empty pf.conf (pass in all, pass out all) with jails enabled -- see if the pf error occurs or not. You have unused variables in pf.conf, try to delete them. Etc..
 
Chris_H I don't think this is related to host resolving, at least not if OP shared the full config files. I checked the source sbin/pfctl/parse.y where the parse error could occur, but I was not able to find out what is wrong with those config files. I tried it myself and it was working just fine. That's why I asked those questions above.
 
Hello, _martin !
Based on my interpretation of "host specification", and (in part) from my own experiences with pf(4) and my build jail(8)s. I hear pf telling me it has no idea what <invalid hostname> is. eg; an arbitrary host name
on an rfc6890 network, and then not (properly) mapping that name internally.
The most likely scenario I see for why it didn't initially work, but then seemed to work, following a (pf) restart;
the order of resolution in /etc/hosts.conf is:
Code: 
dns
hosts
which causes DNS resolution to fail. Given that internal names can't be resolved on the internet. But, after
having had time to consult /etc/hosts for resolution; then succeeds -- assuming the names are listed there, and have a corresponding IP address.
But it could be for a number of similarly related misconfigurations. In short; pf(4) can't resolve
or properly map the name. But can use an IP address -- see; ping it.
That was my take on it, anyway.

--Chris
 
The thing here is that error is about parsing the hostname, not resolving it. In my current sbin/pfctl/parse.y there are few occurrences of this error message 'could not parse host specification' but I didn't find any issue with the above config ; given that it's the complete config that is.

Note that I did the test in my VM with IPs there were not resolved either.
 
first occurance
/usr/src/sbin/pfctl #3102
Code: 
if (($$ = host($1)) == NULL)    {
...
yyerror("could not parse host specification");
translated;
If host(name) <not there|invalid>
return error (message) "could not parse host specification"
where; <not there>, invalid
just as easily means doesn't resolv, or can't find

seems fairly clear (to me, anyway). ;)

--Chris
 
You must log in or register to reply here.
Back
Top