Issue with connection (operation_not_permitted)

A

aonishenko

Hello!

I've installed FReeBSD 11 on my client mashine. On my server I have also FreeBSD 11.
My client has 3G modem which successfully connects through ppp daemon to the cellurar network.
After that my client successfully establish L2TP IPSec connection to my server.
After that I connect to my client from my server through SSH.
Conection succeded, but after some random time delay (during y ssh session) connection drops with message "operation_not_permitted". There is no any error messages, when I start SSH connection with -vvv flag and there is no any messages on server\client side on syslog's. Also, I checked tcpdump -n -e -ttt -i pflog0 - and also there is no any blocked or dropped packets.
I suppose, that there is MTU issue. Cellurar signal fine and pings have only 50 msec's delay.
I tried to set MTU on PPP interface and VPN interface to 960, but disconnections still happens.
Also, by default, ny VPN server set's MTU to 1280.

Could you help with advice, what should I check? It seems, like this is a client issue, because from my phone everythings work fine and if I connect to SSH from client's wi-fi hostpad - there is no any disconnections.
Here is the example of SSH interruption -

Code: 
candy-pi# Fssh_packet_write_poll: Connection to 10.1.20.1 port 22: Operation not permitted

Here is my PF.conf on client side

Code: 
# interfaces
lan_if="ue0"
ext_if="tun0"
wifi_if="wlan0"
vpn_if="ng0"
localnet = "10.1.20.0/24"
my_server= $ext_if:network
vpn_net = "172.16.1.0/24"
locked_jail = "{10.1.20.2}"
icmp_types = "{ echoreq, unreach }"
######################FW##############################
client_tcp = "{1,5,7,18,20,21,22,23,25,29,37,42,53,67,68,80,81,88,109,110,115,118,119,123,137,139,143,150,156,161,443,546,548,554,1701,500,580:994,1701,1723,3000:6100,8123,8118,8080,8081,9050,9051,9091,>=10000}"
client_udp = "{ 47, 53, 67, 68, 123, 135:139, 500, 1701, 1723, 4500, >=6000 }"
external_tcp = "{ http,22 }"
external_udp = "{ 47, 67, 68, 123, 500, 1701, 4500, >=7000 }"
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 169.254.0.0/16, 192.0.2.0/24, 240.0.0.0/4 }"
table <portsentry> persist
table <fail2ban> persist
set limit { states 1000000, frags 80000, src-nodes 100000, table-entries 500000}
set state-policy if-bound
set optimization normal
set debug urgent
set ruleset-optimization profile
set timeout { frag 10, tcp.established 3600, src.track 30 }
set block-policy drop
set skip on { lo0, ue0, ng0, ng1, ng2, ng3, ng4, ng5 }
scrub in all
#### --- QUEUEING ------------------------------------------------------------------ ##
# ext_if IP address could be dynamic, hence ($ext_if)
nat on $ext_if inet from $localnet to any -> ($ext_if)
nat on $ext_if inet from $vpn_net to any -> ($ext_if)
#rdr on {$int_if,ng0,ng1,ng2,ng3,ng4,ng5,ng6} proto tcp from any to !<goodguys>  port 80 -> 10.1.10.1 port 3128
#rdr on {$int_if,ng0,ng1,ng2,ng3,ng4,ng5,ng6} proto tcp from any to !<goodguys> port 443 -> 10.1.10.1 port 3129
antispoof quick for $ext_if
block log quick from <portsentry>
block log quick from <fail2ban>
block in  quick on $ext_if from $martians to any
block out quick on $ext_if from any to $martians
block log  quick from any os NMAP
block quick from $locked_jail to {!$localnet,!$vpn_net}
block log all
pass quick on ng0
pass inet proto  icmp all
pass in on $wifi_if proto tcp from any to any port $client_tcp flags S/SA keep state
pass in on $wifi_if proto udp from any to any  port $client_udp keep state
pass in on $wifi_if proto udp to 255.255.255.255 keep state
pass in on $wifi_if proto udp to $wifi_if:broadcast keep state
pass in on $wifi_if proto igmp all allow-opts
pass in on $wifi_if proto esp
pass in on $wifi_if proto ah
pass in on $wifi_if proto ipencap
pass out on $wifi_if keep state


pass in on $ext_if proto tcp from any to any port $external_tcp flags S/SA keep state (max-src-conn 250)
pass in on $ext_if proto udp from any to any port $external_udp keep state
pass in on $ext_if proto esp
pass in on $ext_if proto ah
pass in on $ext_if proto ipencap
pass out on $ext_if keep state
 
When you say you lose connection, is that the SSH session or the IPSec tunnel?
 
Generally speaking for debugging you need to reduce complexity and configuration lines that produce an unpredicted behavior till the problem hopefully goes away.

In your pf.conf there are some limitations (timeout, states) regarding ssh. Try to remove them all and notice any difference in behavior when adding them step by step. Watch/enable ssh logfile and pflog. Look at the traffic with tcpdump or Wireshark.

BTW keep state is default with PF so there is no need to explicitly declare this.

If the problem persists with a minimalistic pf.conf the problem is not with the PF-configuration.

Make sure that you do not add yourself to failtoban :)
 
getopt said:
In your pf.conf there are some limitations (timeout, states) regarding ssh.
Click to expand...
Yeah, I was thinking about time-outs too. But more on the client side. If the SSH session is idle the firewall state might time-out. You typically solve this by enabling TCPKeepAlive.

aonishenko are these idle connections or are you actively using the session when it cuts off?
 
getopt said:
Generally speaking for debugging you need to reduce complexity and configuration lines that produce an unpredicted behavior till the problem hopefully goes away.

In your pf.conf there are some limitations (timeout, states) regarding ssh. Try to remove them all and notice any difference in behavior when adding them step by step. Watch/enable ssh logfile and pflog. Look at the traffic with tcpdump or Wireshark.

BTW keep state is default with PF so there is no need to explicitly declare this.

If the problem persists with a minimalistic pf.conf the problem is not with the PF-configuration.

Make sure that you do not add yourself to failtoban :)
Click to expand...
Thank you, kepp trying to reduce firewall limitations. tcpdump shows nothing blocked and dropped. Only usual TCP traffic. Maybe window scale?
 
SirDice said:
Yeah, I was thinking about time-outs too. But more on the client side. If the SSH session is idle the firewall state might time-out. You typically solve this by enabling TCPKeepAlive.

aonishenko are these idle connections or are you actively using the session when it cuts off?
Click to expand...
For example, I send command "cat /var/log/messages" and connection drops, like some amount of data exceeds some limit. Also, if I trying to edit, let's say "nano /usl/local/etc/hostapd.conf" and going trough file with Ctrl+V connection also may drop. That's the reason, why I was looking for MTU.
Also, I enabled KeepAlive in SSH and there is no any changes.
 
getopt said:
Let's have a look on your ppp.conf. MTU might be the culprit if PF is not.
Click to expand...
Unable to connect right now.
My PPP has mtu 960 and mru 960 settings. mpd5 has settings set mtu 960 and set mru 960 settings.
I've previosly checked these values with ping -D -s 960 10.1.20.1 from server side. Packet was too long, when size was 1300. I reduced this size several times, trying to find correlation with disconnecting issues.
Also, VPN tunnel is stable. If it would be PPP issue, I suppose, VPN shoud be dropped, or SA should be reestablished. On server side there is no any logs in racoon, only DPD talk each-side.

You gave me an idea, that
Code: 
set timeout { frag 10, tcp.established 3600, src.track 30 }
may be the culprit. Maybe, packet is very defragmented.

Going to check this weekend. Now out of direct connection to the RPi.
But... I set skip for ng0 (VPN i-face). How pf may be involved in this traffic? During my ssh session on the "pf-controlled" interface only IPSec connection to the 4500 UDP port. Also, wifi has pf involved and there is no any disconnections. Only SSH. For example, munin works fine, when my server gathering data from client side through VPN tcp port 4949. Only fckn' ssh)
 
Hello!

Removed optimization and fragmentation from PF.conf. Disconnection only for ssh sessions still happens.
What should I check?
May be, some encrytption bits corrupted?
 
Hi!

If you were interested in. The FreeBSD was built for RPi from crochet. When I rebuild kernel directly on RPi this ssh issue has gone.
May be this may help you.
 
You must log in or register to reply here.
Back
Top