If you want to know the solution, then you should read to the end.
- First half is about my environment, trial & failures.
- There are solutions at the end.
====
Plz help me guys.
* I'm working on PF + Transparent Bridge + Squid(4.13) for 10 weeks.
I've read thousands web pages(including this FreeBSD/Networking forum), but it was not solved & stuck at the end.
This is my first FreeBSD project.
* I want to build the reverse transparent proxy for web-content-filtering system
(not just url-based, but content-based, ie. text/html or .doc, .zip ...)
I've setuped transparent bridge firewall and it worked well, but squid proxy is not working.
I've found numorous articles about 10-14 years on "not working" or "working for me" but thats all.
I have no clue anymore, currently.
If squid transparent proxy is runnable, I'll adapt ICAP & ClamAV for web-content-filtering.
Actually I've already tested these functionalities.
* PF redirect web traffic to 127.0.0.1:3128 but squid can't accept that traffic.
So I've tested with net cat using
Currently, I suspect the bug or unsupported functionality of PF about local forwarding.
Forwarding to remote systems are working well.
So I usally tested various PF rules.
rdr-to(not supported anymore), rdr on, rdr pass, divert-to.
I tested filtering rules with or without route-to lo0 but nothings changed.
* This is my working environment(115.xxx.xxx.0/24 same public-ip-network).
{Internet}
|
{router/gateway - 115.xxx.xxx.1 - ISP provided}
|
{Dev Macine - 115.xxx.xxx.111}
|
{FreeBSD FireWall with OPNsense/squid/PF using transparent bridge - 115.xxx.xxx.62}
|
{Test Web Sever -115.xxx.xxx.64}
* All the 3 machines resides in same network & use same gateway
{FreeBSD FireWall}'s igb1 interface is wired to {Dev Macine - 115.xxx.xxx.111}
{FreeBSD FireWall}'s igb0 interface is wired to {Test Web Sever - 115.xxx.xxx.64}
{FreeBSD FireWall}'s bridge0(115.xxx.xxx.62) interface has igb1 & igb0 interfaces as a member
All the above IPs are public IP.
* kldstat, ipdivert.ko(and ipfw.ko why?) loaded on boot for divert-to rule
>
Id Refs Address Size Name
...
3 1 0x0 f998 if_bridge.ko
4 2 0x0 72a8 bridgestp.ko
...
9 3 0x0 582f0 pf.ko
10 1 0x0 2af8 pflog.ko
11 1 0x0 ebd0 pfsync.ko
12 1 0x0 5708 ipdivert.ko
13 2 0x0 49ab0 ipfw.ko
...
* system options - for pfil_member, pfil_bridge
>
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 1
net.link.bridge.pfil_member: 1
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 1
net.link.bridge.pfil_onlyip: 0
* listen states
>
...
tcp6 0 0 localhost.3128 *.* LISTEN
tcp4 0 0 localhost.3128 *.* LISTEN
tcp4 0 0 OPNsense.3128 *.* LISTEN
tcp6 0 0 localhost.3129 *.* LISTEN
tcp4 0 0 localhost.3129 *.* LISTEN
...
* /dev/pf permission for squid(client(=source) address resolution for PF rdr rule?)
#/dev/pf
perm pf 0640
own pf root:squid
* squid.conf
>
...
# I've tested by changing "intercept" to "tproxy" manually, but not worked.
...
http_port 127.0.0.1:3128 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
http_port [::1]:3128 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
https_port 127.0.0.1:3129 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
https_port [::1]:3129 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
...
* test secenario 1 - without packet redirection - works well.
# on 115.xxx.xxx.111
>
...
* test secenario 2 - with packet redirection - always failed.
# on 115.xxx.xxx.111
>
* Trying 115.xxx.xxx.64...
* TCP_NODELAY set
* connect to 115.xxx.xxx.64 port 80 failed: Timed out
* Failed to connect to 115.xxx.xxx.64 port 80: Timed out
* Closing connection 0
curl: (7) Failed to connect to 115.xxx.xxx.64 port 80: Timed out
* I've tested with 3 rule-sets, and all the rule sets are not working.
method 1 - rdr on ... + pass in ...
method 2 - rdr pass ... and no filtering rule
method 3 - no rdr rull & pass in [route-to lo0] ... divert-to ...
I dumped packets using tcpdump on every interfaces of 3 machines.
Here are test results.
Only SYN packets(with same SEQ number for retry) are deliverd and no accept from 127.0.0.1:3128.
- rule-set 1, 2 error results.
* packets go through .111 -> igb1 -> bridge0 -> ibg0 -> .64 always, bypassing lo0
5 SYN packets fired from .111 to igb1 of .62.
5 SYN packets arrived to bridge0
* only first packet logged to pflog0 and no packet delivered to lo0 - so squid(or nc -l 127.0.0.1 3128) can't accept.
5 SYN packets(DEST IP/PORT changed to 127.0.0.1:3128) arrived to igb0
5 SYN arrived(DEST IP/PORT changed to 127.0.0.1:3128) arrived to .64
all tcp 127.0.0.1:3128 (115.xxx.xxx.64:80) <- 115.xxx.xxx.111:2066 CLOSED:SYN_SENT
- rule-set 3 error result
* packets go through .111 -> igb1 -> bridge0[no further delivery], always bypassing lo0
5 SYN packets fired from .111 to igb1 of .62.
5 SYN packets arrived to bridge0
* only first packet logged to pflog0 and no packet delivered to lo0 - so squid(or nc -l 127.0.0.1 3128) can't accept.
0 SYN packets arrived to igb0 & .64
all tcp 115.xxx.xxx.64:80 <- 115.xxx.xxx.111:2066 CLOSED:SYN_SENT
- currently I'm testing method 3(divert-to) but why ipfw.ko loaded?
ipdivert.ko only works with ipfw?
* Is there any possibility HardenedBSD causes the error?
OS system is FreeBSD but HardenedBSD is applied.
>
FreeBSD OPNsense.localdomain 12.1-RELEASE-p11-HBSD FreeBSD 12.1-RELEASE-p11-HBSD #0 74f1f081a1e(stable/20.7)-dirty: Fri Dec 4 13:40:15 CET 2020 root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
* PF redirection rule file - rules.txt, by running
- First half is about my environment, trial & failures.
- There are solutions at the end.
====
Plz help me guys.
* I'm working on PF + Transparent Bridge + Squid(4.13) for 10 weeks.
I've read thousands web pages(including this FreeBSD/Networking forum), but it was not solved & stuck at the end.
This is my first FreeBSD project.
* I want to build the reverse transparent proxy for web-content-filtering system
(not just url-based, but content-based, ie. text/html or .doc, .zip ...)
I've setuped transparent bridge firewall and it worked well, but squid proxy is not working.
I've found numorous articles about 10-14 years on "not working" or "working for me" but thats all.
I have no clue anymore, currently.
If squid transparent proxy is runnable, I'll adapt ICAP & ClamAV for web-content-filtering.
Actually I've already tested these functionalities.
* PF redirect web traffic to 127.0.0.1:3128 but squid can't accept that traffic.
So I've tested with net cat using
nc -l 127.0.0.1 3128
with squid disabled.Currently, I suspect the bug or unsupported functionality of PF about local forwarding.
Forwarding to remote systems are working well.
So I usally tested various PF rules.
rdr-to(not supported anymore), rdr on, rdr pass, divert-to.
I tested filtering rules with or without route-to lo0 but nothings changed.
* This is my working environment(115.xxx.xxx.0/24 same public-ip-network).
{Internet}
|
{router/gateway - 115.xxx.xxx.1 - ISP provided}
|
{Dev Macine - 115.xxx.xxx.111}
|
{FreeBSD FireWall with OPNsense/squid/PF using transparent bridge - 115.xxx.xxx.62}
|
{Test Web Sever -115.xxx.xxx.64}
* All the 3 machines resides in same network & use same gateway
{FreeBSD FireWall}'s igb1 interface is wired to {Dev Macine - 115.xxx.xxx.111}
{FreeBSD FireWall}'s igb0 interface is wired to {Test Web Sever - 115.xxx.xxx.64}
{FreeBSD FireWall}'s bridge0(115.xxx.xxx.62) interface has igb1 & igb0 interfaces as a member
All the above IPs are public IP.
* kldstat, ipdivert.ko(and ipfw.ko why?) loaded on boot for divert-to rule
>
kldstat
Id Refs Address Size Name
...
3 1 0x0 f998 if_bridge.ko
4 2 0x0 72a8 bridgestp.ko
...
9 3 0x0 582f0 pf.ko
10 1 0x0 2af8 pflog.ko
11 1 0x0 ebd0 pfsync.ko
12 1 0x0 5708 ipdivert.ko
13 2 0x0 49ab0 ipfw.ko
...
* system options - for pfil_member, pfil_bridge
>
sysctl -a | grep -i net.link.bridge
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 1
net.link.bridge.pfil_member: 1
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 1
net.link.bridge.pfil_onlyip: 0
* listen states
>
netstat -a | grep -i listen
...
tcp6 0 0 localhost.3128 *.* LISTEN
tcp4 0 0 localhost.3128 *.* LISTEN
tcp4 0 0 OPNsense.3128 *.* LISTEN
tcp6 0 0 localhost.3129 *.* LISTEN
tcp4 0 0 localhost.3129 *.* LISTEN
...
* /dev/pf permission for squid(client(=source) address resolution for PF rdr rule?)
cat /etc/devfs.conf
#/dev/pf
perm pf 0640
own pf root:squid
* squid.conf
>
cat /usr/local/etc/squid/squid.conf
...
# I've tested by changing "intercept" to "tproxy" manually, but not worked.
...
http_port 127.0.0.1:3128 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
http_port [::1]:3128 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
https_port 127.0.0.1:3129 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
https_port [::1]:3129 intercept ssl-bump cert=/var/squid/ssl/ca.pem dynamic_cert_mem_cache_size=10MB generate-host-certificates=on
...
* test secenario 1 - without packet redirection - works well.
# on 115.xxx.xxx.111
>
curl -v http://115.xxx.xxx.64/index.html
...
* test secenario 2 - with packet redirection - always failed.
# on 115.xxx.xxx.111
>
curl -v http://115.xxx.xxx.64/index.html
* Trying 115.xxx.xxx.64...
* TCP_NODELAY set
* connect to 115.xxx.xxx.64 port 80 failed: Timed out
* Failed to connect to 115.xxx.xxx.64 port 80: Timed out
* Closing connection 0
curl: (7) Failed to connect to 115.xxx.xxx.64 port 80: Timed out
* I've tested with 3 rule-sets, and all the rule sets are not working.
method 1 - rdr on ... + pass in ...
method 2 - rdr pass ... and no filtering rule
method 3 - no rdr rull & pass in [route-to lo0] ... divert-to ...
I dumped packets using tcpdump on every interfaces of 3 machines.
Here are test results.
Only SYN packets(with same SEQ number for retry) are deliverd and no accept from 127.0.0.1:3128.
- rule-set 1, 2 error results.
* packets go through .111 -> igb1 -> bridge0 -> ibg0 -> .64 always, bypassing lo0
5 SYN packets fired from .111 to igb1 of .62.
5 SYN packets arrived to bridge0
* only first packet logged to pflog0 and no packet delivered to lo0 - so squid(or nc -l 127.0.0.1 3128) can't accept.
5 SYN packets(DEST IP/PORT changed to 127.0.0.1:3128) arrived to igb0
5 SYN arrived(DEST IP/PORT changed to 127.0.0.1:3128) arrived to .64
pftop -ss | grep 115.xxx.xxx.64:80
shows below lineall tcp 127.0.0.1:3128 (115.xxx.xxx.64:80) <- 115.xxx.xxx.111:2066 CLOSED:SYN_SENT
- rule-set 3 error result
* packets go through .111 -> igb1 -> bridge0[no further delivery], always bypassing lo0
5 SYN packets fired from .111 to igb1 of .62.
5 SYN packets arrived to bridge0
* only first packet logged to pflog0 and no packet delivered to lo0 - so squid(or nc -l 127.0.0.1 3128) can't accept.
0 SYN packets arrived to igb0 & .64
pftop -ss | grep 115.xxx.xxx.64:80
shows below lineall tcp 115.xxx.xxx.64:80 <- 115.xxx.xxx.111:2066 CLOSED:SYN_SENT
- currently I'm testing method 3(divert-to) but why ipfw.ko loaded?
ipdivert.ko only works with ipfw?
* Is there any possibility HardenedBSD causes the error?
OS system is FreeBSD but HardenedBSD is applied.
>
uname -a
FreeBSD OPNsense.localdomain 12.1-RELEASE-p11-HBSD FreeBSD 12.1-RELEASE-p11-HBSD #0 74f1f081a1e(stable/20.7)-dirty: Fri Dec 4 13:40:15 CET 2020 root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
* PF redirection rule file - rules.txt, by running
pfctl -x loud -f rules.txt
Code:
## DEFs
lan="115.xxx.xxx.111/32"
web="115.xxx.xxx.64/32"
## OPTIONs
set block-policy return
## NORMALIZATIONs
scrub on lo0 all
scrub on igb0 all
scrub on bridge0 all
scrub on igb1 all
#scrub in all
#scrub on igb1 all reassemble tcp
#...
## QUEUEINGs
## NATs/RDRs
# method 1 - rdr on - not working.
#rdr log on igb1 inet proto tcp from $lan to $web port 80 -> 127.0.0.1 port 3128 # rdr 3128 igb1
# method 2 - rdr pass - not working.
#rdr pass log on igb1 inet proto tcp from $lan to $web port 80 -> 127.0.0.1 port 3128 # rdr pass 3128 igb1
# method 3 - no rdr, just use diver-to at below
# FILTERINGs
#set skip on lo0
#antispoof quick for bridge0
# syntax err - rdr-to - not working
#pass in log quick on igb1 route-to lo0 inet proto tcp from $lan to $web port 80 rdr-to 127.0.0.1 port 3128 keep state label "rdr-to 3128" # rdr-to 3128
# method 1 - pair to rdr on - not working.
#pass in log quick route-to lo0 inet proto tcp from any to 127.0.0.1 port 3128 keep state label "to 3128" # to 3128
# method 2 - pair to rdr pass - not working & no rule required
# method 3 - no rdr - not working
pass in log quick inet proto tcp from $lan to $web port 80 divert-to 127.0.0.1 port 3128 keep state label "divert-to 3128" # divert-to 3128
# 22, 8080, 8443 management port
pass in quick inet proto tcp from any to any port {22,8080,8443} keep state label "allow 22, 8080, 8443" # allow 22, 8080, 8443
pass out quick
#block log all label "block all" # block all
Last edited: