FreeBSD 14.1 intermittent freeze

L

lassino

Hello everyone there! I use FreeBSD on my home NAS, and recently, intermittent freeze occurs on the system continuously. The system freezes for serveral minutes and becomes responsive again for serveral minutes and then freezes again. When the system freezes, not only pings and tcp connections to the system times out, when I connect a display through HDMI to the system, the terminal and the X11 GUI are not responsive too.

I noticed this problem recently, when the freezes become more frequent and longer. But I suppose this also existed before because sometimes connection to the system took much longer.

I mainly use the system for seeding and downloading with net-p2p/qbittorrent (in nox flavor), and as a net/samba416 server for network file access. And short before I notice the system freeze, I increased the torrenting workload, seeding around 1000 torrents at the same time (previously around 500 torrents).

I searched about the problem and tried some measures by myself but failed to identify the root cause, as follows:
1. I tried to inspect system load with systat -ifstat and top when the system recover from a previous freeze, but the statistics seems completely normal. The CPU load is around 3.00 when the system is responsive, which does not seem very high. I also use sysctl dev.re.0.stats=1 provided by the net/realtek-re-kmod driver to display network statistics, and no significant abnormal can be observed.
2. I find some discussion saying that Realtek ethernet controlle (which is exactly the NIC on my system) hangs with high UDP traffic, so I turned off µTP on qBittorrent and also limit the upload and download bandwidth (to 30MB/s). I also limits the maxium connection to 200. This seems to mitigate the problem, with each freeze seem to last shorter, but it's still occurring. Plus, I suppose a NIC hang should not cause the complete system unresponsive.

Here are also some possible reasons I suppose, but I don't know how to confirm these suspicions.
1. The CPU my NAS uses is Intel i7-12800H, with hybrid architecture of 6 P-cores and 8 E-cores. Can this be related with the scheduling between E-Cores and P-Cores?
2. The storage I use for torrenting and Samba access is a ZFS RAID-Z pool with 4 8-TB underlying SATA hard drive. Is this related to some filesystem IO?
3. Samba and qBittorrent are running in a jail, which is connected to the network via epair bound to the network controller. Is this related to the jail or epair system?

Do you have any idea what causes the system freeze, and can you suggest what I can do further to identify and solve my problem?
 
My personal feelings are that jails deserve their own interfaces. Vnet passthrough interfaces instead of epair.
Slap in an Intel Gigabit Adapter for your high load jails and use the re0 interface for the host system.

Truthfully you have added on so much fluff any one could be the problem. I like to KISS then migrate.
ZFS
Jails
Realtek
Samba
X11 GUI on a Server with 1000 clients all hitting you up for small files?

I doubt the cores are a problem like this.
 
Tangentially, Do you think this processor is best for a NAS? No ECC support and you are using ZFS?

Intel® Core™ i7-12800H Processor (24M Cache, up to 4.80 GHz) - Product Specifications | Intel

Intel® Core™ i7-12800H Processor (24M Cache, up to 4.80 GHz) quick reference with specifications, features, and technologies.
www.intel.com www.intel.com

With 1000 clients I think the term is File Server and not NAS.
Using Realtek for 1000 clients is probably the problem but you also have a mobile CPU without ECC serving ZFS...
So that is my critique of the build. Sorry to be so brutal.
 
Well, though I seed 1000 torrent files, there are usually only 10-20 seeds actually active. Meanwhile, in qBittorrent I set the maximum connection to 200. I think such a torrenting workload is not a very heavy one.

As for the system spec, you are right that this is not a so ideal one, since I just meant to build one personal home NAS with my old laptop. By the way, is there any severe problem of running ZFS without ECC?
 
lassino said:
I increased the torrenting workload, seeding around 1000 torrents at the same time (previously around 500 torrents).
Click to expand...
So I have to wonder about this.
Was it running OK at 500 torrents?
Problems only start when increasing to 1000 torrent seeds???
If so that points to a problem with the torrent serving program.
If 500 ran OK I would gradually increase to 1000 seeders. See where the threshold is. ect..600, then 700, then 800.
With mininal actual clients hitting the box the problem may be with the torrent server itself.
 
Ethernet hardware? Hard to debug. My dumb suggestion (probably impractical): Borrow a different Ethernet card (Intel for example), see whether the problem goes away.

Networking stack software? One suggestion would be to get rid of the jails.

One more thing: Under extreme intense storage workloads with deep queues, I've seen storage bring the whole kernel to its knees (not on FreeBSD, on Linux, but the principles are the same). That is sort of the exception to Cracauer's (otherwise correct) observation that disk problems wouldn't break ping. To see whether that's happening, keep "iostat -d 1" running (perhaps with a -n option to see all disks), and save the output to a file. By the way, keeping "vmstat 1" running at all times isn't a bad idea either. Then look whether you see any unusual spikes in workload or latency at the time of your freeze.
 
You must log in or register to reply here.
Back
Top