ZFS Advice on configuring existing media server (ZFS)

[iyoti]

Hey all

I've a media server with:

• 2 × 466GB NVMe drives for OS
• 12 × 5TB HDDs for storage
• 1 × 699GB SSD for "performance" (Crucial CT750MX300SSD1, which may be relevant for questions below)
• 32GB RAM
• 12-core CPU

All are SATA drives. I have 2 zpools set up, the first in a mirror for OS which is working great.

For the storage, I set things up this way:

# zpool create media raidz2 ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 ada8 ada9 ada10 ada11 log ada12
# zfs set compress=lz4 atime=off xattr=sa recordsize=1M logbias=throughput dedup=on autotrim=on media

Here I'm using the spare SSD as a log device to improve performance/safety of writes.

This system is replacing an older media server which had a similar HW configuration but only 4 storage drives and 16G RAM and a 4-core CPU, but in this case the "performance" SSD was set up as a cache. I rarely noticed issues other than streaming some very large 4K files.

However, I'm noticing that streaming certain files from this new setup can have some hiccups, with the media player often crashing. The media streaming service on the system has recently started crashing too. It isn't only 4K files that are having problems.

So, I'd like some advice on how to tweak my current setup. Some things I'm wondering:

• Whether I should enable sync=always for the media pool
• Whether I should switch the log drive to be a cache drive (like my prior setup)
• Can I use the "performance" SSD for both log and cache?

Is there anything else I can do to improve the config?

 

[zirias@]

Are you sure the issues are related to storage? Could it be related to network (either physical or the software used)?

Asking because my server only has 4 (spinning) HDDs in a single raidz1 pool, no extra devices, and this machine does everything I need (separated with VMs and jails), including a media server. I can stream 4k content on my tv set without any issues ...

 

[Jose]

Your vdev's stripe width might be too large:

How I Learned to Stop Worrying and Love RAIDZ

The popularity of OpenZFS has spawned a great community of users, sysadmins, architects and developers, contributing a wealth of advice, tips…

www.delphix.com

Can you experiment with two vdevs with six drives each?

 

[iyoti]

Are you sure the issues are related to storage? Could it be related to network (either physical or the software used)?

Asking because my server only has 4 (spinning) HDDs in a single raidz1 pool, no extra devices, and this machine does everything I need (separated with VMs and jails), including a media server. I can stream 4k content on my tv set without any issues ...

It might be… It is a very odd issue, especially since it is difficult to track down or replicate.

The network speed seems fine; from that machine using speedtest-cli I get 400Mbps down and 200Mbps up to the internet, and in general my home network shows around 500Mbps between devices over wifi or over powerline LAN.

I have noticed that my media server can be a little slow to connect via SSH when these issues arise, but then I connect and the SSH session is fine; htop refreshes quickly and without issues.

I'll investigate whether there are any network config changes I can make for my specific NIC. ?

 

[iyoti]

Your vdev's stripe width might be too large:

How I Learned to Stop Worrying and Love RAIDZ

The popularity of OpenZFS has spawned a great community of users, sysadmins, architects and developers, contributing a wealth of advice, tips…

www.delphix.com

Can you experiment with two vdevs with six drives each?

I'm not sure; I think I'd have to disconnect and resilver a drive at a time? There's already a large number of files on the pool and it is very time-consuming to recreate all that from the original media. Not impossible, but I'd like to avoid it if I can.

Can the stripe width be changed on an active pool?

 

[VladiBG]

First you need to determinate where is the bottleneck.
To test that connect to the server using SSH and use systat to monitor your :ifstat while you streaming to see if the network is the problem or not. Then you can try to copy some 4k video on your NVMe disk where the OS is and stream it from there to see if the storage is your bottleneck or not and you can also observe it using systat on :iostat page or :zarc

systat(1)

www.freebsd.org

 

[Jose]

I'm not sure; I think I'd have to disconnect and resilver a drive at a time? There's already a large number of files on the pool and it is very time-consuming to recreate all that from the original media. Not impossible, but I'd like to avoid it if I can.

Can the stripe width be changed on an active pool?

Nope. It's a destructive operation. You would have to recreate from the original media.

 

[iyoti]

Ok, I've attached a selection of screenshots while streaming some 1080p and 4k video from the various disks.

I can't really see any issues with any of the numbers I'm seeing, but I'll be the first to admit that my zfs knowledge is limited.

Any advice is greatly welcomed!

 

[zader]

what "media server" are you using? plex?

you could try tweaking your arc, dirty laundry and hard kernal settings... as listed in my "how to" above .. but tbh this sounds less like a hardware/zpool issue and more like "media server" settings..

for example, are you transcoding? if so, how? how many instances do you allow and are they throttled? are you transcoding with hardware or software? what codec is the source? ie 264 vs 265 are miles apart.
for plex make sure your database is cleaned and kept upto date.

another thing you could do is create a zvol on your nvme drive and put the /temp folder and meta info database on that.. you could also set up transcoding to stream from that nvme device.

4k hdr only requires about 25mb/s .. if you are using un-compressed 7.1+ Dolby audio.. about twice that .. so a "normal" 5GH wifi router should be able to easily stream it..

a "normal" 5600 rpm rust pile can easily do 160MB/s transfer .. 7200 rpm is more like 200-260MB/s so your zpool is most likely not the issue..

 

[iyoti]

what "media server" are you using? plex?

Yep.

you could try tweaking your arc, dirty laundry and hard kernal settings... as listed in my "how to" above .. but tbh this sounds less like a hardware/zpool issue and more like "media server" settings..

for example, are you transcoding? if so, how?

Not for everything, most is via direct-play.

how many instances do you allow and are they throttled?

One, only on home network.

are you transcoding with hardware or software?

Any transcoding happens via software, AFAIK. Some hardware toggles are enabled, but I don't think they have much of an effect. CPU is a AMD Ryzen 5 5600G with Radeon Graphics (amd64, 12 cores).

what codec is the source? ie 264 vs 265 are miles apart.

Usually 264.

for plex make sure your database is cleaned and kept upto date.

All scheduled and working well, as far as I know

another thing you could do is create a zvol on your nvme drive and put the /temp folder and meta info database on that..

Good shout. I've started recreating the meta data there, and I'll move /tmp too.

you could also set up transcoding to stream from that nvme device.

So I have the spare Crucial SSD… it was set up for the SLOG but I've changed it to be a cache. I could pull it out of that pool and set it up on a new mount point purely for metadata & transcoding? Or only for transcoding?

 

[diizzy]

I somewhat doubt it's a storage related issue...

What does crashing mean, logs? Does this box have a Realtek nic? What version of FreeBSD are you running? Is it stable if you disable transcoding?

 

[zader]

sorry missed one tiny thing

Code:
# zpool create media raidz2 ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7 ada8 ada9 ada10 ada11 log ada12
# zfs set compress=lz4 atime=off xattr=sa recordsize=1M logbias=throughput dedup=on autotrim=on media

this is for freenas, the pros/cons & costs directly apply

dedupe

I would not enable dedupe on a media server unless you have a specific reason to. Especially considering all the "extra" temp/junk plex does.