ZFS Recommended configuration for 4 HDDs and 1 SSD

I am in the process of setting up a NAS in my home. Currently, I have 4x WD Purple 4TB HDDs and one old LITEON CV8-8E128 (128GB) SSD.
I could find this thread in the forum, but it seems to be quite old and things may have changed since then, especially for the SSD.
Here is the link: https://forums.freebsd.org/threads/zfs-best-partitioning-scheme-between-ssd-and-caviar-red.38952/

My plan is to install the OS on the SSD and use the HDDs for main data storage.
I am considering configuring the HDDs with RAIDZ2 in the zpool.

And would you recommend encryption for this setup?

Additionally, I am thinking of purchasing 2 more HDDs. What would you suggest I do if I were to buy new drives?
 
My plan is to install the OS on the SSD and use the HDDs for main data storage.
Assuming that your main data storage is not terribly performance critical, that seems like a really good plan. In reality, for most small system usage, the system disk is more relevant to "felt" performance (how long commands take, or how responsive the GUI is) than the data disk. Think about what happens when you edit a file or play a video: You access the data disk once or twice, and the system disk several times (for the executable, shared libraries, shell initialization...). So putting your fast disk on the OS disk workload is sensible.

The one big downside (and my home system has exactly the same issue): There is no redundancy on the OS disk. Should your SSD fail (and you described it as "old"), your machine goes down hard. Now, that is "easily" fixed: Just buy a new SSD, re-install the OS (only takes a few hours or days, depending on how skilled and thorough you are), and you're back online. If you're willing to live with the risk of a minor hardware failure (old SSD dies) turning into a wasted weekend, go for it.

I am considering configuring the HDDs with RAIDZ2 in the zpool.
I like that. It means your 4 disks will give you 2 disk's worth of capacity (an acceptable overhead), and any two disks can fail and your ZFS pool continues functioning, giving you ample time to obtain and install a replacement disk.

In the unlikely case that you are performance constrained on the data disks, I would recommend instead using a 2-way mirror built from 4 disks. Like that you get the bandwidth of 4 disks when doing reads (with RAIDZ2 you only get 2 disk's worth of bandwidth). The price you pay: Only 1 disk at a time can fail. And with older disks, and the high absolute error rates of today's disks, having a pool that can only tolerate a single fault is uncomfortable.

And would you recommend encryption for this setup?
Are there any realistic threats that encryption would protect you from? Is a burglar likely to steal your disks, and then turn the computer back on and try to steal your data? Don't worry about your data being at risk from your computer being hacked (encryption doesn't help against that, since the computer is on and the disks unlocked while the hack is running), and don't worry about nation-state actors. Before you put in the effort required to set up encryption, and the risk of it (what happens if you lose the key?), consider the probability of various attacks and scenarios.

Additionally, I am thinking of purchasing 2 more HDDs. What would you suggest I do if I were to buy new drives?
Having a glass of wine or a beer afterwards, because it is a good idea.

The only problem is: What are you going to do with the new drives? Add them to the existing 4-drive pool? Mixing drives of different capacity and performance in the same pool is usually not satisfying, since the pool will be held to the "lowest common denominator". Create a new pool with just the two new drives? First, that pool will not be very redundant (best you can do is a mirror). And now you have to make the painful decision of what data to move to the new pool.

Here's what I would suggest: Run your new machine for a while, and then figure out what the biggest problem is: Is it lack of capacity? Lack of speed? Insufficient redundancy? No backups? Too much power usage? Then buy the correct hardware to fix the biggest problem. A few ideas: Set up the two new drives (very large capacity, low performance) as a pure back pool. Or buy two SSDs, and a find a way to segregate out the really hot data onto the new SSD pool. Or buy two very large disks (22TB), and find a way to segregate out really cold data (perhaps archives of old baby pictures and videos) onto those.

By the way, my home server setup is 4½ disks: One 250 GB SSD to boot from that has the completely OS install on it, two 3-4 TB spinning disks in a mirror for the data, one 2 TB spinning disk (physically 2m away from the server in a fireproof safe) with the backups. The ½ disk is a joke: There is a spare SSD in the server, and once a day the boot/OS disk is copied onto the spare using dd. And there is are several 1 TB backup disks that are physically stored far away (half hour by car), and updated only every we weeks, as off-site disaster recovery.
 
There is a spare SSD in the server, and once a day the boot/OS disk is copied onto the spare using dd.
this will waste the backup disk in less than 3 years (assuming tlc and same 250GB)
also dd ing a live fs is unsafe
you are better off with rsync or zfs send / recv
 
I started out, more or less, like you, and completely re-built my ZFS server, twice...

Consider the advantages of putting the root on a mirror. If my ZFS goes down, many things break, and the inherent delay required for a scratch rebuild is unacceptable.

I put the tank on a stripe of mirrors because I wanted it to be both reliable and fast. Yes, you get less redundancy than RAIDZ2, but the speed advantage is (really) huge.

I "buy back" some disk redundancy by using different SATA controllers for each side of each mirror.

Consider the environment in which your disks have to operate. Where I live, summer temperatures can get above 30C, and in the winter just below freezing. I also experience power blackouts, and glitches.

The most significant error I made was to think that an "ordinary" case and wall power was "good enough" for a ZFS server. What I learned was that I needed a well designed case with a fan blowing directly onto each and every spinning disk, and a UPS.

Consider holding some spare parts. My rural location means that it can take a week, or more, to replace a disk. I keep spare "disks" on hand, in a box.

Do you need to backup your ZFS server? If so, build the solution into your initial design, because it can change your hardware spec.
 
this will waste the backup disk in less than 3 years (assuming tlc and same 250GB)
It's SLC. Industrial, made by Intel for IBM, as an internal logging device (intended for write-only service!). I think it was spec'ed for 10K write cycles. I got it from the trash pile in the lab.

also dd ing a live fs is unsafe
Yes, I know. Nothing a little fsync won't cure. If my boot disk fails, having to run fsync once is the least of my problems.

you are better off with rsync or zfs send / recv
Those are on my to-do list. As is using ZFS snapshots in the backup program. Sadly, there is always more urgent stuff to do.
 
Thank you all for helping me.

I just got two new same HDDs (forgot to mention I can buy same HDDs because I bought from inventory) so now I have 6 HDDs. But this motherboard only supports 6 sata. So I can't use SSD. I can add some pci card if I want, I'm not sure it's bootable with this card and I'm fine with this configuration.

The most significant error I made was to think that an "ordinary" case and wall power was "good enough" for a ZFS server. What I learned was that I needed a well designed case with a fan blowing directly onto each and every spinning disk, and a UPS.
I also got the UPS, luckily I could get a new one for almost half the price. It's noisy so I should move it outside the room.
Thanks for great advice. I live in middle of city and we almost never had blackout but it can happen and sometimes my surge protector goes down so it's great.

I decided to use stripe of mirrors as my RAID configuration. I considered RAID-Z2 but I chose speed. And I can get the replacement in one day so it will be okay.

My choice was still encryption because I use encryption on every device I can.

Thank you all again.
 
I realized only using one pool with HDDs is not that good idea. I just got pcie to nvme adaptor, luckily my mainboard supports NVMe booting from PCIe.

So it will be 6HDD (stripe of mirrors) with one NVMe SSD.
 
Consider holding some spare parts. My rural location means that it can take a week, or more, to replace a disk. I keep spare "disks" on hand, in a box.

TIP that I learned the hard way. The discs that you keep spare try to occasionaly do a smartcheck -t long . Because I had spare red hard disks in the antistatic bag (brand new) and when I needed them one of these was DOA (which I discovered it after 3 years, but due to the 5year warranty I got it replaced after 3 weeks). Luckily this system was used only to backup my data so I could wait, and for 3 weeks I had no backuped running...
 
Mine is six WD Red Pro as RAIDZ-2.
I boot from a USB thumb drive.
The booted OS creates a ram drive for its chatty stuff, and keeps only the system config on the USB.
This lets me use all six SATA ports on the system board.

It is my understanding there should be an even number of data drives, and one or more stripe drives.
 
Back
Top