And that is achieved by just having one dataset per service on the pool. eg:
storage/nfs, storage_smb and so on, right?I did this 5.5 years ago and wrote about it here. While some of that information is a bit dated, you may find it useful. In particular, it considers backups and replication which are often afterthoughts in these projects.
I did various tuning when I first set up the 'zillas, but as I mentioned that was 5.5 years ago. Things have changed in the meantime (a couple of major FreeBSD kernel versions, switching to OpenZFS, etc.). I'd hesitate to say "this is the one true way".
Yes. Just in 14 drives we are talking $6K USD
www.wiredzone.com
Yes. Just in 14 drives we are talking $6K USD
Here is a SAS drive that I might buy.
WUH721816AL5204
Ok $415x14=$5810 from this random place.
Two spares puts you up around $6.8K.
Then add in $4K chassis we are past $10K.
How much is 256GB DDR4? Zoinkers. ZFS needs lots of RAM.
Here is a nice chassis:
Supermicro SSG-6049P-E1CR24L 4U Storage Barebone Dual Processor
Barebone 4U Rackmount SuperServer, Dual Intel Xeon Scalable Processors Gen. 2, Intel C624 chipset, Up to 4TB DDR4 ECC 2933MHz memory, Broadcom 3008 SAS3 AOC, 24 Hot-swap 3.5in drive bays, Dual 10GBase-T LAN ports, Redundant 1200W Power Supply (CSE-846BE1C-R1K23B + MBD-X11DPH-T)
That isn't necessarily disqualifying. Spinning rust media isn't going to come close to saturating a 12G SAS port. I didn't switch to expanders on my newer 'zilla builds because there wasn't enough of a cost savings to justify having different types of hardware. If I was building systems with more than 16 drives I might have reconsidered. It looks like Supermicro still sells versions of the same SC836 chassis I used, which should be more than sufficient.
That is an often-cited article (probably because of the flashy titleKeep in mind that ZFS stripe width is Fraught:
How I Learned to Stop Worrying and Love RAIDZ
The popularity of OpenZFS has spawned a great community of users, sysadmins, architects and developers, contributing a wealth of advice, tips…www.delphix.com
), but I found this one—dealing with the same topic—much more informative and illuminating:
jro.io
In addition to the expander (BE) and individual (TQ) backplanes, they have the BA multi-lane backplanes. That gets you down to 4 cables. Those may no longer be a "standard configuration" shown on their website, but do seem to be orderable from distributors (for example, CSE-836BA-R920B).
Agreed. The current batch of SuperMicro 3.5" drive 24-bay chassis all seem to use an expander.
I agree its just the thought of spending that much and getting an expander.
There is no one generic "best" configurations - those articles serve as reasonable guidelines to start with, but if maximum performance is a requirement, various configurations will need to be tried to find out what works best on a specific set of hardware and data.That is an often-cited article (probably because of the flashy title
Building a NAS
The use case mentioned in the OP shouldn't need anywhere near that number of drives, fortunately. While 8TB drives were still rather cutting edge when I built my 'zillas, these days drives of up to 20TB each are available.
Some years ago on eBay there were a large number of 60-drive enclosures from (IIRC) HGST. Orientation in other than the "classic" label-on-top is feasible as long as the drive manufacturer explicitly supports that configuration. Not all drives do. Dell had some desktops that mounted the drives connector-on-top and depending on the brand / model of drive, it could be problematic. For any other greybeards here - Atasi 3046
Mounting random SAS expanders and cabling them can be an exercise in frustration (and usually is). Expander backplanes are much better-behaved.
I mostly disagree, but partially agree. For large configurations, and for efficiency, expanders are simply necessary. Buying enough HBAs and cabling to give each disk drive its own SAS lane is expensive, power-hungry and wastes PCIe slots. When engineered well, expanders are not the bottleneck.
Absolutely! If one wants to seriously optimize performance or cost or efficiency or ... then one has to do some good planning and some trial and error.
My personal record for a single host was 18 GB/sec, sustained from disk drives through the CPU to Infiniband. It takes lots of work to get there.
About 6 or 8 years ago, we checked with the big disk drive vendors, and at the time all drives could be mounted in any orientation that was parallel or perpendicular to gravity, while diagonal was probably OK but untested.
Absolutely true. And the effect is more pronounced for cheap drives (the SATA drives from 15-20 years ago), which were not very tolerant of vibration. This is why disk enclosures actually need to be engineered by people who understand mechanical engineering; you can't just bolt disk drives to a random piece of aluminum and expect them to work.
This is a perfect example of: If you use expanders, you need to use expanders with good firmware, and you need to make sure the whole stack firmware/software works well together.
That doesn't surprise me - in order to meet modern performance specs, things need to be pretty well balanced within the drive. The days of a giant unbalanced head driven through a metal ribbon from a stepper motor are long gone.
"Flashy" is one way to describe it. "Authoritative" is another:That is an often-cited article (probably because of the flashy title
Building a NAS
www.delphix.com
Some combination of replication and backup will likely be needed. Here are the sections of my 'zilla articles discussing this:
bugs.freebsd.org
vermaden.wordpress.com
Yes this seems to be an interesting new option on FreeBSD 13 and OpenZFS.
# zfs create nas02/nfs
# zfs create nas02/smb
# zfs create nas02/iscsi
# zfs set recordsize=4k nas02/iscsi