ZFS Building NAS - Hardware Review?

F

frijsdijk

Hey all,

I'm in the process of rebuilding my NAS at home. Currently this is the hardware that I've picked:

  • Mobo: Gigabyte GA-B150M-D3H
  • RAM: 4x Crucial CT8G4DFD8213 (32GB RAM total) (From Gigabyte's QVL)
  • CPU: Intel Core i3-6100T Boxed
  • SATA: Delock PCI Express Card > 10 x internal SATA 6 Gb/s
  • OS Disk: 64GB Kingston SSD
  • Storage Disks: 8*2TB, WD2002FYPS (in raidz1, I have spares)

The only issue I see here is that the NIC (Intel I219V) of this mobo is currently not supported, but Intel has drivers that can be used: https://downloadcenter.intel.com/download/17509/Intel-Network-Adapter-Gigabit-Base-Driver-for-FreeBSD-?product=82186 . Worst case scenario, I'll have a Intel PCI-e NIC that I can use.

The Delock SATA card works pretty well since http://freshbsd.org/commit/freebsd/r309251 . I've been testing this the last 2 weeks in FreeBSD-11. Already copied ~20TB without any issues.

I just wanted to post this and see if perhaps somebody could spot any pit falls that I've missed.

Also, this board is an M2 slot. It woul be nice to fit the board with a 128GB m2 flash drive, and use this as an L2Arc cache for ZFS. But I have no idea how far support is for M2 in FreeBSD.

Any comments? Suggestions?

Thanks!
 
Why not the GA-X150M-PRO ECC instead of the GA-B150M-D3H?

I actually have the former and it works well with FreeBSD, not to mention it supports ECC RAM for data integrity.

Your CPU is a 35 watt part. I don't know if you planned to use the stock cooler but the Alpine 11 - Passive is better in almost every way. Completely silent, no dust accumulation and no need to worry about a failed fan down the road.

Haven't tried an M2 SSD yet. If I were in your shoes I'd also choose a Samsung over a Kingston SSD if they are similar in price.
 
  • Thanks
Reactions: Oko
gofer_touch said:
Why not the GA-X150M-PRO ECC instead of the GA-B150M-D3H?
I actually have the former and it works well with FreeBSD, not to mention it supports ECC RAM for data integrity.
Your CPU is a 35 watt part. I don't know if you planned to use the stock cooler but the Alpine 11 - Passive is better in almost every way. Completely silent, no dust accumulation and no need to worry about a failed fan down the road.
Haven't tried an M2 SSD yet. If I were in your shoes I'd also choose a Samsung over a Kingston SSD if they are similar in price.
Click to expand...

Here are both compared (dutch, but should not be a problem): https://tweakers.net/pricewatch/compare/461156;480875/

The PRO version does not have video out, I don't want to spend a pci-e slot on a video card, specially not for a NAS.
I have considered ECC memory. The D3H does support Registered error correcting RAM, but the price.. - however, you have convinced me to reconsider.

Cooling option is a good tip, will consider that for sure!

The Kingston SSD I had lying around. Otherwise Samsung would have been my preference for sure!
 
Great!

The GA-B150M-D3H supports ECC modules, but the ECC functionality will not work (according to Gigabyte's own information page).

Noted concerning the video card. I just found the cheapest used VGA out that I could find. It ended up being around $10 US or so.
 
gofer_touch said:
Great!

The GA-B150M-D3H supports ECC modules, but the ECC functionality will not work (according to Gigabyte's own information page).

Noted concerning the video card. I just found the cheapest used VGA out that I could find. It ended up being around $10 US or so.
Click to expand...

I yes, I had doubts on the ECC part, didn't exactly know how to interpretate their words about ECC, but since I wasn't considering it, I ignored it.

About the VGA - it's not the card/cost problem (I have several lying around), I don't want to spend the PCI-e slot!

PS: The ZFS/ECC discussion is very active I found.... nice reads!

http://blog.brianmoses.net/2014/03/why-i-chose-non-ecc-ram-for-my-freenas.html
 
Gotcha.

You will go nuts reading all of the information around the Internet concerning the ZFS and ECC RAM debate. There are extremists on both sides. Essentially it boils down to the fact that ZFS is doing a lot for you behind the scenes. Calculating RAID parity, caching filesystem data in RAM and repairing data on the fly where the checksums don't match.

Hardware RAID cards generally come with ECC RAM onboard which helps provide data integrity. However, instead of a hardware RAID card it is your system's CPU and RAM that is now doing what a RAID card normally would (and much more).

In sum ECC is generally recommended for data storage especially for a machine that will be on 24/7. It will give you peace of mind. But there are a host of other issues that can cause your data to disappear which is often neglected in these "ZFS needs ECC RAM" debates.
 
  • Thanks
Reactions: Oko
Take a look at the new Supermicro X10SDV boards. Starting at <400$ you get a really nifty platform with 2x I350 GBit NICs and even 2x 10gE on board.
There are Versions with 4x SATA3 + 16x SAS2/SATA 3 available; although I'd compare the total price to one of the (much cheaper) variants without onboard-SAS-Controller but with 2x PCIe x8 and 2x IBM M1215 SAS controllers (~100$ each). This might be the cheaper and more flexible option.

These boards also have a fully-fledged ASpeed BMC, so you can do proper OOB-management via IPMI/iKVM. If you still want to carry around a VGA monitor, they also still have a VGA header.

As for ECC RAM: I wouldn't save up on RAM in a storage system that is supposed to run 24/7 and keeping my data safe and consistent. As I've seen some weird and annoying failures (mostly very sporadic, hence very time-consuming to trace down) with regular RAM, especially modules targeted for "gaming" or "overclocking", I even started using ECC in my desktop machines if supported.
 
sko said:
Take a look at the new Supermicro X10SDV boards. Starting at <400$ you get a really nifty platform with 2x I350 GBit NICs and even 2x 10gE on board.
There are Versions with 4x SATA3 + 16x SAS2/SATA 3 available; although I'd compare the total price to one of the (much cheaper) variants without onboard-SAS-Controller but with 2x PCIe x8 and 2x IBM M1215 SAS controllers (~100$ each). This might be the cheaper and more flexible option.

These boards also have a fully-fledged ASpeed BMC, so you can do proper OOB-management via IPMI/iKVM. If you still want to carry around a VGA monitor, they also still have a VGA header.

As for ECC RAM: I wouldn't save up on RAM in a storage system that is supposed to run 24/7 and keeping my data safe and consistent. As I've seen some weird and annoying failures (mostly very sporadic, hence very time-consuming to trace down) with regular RAM, especially modules targeted for "gaming" or "overclocking", I even started using ECC in my desktop machines if supported.
Click to expand...

Thanks for the tips! I've checked out the X10SDV-4C-TLN2F .. very nice board indeed. But over my budget I'm afraid. Remember it's a Home NAS. 10GbE would certainly be nice, but I think that's for the next NAS in 3-5 years or so :-)
 
There are also 2-Core variants available. E.g. the X10SDV-2C-TP4F is selling from 380 EUR here in Germany, so should be well under 400$. The X10SDV-2C-7TP4F is slightly above 500EUR and already has 16x SAS onboard (+4 SATA). Just add RAM and disks and you are good to go...

For tight budget builds of 2 gateways/firewalls and a small fileserver earlier this year, I've used ASUS P9D-E/4L boards. They can handle CPUs from e.g. the dirt-cheap Celeron G1820 up to the E3-1200 v3 Xeons.
It's a bit older, hence it "only" got the I210 NICs and only supports DDR3, but DDR3 and socket 1150 CPUs became quite cheap, so for a budget build this might be a viable solution.
There are proprietary ASUS PIKE add-on cards available to enable the additional 8 SATA-Connectors. It's quite a mess though when you need to wire up 14 single SATA-connectors to a Backplane...
The newer P10D Series was quite a bit more expensive when I last checked, but if you want/need more current hardware have a look at those.
 
The recommendations from iX and the FreeNAS forums are to always use ECC. If you are buying new hardware, there is very little reason not to. It is very cheap insurance, especially with the relatively large amounts of RAM used by a NAS.

gofer_touch: I have been curious about the X150M Pro ECC motherboard for a while, it looks like it could be a useful small NAS board. Do you know if it will boot without a video card? That would make the lack of onboard video unimportant. Put in a video card to install the system, remove it and run without it most of the time.
 
Oh, and the recommendation for L2ARC is to not add any until you have put in as much RAM as the motherboard will hold.
 
wblock@ said:
The recommendations from iX and the FreeNAS forums are to always use ECC. If you are buying new hardware, there is very little reason not to. It is very cheap insurance, especially with the relatively large amounts of RAM used by a NAS.

gofer_touch: I have been curious about the X150M Pro ECC motherboard for a while, it looks like it could be a useful small NAS board. Do you know if it will boot without a video card? That would make the lack of onboard video unimportant. Put in a video card to install the system, remove it and run without it most of the time.
Click to expand...

On ECC: still considering. Reason being of course higher price of mobo and RAM. And chances are very small. Non-zero, but small. This is a discussion that's endless, and I'm aware of it.
On ECC board without video: yes, I was thinking the same.

Pro on ECC board: Intel(variant) NIC is already supported in FreeBSD.

Hmmmm.
 
Since this is a home server I'd say ECC isn't that critical but it's all down to how valuable the stored data is for you and if it has to be available to you all the time. Just to note that if you go the non-ECC route there's nothing that will tell you that there's memory related data corruption going on. ZFS is not able to do that because the checksumming only covers what has been written on the disk(s). If the system is feeding ZFS already corrupted data it will just happily store it and calculate the checksums over the already corrupted data.
 
kpa said:
Since this is a home server I'd say ECC isn't that critical but it's all down to how valuable the stored data is for you and if it has to be available to you all the time. Just to note that if you go the non-ECC route there's nothing that will tell you that there's memory related data corruption going on. ZFS is not able to do that because the checksumming only covers what has been written on the disk(s). If the system is feeding ZFS already corrupted data it will just happily store it and calculate the checksums over the already corrupted data.
Click to expand...

I have read otherwise (as in, in most cases, ZFS will notice something's wrong). Again not zero, but it's very small * very small = very small ^2 = very very very small :)
 
frijsdijk said:
I have read otherwise (as in, in most cases, ZFS will notice something's wrong). Again not zero, but it's very small * very small = very small ^2 = very very very small :)
Click to expand...

I have to question the expertise of anyone claiming that to be true. ZFS is a just another filesystem for any application wanting to store data and it has no knowledge of what the data it's asked to store is supposed to be. It does try to guarantee the integrity of its own buffers and caches but if the corruption happens outside it's own data structures it has no method of knowing if the incoming data from an application is corrupted or not. It can only detect errors when it reads back the data from disk and compares the calculated checksum to the stored checksum that was stored when the data was last time written.
 
kpa said:
I have to question the expertise of anyone claiming that to be true. ZFS is a just another filesystem for any application wanting to store data and it has no knowledge of what the data it's asked to store is supposed to be. It does try to guarantee the integrity of its own buffers and caches but if the corruption happens outside it's own data structures it has no method of knowing if the incoming data from an application is corrupted or not. It can only detect errors when it reads back the data from disk and compares the calculated checksum to the stored checksum that was stored when the data was last time written.
Click to expand...

Thanks kpa - I'll take your word for it, as I will take wblock's. I guess you fellas should be quite reliable :-)

GA-X150M-PRO ECC & ECC it is then!
 
It would surely help where to know you live and that Delock card is going to be a bit "living on the edge" at best. You'd probably be much better off (especially given the price) getting a Dell T20 server, ripping out the innards and placing it in a larger case.
http://www.idealo.de/preisvergleich...-t20-xeon-e3-1225-v3-3-2ghz-20-3708-dell.html

The T20 server is pretty much available everywhere in EU and US.

The mobo is mATX and you need a cheap PSU adapter (~10 USD) if you want to use another PSU, all Intel and works great with LSI adapters.

That's why I did at home and it works great with FreeBSD (I have the cheaper G3220 CPU however).
https://translate.google.com/translate?sl=de&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http://www.hardwareluxx.de/community/f101/dell-poweredge-t20-1031138.html&edit-text=&act=url

If you speak German skip the Google translate. :)

wblock@ ^^
 
wblock@ said:
gofer_touch: I have been curious about the X150M Pro ECC motherboard for a while, it looks like it could be a useful small NAS board. Do you know if it will boot without a video card? That would make the lack of onboard video unimportant. Put in a video card to install the system, remove it and run without it most of the time.
Click to expand...

I've asked Gigabyte this question. Let's see how they respond!
 
So, Gigabyte seems to be pretty apathic in their response. None, whatsoever. Moreover, the X150M Pro ECC seems to be out of stock in most shops (only shops still selling it are much more expensive).

So now my focus shifted towards the Fujitsu D3417-B. Board is a bit more expensive, but it's Intel all over, C236 chipset, so ECC support as well. Also video onboard (no hassle with seperate video card), on-board USB3 slot (could come in handy for OS installations on USB sticks).

Any opinions? Would like to hear them!
 
diizzy said:
May I ask why the T20 with the original case or without wouldn't be suitable? :)
Click to expand...

It's not a bad thing, the T20, I have to admit. But I feel it's a bit dated today, compared to the stuff I'm assembling now. Besides that, I have 8 3.5" disks to hold, +2 SSD's, and I think one more 3.5" disk on top of that for local backups. I don't thing that'll fit in a T20.
 
Consider your targets: 1-safety, 2-cost and 3-speed.
Consider your usage profile: home NAS with a low number of clients and multimedia type filestorage (I presume)
for cost you've set your budget.
for speed you've set your budget too: 1Gbit or 100Mbyte/sec. Your hardware (disk and controller) can support that. But think of your latency.
If there is any space in your $budget spend it on safety and reduction of latency.
I second that ECC is the way to go for your datastorage endpoint; a failed non-ecc RAM module will always cost you days of troubleshooting, let alone the data loss it might provoke. Remember the golden rule, 1Gb of RAM per terabyte of disk; don't see much use for 32Gigs in your setup with 16T - 16Gb of RAM is adequate.
Next suggestion: use 2 ssd's and partition them as mirrored os partitions and a log partition (1/2 the size of your RAM). Preferably attach these ssd's to separate controllers (1 on onboard ahci/m2 and 1 on pci-hba) . This way you'll have double pathways on the whole storage backend - from ram to bits on disk. Plus the advantage of near-zero latency for writes.
When budget remains or need arises, add l2arc via onboard sata slots.
Stick in a usb-stick for emergency boot (and remember to keep it uptodate with releases); lean back and enjoy!
But remember, this is your NAS. Try to resist any temptation to use it as your tryout playground.
 
wblock@ said:
The recommendations from iX and the FreeNAS forums are to always use ECC. If you are buying new hardware, there is very little reason not to. It is very cheap insurance, especially with the relatively large amounts of RAM used by a NAS.

gofer_touch: I have been curious about the X150M Pro ECC motherboard for a while, it looks like it could be a useful small NAS board. Do you know if it will boot without a video card? That would make the lack of onboard video unimportant. Put in a video card to install the system, remove it and run without it most of the time.
Click to expand...

Right. So I meant to follow the thread but got tied up with new year stuff.

I can confirm that the X150M Pro ECC will indeed boot without an installed video card. I think I might actually run it like this from now on. I was using an old Sapphire Radeon HD 3450 card that I found secondhand mainly for the install. The card dissipated quite a bit of heat into the chassis.

I haven't checked with a watt meter, but I probably shaved off a good bit of wattage by removing the card.
 
gofer_touch said:
Right. So I meant to follow the thread but got tied up with new year stuff.

I can confirm that the X150M Pro ECC will indeed boot without an installed video card. I think I might actually run it like this from now on. I was using an old Sapphire Radeon HD 3450 card that I found secondhand mainly for the install. The card dissipated quite a bit of heat into the chassis.

I haven't checked with a watt meter, but I probably shaved off a good bit of wattage by removing the card.
Click to expand...

That's good to know!

I'm now trying to decide to go either for the Gigabyte board, or the Fujitsu board.
 
You must log in or register to reply here.
Back
Top