Storage Cube build for serious collectors

[Seani]

Hi there.
I am building a Server/NAS mainly for storage purposes (mainly movies, music and RAWs from my entire family) and would appreciate your input regarding sufficient hardware.

What I own already:

Case: Lian-Li PC-D8000 (enormous cube case)
http://www.lian-li.com/en/dt_portfolio/pc-d8000/
I bought this case since it offers space for up to 20 3,5" HDDs (24 if you insert a 3x 5,25" to 4x 3,5" cage in the front), 2 PSUs and sufficient cooling/upgrade possibilitys.

PSU:
Corsair HX850i
Mainly for the option of connecting up to 24 SATA devices directly, the platin level and decent spare power for spin-on-surges (also, I got it quite cheap).

HDDs:
10x Seagate IronWolf 8TB NAS drives
http://www.seagate.com/www-content/product-content/ironwolf/files/ironwolf-ds1904.5-1609us.pdf
I got them cheap and 3 years warranty seemed insuring to me (otherwise I would not have gone with Seagate). I intend to run them in a single RAID-Z2 pool of 10 drives (since I have 2 separate backups I think I can live with the risks that implies).

Cooling:
I installed 6 120mm radiators (bequiet Pure Wings 2) on the sides of the 2 harddrive cages for a continous airflow from one side to the other and connected them to a BitFenix Recon to adjust the rotation speed/the noise.

What I still need is a motherboard+HBA wich combined support
- a minimum of 20 SATA ports (I will upgrade to 20 drives in a few years|if the mobo alone supports 11x SATA (10 drives + boot device) that would also suffice for now)
- 8 DIMM slots which each support at least 16GB of RAM per DIMM
- IPMI since I intend to use an UPS (I heard good things about this one: http://xtreme.metacomp.de/Shop/DE/Product/Details/3943/880559/false/)

Also a fitting CPU that supports that much RAM and has decent computing power (at least 4 cores with 2,5GHz).
And at last a boot device. I read a lot about using USB drives but I would tend to go with a small SATA-SSD.

After reading the hardware recommondations multiple times I came up with this stuff here. But I have the feeling that this is serious overkill and unnecessary for my purposes (wich are safe storage and media streaming with a max. of 5 people simultaneously accessing data).
mobo:
http://www.supermicro.com/products/motherboard/xeon/c600/x10srh-cln4f.cfm
ram x4:
https://www.amazon.com/dp/B00MDZL2B6/?tag=ozlp-20
cpu:
http://www.mindfactory.de/product_info.php/Intel-Xeon-E5-1630v4--4x-3-70GHz--tray_1110441.html
As a boot device I would use an old 60GB SSD from one of my old PCs.

I am completely open to other (and hopefully a bit cheapter) suggestions and you can always call me a moron if this is serious overkill.

I'd be delighted if you guys could help me to narrow down CPU, MOBO, RAM and boot device. I am a little lost since I never build a NAS/Server before and can't estimate what server-grade gear is overkill and what is needed. My experiences from building windows PCs do not really help me a lot here.

At last: I know that my english skills aren't perfect since I am from germany and therefore not a native speaker.

 

[darkwarrior]

HI there,

interesting project that you have there :D

Seeing how many disks you plan on installing over time, I would definitely take the time to look at a server enclosure that will offer you 24 ( or more) Hot-plug disk slots.
Additionally, the power requirements will be way too much for that Corsair PSU to handle ...
With 10 disks you can already reach 700W peak power draw , so that PSU will suffer (without speaking about the fact that Seasonic would be a better choice).
Have a look at the sticky here for more info on that matter ;)

 

[Seani]

10 disks you can already reach 700W peak current

The sticky implies that one should take 35W per drive to be on the safe side.
Even with 20 drives (I do not plan to use more than that in the next 10 years since the 10 drives I use now already have a decent buffer and sum up my familys data collection of the last ~10 years) only 700W of 850W would be used for spin-up. Therefore I do not really see your point there (despite the fact that I can install a second PSU into my case anytime. If I wanted hot-plug, I would buy the hot-plug backplates that are available for my Lian-Li case. But since downtime is not really an issue for me and also because I don't know of any other advantages of hot-plug (other than not having to open your case and remove two cables and not having to shut down your system before doing so) I don't think that having hot-plug is worth the money.
Still, thank you for your input.

 

[darkwarrior]

The sticky implies that one should take 35W per drive to be on the safe side.
Even with 20 drives (I do not plan to use more than that in the next 10 years since the 10 drives I use now already have a decent buffer and sum up my familys data collection of the last ~10 years) only 700W of 850W would be used for spin-up. Therefore I do not really see your point there (despite the fact that I can install a second PSU into my case anytime. If I wanted hot-plug, I would buy the hot-plug backplates that are available for my Lian-Li case. But since downtime is not really an issue for me and also because I don't know of any other advantages of hot-plug (other than not having to open your case and remove two cables and not having to shut down your system before doing so) I don't think that having hot-plug is worth the money.
Still, thank you for your input.

Your call, nevertheless you should not forget about the power consumption of the rest of the system. ;)

 

[Seani]

Your call, nevertheless you should not forget about the power consumption of the rest of the system. ;)

Since there aren't going to be any power-hungry graphic cards around, I think even with 20HDDs it should be quite safe. This test of the HDD I use says that my HDDs use 2.0A and 12V for spin-up, that leads to 24W power consumption: (http://www.storagereview.com/seagate_nas_hdd_8tb_review) Let's assume 25W for buffer purposes. That makes 500W if I use 20 of those drives. Plus 25W MOBO, plus lets say 50W CPU, 30W RAM, 50W fans and 5W for SSD/USB-stick. Adding up to 660W out of 850W for the few seconds the HDDs need to spin up (I don't intend to shut this thing down often). 77% load shouldn't fry a PSU that was meant for overclocking systems and was therefore designed for higher stability concerning uncommon load scenarios. I thought 850W were an overkill. Funny how opinions differ. With the 10 drives I want to use for the next few years I think it will be perfectly safe and that is my main concern right now.

 

[Ericloewe]

This test of the HDD I use says that my HDDs use 2.0A and 12V for spin-up, that leads to 24W power consumption: (http://www.storagereview.com/seagate_nas_hdd_8tb_review)

That's not part of the test, it's lifted straight from the datasheet. Empirical evidence suggests that HDD datasheet power consumption is not to be taken at face value.

Realistically, 850W are going to be fine, especially given the fact that there's little to choose from between 850W and 1000W. However, I'd definitely recommend a Seasonic Platinum 860, which is probably the best Platinum PSU you can buy. (Seasonic Prime Titanium is not unequivocally better - the fan is a downgrade and transient performance is on the dubious side. And boy is it pricy... Other than that, it's better in every way. Well, except for aesthetics... I much prefer the X-Series/Platinum look over the gaudy Prime look)

 

[Seani]

However, I'd definitely recommend a Seasonic Platinum 860, which is probably the best Platinum PSU you can buy.

It's funny that so far no one has commented on the things I do have questions about but instead everyone doubted the decisions I already made. Both case and PSU are bought and even configured already and therefore it is idle thought to talk about case and PSU. I will try my luck. ;)

 

[Ericloewe]

It's funny that so far no one has commented on the things I do have questions about but instead everyone doubted the decisions I already made. Both case and PSU are bought and even configured already and therefore it is idle thought to talk about case and PSU. I will try my luck. ;)

Not much to say, the build is reasonable. Nobody is going to talk you down to a Xeon E3, since you want to expand to 20ish drives. It's a compromise you can go with, but it has important limitations.

 

[Stux]

The x10 e5 mother boards support 10 Sata devices before accounting for HBAs. If you want 11 devices (Ie with a Sata boot drive), it puts you into a pickle.

Adding the HBA gets you an extra 8 disks. Which is still three short of the 21 disks you plan.

But I guess is perfect for 25 disks when you add an additional HBA

(10 on board + 8 + 8)

850W is marginal for 24 disks system, but will maybe be fine. Will be fine for ten disks

 

[pirateghost]

SAS expander will get you tons of hard drives.

I run 20 on my setup with the 8 onboard SAS ports and an expander

 

[Seani]

If you want 11 devices (Ie with a Sata boot drive), it puts you into a pickle.

Exactly my thoughts. My idea there was that if I use an USB-stick instead of a SSD, some mainboards have an USB 3.0 Typ A onboard wich combined with the 10 SATA ports those mainboards have would fit my setup perfectly until the next upgrade gets scheduled. Only downside is my serious distrust towards USB-sticks. Is it possible to buy lets say 2 or 3 identical USB-sticks and mirror them for the event of a failure (at least 4 USB-sticks have died on me over the last years)?

SAS expander will get you tons of hard drives.

I have no experience whatsoever with SAS in general (up to today never needed a HBA) and therefore also no experience with SAS expanders. Do you have a source where I can read common facts (pro/con of a SAS expander in comparison with a SAS HBA, performance/bandwith issues e.g.)?

850W is marginal for 24 disks system, but will maybe be fine. Will be fine for ten disks

Before the next upgrade I will probably measure how much power my system really consumes while spinning all those drives up. Then (hopefully this won't happen 'till a few years in the future) I will either buy a new PSU or simply install a second PSU (since my case supports it).
24 disks is not really appealing to me (if it has advantages I am oblivious to, please feel free to enlighten me). With 2x 10 disk RAID-Z2 I get the storage of 16 drives and have 4 that can die on me before data loss becomes an issue (though I will keep 2 separate backups anyway). I would feel uneasy with a RAID-Z2 of 12 disks and when going with 3x 8 disks RAID-Z2 I would gain the storage of 2 disks but "lose" 2 more drives, upping the redundand space from 20% to 25% + 10-20% for ZFS to work properly. Right now I have about 40TiB of data that would need to be stored on the NAS. 10x 8TB drives = 73TiB storage - 2*7,3= 58,4TiB of storage with RAID-Z2 *0,8 to leave 20% free = 46,72TiB of usable space before the performance of ZFS starts to become a brake. Since it took me and my family over 10 years to acquire that much data I hope that we won't reach the critical mark of 90%=52,56TiB in the next few years.

 

[Stux]

Exactly my thoughts. My idea there was that if I use an USB-stick instead of a SSD, some mainboards have an USB 3.0 Typ A onboard wich combined with the 10 SATA ports those mainboards have would fit my setup perfectly until the next upgrade gets scheduled. Only downside is my serious distrust towards USB-sticks. Is it possible to buy lets say 2 or 3 identical USB-sticks and mirror them for the event of a failure (at least 4 USB-sticks have died on me over the last years)?


I have no experience whatsoever with SAS in general (up to today never needed a HBA) and therefore also no experience with SAS expanders. Do you have a source where I can read common facts (pro/con of a SAS expander in comparison with a SAS HBA, performance/bandwith issues e.g.)?


Before the next upgrade I will probably measure how much power my system really consumes while spinning all those drives up. Then (hopefully this won't happen 'till a few years in the future) I will either buy a new PSU or simply install a second PSU (since my case supports it).
24 disks is not really appealing to me (if it has advantages I am oblivious to, please feel free to enlighten me). With 2x 10 disk RAID-Z2 I get the storage of 16 drives and have 4 that can die on me before data loss becomes an issue (though I will keep 2 separate backups anyway). I would feel uneasy with a RAID-Z2 of 12 disks and when going with 3x 8 disks RAID-Z2 I would gain the storage of 2 disks but "lose" 2 more drives, upping the redundand space from 20% to 25% + 10-20% for ZFS to work properly. Right now I have about 40TiB of data that would need to be stored on the NAS. 10x 8TB drives = 73TiB storage - 2*7,3= 58,4TiB of storage with RAID-Z2 *0,8 to leave 20% free = 46,72TiB of usable space before the performance of ZFS starts to become a brake. Since it took me and my family over 10 years to acquire that much data I hope that we won't reach the critical mark of 90%=52,56TiB in the next few years.

I only mentioned 24 drives because you did... "24 if you insert a 3x 5,25" to 4x 3,5" cage in the front"

You can ZFS mirror your boot drives and scrub them to detect errors.

 

[Seani]

I only mentioned 24 drives because you did...

That was a misunderstanding. I just pointed out how many drives I COULD easily install in the case and not how many I actually want to install.

Thank you all for your input so far.
I reconsidered a bit and right now I would buy:
Mainboard: Supermicro X10SRH-CLN4F
http://www.mindfactory.de/product_i...11-3-Quad-Channel-DDR4-ATX-Retai_1039998.html
CPU: Intel Xeon E5-2620v4 8x 2,1GHz
http://www.mindfactory.de/product_i...-2620v4-8x-2-10GHz-So-2011-3-WOF_1035601.html
CPU-FAN: Noctua NH-U12DX i4
http://www.mindfactory.de/product_info.php/Noctua-NH-U12DX-i4-Tower-Kuehler_897203.html
RAM x4: Samsung MEM-DR416L-SL01-ER21
https://www.amazon.com/Supermicro-C...ER21-Samsung-Memory/dp/B00MDZL2B6?tag=ozlp-20
Since the mainboard has a SAS3 HBA onboard, maybe I will buy one of those: https://www.heise.de/preisvergleich...4x-sata-sff-8448-kabel-2279800-r-a915530.html to also use my old 60GB SSD as the boot device instead of an usb-stick.

I am still uncertain how potent a UPS I should acquire. Is it enough if the UPS can handle the average load of the server or is it necessary to sustain the peak load?

 

[darkwarrior]

You might not want to consider my opinion, but i'm telling you anyway ;)
The CPU you have chosen is not the best for NAS duties (especially SAMBA) because of the low core speed.
You might be better off with E5-162, 1630, 1650 (almost 4GHz with turbo-boost).:)

Concerning the RAM you can always compare with what Crucial is able to offer. I had a very good experience with their products (and I got a bit scared of the Amazon shipping "care").

 

[Stux]

Crucial guarantees compatibility, and is Micron's retail brand

 

[Seani]

The CPU you have chosen is not the best for NAS duties (especially SAMBA) because of the low core speed.
You might be better off with E5-162, 1630, 1650 (almost 4GHz with turbo-boost).

I thought with the boost of 3GHz the 8-core model would be okay. It just feels so wrong to pay almost the same amount of money for half the amount of cores. Do you know of a sweet spot processor when it comes to the relation between GHz, core amount and price? Thank you guys for advising me.

One of the few crucial memory modules I found that can be purchased not only from crucial itself is this:
http://www.crucial.de/deu/de/x10srh-cln4f/CT7128970
Do you think that this is a better choice than the Samsung DIMMs that need shipping?

 

[diedrichg]

Do you plan to do any video transcoding?

The Guideline

Very roughly speaking, for a single full-transcode of a video, the following PassMark scores are a good guideline for a requirement:

1080p/10Mbps: 2000 PassMark
720p/4Mbps: 1500 PassMark

The CPU Benchmark website is a good resource to see what sort of PassMark score a particular processor received.

Related Page: CPUbenchmark.net

Source:
https://support.plex.tv/hc/en-us/articles/201774043-What-kind-of-CPU-do-I-need-for-my-Server-

 

[Seani]

Do you plan to do any video transcoding?

Since my upload at home is ~17MBit/s it will certainly become interesting once the server/nas is online.

 

[Seani]

Update for my build:
The NAS is now up and running (I had it run 2 passes on memtest86 (20h, no errors, nothing in the log concerning ECC) and then stressed the CPU+RAM for 8 hours straight (AIDA64 Reliability test) It never got more than 39°C at 100% load @16x 2,3GHz consuming 52W.

In the end, I went with:
RAM: 4x Crucial CT16G4RFD4213
Mainboard: Supermicro X10SRH-CLN4F
CPU: Intel Xeon E5-2620v4
and CPU-FAN: Noctua NH-U12DX i4 to complete my build.

I'm just now struggling with the LSI 3008 FW (I crossflashed it to IT but used the newest version (P13), wich is wrong, as FreeNAS tells me as a warning: "WARNING: 22 Jan 2017, 5:09 p.m. - Firmware version 13 does not match driver version 12 for /dev/mpr0. Please flash controller to P12 IT firmware."
So I will have to flash the firmware again (here I stood, thinking: "Jay, no more flashing.").

Then the Hard Drive Burn-In may begin. Thanks for all your help so far.

 

[xekon]

sorry to bump an old thread, however in this case I am curious how your system is doing and how you have liked your Lian D-8000 case since getting it all going back in January? I am considering getting the same case.