BUILD 60TB home server (but will expand to more in the future)

[9ine]

Trying again:

Chassis:

As of right now I cannot decide on a chassis. I will be expanding to more drives in the future when my finances permit me to do so. I don't want to spend $2000 on a chassis but have the capacity to expand and fit in more drives while saving money. In short, I have read (https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/) about the airflow importance, cooking up the drives. I am looking at two options right now with high difference in price. Hoping to get more input from the people who have built high storage server (50TB+) and are still expanding.

1. Supermicro 3U Chassis - Black CSE-836BE16-R920B for $1183.33
2.
Antec Nineteen Hundred Component Cases Nineteen Hundred Red Black/Red for $179.99

I know this wasn't the order in the hardware recommendations thread created but I am going to list my CPU cooler:
Corsair Hydro Series H100i GTX High Performance Liquid CPU Cooler CW-9060021-WW for $109.99

Motherboard:

I am looking to buy:

a. One with ECC RAM only
b. Future expansion
c. IPMI included on the board
d. Intel NIC boards only.

SuperMicro X10DRD-I-B Dual LGA2011 Intel C612 DDR4 SATA3 V&2GbE eATX Server Motherboard for $404.72

CPU:

Intel Xeon E5 Hexa Core Processor 3.5 6 LGA 2011 (BX80644E51650V3) for $597.78

Memory:

Again, I am having hard time selecting a brand. Strictly no no to Kingston. Looking at:

1.
Crucial 32GB Kit (16GBx2) DDR4-2133 MT/s (PC4-2133) CL15 DR x4 ECC RDIMM Server Memory CT2K16G4RFD4213/CT2C16G4RFD4213
for $233.87
2.
Supermicro Certified MEM-DR432L-SL01-LR21 Samsung 32GB DDR4-2133 4Rx4 LP ECC LRDIMM Memory
http://www.amazon.com/Supermicro-Ce...49933664&sr=8-1&keywords=MEM-DR432L-SL01-LR21

Power Supply:

EVGA SuperNOVA 650 P2 80+ PLATINUM, 650W ECO Mode Fully Modular NVIDIA SLI and Crossfire Ready 10 Year Warranty Power Supply for $99.99

Boot Device:

I have already bought this one couple months ago so I am going to stick with it unless it isn't recommended.


Samsung 850 PRO 128GB 2.5-Inch SATA III Internal SSD (MZ-7KE128BW)

Storage:
10 WD 6TB Red

WD Red 6TB NAS Desktop Hard Disk Drive - Intellipower SATA 6 Gb/s 64MB Cache 3.5 Inch - WD60EFRX
for $2490

Total cost: $4212.33

Thanks!

 

[danb35]

Look at eBay for the SuperMicro rack chassis. They'll also typically include a power supply, and often dual (redundant) power supplies. I'd suggest the SC846 line, for 24 bays in 4U, or if you really want expansion space, look at the SC847 line, which gives you 36 bays in the same footprint. You should be able to find one of these with dual power supplies for under $500 IIRC.

I strongly doubt that a liquid-cooled CPU is necessary, or even useful, in this application. I'll bet that Supermicro has a recommended heatsink for your board/chassis combo, and it'll cost much less than $100 too. Looks like it's the SNK-P0050AP4 for a 4U application.

As to RAM, Crucial has better availability IME. If you use their website to pick the modules, they'll guarantee compatibility. Experience with them has been uniformly good AFAIK.

On the boot device, nothing wrong with the SSD, but most of the capacity will be wasted if you use it as your boot device.

 

[9ine]

Look at eBay for the SuperMicro rack chassis. They'll also typically include a power supply, and often dual (redundant) power supplies. I'd suggest the SC846 line, for 24 bays in 4U, or if you really want expansion space, look at the SC847 line, which gives you 36 bays in the same footprint. You should be able to find one of these with dual power supplies for under $500 IIRC.

So with your recommendation I don't need the EVGA Power Supply, correct? I will look into it.

I strongly doubt that a liquid-cooled CPU is necessary, or even useful, in this application. I'll bet that Supermicro has a recommended heatsink for your board/chassis combo, and it'll cost much less than $100 too. Looks like it's the SNK-P0050AP4 for a 4U application.

I've had great experience with Corsair H100i Performance Liquid CPU Cooling System and it is not too expensive considering it is going to last a while and keep the parts from frying up.

As to RAM, Crucial has better availability IME. If you use their website to pick the modules, they'll guarantee compatibility. Experience with them has been uniformly good AFAIK.

That's what I thought too. Will 32GB do justice to streaming over Gigabit LAN to multiple devices while seeding 5000+ torrents?
I am thinking of using Deluge as the torrent client because I can access/control it remotely from my desktop/laptop

On the boot device, nothing wrong with the SSD, but most of the capacity will be wasted if you use it as your boot device.

I know but I had bought it couple months ago and figured I will use it in my home server to install the OS on it.

Apart from all this, what is the best RAID configuration to not lose data? I am guessing you don't recommend hardware RAID.

 

[danb35]

Will 32GB do justice to streaming over Gigabit LAN to multiple devices while seeding 5000+ torrents?

I honestly don't know. After all, 5000 torrents is a lot. Your board should have plenty of room for more RAM, though, if needed.

I am guessing you don't recommend hardware RAID.

I dare say nobody here will recommend hardware RAID. Ten disks in RAIDZ2 will give you the capacity of 8 of them with two disks' worth of redundancy. Hardware-wise, your best configuration is a SAS HBA like the LSI 9211-8i or IBM M1015, coupled with a chassis that has a SAS2 expander backplane (e.g., the SuperMicro SC846E16). The single HBA will drive all 24 (or 36 in an 847 chassis) disks.

 

[9ine]

Online RAID calculator tells me I get 37.786TiB of usable space in my case with RAIDZ2. This RAIDZ2 is software install after the install the FreeNAS OS?

I honestly don't know. After all, 5000 torrents is a lot. Your board should have plenty of room for more RAM, though, if needed.

I will not load 5000 torrents all at once. So, I will have time to add more RAM while not burdening my finances (although they don't cost much)

I dare say nobody here will recommend hardware RAID. Ten disks in RAIDZ2 will give you the capacity of 8 of them with two disks' worth of redundancy.

That is good.

Hardware-wise, your best configuration is a SAS HBA like the LSI 9211-8i or IBM M1015, coupled with a chassis that has a SAS2 expander backplane (e.g., the SuperMicro SC846E16). The single HBA will drive all 24 (or 36 in an 847 chassis) disks.

I don't understand this one. Could you please expand?

 

[danb35]

This RAIDZ2 is software install after the install the FreeNAS OS?

No, there's nothing additional to install--RAIDZ is part of ZFS, the filesystem used by FreeNAS. See https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/ for ZFS basics.

I don't understand this one. Could you please expand?

https://forums.freenas.org/index.ph...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

 

[9ine]

https://forums.freenas.org/index.ph...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

Is it mandatory for my FreeNAS with RAIDZ2 setup?

 

[danb35]

No, but expander backplanes will let you use a single HBA (which would otherwise only handle 8 drives at most) to control lots more (I think the documented limit is 128 drives, but I might have that number wrong). Reduces costs and simplifies cabling.

 

[Ericloewe]

I've had great experience with Corsair H100i Performance Liquid CPU Cooling System and it is not too expensive considering it is going to last a while and keep the parts from frying up.

There's no place where you could even mount it in a Supermicro chassis.

 

[9ine]

There's no place where you could even mount it in a Supermicro chassis.

I thought it was a big enough chassis for fitting it in.

 

[Ericloewe]

I thought it was a big enough chassis for fitting it in.

There are literally no 120mm fan mount points, not to mention 240mm radiator mount points.

 

[9ine]

Stupid question: Is it possible to expand the future storage drives to a new case, say Antec 1900 just for the new HDDs, and connect that to the existing one?

 

[Ericloewe]

Stupid question: Is it possible to expand the future storage drives to a new case, say Antec 1900 just for the new HDDs, and connect that to the existing one?

Yes.

There are a few practical issues that need to be addressed:

• You're forced to use an SAS expander in the external chassis, since SATA signalling is limited to 1m (pretty tightly, too)
• There's an interesting new failure mode where only the disks lose power, which can lead to messy situations

Along with a few, more prosaic difficulties.

 

[9ine]

Yes.

There are a few practical issues that need to be addressed:

• You're forced to use an SAS expander in the external chassis, since SATA signalling is limited to 1m (pretty tightly, too)
• There's an interesting new failure mode where only the disks lose power, which can lead to messy situations

Along with a few, more prosaic difficulties.

I will be expanding in the future for sure. I am leaning towards the Antec 1900 case and moreover, it has room for Corsair H100i GTX 70.7 CFM Liquid CPU Cooler with 12 3.5HDDs and is still cheaper than Supermicro SC846/847. I am unable to trust the supermicro cpu cooler and since I am investing so much in this project, I don't want my drives to be fried and fail very early into the project.

Speculating:
What are my options of expanding without using the SuperMicro's SC 846/847 and using a Antec 1900 case (or any other case that accommodates Corsair H100i GTX 70.7 CFM Liquid CPU Cooler)?

 

[danb35]

I am unable to trust the supermicro cpu cooler

If you're not able to trust the cooler that Supermicro designed for that CPU/board/chassis combo, there are other issues going on, but I guess that's up to you.

 

[9ine]

Bear with me one more speculation:

What if I fill all the 36 slots in the SC 847. How do I expand after that?

 

[INCSlayer]

Bear with me one more speculation:

What if I fill all the 36 slots in the SC 847. How do I expand after that?

Yes.

There are a few practical issues that need to be addressed:

• You're forced to use an SAS expander in the external chassis, since SATA signalling is limited to 1m (pretty tightly, too)
• There's an interesting new failure mode where only the disks lose power, which can lead to messy situations

Along with a few, more prosaic difficulties.

that or you setup a complete second server however since that only needs to serve files it doesnt need very much cpu power

 

[danb35]

What if I fill all the 36 slots in the SC 847. How do I expand after that?

Get one of these: http://www.ebay.com/itm/New-Supermi...158711?hash=item27fb426bb7:g:sU8AAOSw34FVA6OZ

 

[jmccoy555]

What's the noise level of the SC847E16 like, anyone know??

Sent from my Passport using Tapatalk

 

[depasseg]

What if I fill all the 36 slots in the SC 847. How do I expand after that?

Either daisy chain another enclosure of of it, or add a new external HBA in your server and connect to that.