melp
Explorer
- Joined
- Apr 4, 2014
- Messages
- 55
I've been planning and saving for this build for several years now and all the pieces have just about fallen into place. I'm waiting for HDD prices to come down a bit more, but other than that, I'm about ready to do this thing. I've pored over this configuration for endless hours and I think I've got everything just about dialed in, but I would really appreciate some extra (expert) eyes and advice. I'll start with a basic list of components and then list my justifications for each choice down below.
That's about everything. I'm going to replace the fans in the chassis, but that's no biggie. I'll also get a cheap rack on craigslist and mount this and the UPS. Again, thoughts and feedback would be greatly appreciated!
- Chassis: SuperMicro SC846 (used from eBay)
- CPU: Intel Xeon E5-1630v3 (4/8 cores/threads @ 3.7GHz)
- Motherboard:
SuperMicro X10SRASuperMicro X10SRL-F - Memory: 64GB Crucial DDR4 in 4 sticks (ECC/Unbuffered)
- SAS Expander: IBM M1015
- Replacement ATX PSU:
Corsair HX1000i2x SuperMicro PWS-920P-SQ - Replacement Backplane: SuperMicro BPN-SAS2-846EL1
- Boot device:
8GB USB3.0 drive2x Intel 40GB 320 Series SSD - UPS: APC 1500VA (used from eBay)
- vdev1:
12x 8TB WD Red in RAID-Z38x 8TB WD Red in RAID-Z2 - vdev2:
12x 4TB WD Red in RAID-Z38x 8TB WD Red in RAID-Z2 - vdev3: 8x 4TB WD Red in RAID-Z2
Chassis: SuperMicro SC846 (used from eBay) -- Solid case, ~$250 on eBay, not much to say. I'll be swapping the PSUs, backplane, and fans.
CPU: Intel Xeon E5-1630v3 (4/8 cores/threads @ 3.7GHz) -- Highest single core clock speed in this family of Xeons. I'll mostly be serving up files over SMB/CIFS, so this will come in handy. I'll explain LGA 2011 vs. LGA 1151 below.
Motherboard:SuperMicro X10SRA SuperMicro X10SRL-F -- The basic LGA 2011 server board. LGA 1151 would have worked but the SC846 chassis doesn't take micro ATX boards and the full ATX versions of SuperMicro's LGA 1151 boards are like $500. LGA 2011 will also allow me to add more RAM (if I ever need it).
Memory: 64GB Crucial DDR4 in 4 sticks (ECC/Unbuffered) -- This will mostly be static storage, so 64GB should be plenty. As mentioned above, being on LGA 2011 will allow me to throw in another 4 sticks easily. I read the tales of Kingston, so I'll probably go with Crucial, but I'm open to other vendors. Of course I'll check memory/mobo compatibility, etc., before I purchase.
SAS Expander: IBM M1015 -- Flashed to IT mode, of course. I'll only need one for all 24 drives as I'll be connecting the M1015 to the 846EL1 backplane which has its own SAS expander chip. As I understand it, I'll effectively have a 24Gbit/sec link which will be shared between all 24 drives, giving them 1Gbit/sec each. Considering I'll be connecting over 1Gbit/sec ethernet, this will be plenty.
Replacement ATX PSU:Corsair HX1000i 2x SuperMicro PWS-920P-SQ -- The PSUs that come in the SC846 are apparently very loud. I want something quiet, but I was advised I should stick with redundant PSUs, so I'll get pair of PWS-920P-SQs instead.
Replacement Backplane: SuperMicro BPN-SAS2-846EL1 -- All the SC846's I've seen on eBay come with an older backplane (BPN-SAS-846EL1) that doesn't have SAS2 support. Without SAS2, apparently the maximum capacity of the array would be limited and/or it would only recognize drives up to 2 or 3TB; I'm not totally clear on all this. The solution is this SAS2-capable backplane. Another possible option is the BPN-SAS-846A backplane which appears to be a SAS breakout cable baked into a PCB. It doesn't have its own SAS expander chip so it gets around the above SAS/SAS2 limitation. I would however need three of the M1015's, so I'm not sure if there would be any advantage in this option. The 846A backplanes are also like $450 on eBay. Doesn't seem worth it to me
Boot device:8GB USB3.0 drive 2x Intel 40GB 320 Series SSD -- I was advised to go with SSDs over USB drives for reliability.
UPS: APC 1500VA (used from eBay) -- Cheap on eBay, does NUT place nice with APC UPSs? Or do I have to get the APC drivers loaded?
vdev1 & 2:12x 8TB WD Red in RAID-Z3 8x 8TB WD Red in RAID-Z2 -- As soon as these hit $300, I'm ready to go. I was talked out of doing a 12-wide vdev, so I'll go with 8-wide and to Z2.
vdev3: 8x 4TB WD Red in RAID-Z2 -- I already have 8x 4TB WD Red drives, so I'll just use those for the final vdev.
CPU: Intel Xeon E5-1630v3 (4/8 cores/threads @ 3.7GHz) -- Highest single core clock speed in this family of Xeons. I'll mostly be serving up files over SMB/CIFS, so this will come in handy. I'll explain LGA 2011 vs. LGA 1151 below.
Motherboard:
Memory: 64GB Crucial DDR4 in 4 sticks (ECC/Unbuffered) -- This will mostly be static storage, so 64GB should be plenty. As mentioned above, being on LGA 2011 will allow me to throw in another 4 sticks easily. I read the tales of Kingston, so I'll probably go with Crucial, but I'm open to other vendors. Of course I'll check memory/mobo compatibility, etc., before I purchase.
SAS Expander: IBM M1015 -- Flashed to IT mode, of course. I'll only need one for all 24 drives as I'll be connecting the M1015 to the 846EL1 backplane which has its own SAS expander chip. As I understand it, I'll effectively have a 24Gbit/sec link which will be shared between all 24 drives, giving them 1Gbit/sec each. Considering I'll be connecting over 1Gbit/sec ethernet, this will be plenty.
Replacement ATX PSU:
Replacement Backplane: SuperMicro BPN-SAS2-846EL1 -- All the SC846's I've seen on eBay come with an older backplane (BPN-SAS-846EL1) that doesn't have SAS2 support. Without SAS2, apparently the maximum capacity of the array would be limited and/or it would only recognize drives up to 2 or 3TB; I'm not totally clear on all this. The solution is this SAS2-capable backplane. Another possible option is the BPN-SAS-846A backplane which appears to be a SAS breakout cable baked into a PCB. It doesn't have its own SAS expander chip so it gets around the above SAS/SAS2 limitation. I would however need three of the M1015's, so I'm not sure if there would be any advantage in this option. The 846A backplanes are also like $450 on eBay. Doesn't seem worth it to me
Boot device:
UPS: APC 1500VA (used from eBay) -- Cheap on eBay, does NUT place nice with APC UPSs? Or do I have to get the APC drivers loaded?
vdev1 & 2:
vdev3: 8x 4TB WD Red in RAID-Z2 -- I already have 8x 4TB WD Red drives, so I'll just use those for the final vdev.
That's about everything. I'm going to replace the fans in the chassis, but that's no biggie. I'll also get a cheap rack on craigslist and mount this and the UPS. Again, thoughts and feedback would be greatly appreciated!
Last edited: