Supermicro SC846 storage server build

[AlphaG]

I recently bought and modified to some degree a used Supermicro server from theBay. I wanted to post the setup here and hopefully get some guidance on the way forward. The system will be used for whole house AV media serving via a ReQuest media server (bluray rips and audio), as a Time Machine depot, and possibly for other miscellaneous household storage needs as they arise.

I have a 4 miscellaneous drives up to 2Tb that have been removed from a Mac Pro that has been upgraded but got more serious and picked up a total of 7 WD 3Tb Red's over the course of several months. In the meantime I tried and bought FlexRAID, tried the free Unraid trial (also bought a pro key but have not used it), and of course am now settling on the free and more robust FreeNAS and have been experimenting with it over the past few weeks.

The current setup:

Chassis: Supermicro SC846 -- 24 bays
Motherboard: Supermicro H8DME-2
CPU's: AMD Opteron Hex Core 2419 @ 1.8GHz
RAM: 64GB using 16x4GB ECC DIMMs
IPMI Card: Kira 100
HD Controllers: 3xSAT2-MV8 PCI-X
LAN: 1 of 2 on-board nVidia Gb ports on a Gb LAN

Two initial questions I have are about RAM and the HD controller.

The RAM I only recently upgraded because I was able to find one of the registered Micron DIMMs being sold on eBay for $10 each. The machine came with 4 1Gb sticks. When I populate the last four memory slots going from 48 to 64 Gb, the motherboard slows the memory down from DDR400 to DDR266. I'm not sure the slowdown is worth it but wanted to ask here before cutting the RAM back to 48Gb.

The HD controller supports 3TB drives but for the future will probably not support larger drives that come available. I picked 3Tb as they are currently in the sweet spot at lowest cost per Tb when pricing out through Amazon. But ... As I look to the future I want to be able to add to the drive pool with new vdev's made of larger drives or swap out the 3's when they need to be upsized and resilver. To that end I have been looking at other controllers that will support the larger drives. One such controller that seems to come up is the M1015 but I only have two PCIEx8 slots and would need three of those controllers. Another that would be neater and cheaper is a single LSI 3ware 9650SE-M24 card. Would there be a significant hit in performance when scaling up to 24 drives on a single PCIEx8 lane card?

 

[Ericloewe]

I recently bought and modified to some degree a used Supermicro server from theBay. I wanted to post the setup here and hopefully get some guidance on the way forward. The system will be used for whole house AV media serving via a ReQuest media server (bluray rips and audio), as a Time Machine depot, and possibly for other miscellaneous household storage needs as they arise.

I have a 4 miscellaneous drives up to 2Tb that have been removed from a Mac Pro that has been upgraded but got more serious and picked up a total of 7 WD 3Tb Red's over the course of several months. In the meantime I tried and bought FlexRAID, tried the free Unraid trial (also bought a pro key but have not used it), and of course am now settling on the free and more robust FreeNAS and have been experimenting with it over the past few weeks.

The current setup:

Chassis: Supermicro SC846 -- 24 bays
Motherboard: Supermicro H8DME-2
CPU's: AMD Opteron Hex Core 2419 @ 1.8GHz
RAM: 64GB using 16x4GB ECC DIMMs
IPMI Card: Kira 100
HD Controllers: 3xSAT2-MV8 PCI-X
LAN: 1 of 2 on-board nVidia Gb ports on a Gb LAN

Two initial questions I have are about RAM and the HD controller.

The RAM I only recently upgraded because I was able to find one of the registered Micron DIMMs being sold on eBay for $10 each. The machine came with 4 1Gb sticks. When I populate the last four memory slots going from 48 to 64 Gb, the motherboard slows the memory down from DDR400 to DDR266. I'm not sure the slowdown is worth it but wanted to ask here before cutting the RAM back to 48Gb.

The HD controller supports 3TB drives but for the future will probably not support larger drives that come available. I picked 3Tb as they are currently in the sweet spot at lowest cost per Tb when pricing out through Amazon. But ... As I look to the future I want to be able to add to the drive pool with new vdev's made of larger drives or swap out the 3's when they need to be upsized and resilver. To that end I have been looking at other controllers that will support the larger drives. One such controller that seems to come up is the M1015 but I only have two PCIEx8 slots and would need three of those controllers. Another that would be neater and cheaper is a single LSI 3ware 9650SE-M24 card. Would there be a significant hit in performance when scaling up to 24 drives on a single PCIEx8 lane card?

You can use a single card plus an expander. Performance with mechanical disks shouldn't be affected.

 

[AlphaG]

So something like this:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207
... Along with an M1015?

A 24 port 9650SE can be had on eBay for half the cost though....

 

[Ericloewe]

So something like this:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207
... Along with an M1015?

Exactly.

A 24 port 9650SE can be had on eBay for half the cost though....

...and it's ancient, not known to work well (if it even works) and it's a RAID controller with no guarantee that it will pass anything through. It's your money, but if something goes wrong, the first piece of advice you'd get would probably be "get a proper HBA".

 

[mjws00]

The 9650se doesn't have a pure IT mode. You can make jbod work, but it isn't great. I know @cyberjock had one and punted it. I think primary concern is lack of SMART... maybe he will elaborate. I have one as well, but haven't even bothered with using it on FreeNAS.

 

[jgreco]

I don't believe the 9650 supports HBA style use. The 9550 doesn't for sure. You really want something that can monitor the health of your drives. On FreeNAS, that's generally going to be S.M.A.R.T.

The AOC-SAT2-MV8 is based on the Marvell 88SX6081 and as far as I know doesn't have a 2TB limit, which implies that it probably works with 4TB or 6TB drives. That said, it could be slowish.

The RES2SV240 is fairly well-respected and would be a fine way to connect your drives.

 

[AlphaG]

Ok ... Enough said. M1015 with expander seems to be the more solid and better supported way to go.

How about the RAM? More RAM at a slower speed or a little less RAM running slightly faster?

I have chosen not to use ZIL and L2ARC to keep things simple and I think the upgrade in RAM at either 48 or 64Gb is the way to go instead. That's not my thought actually, I learned that here from you guys. I actually have two spare Intel X25 80Gb drives that I was considering using for ZIL but not bought for that purpose. I was using them as Windows boot disks when running FlexRAID.

 

[jgreco]

Gobs of slightly slower RAM will cache more. If you are network limited to 1Gbps you are much better off with the RAM at least for general purpose fileserver use.

 

[AlphaG]

Will I be able to seamlessly change the controllers out to the M1015 + expander topology? Shouldn't be an issue if I already have FreeNAS configured and running right? As the drives will be JBOD there will be no migration involved when I do decide to make the upgrade later right? I will probably put that off for now until or if I need better performance.

 

[cyberjock]

The 9650se doesn't have a pure IT mode. You can make jbod work, but it isn't great. I know @cyberjock had one and punted it. I think primary concern is lack of SMART... maybe he will elaborate. I have one as well, but haven't even bothered with using it on FreeNAS.

That was precisely it. There's a hack that can make it work in something like IT mode, but still doesn't do SMART. Not a good card if your data is important. ;)