40 TB storage with commodity hardware

[tyjgHoVeqqNeAJUr]

Hello,

I have a machine with 6 SATA ports. How can I achieve 40 TB of storage?

a) Wait for 7 TB SATA hard drives to come out. Buy 6 and plug one into each SATA port

b) Buy 6 4 TB SATA HDD and plug them into your SATA ports. Then buy 4 external USB HDDs and plug them into your USB ports. Tell FreeNAS to create a pool with those 6 internal and 4 external HDDs

c) Install a PCI SATA card with 4 ports. Buy 10 4-TB HDD. Plug them all in and create a pool.

d) [your idea here]

Thank you,

Chris

 

[david kennedy]

Do you care about performance?

If not you can look at some sort of "sas expander" route (your bandwidth to the disks is obviously multiplexed).

Depending on your use case this might work.

 

[diehard]

This seems like a very odd request. If you have the money for ten 4tb drives.. buy a proper enclosure. Seriously you can get older HP or Dell servers on ebay for extremely cheap.

Also note that if you want any sort of redundancy and 40tb of actual storage it will be more than 10 drives.

 

[Michael Wulff Nielsen]

40 TB of storage would mean somewhere between 13 and 16 4TB drives. That means special enclosures and server grade components. I would say a nice SuperMicro board that can take more than 32GB of ram and some IBM M1015 SATA PCI-E controllers.

You're not looking at cheap and cheerful with those storage requirements, and at that amount of storage I would suggest the best ecc ram money and psu that money can buy.

 

[tyjgHoVeqqNeAJUr]

buy a proper enclosure

Thank you for your replies everyone.

Please forgive my ignorance, but what is a "proper enclosure"? For that matter, what is an "enclosure"? I am currently running FreeNAS on a home-built computer with an ASUS mobo sporting 6 SATA connectors... so that is the only "enclosure" that I am familiar with.

If possible, please provide a link to dell.com or ebay.com that provides an example of a "proper enclosure". I assume you are talking about a rack-mount gizmo that Dell lists under the "server/storage" section.

Thank you very much,

Chris

 

[Michael Wulff Nielsen]

All right, lets assume that you go with a raidz3. That means that you need to place 13 disks inside the case and provide adequate cooling.

Something like this supermicro case could potentially hold all the drives and provide enough power for the system. I seriously doubt that you can find a standard tower that will take 13 harddrives.

Then you would neeed an appropriate motherboard, ecc ram, a xeon cpu and some ibm m1015 sata controllers.

What on earth are you doing that you need 40 TB of space?

 

[tyjgHoVeqqNeAJUr]

What on earth are you doing that you need 40 TB of space?

Michael,

I am post processing radar data. My team wants to spend a lot of money on some COTS product but I figured I can do the same thing with FreeNAS, as I do for my other data. We are not opposed to spending money and I did not intend for "commodity hardware" to be the focus of the thread (despite the subject... I should have used a different subject line).

Please correct me where I am wrong:

The chassis you linked is basically a case (and PS) that has a lot of drive bays. I will still need to purchase a mobo (and CPU/memory) to fill the case. I will need to make sure to buy a mobo that has 16 SATA ports if I want to make use of all 16 drive bays (not sure what is a 6Gb/s SAS2 Expander... looking it up now).

Thank you again,

Chris

 

[Michael Wulff Nielsen]

Right, your subject line did throw me there. We see a lot of people on these forums that show up with desktop hardware and expect server-grade performance. Honestly if you are not experienced in building server-grade hardware it is quite a jungle.

You might do well to hire CyberJock as a consultant for a few hours and let him put together a system specification for you.

But no motherboard has 16 sata ports, so you will need some sata controllers.

 

[diehard]

The motherboard won't need 16 SATA ports, or really any at all.. the M1015 he mentioned is a SAS PCI-E Controller that will connect via a single cable (or two for redundancy) into the backplane in the supermicro chassis.

But no motherboard has 16 sata ports, so you will need some sata controllers.

Believe it or not ASROCK makes one with 22 pors.

 

[Michael Wulff Nielsen]

Cool, didn't a board like that existed. But I am gonna exit this discussion as we are getting into serious server hardware and that is really not my speciality. :)

 

[jgreco]

It isn't that hard to do. 12 drive 2U chassis, socket 1150 board, 12 4TB drives in RAIDZ2. Done.

 

[Michael Wulff Nielsen]

@jgreco: with 12 disk in the pool would 2 spares be considered adequate?

 

[jgreco]

For postprocessing radar data? Assuming that long term retention isn't the goal, quite reasonable. RAID6 arrays are built that way often enough.