Enterprise Configuration

[Xylex]

I work for a server hosting, co-location and ISP company. We were recently requested a Large scale nas storage system that can start as small as 50TB and grow as they needed storage to well over 200TB.

My question... In a zfs array of 16 Disks of 4TBs, what would the best setup be.
Factors:
Data loss is NOT an option.
Speed is not critical, but It cant be slow either.

Any sugestions?

 

[Stephens]

Read cyberjock's guide (linked to in his signature).

Data loss is always an option unless you're backing up the data somewhere. RAID isn't backup. A power supply can explode and take out your whole array. There are several things you need to consider. Here are some:

- If you have to replace a disk, how long do you want to run with reduced functionality? If you have 4TB discs, reaplacing a disk (resilvering) is going to take a LONG time. The bigger the vdev, the longer it'll take. For that reason, folks commonly try to keep vdevs down to 6 drives.

- Efficient RAID-Z2 (can lose up to 2 drives and still have your data) layouts are 4, 6, and 10 drives. But be aware of the aforementioned rebuild times if you lose a drive in a 10-drive vdev.

- Losing more drives than your vdev can tolerate results in all data in the POOL being lost, not just that vdev.

How'd you settle on 16 drives? Is 18 an option? (3@6x4TB in RAID-Z2) You can also do 2@6x4TB + 1@4x4TB in RAID-Z2.

Remember when calculating available storage that to covert the number of data drives you'll have (minus the parity drives) to TiB by multiplying by 0.90. So a 4TB drive becomes 3.6TB, minus more space for ZFS formatting and overhead (swap space).

 

[cyberjock]

Don't forget you can do RAIDZ3 too. If data loss isn't an option then you'll need a second independent server with backups, ideally using snapshots.

By the way, when you start talking about servers of the size you are describing, you are talking BIG money. Backblaze has an "open source" case they nicknamed a "StoragePod" that can hold up to 45 drives. The "StoragePod" parts without hard drives cost about 5k and are all COTS parts.

So you are talking big money for this.

 

[Xylex]

Backblaze has an "open source" case they nicknamed a "StoragePod" that can hold up to 45 drives. The "StoragePod" parts without hard drives cost about 5k and are all COTS parts.
So you are talking big money for this.

I saw that project, and I did significant research on it. About the only thing really special about it is the custom case.
Currently, I have a setup that will be about $2500 without drives that can go to 144TB (Without parity)

Don't forget you can do RAIDZ3 too. If data loss isn't an option then you'll need a second independent server with backups, ideally using snapshots.

Ive thought hard about a RAIDZ3 and we might go in that direction.

Now to be clear about what I need to know.
Lets say I have one server, with 32 Drives. Would it be best to create 1 RAIDZ2 volume? 4 RAIDZ2 volumes?

I have a room mate who works with a large data storage company. He advises that a volume should not be any larger than 6-12 Drives.
as the more drives you have in any RAID environment, the higher your chances are in loosing files.

I am asking for a second opinion on this.

Final question. In what circumstances can a Power supply explode?
How can it be prevented?

 

[Stephens]

Lets say I have one server, with 32 Drives. Would it be best to create 1 RAIDZ2 volume? 4 RAIDZ2 volumes?

4. Make sure you understand why.

I have a room mate who works with a large data storage company. He advises that a volume should not be any larger than 6-12 Drives.

He's correct. In the case of RAIDZ2, it's 6 or 10. In the case of RAIDZ1, it'd be 7 or 11.

as the more drives you have in any RAID environment, the higher your chances are in loosing files.

I don't know that I'd say it that way. I'm not saying that isn't true, just that it wouldn't be the overriding factor for me. Rather, the fact that if I lose a drive it'll take forever+a day to get the pool back to a healthy state makes it a no-brainer to use a manageable vdev size.

Final question. In what circumstances can a Power supply explode? How can it be prevented?

Lightning. I assume you have a UPS. That may help, but it may not. Some people get bottom end UPS's and are surprised that they don't create cyber utopia. I believe cyberjock has some lightning stories posted to this forum. Anyway, you can also get a faulty power supply. It's an imperfect world. I couldn't in good conscience tell you how to better pretend it isn't. I'd rather say accept that it's an imperfect world and plan accordingly. Stuff breaks.

 

[gpsguy]

So, I've been watching the 2 threads that you posted recently and wanted to look at it from 30,000' up.

Who will support this 50-200+TB "Enterprise" solution running FreeNAS? It sounds like you're on the verge of loosing some data (in the other thread) and don't have backups for it. If you're the only one in your company who can support it, then I'd probably walk away from the deal - unless you want to consider a real "enterprise solution". As cyberjock said, you're talking "BIG money". And, the Backblaze solution that he mentioned is probably peanuts (@11k+) for an enterprise solution. IIRC, they were using consumer drives (buy cheap external drives from Costco [black Friday?] and send them to us), and their NOS runs in 8Gb RAM, with JFS. Data is only available via https.

I *think* you'd need to spend "VERY BIG money". Though I'm not an enterprise storage architect, I would look at a solution that offered at least 2 controllers and 2 storage arrays all interconnected for redundancy on the primary system and replicated the data to an offsite location.

You mention that your roommate works with a large data storage company. Does he/she think that FreeNAS is a good solution for your customer?

I work for a server hosting, co-location and ISP company. We were recently requested a Large scale nas storage system that can start as small as 50TB and grow ...
...
I have been using ZFS for quite some time now. I understand the simple zpool commands and how to scrub and manually mount/import a pool. I cannot loose this data, any suggestions?

 

[louisk]

How much RAM are you allocating for this server? Keep in mind that best practice says 1G per 1T, if you want dedupe, 5G per 1T.

I would start by backing up and figuring out the requirement for this storage solution.

Does it require redundant controllers?
What are the IOPS required? How does this effect the type of media that will be chosen?
What level of redundancy is required for the data?
How do the hosts need to be connected to it? What throughput is required (how is this different from IOPS required)? What happens if a host loses its connection with the storage?
How many people will be on call to support this?
How will you be backing up this data? Does this effect the ways that hosts need to connect to it? Can you actually backup 200T in the necessary time window?
How will you be managing the storage over time? Will you have people trained on how to replace disks? Will they know which disk ad8 is?
How will the number of interrupts from all the drives effect the system? Will they all be handled by HBAs? Will they be handles by software? What does this mean?

There may be more questions that are appropriate for your setup. This is what I thought of off the top of my head. I would not write up a proposal w/o knowing the answers to all of these questions.

 

[survive]

Hi Xylex,

I second the advice given by gpsguy & louisk. First thing you need to do is define your problem much more clearly & in more detail. First thing I would tell you is don't get stuck on just the server itself.

I'm assuming this is a "bet your business" type of project and when your boss gave it to you it became a "bet your job" kind of situation for you. Best thing you can do is define the scope of the project, consider the risks and present the boss with some solutions. There are plenty of experienced datacenter guys who come through here who would be happy to help you come up with your worst nightmare situations and offer plenty of advice on how to capture them & present it to management

I certainly wouldn't say FreeNAS wouldn't be a fine solution to the storage problem but i would encourage you to contact iX Systems and get a proper TrueNAS solution & quote as well. I would even go so far as to contact some of the big storage vendors like EMC & Netapp and get a price on a solution from them if for no other reason that to give some idea of the scale of what some big brand gear costs.

-Will