Large Backup Storage Appliance

[G1Chris]

I'm in the midst of designing a storage appliance to house client offsite backups which will replicate to one another bi-coastal (freenas replication). The files housed on the storage appliance would range in 10GB - 50GB in nature, typically no smaller and are 95% writes versus reads. There is likely a ingestion of 5 terabytes per day. There are no VM's connected to this storage its only for offsite backup targets and replication.

Here is the hardware I'm currently looking at:

Supermicro 5049P-E1CR45L
(1) Intel® Xeon® Gold 6128
(6) 32GB DDR4-2666 2Rx4 LP ECC REG DIMM - 192GB Total
(2) 120-200GB OS DISK
(42) 12TB SAS 7.2k
(2) 400-800GB SAS SSD - 10DWPD (SLOG - recommendations here)

Open to all suggestions on layout, currently not looking to use deduplication only compression. My only main requirement is around 300TB of usable storage space.

One of my design thoughts: Raid Z-1 w/5 Groups could be 7 disks with an 8th disk as parity.

Thanks for everyones input in advance.

 

[Chris Moore]

The guidance is not to use RAIDz1 with drives larger than 1TB. You need to be thinking RAIDz2 and you will need more drives with smaller vdevs.
At work, I have one 60 bay chassis and another one is being processed through purchasing. They are much better on the rack units consumed per TB of storage.
To hit the performance goal, 5TB per day... I have to do some math on this.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[G1Chris]

Im mentally struggeling with the loss of so many disks. Am I getting closer with this layout ideas:

- 7 vdevs w/ 6 drives ~ 294TB Formatted/Usable - 14 disk parity on 42 disks
- 6 vdevs w/ 7 drives ~300TB Formatted/Usable - 12 disk parity on 42 disks
- 5 vdevs w/ 8 drives ~ 294TB Formatted/Usable - 10 disk parity on 40 disks

I'm not as concerned with IOps as I'm with Throughput and Formatted footprint. Thanks again for responding.

 

[Chris Moore]

I am not sure that your math works out.

- 6 vdevs w/ 7 drives ~300TB Formatted/Usable - 12 disk parity on 42 disks

Your raw capacity (with 12TB drives) would be 456TiB and at RAIDz2 it would leave you with 294TiB of usable space, but you have to keep a buffer of free space because of the copy on write nature of the file system, so you would only be able to put 264Tib in that pool. With compression, you will be able to get more out of it, depending on the compression you use.

I'm not as concerned with IOps as I'm with Throughput and Formatted footprint. Thanks again for responding.

I have 265TB stored in my older server using 80 x 4TB drives in 10 vdevs of 8 drives each in RAIDz2 and setting the compression to gzip=9. The total storage is 263.9TiB, but after parity it is 180.9TiB usable. I was able to shove so much data into it by cranking the compression but it makes the CPU work hard.
The high compression slows the system down but with those newer 12TB drives you will probably get enough performance out of it to make 5TB a day. My older system is CPU bound because of the compression and won't quite do 5TB a day of transfer, but your spec is for a higher end CPU, so it may not be a problem.
I suggest doing some testing of different configurations before you make it live.

 

[Chris Moore]

PS. Is your data highly compressible? Mine is. If you are not going to get a lot of compression, you will need more drives. That, and the transfer speed you wanted 5TB a day, is the reason I was saying to step up to the 60 bay chassis.
We just put a second one of those on order.

 

[Chris Moore]

To try and clarify, at work I manage (among other systems) a FreeNAS server with 80, 4TB drives; another with 60, 6TB drives; and we just placed an order for another with 60, 12TB drives. I got it specked out for under $40k.

 

[G1Chris]

I have the funding to do a larger setup like your 60x 12TB option. What layout are you using and are you using any SSD for SLOG? Do you like the 6vdevs with 7 drives? Also after futher calculation we're ingesting about 1TB a night. I was way off there.

No our data isn't extremely compressible, Im wondering if for what we're doing ZFS is even the right solution. At the end of the day we need the ability to replicate data at a block-level between our datacenters. Figured I'd look at building our own ZFS, we have the skills internally to manage this rather then spend 1.5x on some product on the market.

 

[G1Chris]

What would ultimately be cool is if we could copy-on-write out to a second array in near-real-time but I doubt thats possible due to latency? California to Virginia is around 80ms Latency, we have a 1gig available.

 

[Chris Moore]

I will be happy to give more details later, when I am not on the road, but the new system is for a file database that has millions of small (kilobyte range) files and a lot of random IO. Quite different from your needs.
It will be configured as a pool of mirrors, 30 vdevs, to maximize IOPS and it will be equipped with SLOG and L2ARC.
I will run some numbers this evening and get back to you but I think it might be reasonable to do 6 vdevs of 10 drives.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[G1Chris]

Yeah your situation is totally different. we're really ingesting data that doesn't require a lot of random IO.

 

[G1Chris]

On another note I'm going to build a test/dev freenas box (has some spare hardware) w/dedupe and ingest some of these customer backups to see if there is any reasonable data reduction to be had with dedupe on. If its more than 1x i'd consider doing smaller Z-2 vdevs knowing we'll get the dedupe synergies (if that would be better). So as not to complicate this thread more what would you recommend be reasonable amount of RAM for 300TB :)

 

[twinscroll]

This stuff is fascinating...I'm just reading along...don't mind me.

 

[Chris Moore]

On another note I'm going to build a test/dev freenas box (has some spare hardware) w/dedupe and ingest some of these customer backups to see if there is any reasonable data reduction to be had with dedupe on. If its more than 1x i'd consider doing smaller Z-2 vdevs knowing we'll get the dedupe synergies

The thing about deduplication is that it requires a lot of RAM. This is not the technical answer, kind of a paraphrase to simplify, but the system keeps a table of the duplicates in memory so the more duplicates, the more RAM is needed. The guidance I have seen is that compression is more useful than deduplication unless you have a highly repetitive type of data where you know you will save a lot of room from the deduplication.

So as not to complicate this thread more what would you recommend be reasonable amount of RAM for 300TB

The 60 drive server we have at work is configured with 10 drives per vdev, 6 vdevs in RAIDz2. That gives us 327TiB of raw capacity and 249TiB of usable capacity, but with compression we are able to get over 265TB of data in there and the system is reporting 160 ish TB of free space.
This server is configured with 256GB of system memory and it works fairly well, but we are going to the new server to improve performance because this drive configuration does not give enough IOPS to make the database responsive and adding RAM didn't make a significant difference in our situation. Memory usage was only between 50% and 80% most of the time.

 

[Chris Moore]

What would ultimately be cool is if we could copy-on-write out to a second array in near-real-time but I doubt thats possible due to latency? California to Virginia is around 80ms Latency, we have a 1gig available.

I don't think you could get near real time, but I expect you could push hourly updates. Possibly even more often than that.

 

[Chris Moore]

What layout are you using and are you using any SSD for SLOG?

The new server will have PCIe SSD drives, one for SLOG and one for L2ARC, but we don't have any in the current server and that is definitely reducing the performance we are able to get out of the system. The existing server was ordered by a manager that had no understanding of how their own software worked.

 

[Chris Moore]

Do you like the 6vdevs with 7 drives? Also after futher calculation we're ingesting about 1TB a night. I was way off there.

If you use 6 vdevs of 10 drives, that gives you 12 parity drives, and even with a 20% reservation of free space, you will have 380TB of usable space.

 

[Stux]

I don't think you could get near real time, but I expect you could push hourly updates. Possibly even more often than that.

I think you could actually get away with per minute replications. If a replication is already running, afaik, the next replication won't begin until the previous completes. And after the first replication, they should complete pretty quickly as only the incoming data in the last minute will be replicated... it might fall behind... but it'd catch up eventually.

Of course, it might make more sense to run the replications in the off-hours.

The main thing is you can't bidrectionally replicate, rather you'd have a primary dataset on one server, say "east" and "west" on the other, and then you'd replicate "east" to the west server, and "west" to the east server.

As a large backup repository, I'd say the sweet spot is probbaly RaidZ2 with 8-10 drives per Vdev. I don't think SLOG would make much of a difference, unless you have a lot of sync writes when storing. And L2ARC would probably not matter much either, since you don't expect to be doing a lot of reading.

So, sounds like just a large amount of relatively wide vdevs with 20-25% parity usage.

Also, if you get the initial setup right, then you can add additional disk shelves as your usage grows.

You can use up to 90% of the space before things slow down critically, and for continuous sequential additions, that should be fairly performant... I would normally advocate capacity expansion planning when you get to 80% (and the warnings start) and before you get to 90% utilization.

 

[Chris Moore]

Figured I'd look at building our own ZFS, we have the skills internally to manage this rather then spend 1.5x on some product on the market.

It is the spending 1.5x that makes me crazy. The solution quote from Dell/EMC was $130k and was 10 rack units. The solution we decided on is $40k and only 4 rack units. I spent a lot of time figuring out what we needed and I had a tough time getting any vendors to quote the hardware I was asking for. The HP quote was for an HA pair of servers and over $150k. I get that they are trying to make a living, but we have a hard limit of how much we can spend to get the job done. The HP quote actually had the individual component prices broken down and they were asking over $1200 for each hard drive.
Sorry, a little bit of a rant. If you have the knowledge to put it together, you can make a perfectly good solution for much less.

Sent from my SAMSUNG-SGH-I537 using Tapatalk