Build for Backup repo sanity check

[FPBrian]

Hello,

I have inherited a backup system at my Firm that has outgrown our available storage. Long story short, I have an HP DL380P G8 with lots of 2.5" slots and I was thinking of populating those slots with ~18 2TB WD Red SSD drives. The intention is having 32TB of available storage on the one 'server' device. The use is simple local backup storage. Locals are taken to here, and fed to the cloud from here. The big questions are:

Do WD Reds work with the HP DL380p with P420i controller reliably?

Is 18 x 2TB drives in RAID 6 unwise (# of drives in the array etc.) or..? My largest VM is ~10TB in size and I need to handle that plus allow room for dailies, working room for synthetic fulls, and of course, the rest of the infrastructure I am backing up. So I need 32TB free space. Also, I went 2TB over bigger thinking of rebuild times should a failure occur but let me know if that thinking has issues.

Does TrueNAs work on this server well?

I have seen some older posts on places that talk about heat sensing and the fans running fast when using older HP servers but that seems to be only with earlier SSD drives. Newer SSD firmware seems to address this issue.

I have seen posts about the controller IOPS being a limiting factor in some HP controllers of this generation but that seems not to be the case for the P420i family.

I cannot find any post that says "Yes, I have run SSD drives in my DL380P with the P420i controller for years with no issues" so if you have been, speak up! :)

This build doesn't necessarily need to be running TrueNAS as it is a simple VEEAM repo. So I can do TrueNAs and an NFS/SMB share or build it Windows and have it be one big drive. I honestly don't care. It will never be used for any other purpose than local backup repo. I favor TrueNAS though as it allows more configuration options going forward.

I am open to other suggestions re SSD drives to use. Reds just seem to be what WD suggests be used in this application.

DL380p G8:

2xXeon ES-2650 v2 @ 2.6Ghz
96GB RAM
2x 10GB Ethernet
HP Smart Array P4201 Controller
Current boot drives: 2 x 146GB 10K HP spinners, RAID 1

Thanks in advance for any and all opinions and assistance!!

BPBrian

 

[jgreco]

Do WD Reds work with the HP DL380p with P420i controller reliably?

I'm sure they do, but you've asked the wrong question.

Does TrueNAs work on this server well?

Fortunately, this approaches the correct question, which should be something along the lines of "Is the P420i usable with TrueNAS", to which the answer is hella-no.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

The Windows solution may be the easier solution in the short term, simply due to the hardware support already existing and being a known quantity. There are recommendations on the forums for TrueNAS-compatible HBA's to replace the P420i with as well, some searching should yield results. I don't really track the HP hardware given their stance on firmware updates.

There is also specific advice floating around out there for Veeam. I don't recall what it is. Be aware that ZFS has two primary strengths -- one is sequential file storage with RAIDZ{1,2,3} and the other is mirror storage for random access data like VM block storage or databases. Picking RAIDZ when you really needed mirrors is typically a project killer, whereas picking mirrors where you might have picked RAIDZ results in you having less storage available (but lots of performance).

For block storage, please see

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

And note that you really do need some significant amount of free space overhead on the pool; if you want 32TB of usable space on the pool, you should shoot for maybe 48-64TB of pool space. This becomes difficult in a 24 bay chassis because 24 bays times 4TB is 96TB raw space, which is just barely 48TB of mirrored space.

 

[ChrisRJ]

Unless VEEAM has some very specific requirements for random I/O, why are you looking at SSDs for backup? If it is the case, given its age I would look at getting something else (similar age) but with LFF slots for disks and put 10+ TB HDDs in.

For additional background on TrueNAS and ZFS, please have a look the "Recommended readings" in my signature.

 

[jgreco]

why are you looking at SSDs for backup?

Because the system is inherited, presumably.

I would look at getting something else (similar age) but with LFF slots for disks and put 10+ TB HDDs in.

But that's a great point.

 

[FPBrian]

Hello, and thanks for engaging! I appreciate your thoughts on this!

For my choice of drives I am looking at it this way: I have ~23 usable slots in the server. I backup just under 20TB now and it will never get smaller (as always!). Growth is about 1TB per year. As noted, the role needs good working space and also growth room so I have targeted 32TB for the build starting usable space.

I chose the 2TB drive size because it gives me what I need while leaving a decent amount of 'growth slots' in the server should I need them. Drawback is number of drives needed (~18) in the array increases the chance of a failure.

Performance is also of concern. I don't have 12 hours to backup, I have about 6. This includes weekends so can I get a full 20TB back up in 6 hours using large 7.2K drives? The current backups go to 600GB 10K drives but on multiple units so jobs run concurrently. I wish to conglomerate to the one unit and am worried that concurrent backup jobs to the one repo might overwhelm 7.2K drives, esp if I have a small set of huge drives. FYI networking is 10Gb, sources are a mix of SSD and 10K drives.

The SSD and the 2TB size are also due to a thought about drive rebuild time. This isn't end-user facing 'production' but it is 'production' for IT. There is also no tolerance for missed backups. So array degradation must be minimized, with return to full service prioritized. So my thought is going with SSD drives that are 'big enough but not too big' even with using RAID 6.

If you wanna throw the flag on any of my reasoning, please do! Let's discuss: Am I underestimating the speed of the backup when using say, a 4TB or 6TB 7.2K RPM drive? Same for the rebuild time? I've been wrong before........ :)

Best,

BPBrian

 

[jgreco]

I have targeted 32TB for the build starting usable space.

Using what math?

24 bays with 2TB drives probably works out to one of the following designs:

4 RAIDZ2 vdevs of 6 drives (8TB per), which gives you 32TB of pool space, or
3 RAIDZ2 vdevs of 8 drives (12TB per), which gives you 36TB of pool space, or
2 RAIDZ2 vdevs of 12 drives (20TB per), which gives you 40TB of pool space, or
12 mirror vdevs of 2 drives (2TB per) for 24TB of pool space.

Now, within a pool, you still need to leave a certain amount of free space, such as the frequently quoted 80% full/20% free, though this can maybe be shrunk to 92% full/8% free or even a bit more with corresponding loss of performance.

I've selected RAIDZ2 here because in this case you've said

So array degradation must be minimized, with return to full service prioritized.

The paranoid among us run RAIDZ3 for important pools with hard to replace data, but you're trading off performance and raw disk space to get that "extra 9" (referring to the love of IT to work for 9's in 99.99blabla). I personally run mirrors for backup storage, because I see it as block storage, want it to be fast as possible, and damn the extra cost.

can I get a full 20TB back up in 6 hours using large 7.2K drives?

VEEAM offers a variety of options, full vs incremental, synthetic, etc., so it's really hard to say. The thing I'll tell you is that 7.2K drives are basically a fool's errand. They really only do something for you in an environment where you're being forced to seek alot, something that ZFS uses different tricks to mitigate. For example, having a ZFS pool with gobs of ARC+L2ARC can result in a server that rarely even tries to read from HDD. For writes, having large amounts of free space on the pool minimizes the amount of time ZFS has to work to find free space, so if you gave me a 10TB HDD that was 60% full or a 20TB HDD that was 30% full, the 20TB drive will effectively write much faster because ZFS tends to write its transaction groups contiguously (meaning also sequentially). For pools that are relatively empty, a HDD based pool can feel like it is made out of SSD. Check this out from our friends at Delphix.

The trick to HDD pool write performance is keeping a pool empty, or, put differently, using much larger hard drives (which has the same effect). As you get up towards 80%, your speeds drop due to the number of seeks incurred. Yes, a higher RPM HDD can still go somewhat faster, but I'd be looking at larger drives rather than faster spindle if I could.