Debating a 5 disk raidz2

[joerawr]

Why would I consider a 5 disk raidz2 (5x3TB WD Reds)? Only one reason really, my mobo only has 6 sata ports. So I can build a 5 disk raidz2, and have one port left over to copy the data back to the pool from my backup. And I can use the 6th sata port for zfs replicating snapshots of the pool to a pair of very large drive I will rotate offsite. I keep reading that with lz4 enabled, the "non-optimal" sizing of a odd numbered raidz2 goes away. But I haven't successfully googled why this is. So I'm trying to figure out if I am missing something with 5 disks. I've seen no issues with 5 20GB disks in my 9.2.1.7 VM. But that's not the real world.

My purposes are a very lightly used home nas. Backups of the wife's laptop, camera video and picture files, occasionally streaming video or music, and storing vmdks that I'd copy elsewhere before booting. 99% compressed data.

Current System (over 2 years old):
Fujitsu MX130-S2
AMD 610e Athlon II (quad 2.4GHz)
8GB ECC Crucial memory
6xsata ports on mobo
FreeNAS 9.2.1.7
basic gigbit consumer wired and AC wireless networking
2x3TB WD Reds Mirror.

After upgrade I'd have:
5x3TB Reds RaidZ2
16GB ECC Crucial memory (maxed out)

Performance wise, my benchmark is the 2 disk mirror, so I'm pretty sure I'll be happy with the bump I suspect the 5disk raidz2 will provide.

The alternative that I can see, is getting a 2 or 4 port sata card for some additional dollars, and a 6th 3TB drive. But I won't need THAT much space for a long time. I think when I need more space than 3 x3TBs (~6TB usable @ under 80% utilized) I'll be wanting to upgrade the whole box to something with a CPU that can transcode plex (the 610e can NOT), and maxes out at 32GB or more memory. So I'm not a fan of the extra sata card (cheap ones are cheap, good ones are more than a HD), and the extra drive (rather have it as a cold stand by anyways). And with just 5 drives I think it's more K.I.S.S.

However... The interwebs really really really recommend a 6 disk RaidZ2. But then most of those recommendations are based on pre- "compression is on by default" zfs. I've read on these forums that when 9.3 comes out, it will drop the "non-optimal" warning when choosing 5 disks in a RaidZ2. And CyberJock has mentioned several times that with compression the optimal/non-optimal math goes out the window (very rough paraphrase ;-) ).

So what you say?

 

[cyberjock]

I'm personally still doing the "optimal" until there is more hard evidence. I just created my new pool and it's a 10 disk RAIDZ2. Personally I'd recommend you go with a 6 disk RAIDZ2 and just copy the data back to the server over the network. Yes, it will be limited to Gb LAN, but it's not the end of the world. It's a 1-time thing.

 

[9C1 Newbee]

I'm curious about how a non-optimal configuration would run.

I just created my new pool and it's a 10 disk RAIDZ2.

I also just did this. Quite impressed how smoothly it went.

So far I have these configs under my belt:

3 Disk RAIDz1
6 Disk RAIDz2
10 Disk RAIDz2

I haven't noticed any speed differences. Probably due to the lan speed limitation. Not to mention, all of my configs were "optimal" Curious if gigabit will continue to be the bottle neck with the 5 disk configuration.

I know the 6 disk works. I'd spring for a $100 3tb WD green. But hell, that's just me.

 

[joerawr]

Thanks guys.

I built the 6 disk RaidZ2. And I picked up a SYBA SI-PEX40064 to do the copying from the mirror set, allowing all 6 drives to use the internal sata ports. Data copied over at 80-100MB from each drive in the mirror to the RaidZ2, which ain't shabby.

Swapped the volume names, and all my jails, shares, and stuff looks like it's worky snazzy. I need to pick up two more reds or greens to swap out the two borrowed Constellations I used to do the rebuild.

Compression was completely unimpressive at 1.01. But I'm not surprised with my data being media, and compressed backups.

 

[9C1 Newbee]

My data ended up being x1.00 with lz4. Seems like it just adds another layer of complexity.

Never heard of a SYBA SI-PEX40064. The price sure seems right. Glad it worked/working out for ya.

 

[joerawr]

Syba and IO Crest are the same card, by the same manufacturer. I got the Syba from NewEgg same day via will call. Both use the Marvell 88SE9215 chip. Works in a PCI-E x1 slot, which is fine for me since I'll only be using it for backups.

IO Crest is on Amazon: View: http://www.amazon.com/dp/B00AZ9T3OU

Syba is on NewEgg: http://www.newegg.com/Product/Product.aspx?Item=N82E16816124064

This guy's blog post turned me onto the card:
http://www.dslrfilmnoob.com/2014/02/07/freenas-pci-e-4-port-sata-upgrade-revisited/

Compression: the cool thing about compression is that you can turn it on and off on the fly. Let's say you have some data to archive on your NAS that should compress really well with Gzip, and you won't mind A) the copy taking longer, and B) the cpu hit won't impact you: Enable Gzip maximum and let it run. Assuming you really don't access that data often, you'll save disk space only for a time penalty on copy. Switch it back to lz4 or off when done with the archive. Same goes for copying uncompressable data. lz4 is really smart about bailing on attempting to compress uncompressable, but you can probably get a bit faster of a copy of uncompressable data by turning it off.

I knew I was only going to get 1.0x so I probably should have turned off compression for my zfs recv to the new raidz2. Probably would have run a bit faster, but didn't really bother me. I've had great experiences with lzjb on our Oracle ZFS at work, so I figured I'd just leave it at the "recommended" lz4 settings and not stress.

 

[cyberjock]

9C1 Newbee, all of those can probably do double what Gb can do. So yeah, definitely bottlenecked at the LAN port. But when doing things like ZFS replication more disks means MUCH higher transfer rates if doing 10Gb LAN.

 

[9C1 Newbee]

I didn't look past the price tag. I saw the SATA3 and just assumed it would be good. I know the M1015 is x16 vs x1 of the Syba. I assume the x1 is the bottleneck. Another thing I noticed is the Syba has half the ports as well. $$$/port is not such a huge bargain vs the M1015. However, if you needed to just "get by", or add a few drives on the cheap, this might just be the ticket in my mind. But it HAS to be dead on balls reliable.

Great information here! My search didn't yield too much info on the Syba.

 

[solarisguy]

Syba and IO Crest are the same card, by the same manufacturer. I got the Syba from NewEgg same day via will call. Both use the Marvell 88SE9215 chip. Works in a PCI-E x1 slot, which is fine for me since I'll only be using it for backups. [----]

I think it would only properly work in x1 slot. If you start using it in x4 or x16, you might get errors... I misplaced the bookmark to the technical detail, I will add it here when I find it.

P.S.
The errors would be like the ones mentioned here
http://forums.freenas.org/index.php?threads/smart-errors-but-all-smart-tests-pass.15620/