Pool Layout for 14x HDD's?

[TXAG26]

Quick question regarding pool construction for 14x HDD's. I'm putting together a cheap back-up server using 14x old 2TB HDD's. I am deciding whether to set it up as follows:

Either:
2x 7 drive Raid-Z2 vdev's and then stripe the vdev's together for a total of 18.2 TB usable
or
set up one large 14 drive Raid-Z3 for a total of 20.0 TB usable.

This is only on a gigabit network and this machine will actually be turned off 95% of the time. Speed isn't an issue since once the initial backup completes, everything else will be either small weekly or monthly incremental backups of some networked machines and file shares.

 

[cyberjock]

I recommend against the wider setup. I'd do 2 vdevs.

 

[HoneyBadger]

With them being "old 2TB" if they've got a lot of miles on them, how about 2 7-drive RAIDZ3 for about 12TB total, then aggressively compress the backup sets to make up for the extra space lost to parity?

Depends on your own personal level of risk aversion of course. Two 7-drive RAIDZ2 is probably fine, but definitely don't do the single ultra wide vdev.

 

[TXAG26]

That sounds good, I like the idea of two striped x7 disk Raid-Z2 vdev's. Using 4 drives for parity is fine, but killing 6 is probably more than necessary since this is a secondary backup server. I might see about picking up two more on the cheap from eBay and bump them up to 8 drives in each vdev as that works well for raid-z2.

 

[TXAG26]

Update: I found two more HDD's bringing the total to 16x 2TB HDD's for the file backup server. Interestingly, ended up with 8x 5900rpm Seagate and 8x 5400rpm Samsung drives (8x RaidZ2 vDev for each).

Playing around with the 4 open 8087 ports on the Intel Expander card (hooked up via 2 8087 ports from an LSI based Supermicro S2308L card), I saw really strange transfer rates when two Seagate and two Samsung drives shared the same 8087 break-out cable (e.g P0 & P1 Seagate, P3 & P4 Samsung). After about a minute or two, network file transfers would go from 110MB/s down to zero. It appears that most of the writes for the Samsung disks were instead being cached in system ram, and when the ram would fill up, everything would grind to a halt while the cache was emptied off to the Samsung disks. Quite the head scratcher, but I'm confident this is what was occurring after tracking everything down. Separating the disks to two different 8087 ports completely made this issue go away. Everything is humming along just fine now. No smart errors on either drive and indicidually, using DD read/write tests, all drives test as expected.

I know that each disk is separately addressable using an expander, and in theory, it shouldn't matter if different brands of HDD's are on the same break-out cable from the same 8087 port, but that's what was observed. I even swapped out cables, changed drivebays, recreated the vdev, and nothing worked until moving the Seagate and Samsung drives to different 8087 breakout cables. Just an FYI if someone runs across a similar issue! YMMV.

 

[Arwen]

...
I know that each disk is separately addressable using an expander, and in theory, it shouldn't matter if different brands of HDD's are on the same break-out cable from the same 8087 port, but that's what was observed. I even swapped out cables, changed drivebays, recreated the vdev, and nothing worked until moving the Seagate and Samsung drives to different 8087 breakout cables. Just an FYI if someone runs across a similar issue! YMMV.

I wonder how expanders work with different speed devices?

You might check the specifications for the 2 models/vendors. If one was 3Gbps SATA II and the other
was 6Gbps SATA III, that would be one explanation. (Or worse, 1.5Gbps SATA I verses higher...) If it
does the top speed available for each drive, but has to change speeds for different drives models, the
retraining time may be what is biting you.

You know you've been in the computer industry too long, when you start grasping at straws and some
times your right :-(.

 

[TXAG26]

Good info, I was definitely drawing at straws and checking all sorts of stuff, including the interface speeds, but all of the drives are 3Gbps (SATA II). I really thought I had two or three failing drives, but luckily that doesn't appear to be the case. Everything is still running well. I've had mixed speed interface drives on this LSI based expander before (3 & 6Gbps), but they were all the same brand (Seagate).

 

[cyberjock]

I wonder how expanders work with different speed devices?

You might check the specifications for the 2 models/vendors. If one was 3Gbps SATA II and the other
was 6Gbps SATA III, that would be one explanation. (Or worse, 1.5Gbps SATA I verses higher...) If it
does the top speed available for each drive, but has to change speeds for different drives models, the
retraining time may be what is biting you.

You know you've been in the computer industry too long, when you start reaching for straws and some
times your right :-(.

Well, considering that SAS expanders aren't for SATA... the question itself is non-sequitur.

 

[TXAG26]

Well, considering that SAS expanders aren't for SATA... the question itself is non-sequitur.

I wonder if we should tell Intel and LSI that they're not??? :D

 

[Ericloewe]

They work with SATA drives attached, not SATA controllers.

 

[jgreco]

They work with SATA drives attached, not SATA controllers.

Interestingly enough if you read the product overview shown, it actually does say "SFF8087 SAS/SATA connectors for attaching up to 24 targets or initiators"

(((mutters something about wanting to beat head)))

 

[Ericloewe]

Interestingly enough if you read the product overview shown, it actually does say "SFF8087 SAS/SATA connectors for attaching up to 24 targets or initiators"

(((mutters something about wanting to beat head)))

*sigh*

I hate imprecise marketing.

 

[cyberjock]

Yep.. I was responding to the comment about SATA-I connecting to something higher. That's totally impossible since you'd never connect SATA-anything to a SAS expander. It won't work.