2x Mirror vDev vs 1x Mirror vDev

[ryanpr]

I'm receiving components today for my new build and need some drives (I'm short of space anyways) which I'll pickup from Micro Center at lunch today.

I'm looking at 6TB WD Reds. I have 2x 4TB drives (existing mirror). 2x 6TB would be sufficient. However, I'm considering adding the 2x 4TB back to the pool. The idea is that I'll get better performance. I know that sequential reads/writes can exceed the Gigabit network (I think the drives run around 115MB/s). But I'd imagine the smaller files (photos / computer backups) will be faster with two mirrors in the pool.

The downsides would be 4 disks running which can fail and take power to run.

I'm curious to know if what I should expect performance wise moving to 2 mirrors and if it's worth it? I want to figure this out today as if I move the data to the 2x 6TB mirror without the 2x 4TB mirror and add the 2x 4TB later, then the data will be almost entirely on the 6TB mirror so I won't get the performance boost, except for new data.

Thanks!
Ryan

 

[Arwen]

Keep in mind that if you have data on one Mirrored vDev, and then add a second Mirror vDev, the existing data won't magically move. You have to write the data as a striped Mirror to get the performance and data balancing of 2 vDevs. Further, when writing, ZFS will prefer vDevs with more free space.

You could create a new pool of 2 disks striped, 1 x 4TB and 1 x 6TB, then copy the data from the other 4TB to this new pool. Whence done, you can then add in the other disks as mirrors. But, I would not do this unless I had good backups.

If you truly want more redundancy, you can go with a 4 disk RAID-Z2. That way you can loose any 2 of the disks. But of course you only get the size of the smallest disk, 4TB. So the 2 x 6TB disks aren't fully utilized unless / until you replace both the 4TB disks later.

 

[ryanpr]

Yeah, I get the lack of balance created by adding later. I'm wondering, even if I can balance the mirrors, is it worth it? I'm having a hard time finding performance numbers for 1 v 2 mirrors.

Thanks,
Ryan

 

[Chris Moore]

I'm looking at 6TB WD Reds

I have some WD Red drives in my system... Here is a copy of part of one of the reports the system sends me on a daily basis:

Code:

+------+----------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
|Device|Serial		  |Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek  |Total	 |High  |Command|Last|
|	  |Number		  |	|On   |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks	 |Fly   |Timeout|Test|
|	  |				|	|Hours|Count|Count|	   |Sectors|Sectors |	  |		  |Writes|Count  |Age |
+------+----------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
|da18 ?|K1JX74		  | 45 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da21 ?|K1JX79		  | 44 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da20 ?|K1JX1M		  | 43 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da19 ?|K1JX7N		  | 42 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da37  |Z301V2		  | 29 | 1159|   11|	0|	  0|	  0|	   0|	 0|   3484222|	 5|	  0|   1|
|da36  |S3001L		  | 29 | 1435|   36|	0|	  0|	  0|	   0|	 0|   3552013|	 6|	  0|   1|
|da38  |Z301V2		  | 29 | 1159|   11|	0|	  0|	  0|	   0|	 0|   3568398|	 7|	  0|   1|
|da39  |Z301XC		  | 29 | 1176|   23|	0|	  0|	  0|	   0|	 0|   3565461|	 7|	  0|   1|

If you notice the number in the "Temp" column... The WD drives run significantly hotter than the Seagate drives. Unless you have some serious problem with Seagate, I suggest using them instead. Less heat to deal with.

 

[Chris Moore]

Yeah, I get the lack of balance created by adding later. I'm wondering, even if I can balance the mirrors, is it worth it? I'm having a hard time finding performance numbers for 1 v 2 mirrors.

Thanks,
Ryan

What kind of speed are you getting from your NAS now? Speaking of which, what hardware are you using? Going from one vdev to two, is like going from one drive to a RAID-0 set of drives. Theoretically, the performance should double, but it isn't perfect because there is some overhead. Also, if you have a 1Gb network, the network may cap your potential speed, however a pair of vdevs should be better at handling random read tasks than a single vdev.

 

[ryanpr]

If you notice the number in the "Temp" column... The WD drives run significantly hotter than the Seagate drives. Unless you have some serious problem with Seagate, I suggest using them instead. Less heat to deal with.

Wow, that's intense! In the past the recommendation that seemed to be consistent was WD Reds. However, definitely open to Seagates. My WD Reds actually ran right on par with how yours run. I'll go Seagates this go around. I assume this is the guy I want? http://www.microcenter.com/product/...rpm-sata-iii-6gb-s-35-internal-nas-hard-drive

What kind of speed are you getting from your NAS now? Speaking of which, what hardware are you using? Going from one vdev to two, is like going from one drive to a RAID-0 set of drives. Theoretically, the performance should double, but it isn't perfect because there is some overhead. Also, if you have a 1Gb network, the network may cap your potential speed, however a pair of vdevs should be better at handling random read tasks than a single vdev.

I'm presently on an x9scm-f with an e3-1230 v1. I'm moving to the same w/ an e3-1240 v2 (1230 was more $ on ebay). Performance isn't great presently, but I think it's because the existing motherboard has some issues. (build thread: https://forums.freenas.org/index.php?threads/buy-v-build.68962/). I fully expect the 1Gb network to cap sequential reads. I'm wondering if the extra mirror in the pool will help the small / random reads / writes which don't fill the network pipe.

FYI: My data is fully backed up, so I'm not concerned. If I need to degrade a vdev like Arwen suggested, I'm ok with that.

 

[Chris Moore]

I'm wondering if the extra mirror in the pool will help the small / random reads / writes which don't fill the network pipe.

It will improve some, but it will still churn a bit with small files. One of the pools in my NAS has eight mirrors (16 drives) striped together in an effort to get the speed up for small random file access and it is better but still not as much as I had hoped for. Small, random file access is the toughest thing ever. Just about the only thing that does it well is a massive number of drives or switch to SSD.
What kind of workload is it?

FYI: My data is fully backed up, so I'm not concerned. If I need to degrade a vdev like Arwen suggested, I'm ok with that.

The way @Arwen was suggesting that you upgrade the pool would be a way to get your data spread across both vdevs. A couple of the steps would have to be done from the command line because the GUI doesn't have a way to do it. The easy way might be to reconfigure the pool and then load the data back in from the backup. If you had a spare (5th) drive you could put in the syste that is large enough to hold all your data, you could do a zfs send | zfs receive to copy the data to the spare drive, then rebuild the pool through the GUI as two mirror vdevs, then do the zfs send | zfs receive to bring the data from the single drive back to the pool. The speed of this might be faster than any other kind of backup and restore.

 

[pro lamer]

The way @Arwen was suggesting that you upgrade the pool would be a way to get your data spread across both vdevs. A couple of the steps would have to be done from the command line because the GUI doesn't have a way to do it. The easy way might be to reconfigure the pool and then load the data back in from the backup. If you had a spare (5th) drive you could put in the syste that is large enough to hold all your data,

Or maybe just replace one of the 4TB drives with a 6TB, the mirrored vdev will not grow, then you have one 6TB and one 4TB HDD available to build a stripe...

Sent from my mobile phone

 

[Stux]

Keep in mind that if you have data on one Mirrored vDev, and then add a second Mirror vDev, the existing data won't magically move. You have to write the data as a striped Mirror to get the performance and data balancing of 2 vDevs. Further, when writing, ZFS will prefer vDevs with more free space.

You could create a new pool of 2 disks striped, 1 x 4TB and 1 x 6TB, then copy the data from the other 4TB to this new pool. Whence done, you can then add in the other disks as mirrors. But, I would not do this unless I had good backups.

If you truly want more redundancy, you can go with a 4 disk RAID-Z2. That way you can loose any 2 of the disks. But of course you only get the size of the smallest disk, 4TB. So the 2 x 6TB disks aren't fully utilized unless / until you replace both the 4TB disks later.

Interesting approach. Actually a good solution.

Or you could just not worry about it and add the drives as
Mirrros as you get them ;)

 

[Stux]

I have some WD Red drives in my system... Here is a copy of part of one of the reports the system sends me on a daily basis:

Code:

+------+----------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
|Device|Serial		  |Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek  |Total	 |High  |Command|Last|
|	  |Number		  |	|On   |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks	 |Fly   |Timeout|Test|
|	  |				|	|Hours|Count|Count|	   |Sectors|Sectors |	  |		  |Writes|Count  |Age |
+------+----------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+----+
|da18 ?|K1JX74		  | 45 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da21 ?|K1JX79		  | 44 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da20 ?|K1JX1M		  | 43 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da19 ?|K1JX7N		  | 42 |  456|	3|	0|	  0|	  0|	   0|   N/A|	   N/A|   N/A|	N/A|   2|
|da37  |Z301V2		  | 29 | 1159|   11|	0|	  0|	  0|	   0|	 0|   3484222|	 5|	  0|   1|
|da36  |S3001L		  | 29 | 1435|   36|	0|	  0|	  0|	   0|	 0|   3552013|	 6|	  0|   1|
|da38  |Z301V2		  | 29 | 1159|   11|	0|	  0|	  0|	   0|	 0|   3568398|	 7|	  0|   1|
|da39  |Z301XC		  | 29 | 1176|   23|	0|	  0|	  0|	   0|	 0|   3565461|	 7|	  0|   1|

If you notice the number in the "Temp" column... The WD drives run significantly hotter than the Seagate drives. Unless you have some serious problem with Seagate, I suggest using them instead. Less heat to deal with.

Are both drives the same rotational speed?

 

[Chris Moore]

Are both drives the same rotational speed?

If I recall correctly, the Seagate drives are 5900 RPM where the WD drives are 7200 RPM, so that accounts for part of the difference I am sure.

 

[Johnnie Black]

WD drives are 7200 RPM

Then they are RED PRO, REDs are 5400rpm

 

[ryanpr]

It will improve some, but it will still churn a bit with small files. One of the pools in my NAS has eight mirrors (16 drives) striped together in an effort to get the speed up for small random file access and it is better but still not as much as I had hoped for. Small, random file access is the toughest thing ever. Just about the only thing that does it well is a massive number of drives or switch to SSD.
What kind of workload is it?

... you could do a zfs send | zfs receive ...

Workload is a good point. For the primary data that I'm looking to speed up I could use 2x SSDs at a reasonable price (500GB would leave me plenty of room to grow). I'll see how the two mirrors work and keep this as an option.

For now I'd go w/ 2 mirrors as it'll allow me to get 4TB drives instead of 6TB drives which will save $120 for now and will still double my storage, bringing me from ~82% used to only 41% used. And I'll get a small performance boost.

 

[ryanpr]

Then they are RED PRO, REDs are 5400rpm

Mine are WD Reds (non-pro) at 5,400 RPM

 

[Stux]

Then they are RED PRO, REDs are 5400rpm

I have Seagate ironwolf 4 and 8TB, the 8TB are 7200rpm

Also WD Red and Red Pro 3TBs, the pros are 7200 rpm

All the 7200 rpm drives run significantly hotter than. 5200 rpm drives.

I suspect the defining characteristic for heat is speed not brand.

 

[Bidule0hm]

Yep, I only see (maybe) a 1 °C difference between the WD Red and Seagate NAS drives.

 

[ryanpr]

Thanks for the replies everyone. Here's my thoughts on moving the data. I've tried it out on a FreeNAS VM except for the zfs send | zfs receive as I didn't want to create a second FreeNAS VM just to test that part.

It looks like it'd work. Anything I'm missing?

1. offline 1 drive from old server
2. install 2x new drives + 1x old drive in new server
3. create pool: mirror of 2x new drives
4. offline 1 drive from pool
5. zpool add <poolname> mirror <offlinedDrive> <oldDrive1> (may require -f if the WD Red is diff size from the Seagate)
6. offline new drive from new mirror
7. replace offlined drive in original pool with offlined drive
8. should have 1x mirror ONLINE with new drives & 1x mirror DEGRADED with 1x old drive
9. zpool send | zpool receive datasets
10. offline second old drive from old server destroying the original pool and decommissioning old server
11. add old drive to new server
12. replace missing drive in DEGRADED mirror set with the second old drive
13. wait for resilvering to finish

 

[ryanpr]

I did this whole cycle on my VM test machines (with the zfs send | receive this time). I noticed at the end that one mirror set has da3 & da4 the other has gptid/GUID & gptid/otherGUID.

state: ONLINE
scan: resilvered 555M in 0 days 00:00:01 with 0 errors on Sat Aug 11 12:23:01 2018
config:

NAME STATE READ WRITE CKSUM
origPool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/0716a8af-9d7d-11e8-9ea9-00155d0a9001 ONLINE 0 0 0
gptid/9a34d366-9d7e-11e8-9ea9-00155d0a9001 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da4 ONLINE 0 0 0
da3 ONLINE 0 0 0

What's the difference there and will it negatively impact my real server if I take the same approach?