Advice on adding drives/expanding a pool

[Amsoil_Jim]

currently I have 6 3TB WD reds in a pool in my 24 bay server. I just bought 6 WD 10TB drives that I want to add.
I’m pretty sure the 3TB drives are setup in raidz1. I know you can’t just expand the pool by adding the drives.
Can I create another vdev with the 10TB drives in raidz1 and add it too the pool? Or does that not work either because the original vdev was created using the 3TB drives?
I know another option is to swap out the 3TB drives one at a time with the 10’s and resilver them but then do I just create the second vdev using the 3 TB drives?
Just trying to figure this out and have the most usable space.

 

[Scharbag]

Good work on the 6 10TB disks!!

First advice would be to NOT use RaidZ1 with 3TB drives - certainly not with 10TB drives.

If I were you, I would create a separate 6 disk RaidZ2 vDev with the new 10TB disks (recommend burning in the drives). Once this is ready, copy all of your data to this new pool. Also, copy your jails etc. to the new pool. I know this will take some time but bear with me. Once you have all of your data copied, and your system is effectively not using the old pool, destroy the old 3TB disk pool and then add the 6 3TB disks as a second RaidZ2 vDev to the new 10TB pool.

You will end up with a 12 disk pool with a 6@10TB vDev and a 6@3TB vDev. Yes, 4 drives are lost for parity but it is totally worth it.

Also, as a public service announcement, make sure you have backups!!!

Cheers,

 

[Amsoil_Jim]

Good work on the 6 10TB disks!!

First advice would be to NOT use RaidZ1 with 3TB drives - certainly not with 10TB drives.

If I were you, I would create a separate 6 disk RaidZ2 vDev with the new 10TB disks (recommend burning in the drives). Once this is ready, copy all of your data to this new pool. Also, copy your jails etc. to the new pool. I know this will take some time but bear with me. Once you have all of your data copied, and your system is effectively not using the old pool, destroy the old 3TB disk pool and then add the 6 3TB disks as a second RaidZ2 vDev to the new 10TB pool.

You will end up with a 12 disk pool with a 6@10TB vDev and a 6@3TB vDev. Yes, 4 drives are lost for parity but it is totally worth it.

Also, as a public service announcement, make sure you have backups!!!

Cheers,

I understand the risk of using raidz1,none of the data is irreplaceable, it’s just media.
I do plan on doing burnin on them before using them in a new vdev/pool.
Your recommendations are what I was thinking I would have to do in order to add the larger drives to an existing pool,basically.
The drives are easystores that have to be schucked so I was thinking I would do burnin before schucking them that way I could easily exchange it if there’s an issue with one.

 

[Scharbag]

I understand the risk of using raidz1,none of the data is irreplaceable, it’s just media.
I do plan on doing burnin on them before using them in a new vdev/pool.
Your recommendations are what I was thinking I would have to do in order to add the larger drives to an existing pool,basically.
The drives are easystores that have to be schucked so I was thinking I would do burnin before schucking them that way I could easily exchange it if there’s an issue with one.

If you can tolerate the risk, then your plan is good to go.

Cheers,

 

[Amsoil_Jim]

Is there a way to tell if a SMART test currently running or not? After 140 hours the badblocks test completed and then I ran a long SMART test and I started to run a captive test but it looks like the system threw some error and the status shows as "interrupted (host reset)" so I started the long offline test and it says it take 1134 minutes to complete. Now I'm not sure if its still running or not. The other 5 disk show as completed without error

 

[Kevin Horton]

Is there a way to tell if a SMART test currently running or not? After 140 hours the badblocks test completed and then I ran a long SMART test and I started to run a captive test but it looks like the system threw some error and the status shows as "interrupted (host reset)" so I started the long offline test and it says it take 1134 minutes to complete. Now I'm not sure if its still running or not. The other 5 disk show as completed without error

You can see that in the output of smartctl -a
Do a smartctl -a /dev/daXX | grep -A 3 "Self-test execution status:"

 

[Constantin]

FWIW, unless you have an immediate use for the 3TB drives at this point, I'd transfer the data as suggested to the 10TB-disk pool and then pull the 3TB drives. Put them on a shelf. If an when a use case arises, you have the option to add them to the running 10TB-disk pool, create a new one, etc.

Or.. you might decide to add more 10TB drives. In the meantime, you don't waste electricity running the likely obsolete 3TB drives for essentially no purpose.

 

[Amsoil_Jim]

FWIW, unless you have an immediate use for the 3TB drives at this point, I'd transfer the data as suggested to the 10TB-disk pool and then pull the 3TB drives. Put them on a shelf. If an when a use case arises, you have the option to add them to the running 10TB-disk pool, create a new one, etc.

Or.. you might decide to add more 10TB drives. In the meantime, you don't waste electricity running the likely obsolete 3TB drives for essentially no purpose.

I did not think of that, I was just thinking "I was almost out of space and now I have way more! woohoo!." But I guess I will still have triple the space with just swapping out the 3's for 10's.
Thanks for the suggestion, always looking for ways to keep the power usage down.
Either way I have 24 bays...

 

[Constantin]

I hear you. I'm about to embark on a Motherboard transfer to a new case, etc. and will be changing the pool also.... going from 9x3TB drives to 8x10TB drives. The new 10TB drives also draw less power than the old 3TB units. I would have even considered a 7-drive config (this is a Z3 pool) were it not for the "don't fill a ZFS pool more 80%" rule and the fact that my backup array is a RAID5 5-drive unit. Going for a 7-disk 10TB Z3 pool and 8TB drives for the backup array would have aligned respective "allowed" storage capacities.

But, I'm a big fan of only using one capacity of HDD in my arrays, that way a drive failure in either the FreeNAS or the backup can be dealt with using a single on-site spare. Interestingly, it's always the drives in the backup that have gone over the years, never the FreeNAS HGST 7k4000's... they may run hot, but they haven't failed me yet. Soon, they will be obsolete.

 

[Amsoil_Jim]

You can see that in the output of smartctl -a
Do a smartctl -a /dev/daXX | grep -A 3 "Self-test execution status:"

is this normal that after 21 hours running this long test, the status shows as
root@FreeNAS:~ # smartctl -a /dev/da6 | grep -A 3 "Self-test execution status:"
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 93) seconds.
This Drive did complete a short test and long test before I ran the badblocks test.
Edit: just noticed I received an email when I started replacing one the the disk that showed:
New alerts:
* The volume Media state is ONLINE: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.
* smartd_daemon is not running.
Then 1 minute later:
Gone alerts:
* smartd_daemon is not running.


 

[Amsoil_Jim]

After starting a Long test on one of the 3TB drives that have been replaced already I see that test should not be stuck at 90% after 21 hours.
I have removed this disk from the server for further testing connected to my computer using the WD Drive Utility and it has failed

 

[Constantin]

That's a stinker. I hope you won't be held by WD to return a "unshucked" drive for RMA? I've been breaking all the plastic tabs pretty consistently.

 

[Amsoil_Jim]

That's a stinker. I hope you won't be held by WD to return a "unshucked" drive for RMA? I've been breaking all the plastic tabs pretty consistently.

I was very careful when taking these apart and I may have only snapped 1 tab out of the 6 drives. But I was just going to try and take it back to bestbuy and swap it out first

 

[Constantin]

Swap the white drive or return the white drive to its plastic coffin and then swap?

I presume it's the latter since the folk at Best Buy likely don't have white WDs hanging around to give out on a RMA basis?

 

[Amsoil_Jim]

Yes I completely reassembled and reboxed it
Edit: Exchanged at BestBuy no problem. Now for another 7+ days of testing for the new drive
long smart test = 19hrs
badblocks = 140hrs
long smart test = 19hrs
total = 7 days 10hrs
hopefully this ones good...

 

[Amsoil_Jim]

What happens to the data thats on the original drives after the replacement processes has completed? It looks like it just mirrors the old drive and the new drive then removes the old drive from the pool/vdev. So does that mean all the data is still on the old drive and they need to be erased to be reused?

 

[danb35]

they need to be erased to be reused?

Depends how paranoid you are. Wiping the partition table would make it perfectly reusable. But if you were disposing of the disk, you might want to run a pass of badblocks, or even fire up DBAN.

 

[Amsoil_Jim]

Depends how paranoid you are. Wiping the partition table would make it perfectly reusable. But if you were disposing of the disk, you might want to run a pass of badblocks, or even fire up DBAN.

I won't be disposing of these, just wondering for reuse purposes. Thanks