Problems replacing drive(s) 13.0U2

[grocerylist]

Hi,
I'm having some difficulty replacing/upgrading my drives. I'm trying to replace my 8x4TB drives with 8x14TB drives. Originally I had Truenas 13.0 and I read that currently the only way to replace drives is through CLI and I did that on two drives successfully. On my 3rd drive however I ran into an error when running the 'replace_disk.py' script. I'm not sure if I somehow created this problem, but when running the script, I input the wrong ada#. I input ada2 when (after looking more closely at s/n and gptid) I should have input ada5. The initial error I got was: "Unused disk not found." Then I attempted the script again with the correct ada5 but this time I got the following error:

root@freenas:~ # python3 replace_disk.py FreeNAS gptid/063724c9-f3ee-11e6-9814-d05099c185ff ada5
Traceback (most recent call last):
File "/root/replace_disk.py", line 44, in <module>
main()
File "/root/replace_disk.py", line 40, in main
print('Replace initiated.')
File "/root/replace_disk.py", line 39, in main
c.call('pool.replace', pool[0]['id'], arg, job=True)
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 457, in call
return jobobj.result()
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 200, in result
raise ValidationErrors(job['exc_info']['extra'])
middlewared.client.client.ValidationErrors: [EINVAL] options.force: Disk is not clean, partitions were found.

I've connected these drives to another computer to ensure they're not written to. No folders, files but they are formatted NTFS.

Since then I've replaced with my old ada5 drive and the pool is healthy.

I've tried moving on past ada5 and tried replacing ada4 but in this case I got the following error:

root@freenas:~ # python3 replace_disk.py FreeNAS gptid/0540ba03-f3ee-11e6-9814-d05099c185ff ada4
Traceback (most recent call last):
File "/root/replace_disk.py", line 44, in <module>
main()
File "/root/replace_disk.py", line 40, in main
print('Replace initiated.')
File "/root/replace_disk.py", line 19, in main
assert '13.0-RELEASE' in c.call('system.version')
AssertionError

Since then I had read that 13.0U1 might have fixed the GUI replace drive option so I upgraded to 13.0U2 but now when I click to replace a drive, the drop down doesn't list any drives to be replaced. It just shows '-'.

Did my error inputting ada2 instead of ada5 cause this? And if so, what can be done to remedy? If it's unrelated, any idea what would be causing this and how to proceed?

I appreciate any help, thanks!

Show : See My freeNAS system

ASRock C2750D4I
32GB DDR3 ECC
Mirrored 32GB USB drives
8 WD Red Plus drives
20TB pool raidz2

 

[Samuel Tai]

You don't replace drives from the Storage->Disks screen. You need to navigate to Storage->Pools, and click the gear to pull down to Status to reach the zpool status screen, where disk replacement is possible. The script was only needed for 13.0-RELEASE, and isn't needed in 13.0-U1 or later.

 

[Samuel Tai]

Disk Replacement

Describes how to replace a disk and restore the hot spare in TrueNAS CORE.

www.truenas.com

 

[grocerylist]

Sorry, I didn't mean to confuse with the picture of my drives. I'm aware of where to replace drives in the GUI. But when I attempt to replace a drive, there are no drives to select to replace.

 

[Samuel Tai]

OK, that's because the Pool Manager expects drives without any partitions, which are also not in any pool. First, use the 3 dots pull down next to ada5 to remove ada5. This will put your pool in DEGRADED state. Next, physically remove ada5, and replace with the new ada5. Use the 3 dots pull down next to the missing ada5 and select replace. Since the new disk isn't part of any pool, it will appear in the pulldown. Once selected, click Replace Disk, and the pool will start resilvering.

 

[grocerylist]

I've tried that. What I'm trying to say is that it doesn't work. My first post details that I originally was on 13.0 so I had to do it via CLI. The first 2 disks that I replaced worked fine but I'm guessing my input error (ada2 instead of ada5) might have somehow messed things up? I eventually upgraded to 13.0U2 so I could attempt replacing disks via the GUI. But that fails as I detailed in my original post where I posted the errors I received in CLI. And when I attempt in GUI there's no disk to select. Please see below.

 

[Samuel Tai]

OK, like I said, the Pool Manager expects replacement disks to be factory, without a partition table or partitions. Try wiping your replacement disk so it's like factory.

 

[grocerylist]

The drives are factory, no partition, no files or folders, have never been used other than to confirm they're working as I said in my first post.

 

[Samuel Tai]

OK, we'll have to do it the old-fashioned way. Have you tried from Shell zpool replace FreeNAS gptid/063724c9-f3ee-11e6-9814-d05099c185ff /dev/ada5?

 

[grocerylist]

Yes, like I said in my original post I had to do it that way when I was on v13.0. If you read my first post it explains the problem and I possibly messed something up when I used the replace_disk.py script and put ada2 when I should have put ada5. I don't know if that messed things up seriously or not.

 

[Samuel Tai]

I don't think your typo affected your pool. You could reboot if you're worried about a lingering job.

 

[Patrick M. Hausen]

OK, we'll have to do it the old-fashioned way. Have you tried from Shell zpool replace FreeNAS gptid/063724c9-f3ee-11e6-9814-d05099c185ff /dev/ada5?

One should not do that. You need to create a partition table and use the GPTID as a replacement device.

 

[grocerylist]

I don't think your typo affected your pool. You could reboot if you're worried about a lingering job.

I did reboot to see if that would help but I still get the same results above. And of course it rebooted after I updated to 13.0-U2 also.

 

[Samuel Tai]

OK, now I'm confused. In the replace drive dialog, did you pull down on the - to see if the new drive was there?

 

[grocerylist]

Yes, there are no drives listed to select to replace as the images I uploaded show.

 

[Samuel Tai]

When you inserted the new ada5, do you get any output in dmesg?

 

[grocerylist]

This is odd... I'm not sure what happened between last night and today. I left the new ada5 drive in 'offline' state over night and just now set it to 'online' then clicked 'Replace' and when before nothing would populate, now ada5 was available.

I'll have to wait until at least tomorrow for the resilver to complete. Hopefully whatever was wrong is now fixed and the rest of the disk can be replaced. I'll update this post again if I encounter any further issues with the remaining disks.

Once I've replaced all my 4TB drives with 14TB drives, will TrueNAS automatically expand the pool or is that something I'll have to do manually?

 

[Samuel Tai]

No, the pool will auto-expand after all the drives in all the VDEVs are replaced with larger ones.