How to moving data from one disk to another entirely?

[truenast3x]

The pool layout:

1. vdev1 (multi disks with hardware raid5)
2. vdev2 (multi disks hardware raid6)
3. vdev3 (single ssd)
4. vdev4 (single ssd)

No software raid on any of these device.

Wha I want:

1. moving data on vdev3 & vdev4 to vdev1/vdev2
2. removing vdev3 & vdev4 safely without any data loss

Why I want to do this:

1. I made mistake to create cachey vdev for this pool, the two ssd now are normal vdev but not cache
2. add there is hardware issue on one of the ssd
3. the broken ssd cause the whole pool down sometimes

Now the system is running well, but I don't know when will it break down. It seems that every two days the issue will be back...

I've searched the net, but can't find the solution.

Thank you very much!

 

[jgreco]

Wow. Well, that's a trainwreck. For better or for worse, the addition of the SSD's guarantees you will now need to start over, because that is an irreversible error.

Please start by reading

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

which will explain why

1. vdev1 (multi disks with hardware raid5)
2. vdev2 (multi disks hardware raid6)

is the next major problem, and why

No software raid on any of these device.

is nothing to be proud of. When you're done reading that article, please come back, post a summary of the actual hardware in use, and then we may be able to provide some useful advice on how to rebuild your system.

 

[truenast3x]

Wow. Well, that's a trainwreck. For better or for worse, the addition of the SSD's guarantees you will now need to start over, because that is an irreversible error.

Please start by reading

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

which will explain why



is the next major problem, and why



is nothing to be proud of. When you're done reading that article, please come back, post a summary of the actual hardware in use, and then we may be able to provide some useful advice on how to rebuild your system.

Thank you for your reply!

I'll remove hardware raid in new TrueNAS device.

My hardware raid info:

truenas# MegaCli -LDInfo -Lall -aALL | egrep 'Adapter|Size' | grep -v Strip
Adapter 0 -- Virtual Drive Information:
Size : 14.553 TB
Sector Size : 512
Parity Size : 7.276 TB
Size : 80.044 TB
Sector Size : 512
Parity Size : 14.553 TB
Size : 21.830 TB
Sector Size : 512
Parity Size : 7.276 TB

Each physical disk is 8TB SATA HDD.

The disks which are using in the pool:

da1 & da2 are two logic disks made by hardware raid.

nvd0 & nvd1 are two M.2 SSD connected by a PCIE adapter card.

All of these disks were added directly in the pool without data protection provided by TrueNAS.

I thoughout hardware raid could work well ,and SSDs were cache layer, there is no need to use redundancy. But I didn't know why the vdev type didn't work as I expected, I was sure I've chosen the cache as vdev type for new added SSD.

If what I provided isn't enough, sorry for my poor understanding of TrueNAS.

Thank you!

 

[c77dk]

I'm confident that what @jgreco wanted to know regarding the hw raid was which vendor and model it is. Some cards can be flashed to HBA in IT-mode, and others have to be replaced with a HBA if possible.

As you'll see in the link to the HBA/RAID you won't get the great benefits of ZFS using a RAID card.

 

[jgreco]

How much memory is in this system? 256GB? 512GB?

 

[truenast3x]

I'm confident that what @jgreco wanted to know regarding the hw raid was which vendor and model it is. Some cards can be flashed to HBA in IT-mode, and others have to be replaced with a HBA if possible.

As you'll see in the link to the HBA/RAID you won't get the great benefits of ZFS using a RAID card.

Thank you for your explanation!

That's my RAID card model:

truenas# MegaCli -AdpAllinfo -aAll

Adapter #0

==============================================================================
Versions
================
Product Name : AVAGO MegaRAID SAS 9361-24i
FW Package Build: 24.22.0-0071

 

[truenast3x]

How much memory is in this system? 256GB? 512GB?

Only 64GB, I wanted bigger memory, but that's only I got...

 

[jgreco]

AVAGO MegaRAID SAS 9361-24i

Well, the upside here is that it probably hasn't actually damaged the pool yet. Probably using MRSAS driver.

I thoughout hardware raid could work well

It doesn't.

SSDs were cache layer

They are. But you can only have about 4x-5x your system RAM, so for your given size of 64GB, you are limited to about 240GB of SSD, not 2TB.

there is no need to use redundancy.

Okay. But without redundancy, you can have a catastrophic issue, and there may be no recovery from it. If ZFS detects an error in the data stored on disk but cannot rebuild it from redundancy, that data is gone. Period. ZFS needs to manage the storage redundancy. It should never be hidden from ZFS. ZFS has no "fsck" or "chkdsk" tools, so once damage is done, it may be permanent.

 

[truenast3x]

Well, the upside here is that it probably hasn't actually damaged the pool yet. Probably using MRSAS driver.



It doesn't.

Now I know I've made a mistake here...

They are. But you can only have about 4x-5x your system RAM, so for your given size of 64GB, you are limited to about 240GB of SSD, not 2TB.

The SSD disks are not shown in cache list in dashboard.

Okay. But without redundancy, you can have a catastrophic issue, and there may be no recovery from it. If ZFS detects an error in the data stored on disk but cannot rebuild it from redundancy, that data is gone. Period. ZFS needs to manage the storage redundancy. It should never be hidden from ZFS. ZFS has no "fsck" or "chkdsk" tools, so once damage is done, it may be permanent.

What should I do for saving this pool? One of the SSD is detached again, the pool cannot response again... I must remove the two SSD right now but without data loss.

 

[Ericloewe]

Your only hope of removing the failing SSD, in the absence of a replacement disk, is device removal. I don’t know if the GUI supports this, so you may need to use the CLI. Definitely not something to jump head-first into.

Also, telling us that you added the disks - somehow - as regular vdevs instead of L2ARC and then not showing us either the output of zpool status or the GUI’s pool status page is bizarre.

 

[jgreco]

The SSD disks are not shown in cache list in dashboard.

Because you seem to have added them as pool vdevs. However, that wasn't my point. The point was that they are an order of magnitude larger than the amount of cache your system can reasonably support, even had they been added properly. But they won't show up in the cache list in the dashboard because they're not being used as cache.

 

[danb35]

device removal. I don’t know if the GUI supports this

It does in TN12.

 

[truenast3x]

Your only hope of removing the failing SSD, in the absence of a replacement disk, is device removal. I don’t know if the GUI supports this, so you may need to use the CLI. Definitely not something to jump head-first into.

Thanks for your advice!

Do you mean I should use zpool remove ? The data on unstable disk (it works fine in present, but will be detached often) will be transferred to other disks on the same pool if I do the remove?

Also, telling us that you added the disks - somehow - as regular vdevs instead of L2ARC and then not showing us either the output of zpool status or the GUI’s pool status page is bizarre.

That's the output of zpool status

pool: tesData
state: ONLINE
scan: scrub repaired 0B in 19:35:55 with 0 errors on Sun Jan 2 19:35:55 2022
config:

NAME STATE READ WRITE CKSUM
tesData ONLINE 0 0 0
gptid/626b7560-c4fb-11eb-8d79-90e2ba450e8c ONLINE 0 0 0
gptid/8c2f7ad2-6d53-11ec-b7f7-90e2ba450e8c ONLINE 0 0 0
gptid/d66774e2-75ac-11ec-949b-90e2ba450e8c ONLINE 0 0 0
gptid/d66985fe-75ac-11ec-949b-90e2ba450e8c ONLINE 0 0 0

errors: No

 

[truenast3x]

that's zpool remove output, it seems that only few memory needed to do the remove.

Is it safe to remove this disk?

truenas# zpool remove -n tesData gptid/d66774e2-75ac-11ec-949b-90e2ba450e8c
Memory that will be used after removing gptid/d66774e2-75ac-11ec-949b-90e2ba450e8c: 1.72M

 

[truenast3x]

In the remove process, read error happened. Now pool is offline, I'll buy a new adapter then try it again