SOLVED RAIDZ2 - hung after zvol delete

[joshg678]

I supposes that's true, just seems crazy how the entire system becomes unresponsive.
It reminds me of work where Centos systems with 128GB of ram run out because everyone and their mother is running some java app. the Video card stops displaying stuff too, hard reset is the only thing to bring it alive again.

 

[joshg678]

So funny story (well not really)

I had a second zvol that needed to be deleted (only 500gb this time) and after i got everything imported from last time with the 16gb of ram installed.
So when i went to delete this one, it did the exact same thing, crashed the system. Upon reboot it would fail boot because it would also run out of memory. Same thing if i attempted to import from command line.

Thankfully i had the old ram laying around i could add to give me 24gb of ram. After i installed that i was able to import the pool. Took about 45 minutes, and used about 17GB of ram. before it completed. But after, an export/import was instant.
I took out that extra ram to keep it around encase i try to mess with this again.

I wonder if there is some kind of bug when you delete a vzol?

 

[cyberjock]

Maybe if you delete the contents of the zvol first? I don't know.... Really odd and you are the only person to ever have this problem.

 

[joshg678]

The zvol were used specifically for iSCSI mounts, and the data was removed, alone with the partition on the remote system.

These zvol's were created a long time ago, before most of my data was on the disks, wonder if that has anything to do with it?

 

[cyberjock]

It could be that ZFS needs lots of RAM on zvol deletes that are full because it tries to delete the zvol in a single transaction. Obviously a 1TB+ zvol could take up quite a bit of space. I just find it odd that this would have never been a problem anyone has ever reported before.

 

[Attila]

A similar problem. After deleting the zvol (650 GB), the freenas hungs. It was an ISCSI target previously. I tried to restart FreeNAS server, but cannot mount local file systems. I the following error message is displayed, and nothing happens after several hours:

Solaris: WARNING: can't open objset for [The deleted volume name]

 

[joshg678]

I had the same exact issue, Deleting a zvol that was a iSCSI target.

My problem was that i did not have enough memory for it to complete its process before crashing.

Be sure to test your RAM with memtest before you start messing with anything to make sure that its in good shape first.

Try adding memory (and run memtest again) and then see if its able to complete its process. For me when the hard drives would stop having activity is when the system would crash. (except after i upgraded memory)


I don't know if its trying to flush some logs or what, but it always wants more ram then you have. I upgraded to 16gb from 8gb after i deleted my 1TB zvol to get it to mount again.
However, after it was deleted, i tried to delete a 500GB zvol (also iscsi extend) and the system did the exact same thing. So i had to take the memory that i took out, back in my unoccupied slots to get me to 24GB to get the second zvol to delete successfully.

If we were taking bets, i probably could have put my memory back to 8gb, attempted to delete the second zvol and then upgraded to 16gb and it still would have worked. But i have no idea, just speculation at this point.

 

[Attila]

In this configuration I have 24Gigs of RAM (this is an Dell R610 Server with ECC memory configuration) It is possible to add more RAM, but the configuration is 80km far to me :-(

 

[Attila]

Hi, the problem is solved
I put the server 24 GB of RAM (Sum 48GB)
I Restarted the server.
After a few hours the import was successful. While the server is started
While the server starts the server in about 30 GB of RAM used up.

Having consulted with several colleagues experienced in ZFS, I changed the value of the kernel parameter vfs.zfs.arc_max to 16 GB

/boot/loader.conf
vfs.zfs.arc_max="16G"

 

[joshg678]

Interesting to see the same fix worked for you too.

Do you know if that kernal parameter would fix the issue without upgrading the ram?

What does that Parameter do?

 

[Attila]

Hi,

Yes, I think when I modify the parameter before increasing of memory It would have been possible. The memory upgrade was before I consulted with my colleagues

The interested parameters are:
zpool status -v | camcontrol devlist

Code:

<iDRAC LCDRIVE 0323>              at scbus0 target 0 lun 0 (pass0,da0)
<iDRAC Virtual CD 0323>            at scbus1 target 0 lun 0 (cd0,pass1)
<Kingston DT Rubber 3.0 PMAP>      at scbus2 target 0 lun 0 (pass2,da1)

gpart show | glabel status

Code:

                                      Name  Status  Components
gptid/00d812c3-e8f4-11e2-83f8-00219b915783    N/A  mfid0p2
                            ufs/FreeNASs3    N/A  da1s3
                            ufs/FreeNASs4    N/A  da1s4
                            ufs/FreeNASs1a    N/A  da1s1a

mount | smartctl -q noserial -a /dev/adaX

Code:

can not be interpreted because the hardware raid controller