System failure/crashing on reboot

[bieringm]

Hey all,

I upgrade my Truenas machine about 3-4 weeks ago and all was good. Fast forward to yesterday I got an email alert stating "Pool ... state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected." I went to go log into the dashboard, but was unable to connect to the system and I was also unable to ping the system. I then restarted the machine, got the email alert again (a few minutes after boot) and was still unable to log in. I then plugged in a monitor to my GPU and was greeted with the following screen shots:

At this point I cannot SSH or ping the system and it just sits in this state. I am not sure what the cause of this error is.

The system is as follows:
Ryzen 5 3600
Asrock B550 Phantom Gaming mobo
Mushkin Redline 64GB DDR4 3200 RAM
4x Intel 670p 2TB M.2 running on Dell 6N9RH (Raid Z1)
4x Seagate IronWolf 4TB HDDs (Raid Z1)
1x Kingston A400 240GB Sata SSD (Cache Drive)
1x Kingston A400 120GB Sata SSD (Install Drive)

The Intel drives were added on with the most recent rebuild.

 

[sretalla]

4x Intel 670p 2TB M.2 running on Dell 6N9RH (Raid 5)

I think this is already the source of your problem and unfortunately it's going to be too late to avoid data loss.

Please read this:

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

If you've used a RAID5 configuration in your RAID card to present a volume to ZFS, it's not possible for ZFS to control the data in a way that prevents (and can correct) corruption.

It can, however, detect such corruption, which is what you're now seeing.

I don't think you've attached your screenshots properly, so I have not seen what you're sharing there.

 

[bieringm]

I think this is already the source of your problem and unfortunately it's going to be too late to avoid data loss.

Please read this:

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

If you've used a RAID5 configuration in your RAID card to present a volume to ZFS, it's not possible for ZFS to control the data in a way that prevents (and can correct) corruption.

It can, however, detect such corruption, which is what you're now seeing.

I don't think you've attached your screenshots properly, so I have not seen what you're sharing there.

I was afraid of that. Thankfully I have a CRON job that backs up that pool to the main SATA pool. I have attached the screenshots this time so hopefully they can be viewed.

 

[bieringm]

I think this is already the source of your problem and unfortunately it's going to be too late to avoid data loss.

Please read this:

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

If you've used a RAID5 configuration in your RAID card to present a volume to ZFS, it's not possible for ZFS to control the data in a way that prevents (and can correct) corruption.

It can, however, detect such corruption, which is what you're now seeing.

I don't think you've attached your screenshots properly, so I have not seen what you're sharing there.

With that being said my understanding of the 6N9RH is it is just a bifurcation card and has no real controller on it. All the RAID is done in software via Truenas.

 

[sretalla]

OK, so the "controllers" are all NVME, so not hardware RAID... all good in that area then.

smartctl -a on each of the drives would maybe point to any sources of issue.

 

[bieringm]

I was able to get the machine to boot into the GUI along with allowing SSH. I seem to have 2 failing drives in my HDD array, so I think that was causing most of the issues. Thanks sretalla for the help.