Hi everyone,
I've been running TrueNAS Core for about 5 years, and relatively recently upgraded to a pool of 4x 8TB drives in two mirrored VDEVs. It's been running smoothly for a few weeks, but unfortunately got a pretty scary alert this evening:
Here is my current pool status:
I ran a zpool clear, but the pool is still suspended. I can traverse the directory tree via the shell, but otherwise can't read/write. I didn't want to try to much beyond that before checking in here so I don't make things worse.
The system is currently powered on, and several other pools are still up and running (Including another pool on the same SAS controller).
Some additional error logs:
My current hardware:
- Motherboard: ASUS H87M-PLUS
- CPU: Intel Core i5-4750
- RAM quantity: 8GB (2x 4GB)
- Hard Drives:
- Pool 'NASMirror': 4x HGST 8TB SAS drives, 2x Mirrored VDEV, Connected via Supermicro controller
- Pool 'PlexVol': 1x Patriot Burst 120GB SATA SSD, Connected via main board
- Pool 'SafeMirror': 1x Seagate 1TB 2.5" SATA and 1x Western Digital 1TB 2.5" SATA, Connected via main board
- Pool 'TempMirror': 2x HGST 8TB SAS drives, Mirrored VDEV, Connected via Supermicro controller (I had recently connected these drives an was running smart tests, just put them into a pool to transfer data over if I can read from NASMirror).
- Pool 'Boot': 2x SanDisk Cruiser 16GB USB drives, Mirrored VDEV
- Disk Controller: Supermicro 9207-8I
- Network: Built-in
This is just a hobby machine and nothing critical is on it that isn't also stored somewhere else, but it would be convenient not to lose the pool if possible (and also to figure out what went wrong since it seems to have impacted so many disks all at once).
Thanks everyone!
*Edit: Cleaned up code tags.*
I've been running TrueNAS Core for about 5 years, and relatively recently upgraded to a pool of 4x 8TB drives in two mirrored VDEVs. It's been running smoothly for a few weeks, but unfortunately got a pretty scary alert this evening:
WARNING: Pool 'NASMirror' has encountered an uncorrectable I/O failure and has been suspended.
Here is my current pool status:
Code:
root@ateamnas:~ # zpool status NASMirror pool: NASMirror state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-JQ scan: scrub repaired 0B in 03:39:53 with 0 errors on Fri Jun 30 23:05:27 2023 config: NAME STATE READ WRITE CKSUM NASMirror ONLINE 0 0 0 mirror-0 ONLINE 24 8 0 gptid/af89a88d-128b-11ee-9eea-7824af32148b ONLINE 13 8 0 gptid/6f2ec921-1221-11ee-ad59-7824af32148b ONLINE 9 8 0 mirror-1 ONLINE 33 12 0 gptid/99be896b-1614-11ee-b28a-7
I ran a zpool clear, but the pool is still suspended. I can traverse the directory tree via the shell, but otherwise can't read/write. I didn't want to try to much beyond that before checking in here so I don't make things worse.
The system is currently powered on, and several other pools are still up and running (Including another pool on the same SAS controller).
Some additional error logs:
Code:
... Jul 20 21:13:14 ateamnas mps0: Controller reported scsi ioc terminated tgt 3 SMID 1455 loginfo 31120100 Jul 20 21:13:14 ateamnas (da1:mps0:0:3:0): READ(10). CDB: 28 00 24 40 5c b0 00 01 00 00 Jul 20 21:13:14 ateamnas (da1:mps0:0:3:0): CAM status: CCB request completed with an error Jul 20 21:13:14 ateamnas (da1:mps0:0:3:0): Error 5, Retries exhausted Jul 20 21:13:14 ateamnas mps0: Controller reported scsi ioc terminated tgt 4 SMID 1044 loginfo 31170000 Jul 20 21:13:14 ateamnas (da2:mps0:0:4:0): READ(10). CDB: 28 00 24 40 51 38 00 01 00 00 Jul 20 21:13:14 ateamnas (da2:mps0:0:4:0): CAM status: CCB request completed with an error Jul 20 21:13:14 ateamnas (da2:mps0:0:4:0): Retrying command, 3 more tries remain Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): READ(10). CDB: 28 00 24 40 51 38 00 01 00 00 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): CAM status: SCSI Status Error Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): SCSI status: Check Condition Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): SCSI sense: UNIT ATTENTION asc:29,1 (Power on occurred) Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Field Replaceable Unit: 22 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Retrying command (per sense data) Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): READ(10). CDB: 28 00 24 40 51 38 00 01 00 00 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): CAM status: SCSI Status Error Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): SCSI status: Check Condition Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): SCSI sense: NOT READY asc:4,11 (Logical unit not ready, notify (enable spinup) required) Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Field Replaceable Unit: 83 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Command Specific Info: 0 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Descriptor 0x80: f5 53 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Descriptor 0x81: 00 00 00 00 00 00 Jul 20 21:13:15 ateamnas (da2:mps0:0:4:0): Polling device for readiness
My current hardware:
- Motherboard: ASUS H87M-PLUS
- CPU: Intel Core i5-4750
- RAM quantity: 8GB (2x 4GB)
- Hard Drives:
- Pool 'NASMirror': 4x HGST 8TB SAS drives, 2x Mirrored VDEV, Connected via Supermicro controller
- Pool 'PlexVol': 1x Patriot Burst 120GB SATA SSD, Connected via main board
- Pool 'SafeMirror': 1x Seagate 1TB 2.5" SATA and 1x Western Digital 1TB 2.5" SATA, Connected via main board
- Pool 'TempMirror': 2x HGST 8TB SAS drives, Mirrored VDEV, Connected via Supermicro controller (I had recently connected these drives an was running smart tests, just put them into a pool to transfer data over if I can read from NASMirror).
- Pool 'Boot': 2x SanDisk Cruiser 16GB USB drives, Mirrored VDEV
- Disk Controller: Supermicro 9207-8I
- Network: Built-in
This is just a hobby machine and nothing critical is on it that isn't also stored somewhere else, but it would be convenient not to lose the pool if possible (and also to figure out what went wrong since it seems to have impacted so many disks all at once).
Thanks everyone!
*Edit: Cleaned up code tags.*
Last edited: