i experience an very unpleasent error since i started my Truenas Scale journey.
My NAS is in the signature.
It started very early with Read / Write Errors on the 4TB SSDs.
My first thought was okay well then bad luck with the disks.
I changed both of them because both showed errors.
The replacement didnt work fine because also the new once got errors.
The Smart Test didnt showed any serious error.
My new plan was it, because the pool did get already degraded, to start with a new disk brand and i ordered 4x 4TB Samsung Q870 SSDs for that.
I changed the disks and experienced one day after again read errors on the Samsungs.
I checked again Smart and it didnt showed sektor errors.
In the process i changed also the Sata cabels and for trancend disks i did put them also directly to the sata port because the jonsbo case has a hot plug mount for the disks with a separate port plate.
All of this didnt gave me a stable system.
Now im in day 4 and i tried a scrub and within the process i got a fault error on one of the new disks which degraded again the mirror vdev.
Config now is by the way a 2-Mirror.
Do you have any ideas for me whats going on here?
Disk changed, cable changed, get rid of all adapters.
Mainboard? the 32GB ECC Ram which is certified by the mainboard manufacturer? Write Errors? Possible... but reads?
I have no clue ... and it is so much annoying for my first experience with turenas and a self build...
Thanks for your replies!
EDIT:
1 Disk shows CRC Errors ... well that could be the write error in the gui ...
But another Disk shows a POR Recovery Error ... well i didnt do a hard poweroff ... the power supply is way to oversized but i used the same 12V cable to connect both 12V Ports on that Sata PCB from Jonbo. I tought one 12V with from that kind of supply is enough to handle 4 SSDs ... am i wrong here hmm... i will change that qickly
by the way instead of just read and write errors with the transcend i get now also a lot of checksum errors ... interessting. all after scrub.
i experience an very unpleasent error since i started my Truenas Scale journey.
My NAS is in the signature.
It started very early with Read / Write Errors on the 4TB SSDs.
My first thought was okay well then bad luck with the disks.
I changed both of them because both showed errors.
The replacement didnt work fine because also the new once got errors.
The Smart Test didnt showed any serious error.
My new plan was it, because the pool did get already degraded, to start with a new disk brand and i ordered 4x 4TB Samsung Q870 SSDs for that.
I changed the disks and experienced one day after again read errors on the Samsungs.
I checked again Smart and it didnt showed sektor errors.
In the process i changed also the Sata cabels and for trancend disks i did put them also directly to the sata port because the jonsbo case has a hot plug mount for the disks with a separate port plate.
All of this didnt gave me a stable system.
Now im in day 4 and i tried a scrub and within the process i got a fault error on one of the new disks which degraded again the mirror vdev.
Config now is by the way a 2-Mirror.
Do you have any ideas for me whats going on here?
Disk changed, cable changed, get rid of all adapters.
Mainboard? the 32GB ECC Ram which is certified by the mainboard manufacturer? Write Errors? Possible... but reads?
I have no clue ... and it is so much annoying for my first experience with turenas and a self build...
Thanks for your replies!
EDIT:
1 Disk shows CRC Errors ... well that could be the write error in the gui ...
But another Disk shows a POR Recovery Error ... well i didnt do a hard poweroff ... the power supply is way to oversized but i used the same 12V cable to connect both 12V Ports on that Sata PCB from Jonbo. I tought one 12V with from that kind of supply is enough to handle 4 SSDs ... am i wrong here hmm... i will change that qickly
by the way instead of just read and write errors with the transcend i get now also a lot of checksum errors ... interessting. all after scrub.
Puzzling... power issues might explain behaviour, but I'd expect that there would be some alerts.
Do you have actual data on the drives?
If not, I'd suggest running tests on individual drives...
Perhaps share the SMART data on a drive with errors.
If you are more familiar with CORE... see if there's any difference in behaviour (I doubt it, but it would tend to confirm some obscure hardware issue).
Well i have to look if i got another 12V Cable with an old adapter (4 Pin) for the supply.
I mean its semi passive and the fan didnt even started but who knows ... overall load shouldnt be the problem.
Yes i have data on it.
The curious thing is after a reboot for power thing all disks are back up and the pool dont show a degraded status. all checksum errors got cleared (around 60 after scrub) but still read write errors.
For now im not sure if i should send the old disks back because it doesnt seemed a hardware problem.
And no im not familiar with CORE. Scale is my fist NAS OS i choosed.
Can you explain me how SCALE work for Data Read Write? Im not quite sure but i guess i read that it takes always the way over the RAM for every operation?
Could the Boot / OS Disks also the cause of that problem or not because the OS Disk are only important to load it into the ram?
I have a boot mirror from two M2 SSDs but didnt run a Smart Test on it for now.
And by the way while scrubbing i uploaded at the same time some files on my nextcloud were the checksum errors occurred. Perhaps that brings up some ideas...
Well lesson learned and perhaps i was that smart before but bought QVC SSDs wasnt not a good idea. My Disks before were TLC and the other Disk i had in mind too but that one had other problems perhaps with older firmwares i read in SCALE Topics ... its just a mess really to find a good fitting SSD.
HDDs are way to loud where i want to use the server sadly ... no extra room for it ...
But that is something which doesnt have to do with the problem - just saying.
EDIT:
Lol the Errors are gone now and i see some checksum errors ... well perhaps i should rebuild that pool and data. perhaps i got some errors because the data was already corrupt before the migration? i mean yeah i had a mirror and one device was degraded before i started but should that gerneate write errors on the new disks? i dont think so right?
Should we go through the bios settings as well?
EDIT2: After some reboots the degraded vdev changes from one to another ... again read errors ... but the fault disk from the runtime before has now 0 errors ... however i checked my last disk (transcend tlc) and cant find any serious smart error ... so i will put them back in again ... however i run a little bit out of options ... for testing i will connect them without the 2,5 to 3,5 bracket i bought (tested before) and without the jonsbo pcb ...
EDIT3: Here are some log parts from syslog ... doesnt log that good
Dec 18 13:36:09 truenas kernel: ata4.00: exception Emask 0x11 SAct 0x3c000 SErr 0x680100 action 0x6 frozen
Dec 18 13:36:09 truenas kernel: ata4.00: irq_stat 0x48000008, interface fatal error
Dec 18 13:36:09 truenas kernel: ata4: SError: { UnrecovData 10B8B BadCRC Handshk }
Dec 18 13:36:09 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:09 truenas kernel: ata4.00: cmd 60/00:70:28:e2:90/01:00:97:00:00/40 tag 14 ncq dma 131072 in
res 40/00:80:28:ea:90/00:00:97:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:09 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:09 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:09 truenas kernel: ata4.00: cmd 60/00:78:28:e9:90/01:00:97:00:00/40 tag 15 ncq dma 131072 in
res 40/00:80:28:ea:90/00:00:97:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:09 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:09 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:09 truenas kernel: ata4.00: cmd 60/00:80:28:ea:90/01:00:97:00:00/40 tag 16 ncq dma 131072 in
res 40/00:80:28:ea:90/00:00:97:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:09 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:09 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:09 truenas kernel: ata4.00: cmd 60/00:88:28:ef:90/01:00:97:00:00/40 tag 17 ncq dma 131072 in
res 40/00:80:28:ea:90/00:00:97:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:09 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:09 truenas kernel: ata4: hard resetting link
Dec 18 13:36:09 truenas kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 18 13:36:09 truenas kernel: ata4.00: supports DRM functions and may not be fully accessible
Dec 18 13:36:09 truenas kernel: ata4.00: supports DRM functions and may not be fully accessible
Dec 18 13:36:09 truenas kernel: ata4.00: configured for UDMA/133
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#14 Sense Key : Illegal Request [current]
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#14 Add. Sense: Unaligned write command
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#14 CDB: Read(16) 88 00 00 00 00 00 97 90 e2 28 00 00 01 00 00 00
Dec 18 13:36:09 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2542854696 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:09 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1299794055168 size=131072 flags=180980
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#15 Sense Key : Illegal Request [current]
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#15 Add. Sense: Unaligned write command
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#15 CDB: Read(16) 88 00 00 00 00 00 97 90 e9 28 00 00 01 00 00 00
Dec 18 13:36:09 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2542856488 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:09 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1299794972672 size=131072 flags=180980
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#16 Sense Key : Illegal Request [current]
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#16 Add. Sense: Unaligned write command
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#16 CDB: Read(16) 88 00 00 00 00 00 97 90 ea 28 00 00 01 00 00 00
Dec 18 13:36:09 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2542856744 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:09 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1299795103744 size=131072 flags=180980
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#17 Sense Key : Illegal Request [current]
Dec 18 13:36:09 truenas kernel: sd 3:0:0:0: [sdc] tag#17 Add. Sense: Unaligned write command
Dec 18 13:36:10 truenas kernel: sd 3:0:0:0: [sdc] tag#17 CDB: Read(16) 88 00 00 00 00 00 97 90 ef 28 00 00 01 00 00 00
Dec 18 13:36:10 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2542858024 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:10 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1299795759104 size=131072 flags=180980
Dec 18 13:36:10 truenas kernel: ata4: EH complete
Dec 18 13:36:10 truenas kernel: ata4.00: Enabling discard_zeroes_data
Dec 18 13:36:10 truenas zed[102047]: eid=13 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1299795759104 priority=0 err=5 flags=0x180980 delay=667ms bookmark=826:441949:0:23
Dec 18 13:36:10 truenas zed[102048]: eid=14 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1299795103744 priority=2 err=5 flags=0x180980 delay=662ms bookmark=826:441949:0:19
Dec 18 13:36:10 truenas zed[102051]: eid=16 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1299794055168 priority=2 err=5 flags=0x180980 delay=632ms bookmark=826:441949:0:16
Dec 18 13:36:10 truenas zed[102049]: eid=15 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1299794972672 priority=2 err=5 flags=0x180980 delay=647ms bookmark=826:441949:0:18
Dec 18 13:36:12 truenas kernel: ata4.00: exception Emask 0x11 SAct 0x70 SErr 0x600100 action 0x6 frozen
Dec 18 13:36:12 truenas kernel: ata4.00: irq_stat 0x48000008, interface fatal error
Dec 18 13:36:12 truenas kernel: ata4: SError: { UnrecovData BadCRC Handshk }
Dec 18 13:36:12 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:12 truenas kernel: ata4.00: cmd 60/00:20:68:c1:94/01:00:95:00:00/40 tag 4 ncq dma 131072 in
res 40/00:28:68:c2:94/00:00:95:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:12 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:12 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:12 truenas kernel: ata4.00: cmd 60/00:28:68:c2:94/01:00:95:00:00/40 tag 5 ncq dma 131072 in
res 40/00:28:68:c2:94/00:00:95:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:12 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:12 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:12 truenas kernel: ata4.00: cmd 60/00:30:68:c7:94/01:00:95:00:00/40 tag 6 ncq dma 131072 in
res 40/00:28:68:c2:94/00:00:95:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:12 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:12 truenas kernel: ata4: hard resetting link
Dec 18 13:36:12 truenas kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 18 13:36:12 truenas kernel: ata4.00: supports DRM functions and may not be fully accessible
Dec 18 13:36:12 truenas kernel: ata4.00: supports DRM functions and may not be fully accessible
Dec 18 13:36:12 truenas kernel: ata4.00: configured for UDMA/133
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#4 Sense Key : Illegal Request [current]
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#4 Add. Sense: Unaligned write command
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 95 94 c1 68 00 00 01 00 00 00
Dec 18 13:36:12 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2509554024 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:12 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1282744111104 size=131072 flags=180980
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#5 Sense Key : Illegal Request [current]
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#5 Add. Sense: Unaligned write command
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#5 CDB: Read(16) 88 00 00 00 00 00 95 94 c2 68 00 00 01 00 00 00
Dec 18 13:36:12 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2509554280 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:12 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1282744242176 size=131072 flags=180980
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#6 Sense Key : Illegal Request [current]
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#6 Add. Sense: Unaligned write command
Dec 18 13:36:12 truenas kernel: sd 3:0:0:0: [sdc] tag#6 CDB: Read(16) 88 00 00 00 00 00 95 94 c7 68 00 00 01 00 00 00
Dec 18 13:36:12 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2509555560 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:12 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1282744897536 size=131072 flags=180880
Dec 18 13:36:12 truenas kernel: ata4: EH complete
Dec 18 13:36:12 truenas kernel: ata4.00: Enabling discard_zeroes_data
Dec 18 13:36:12 truenas zed[102799]: eid=17 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1282744897536 priority=0 err=5 flags=0x180880 delay=654ms bookmark=826:443029:0:15
Dec 18 13:36:12 truenas zed[102798]: eid=18 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1282744111104 priority=2 err=5 flags=0x180980 delay=628ms bookmark=826:443029:0:9
Dec 18 13:36:12 truenas zed[102801]: eid=19 class=io pool='DataPool' vdev=ad40036a-2bf5-4572-a75a-fb75994dbe45 size=131072 offset=1282744242176 priority=2 err=5 flags=0x180980 delay=647ms bookmark=826:443029:0:11
Dec 18 13:36:24 truenas kernel: ata4: limiting SATA link speed to 3.0 Gbps
Dec 18 13:36:24 truenas kernel: ata4.00: exception Emask 0x11 SAct 0xe00 SErr 0x680100 action 0x6 frozen
Dec 18 13:36:24 truenas kernel: ata4.00: irq_stat 0x48000008, interface fatal error
Dec 18 13:36:24 truenas kernel: ata4: SError: { UnrecovData 10B8B BadCRC Handshk }
Dec 18 13:36:24 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:24 truenas kernel: ata4.00: cmd 60/00:48:38:50:34/01:00:9d:00:00/40 tag 9 ncq dma 131072 in
res 40/00:58:38:56:34/00:00:9d:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:24 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:24 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:24 truenas kernel: ata4.00: cmd 60/00:50:38:52:34/01:00:9d:00:00/40 tag 10 ncq dma 131072 in
res 40/00:58:38:56:34/00:00:9d:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:24 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:24 truenas kernel: ata4.00: failed command: READ FPDMA QUEUED
Dec 18 13:36:24 truenas kernel: ata4.00: cmd 60/00:58:38:56:34/01:00:9d:00:00/40 tag 11 ncq dma 131072 in
res 40/00:58:38:56:34/00:00:9d:00:00/40 Emask 0x10 (ATA bus error)
Dec 18 13:36:24 truenas kernel: ata4.00: status: { DRDY }
Dec 18 13:36:24 truenas kernel: ata4: hard resetting link
Dec 18 13:36:24 truenas kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Dec 18 13:36:24 truenas kernel: ata4.00: supports DRM functions and may not be fully accessible
Dec 18 13:36:24 truenas kernel: ata4.00: supports DRM functions and may not be fully accessible
Dec 18 13:36:25 truenas kernel: ata4.00: configured for UDMA/133
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#9 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#9 Sense Key : Illegal Request [current]
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#9 Add. Sense: Unaligned write command
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 00 9d 34 50 38 00 00 01 00 00 00
Dec 18 13:36:25 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2637451320 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:25 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1348227526656 size=131072 flags=180980
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#10 Sense Key : Illegal Request [current]
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#10 Add. Sense: Unaligned write command
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#10 CDB: Read(16) 88 00 00 00 00 00 9d 34 52 38 00 00 01 00 00 00
Dec 18 13:36:25 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2637451832 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:25 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1348227788800 size=131072 flags=180980
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#11 Sense Key : Illegal Request [current]
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#11 Add. Sense: Unaligned write command
Dec 18 13:36:25 truenas kernel: sd 3:0:0:0: [sdc] tag#11 CDB: Read(16) 88 00 00 00 00 00 9d 34 56 38 00 00 01 00 00 00
Dec 18 13:36:25 truenas kernel: blk_update_request: I/O error, dev sdc, sector 2637452856 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
Dec 18 13:36:25 truenas kernel: zio pool=DataPool vdev=/dev/disk/by-partuuid/ad40036a-2bf5-4572-a75a-fb75994dbe45 error=5 type=1 offset=1348228313088 size=131072 flags=180980
Dec 18 13:36:25 truenas kernel: ata4: EH complete
Dec 18 13:36:25 truenas kernel: ata4.00: Enabling discard_zeroes_data
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 QVO 4TB
Serial Number:
LU WWN Device Id: 5 002538 f43a1d421
Firmware Version: SVQ02B6Q
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Dec 18 11:10:45 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 320) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 55 -
# 2 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 QVO 4TB
Serial Number:
LU WWN Device Id: 5 002538 f43915539
Firmware Version: SVQ02B6Q
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Dec 18 11:12:38 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 320) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 55 -
# 2 Extended offline Completed without error 00% 52 -
# 3 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 QVO 4TB
Serial Number:
LU WWN Device Id: 5 002538 f43914d4c
Firmware Version: SVQ02B6Q
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Dec 18 11:13:07 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 320) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 55 -
# 2 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 QVO 4TB
Serial Number:
LU WWN Device Id: 5 002538 f43914db9
Firmware Version: SVQ02B6Q
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Dec 18 11:15:32 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 320) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 55 -
# 2 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
256 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Oh God - Samsung QVO - they are shit
[I will admit to a mildly irrational hatred of Samsung QVO drives in particular]
Assuming async writes (normal) writes to the pool are held in RAM for up to 5 seconds as a trasaction group which is then flushed to disk whilst another transaction group is built. The nature of a COW filing system means that the data can be laid down consecutively on the disk (fast) whithout much seeking and then the metadata gets updated.
[I hope - cos thats the way I understand it]
What is important to realise is that once the TX group is built and being flushed, if the next TX group gets filled and can't immediately be flushed then the NAS will tell the sending apps to back off. Thus whilst peak write speeds can be reasonably high (due to being held in memory briefly), your longer term write can never be higher than the speed of the pool being written to. It also explains why, when a pool gets full, it slows down as ZFS can no longer flush a TX Group to consecutive blocks on the pool and has to seek which is slow
I guess the Syslog Infos in my Post are the best indicator for now ... any ideas what happend here?
That he changed even the sata speed is interesting.
yeah for now im in the mood to send these samsung drives back unfortunately ... stick with my transcend and will another crusial tls to mix the brands a little bit and get again up to 4 but first things first i have to solve this ...
So the eec ram could also be a reason here? well lucky me its the cheapest part in the build so not much of a high cost test to order another one...
i switched back to my old TLC SSDs which i think now are totally fine.
I mounted them directly to the Sata Ports with the new delock Flex Cables (they are by the way the best i ever had), and after i got so many Errors (Checksum) in my Pool and also the Apps got effected i decided to restore my backup from the external USB Drive.
First i experienced that if i only restored the ix-applications where i ran in the same errors then before i made the choice to restore the hole pool.
Well at least i made my first experience with restore data.
What i can say for now is that i DONT experienced any error while restoring where the disks gots heavy load.
Now lets see what happens if my apps running again and if then came again some errors ... could be interesting if perhaps nextcloud cause it at least.
If not then the SATA Cables could it be (many topics called it before at CRC Errors in SmartCTL). I already did changed them but who knows ...
EDIT: This OS fucked me up more and more ... no while unlocking the Child Datasets none of my Key Files worked ... i was nearly to delete everything ... then i thought okay lets just put the key into it instead of the file and tada ... it works ... wtf is going on with that thing ...
After restoring everything my apps wont load by choosing the new pool.
I got an error called: filesystem 'ix-applications/k3s/kubelet' cannot be mounted using 'mount'. Use 'zfs set mountpoint=legacy' or 'zfs mount /ix-applications/k3s/kubelet'.
First i tried: "zfs set mountpoint=/DataPool/ix-applications/k3s/kubelet DataPool/ix-applications/k3s/kubelet" with no success
but if i use: "zfs set mountpoint=legacy DataPool/ix-applications/k3s/kubelet" it works just fine ....
Why is that so? Why happens that after a recovery?
What is the difference between legacy and a straight mountpoint?
As i read more ... and that wasnt said by anyone when talking about replication tasks to back up on a usb drive ... its quite better to select choose pool in apps to replicate the apps on the new dataset and then rechoose the originally ... but i dont get why ...
EDIT okay i give up ....
next errors ... apps arent operable ... i have no clew wharts going on .... acl problems? i dont know ... i have seen many apps/ os but thats is just a mess whats going on ...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
