KristijanL
Dabbler
- Joined
- Jun 28, 2013
- Messages
- 13
Hi,
i have been running personal storage from FreeNAS 9 up to TrueNAS Core latest version, then imported the same zfs pool into a fresh install of TrueNAS SCALE 21.06 BETA, i was enjoining the beta, no issues in the general usage ( iscsi shares, SMB shares, few users, few docker apps ), but when one of my drive failed i bought a new one and try to replace it, but i got weird errors.
The drive i am trying to replace is shown as SDD and there are a lot of errors of this type
i have try to replace the new drive with another new drive, same issues, any suggestions?
the only thing that i could find is diffrent from other drives is block size
current status of my pool
dmesg logs
i have been running personal storage from FreeNAS 9 up to TrueNAS Core latest version, then imported the same zfs pool into a fresh install of TrueNAS SCALE 21.06 BETA, i was enjoining the beta, no issues in the general usage ( iscsi shares, SMB shares, few users, few docker apps ), but when one of my drive failed i bought a new one and try to replace it, but i got weird errors.
The drive i am trying to replace is shown as SDD and there are a lot of errors of this type
Code:
blk_update_request: I/O error, dev sdd, sector 7814036920 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
i have try to replace the new drive with another new drive, same issues, any suggestions?
the only thing that i could find is diffrent from other drives is block size
Code:
truenas# blockdev --getbsz /dev/sdd 4096 truenas# blockdev --getbsz /dev/sdd1 512 truenas# blockdev --getbsz /dev/sdd2 512 truenas# blockdev --getbsz /dev/sdg 4096 truenas# blockdev --getbsz /dev/sdg1 4096 truenas# blockdev --getbsz /dev/sdg2 4096
current status of my pool
Code:
NAME STATE READ WRITE CKSUM vault DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 893ae457-7120-11eb-9ce5-000c2903bd67 ONLINE 0 0 0 d3c182f5-7876-11eb-bb89-000c2903bd67 ONLINE 0 0 0 eeb35ab5-e897-11e5-971d-0026558310c6 ONLINE 0 0 0 504962f6-188b-11e9-a15d-000c2973b683 ONLINE 0 0 0 0ca0a654-6b22-11eb-a445-000c2903bd67 ONLINE 0 0 0 4581ec09-1554-4a61-8309-5226187393e6 FAULTED 0 155 0 too many errors (resilvering) logs mirror-1 ONLINE 0 0 0 9db4f4d0-189c-11e9-a15d-000c2973b683 ONLINE 0 0 0 a1e2a28f-189c-11e9-a15d-000c2973b683 ONLINE 0 0 0 cache a45001f9-189c-11e9-a15d-000c2973b683 ONLINE 0 0 0 a6b4d552-189c-11e9-a15d-000c2973b683 ONLINE 0 0 0
dmesg logs
Code:
[1445357.177300] end_device-3:8: add: handle(0x000a), sas_addr(0x4433221101000000) [1445357.181921] sd 3:0:8:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) [1445357.183837] sd 3:0:8:0: [sdd] 4096-byte physical blocks [1445357.190670] sd 3:0:8:0: [sdd] Write Protect is off [1445357.192464] sd 3:0:8:0: [sdd] Mode Sense: 7f 00 10 08 [1445357.194117] sd 3:0:8:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [1445357.274387] sdd: sdd1 [1445357.284463] sd 3:0:8:0: [sdd] Attached SCSI disk [1445404.929893] sdd: sdd1 [1445415.885731] sdd: sdd1 [1445421.140129] sdd: sdd1 sdd2 [1455131.570902] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1455131.570968] sd 3:0:8:0: [sdd] tag#2688 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s [1455131.572823] sd 3:0:8:0: [sdd] tag#2751 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=6s [1455131.576075] sd 3:0:8:0: [sdd] tag#2688 CDB: Write(16) 8a 00 00 00 00 00 cc bb 97 c0 00 00 00 80 00 00 [1455131.579440] sd 3:0:8:0: [sdd] tag#2751 Sense Key : Illegal Request [current] [1455131.579449] sd 3:0:8:0: [sdd] tag#2751 Add. Sense: Logical block address out of range [1455131.582882] blk_update_request: I/O error, dev sdd, sector 3434846144 op 0x1:(WRITE) flags 0x700 phys_seg 16 prio class 0 [1455131.584753] sd 3:0:8:0: [sdd] tag#2751 CDB: Write(16) 8a 00 00 00 00 00 cc bb 97 80 00 00 00 38 00 00 [1455131.586607] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=5 type=2 offset=1756493676544 size=65536 flags=40080caa [1455131.590137] blk_update_request: critical target error, dev sdd, sector 3434846080 op 0x1:(WRITE) flags 0x700 phys_seg 7 prio class 0 [1455131.601170] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=121 type=2 offset=1756493643776 size=28672 flags=40080caa [1455139.221154] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1455139.223252] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1455139.225214] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1455139.227243] sd 3:0:8:0: [sdd] tag#2724 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s [1455139.230915] sd 3:0:8:0: [sdd] tag#2724 Sense Key : Illegal Request [current] [1455139.232789] sd 3:0:8:0: [sdd] tag#2724 Add. Sense: Logical block address out of range [1455139.234717] sd 3:0:8:0: [sdd] tag#2724 CDB: Write(16) 8a 00 00 00 00 00 cc bb 98 48 00 00 00 38 00 00 [1455139.238288] blk_update_request: critical target error, dev sdd, sector 3434846280 op 0x1:(WRITE) flags 0x700 phys_seg 7 prio class 0 [1455139.242096] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=121 type=2 offset=1756493746176 size=28672 flags=40080caa [1455149.832535] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1455149.832563] sd 3:0:8:0: [sdd] tag#2737 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s [1455149.834679] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1455149.838717] sd 3:0:8:0: [sdd] tag#2737 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 bb b8 00 00 00 08 00 00 [1455149.845009] blk_update_request: I/O error, dev sdd, sector 7814036408 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1455149.849114] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=5 type=2 offset=3998639091712 size=4096 flags=180ac0 [1455149.853399] sd 3:0:8:0: [sdd] tag#2738 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s [1455149.857470] sd 3:0:8:0: [sdd] tag#2738 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 bd b8 00 00 00 08 00 00 [1455149.861619] blk_update_request: I/O error, dev sdd, sector 7814036920 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1455149.865828] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=5 type=2 offset=3998639353856 size=4096 flags=180ac0 [1455149.870252] sd 3:0:8:0: [sdd] tag#2736 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s [1455149.874416] sd 3:0:8:0: [sdd] tag#2736 Sense Key : Illegal Request [current] [1455149.876627] sd 3:0:8:0: [sdd] tag#2736 Add. Sense: Logical block address out of range [1455149.878843] sd 3:0:8:0: [sdd] tag#2736 CDB: Write(16) 8a 00 00 00 00 00 00 40 03 b8 00 00 00 08 00 00 [1455149.882866] blk_update_request: critical target error, dev sdd, sector 4195256 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1455149.887150] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=121 type=2 offset=421888 size=4096 flags=180ac0 [1456001.139496] INFO: task txg_sync:4797 blocked for more than 120 seconds. [1456001.141781] Tainted: P OE 5.10.42+truenas #1 [1456001.144031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1456001.146292] task:txg_sync state:D stack: 0 pid: 4797 ppid: 2 flags:0x00004000 [1456001.148651] Call Trace: [1456001.150675] __schedule+0x282/0x870 [1456001.152746] schedule+0x46/0xb0 [1456001.154708] cv_wait_common+0x14e/0x290 [spl] [1456001.156730] ? add_wait_queue_exclusive+0x70/0x70 [1456001.158868] arc_read+0x1d0/0x19d0 [zfs] [1456001.160937] ? arc_can_share+0x90/0x90 [zfs] [1456001.162823] ? __kmalloc_node+0x141/0x2b0 [1456001.164727] ? spl_kmem_alloc_impl+0x75/0xc0 [spl] [1456001.166708] dsl_scan_visitbp+0x4ff/0xd60 [zfs] [1456001.168578] ? kfree+0xc3/0x480 [1456001.170438] ? dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.172371] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.174221] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.176069] dsl_scan_visitbp+0x7a9/0xd60 [zfs] [1456001.177810] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.179563] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.181257] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.182885] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.184502] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.186002] dsl_scan_visitbp+0x591/0xd60 [zfs] [1456001.187509] dsl_scan_visitbp+0x3a6/0xd60 [zfs] [1456001.188905] dsl_scan_visit_rootbp+0xf5/0x1e0 [zfs] [1456001.190279] dsl_scan_visitds+0x140/0x450 [zfs] [1456001.191513] ? spl_kmem_alloc_impl+0x75/0xc0 [spl] [1456001.192700] ? sched_clock+0x5/0x10 [1456001.193795] ? sched_clock_cpu+0xc/0xa0 [1456001.194904] ? __smp_call_single_queue+0x23/0x40 [1456001.196051] ? x2apic_send_IPI+0x49/0x50 [1456001.197083] ? __kmalloc_node+0x141/0x2b0 [1456001.198080] ? spl_kmem_alloc_impl+0x75/0xc0 [spl] [1456001.199102] ? __raw_spin_unlock+0x5/0x10 [spl] [1456001.200128] ? tsd_hash_search+0x75/0xa0 [spl] [1456001.201067] ? __list_add+0x12/0x30 [spl] [1456001.201965] ? __raw_spin_unlock+0x5/0x10 [spl] [1456001.202866] ? tsd_hash_add+0x14a/0x200 [spl] [1456001.203931] dsl_scan_visit+0x1b4/0x340 [zfs] [1456001.204912] dsl_scan_sync+0x5ed/0xb20 [zfs] [1456001.205891] ? spa_sync_iterate_to_convergence+0x12e/0x310 [zfs] [1456001.206982] spa_sync_iterate_to_convergence+0x139/0x310 [zfs] [1456001.208117] spa_sync+0x2e6/0x860 [zfs] [1456001.209121] txg_sync_thread+0x2ef/0x440 [zfs] [1456001.210155] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs] [1456001.211063] thread_generic_wrapper+0x78/0xb0 [spl] [1456001.212025] ? IS_ERR+0x10/0x10 [spl] [1456001.212872] kthread+0x11b/0x140 [1456001.213684] ? __kthread_bind_mask+0x60/0x60 [1456001.214551] ret_from_fork+0x22/0x30 [1528186.139816] sd 3:0:8:0: device_block, handle(0x000a) [1528187.639883] sd 3:0:8:0: device_unblock and setting to running, handle(0x000a) [1528187.644797] sd 3:0:8:0: [sdd] Synchronizing SCSI cache [1528187.645856] sd 3:0:8:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK [1528187.712981] mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221101000000) [1528187.714327] mpt2sas_cm0: removing handle(0x000a), sas_addr(0x4433221101000000) [1528187.715575] mpt2sas_cm0: enclosure logical id(0x5003048000000001), slot(2) [1528335.641931] scsi 3:0:9:0: Direct-Access ATA WDC WD40EFAX-68J 0A83 PQ: 0 ANSI: 6 [1528335.643444] scsi 3:0:9:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x0000000000000000) [1528335.645991] scsi 3:0:9:0: enclosure logical id (0x5003048000000001), slot(2) [1528335.647741] scsi 3:0:9:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) [1528335.649249] scsi 3:0:9:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) [1528335.653608] sd 3:0:9:0: Attached scsi generic sg3 type 0 [1528335.653677] sd 3:0:9:0: Power-on or device reset occurred [1528335.655191] end_device-3:9: add: handle(0x000a), sas_addr(0x4433221101000000) [1528335.659191] sd 3:0:9:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) [1528335.660747] sd 3:0:9:0: [sdd] 4096-byte physical blocks [1528335.665504] sd 3:0:9:0: [sdd] Write Protect is off [1528335.666954] sd 3:0:9:0: [sdd] Mode Sense: 7f 00 10 08 [1528335.667971] sd 3:0:9:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [1528335.749868] sdd: sdd1 [1528335.759338] sd 3:0:9:0: [sdd] Attached SCSI disk [1528380.440457] sdd: sdd1 [1528391.413781] sdd: sdd1 [1528391.509664] sdd: sdd1 [1528391.522038] sdd: sdd1 sdd2 [1528396.516379] sdd: sdd1 sdd2 [1535906.783268] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535906.783295] sd 3:0:9:0: [sdd] tag#2124 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s [1535906.784959] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535906.787910] sd 3:0:9:0: [sdd] tag#2124 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 bc 30 00 00 00 08 00 00 [1535906.792653] blk_update_request: I/O error, dev sdd, sector 7814036528 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1535906.795948] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=3998639153152 size=4096 flags=180ac0 [1535906.799715] sd 3:0:9:0: [sdd] tag#2125 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s [1535906.803195] sd 3:0:9:0: [sdd] tag#2125 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 be 30 00 00 00 08 00 00 [1535906.806624] blk_update_request: I/O error, dev sdd, sector 7814037040 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1535906.810148] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=3998639415296 size=4096 flags=180ac0 [1535906.813890] sd 3:0:9:0: [sdd] tag#2123 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s [1535906.817463] sd 3:0:9:0: [sdd] tag#2123 Sense Key : Illegal Request [current] [1535906.819404] sd 3:0:9:0: [sdd] tag#2123 Add. Sense: Logical block address out of range [1535906.821318] sd 3:0:9:0: [sdd] tag#2123 CDB: Write(16) 8a 00 00 00 00 00 00 40 04 30 00 00 00 08 00 00 [1535906.824910] blk_update_request: critical target error, dev sdd, sector 4195376 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1535906.828671] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=483328 size=4096 flags=180ac0 [1535922.556636] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535922.558883] sd 3:0:9:0: [sdd] tag#2117 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=8s [1535922.558910] sd 3:0:9:0: [sdd] tag#2135 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s [1535922.562831] sd 3:0:9:0: [sdd] tag#2117 Sense Key : Illegal Request [current] [1535922.566913] sd 3:0:9:0: [sdd] tag#2135 CDB: Write(16) 8a 00 00 00 00 00 24 56 47 30 00 00 01 38 00 00 [1535922.569137] sd 3:0:9:0: [sdd] tag#2117 Add. Sense: Logical block address out of range [1535922.573356] blk_update_request: I/O error, dev sdd, sector 609634096 op 0x1:(WRITE) flags 0x700 phys_seg 39 prio class 0 [1535922.575661] sd 3:0:9:0: [sdd] tag#2117 CDB: Write(16) 8a 00 00 00 00 00 24 56 44 08 00 00 02 88 00 00 [1535922.580070] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=309985107968 size=159744 flags=40080caa [1535922.584294] blk_update_request: critical target error, dev sdd, sector 609633288 op 0x1:(WRITE) flags 0x700 phys_seg 81 prio class 0 [1535922.593171] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=309984694272 size=331776 flags=40080caa [1535930.133065] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535930.133144] sd 3:0:9:0: [sdd] tag#2127 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s [1535930.135469] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535930.139787] sd 3:0:9:0: [sdd] tag#2127 CDB: Write(16) 8a 00 00 00 00 00 24 56 49 98 00 00 00 78 00 00 [1535930.142093] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535930.142097] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535930.150909] blk_update_request: I/O error, dev sdd, sector 609634712 op 0x1:(WRITE) flags 0x700 phys_seg 15 prio class 0 [1535930.155234] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=309985423360 size=61440 flags=40080caa [1535930.159836] sd 3:0:9:0: [sdd] tag#2125 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s [1535930.164216] sd 3:0:9:0: [sdd] tag#2125 Sense Key : Illegal Request [current] [1535930.166509] sd 3:0:9:0: [sdd] tag#2125 Add. Sense: Logical block address out of range [1535930.168748] sd 3:0:9:0: [sdd] tag#2125 CDB: Write(16) 8a 00 00 00 00 00 24 53 27 10 00 00 00 08 00 00 [1535930.172938] blk_update_request: critical target error, dev sdd, sector 609429264 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1535930.177261] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=309880233984 size=4096 flags=1808aa [1535952.457084] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000) [1535952.457163] sd 3:0:9:0: [sdd] tag#2158 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s [1535952.459489] sd 3:0:9:0: [sdd] tag#2142 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=9s [1535952.463692] sd 3:0:9:0: [sdd] tag#2158 CDB: Write(16) 8a 00 00 00 00 00 24 5b c2 48 00 00 00 08 00 00 [1535952.467952] sd 3:0:9:0: [sdd] tag#2142 Sense Key : Illegal Request [current] [1535952.472123] blk_update_request: I/O error, dev sdd, sector 609993288 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0 [1535952.474365] sd 3:0:9:0: [sdd] tag#2142 Add. Sense: Logical block address out of range [1535952.478602] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=310169014272 size=4096 flags=1808aa [1535952.480857] sd 3:0:9:0: [sdd] tag#2142 CDB: Write(16) 8a 00 00 00 00 00 24 5b c1 88 00 00 00 38 00 00 [1535952.489342] blk_update_request: critical target error, dev sdd, sector 609993096 op 0x1:(WRITE) flags 0x700 phys_seg 7 prio class 0 [1535952.493633] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=310168915968 size=28672 flags=40080caa [1535960.088166] sd 3:0:9:0: [sdd] tag#2162 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s [1535960.092453] sd 3:0:9:0: [sdd] tag#2162 Sense Key : Illegal Request [current] [1535960.094758] sd 3:0:9:0: [sdd] tag#2162 Add. Sense: Logical block address out of range [1535960.097023] sd 3:0:9:0: [sdd] tag#2162 CDB: Write(16) 8a 00 00 00 00 00 24 5b c2 e0 00 00 00 18 00 00 [1535960.101229] blk_update_request: critical target error, dev sdd, sector 609993440 op 0x1:(WRITE) flags 0x700 phys_seg 3 prio class 0 [1535960.105555] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=310169092096 size=12288 flags=40080caa