ctl_datamove errors

[Cruiseader]

I'm getting some iSCSI errors with my TrueNAS 13-U3.1 and am trying to narrow down the cause. I'm seeing these errors in the logs:

Feb 10 05:42:04 drs0-pp-truenas01 ctl_datamove: tag 0x16012b00 on (12:3:0) aborted
Feb 10 05:42:04 drs0-pp-truenas01 ctl_datamove: tag 0x95002b00 on (9:3:0) aborted
Feb 10 05:42:04 drs0-pp-truenas01 ctl_datamove: tag 0x14012b00 on (12:3:0) aborted
Feb 10 08:52:25 drs0-pp-truenas01 ctl_datamove: tag 0xa6833400 on (9:3:0) aborted
Feb 10 08:52:25 drs0-pp-truenas01 ctl_datamove: tag 0x26843400 on (5:3:0) aborted
Feb 10 08:52:25 drs0-pp-truenas01 ctl_datamove: tag 0x7a853400 on (8:3:0) aborted

These align with this error on my Windows server:

"Initiator could not find a match for the initiator task tag in the received PDU. Dump data contains the entire iSCSI header."

1. Any suggestions on what might be causing these errors?
2. What do the numbers in brackets represent? (9:3:0) for example

We have TrueNAS running on a Dell Poweredge R720 with 384GB of RAM. We have 36 disks split into three RAIDZ2 vdevs (12 disks each).

 

[NickF]

Without knowing a little bit more about the workload I'm not sure I could help. I can absolutely tell you that this error happens when there is alot of latency between the initiator and the target LUN. This could be caused by either the network in between or it could be because you are hitting an IOP limitation from your workload. 3 RAIDZ2s are not exactly racecar status, but I'm not sure what it is you are doing. There is certainly a reason I changed my production homelab (oxymoron?) box to mirrors.

 

[mav@]

@NickF is right, it is likely caused by latency, when initiator gives up waiting for response and aborts the requests. Considering use of too wide RAIDZ2, your pool is likely just unable to provide required IOPS performance. The numbers in brackets mean initiator, port and lun respectively for the command aborted.

 

[jgreco]

We have TrueNAS running on a Dell Poweredge R720 with 384GB of RAM. We have 36 disks split into three RAIDZ2 vdevs (12 disks each).

You should not be using RAIDZ for block storage. Please see

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Additionally, the PowerEdge servers typically come with RAID controllers rather than HBA's. What are you using to attach your HDD's in this system?