Error - Unsupported SCSI OP

[downingjosh]

This is a brand new server with the following specs:

FreeNAS 8.3 p1 x64
Dell T620
1x Xeon E5-2620
32 GB DDR3 1600Mhz
Network - 1x Intel I350 DP (not in use due to driver issues)
Network - 1x Intel dual-port onboard (chipset unknown)
Storage - PERC H310 RAID Controller
4x WD Red 3 TB SATA drives (RAID 10)
4x Samsung 840 Pro 256 GB drives (RAID 10)
Connecting to a vSphere 5.0 server

This is a post for an issue discovered after some troubleshooting in this Networking thread:
http://forums.freenas.org/showthread.php?11424-FreeNAS-resets-network-connections-once-per-minute

We were having issues with the I350 driver resetting every minute, so we tried the onboard NIC instead, but the main symptom is still there. The previous driver resets are gone, but now, we're seeing this entry in 'messages':

Feb 26 19:28:50 freenas istgt[2405]: istgt_lu_disk.c:6737:istgt_lu_disk_execute: ***ERROR*** unsupported SCSI OP=0x4d
Feb 26 19:28:50 freenas istgt[2405]: istgt_lu_disk.c:6737:istgt_lu_disk_execute: ***ERROR*** unsupported SCSI OP=0x4d
Feb 26 19:30:14 freenas last message repeated 2 times
Feb 26 22:05:33 freenas istgt[2405]: Login from iqn.1998-01.com.vmware:localhost-393433a0 (10.1.1.93) on iqn.2011-03.example.org.istgt:datadrive LU2 (10.1.1.91:3260,1), ISID=23d000001, TSIH=2, CID=0, HeaderDigest=off, DataDigest=off

The primary reason we started looking into this issue is that our VMware host (vSphere 5.0) was losing connection to our 'datadrive' iSCSI target. Of the 2 iSCSI targets on the server, only 1 of them is experiencing this problem. 1 is 4 SSDs (Samsung 840 Pro) and the other is 4 rotational drives (Western Digital Reds); the problem only occurs on the Western Digital Red LUN (called 'datadrive' above). The SSDs hold the guest OSes and the WD Reds are given to the file server VM to use as a data drive. This makes me think that the issue may not be strictly with the network card, but perhaps with the drives or their configuration.

I found this thread here:
http://forums.freenas.org/showthread.php?8437-unsupported-SCSI
but we have no monitoring services on the network.

My guess is that it has something to do with the behavior of those Red drives, but I just don't know what. Please let me know what further info I can give you to help. Thanks for everything so far!

 

[jgreco]

It's a SCSI feature that istgt doesn't support. It isn't an actual problem, except if perhaps you've got something that depends on the feature. Are you experiencing any operational difficulties beyond the natural fear of an ominous-sounding error message in the FreeNAS logs?

 

[downingjosh]

It's a SCSI feature that istgt doesn't support. It isn't an actual problem, except if perhaps you've got something that depends on the feature. Are you experiencing any operational difficulties beyond the natural fear of an ominous-sounding error message in the FreeNAS logs?

I wondered if the logging error might be benign (since it occurred 2.5 hours before the main issue occurred). The main issue is that the VMware host keeps losing connection to the 'datadrive' iSCSI target. The VMware error occurred at the same time this FreeNAS log entry:

Feb 26 22:05:33 freenas istgt[2405]: Login from iqn.1998-01.com.vmware:localhost-393433a0 (10.1.1.93) on iqn.2011-03.example.org.istgt:datadrive LU2 (10.1.1.91:3260,1), ISID=23d000001, TSIH=2, CID=0, HeaderDigest=off, DataDigest=off

It does not lose connectivity to the SSD iSCSI target, only the 'datadrive'. I think it may be something to do with the WD Red drives, but I have nothing else go on at the moment. Is there a particular set of log files you would find useful to help diagnose the issue?

 

[jgreco]

Check the VMware ESXi logs to see what it is disconnecting over. ZFS is marvelous at some things, but not at others, and being responsive is sometimes problematic. The log message you quote is ESXi logging back IN to the iSCSI service which implies it disconnected. Bug 1531 discusses responsiveness issues with ZFS and possible strategies to diagnose and mitigate. Follow it through - do NOT just try making random changes mentioned within because you can easily break things further.

 

[downingjosh]

Check the VMware ESXi logs to see what it is disconnecting over. ZFS is marvelous at some things, but not at others, and being responsive is sometimes problematic. The log message you quote is ESXi logging back IN to the iSCSI service which implies it disconnected. Bug 1531 discusses responsiveness issues with ZFS and possible strategies to diagnose and mitigate. Follow it through - do NOT just try making random changes mentioned within because you can easily break things further.

I tried looking through the VMware logs but did not find anything to point me in the right direction. I've attached the logs as well as a screenshot of the vSphere host events at the time of the error. View attachment VMware Host Log.zip

 

[jgreco]

I don't have an example handy, and I'm not in a good position to look at this in more detail right now (world's slowest Internet connection here). There are examples of ESXi iSCSI disconnect messages mentioned on this forum, I'm pretty sure.