FreeNAS 11.2u7 newly created iSCSI device not showing up in ESXi 6.7u3 - duplicate extent serial number

[tomobott]

Hi,
I have a configured and working FreeNAS 1U certified storage device (purchased from iXSystems in 2019), that has been running well with dual 10GiE connections, and iSCSI shares. This device has had two iSCSI shares for several month now without issue.

I recently had a new requirement for storage, and configured two new ZVOLs on my zpool for the FreeNAS, and configured the ZVOLs for iSCSI following the pattern of my two other ZVOLs. Unfortunately, my ESXi hosts (there are 3) cannot connect to one of the two new devices.

On the ESXi side the 2 newly defined targets are visible, and show up in the list of available paths (seen using esxcli storage core path list), however the device does not show up in ESXi. On investigation in the vmkernel.log file shows errors registering a name to one of the two devices. Here's an excerpt of the log:

Code:

2020-04-05T19:37:06.256Z cpu4:2099602 opID=e456f97e)VMWARE SCSI Id: Id for vmhba64:C0:T7:L0 0x62 0x34 0x39 0x36 0x39 0x31 0x34 0x36 0x64 0x32 0x30 0x34 0x30 0x31 0x00 0x00 0x69 0x53 0x43 0x53 0x49 0x20
2020-04-05T19:37:06.256Z cpu4:2099602 opID=e456f97e)ScsiUid: 435: Existing device naa.6589cfc0000002281ce7b4c311cdf6f2 already has uid vml.010000000062343936393134366432303430310000695343534920
2020-04-05T19:37:06.256Z cpu4:2099602 opID=e456f97e)ScsiDevice: 4925: Failing registration of device 'naa.6589cfc0000003d76064531dc405b977'

When I look closely at the FreeNAS iSCSI config under the Extends configuration, it shows that two of my devices were assigned the same Serial Number. Below there is a snippet of my /etc/ctl.conf file from my FreeNAS showing the extent definitions with the same ID.

Is this duplicate serial number the cause of the duplicate vml name that ESX is creating? Can I resolve this by modifying the serial number of the offending device?
Of course, any other suggestions are welcome!

Code:

root@freenas[~]# cat /etc/ctl.conf
portal-group "default" {
}

portal-group "pg1" {
        tag "0x0001"
        discovery-filter "portal-name"
        discovery-auth-group "no-authentication"
        listen "192.168.2.4:3260"
        listen "192.168.1.4:3260"
        option "ha_shared" "on"
}

## Here's one of the two disks with duplicate serial -- this is the "new" one
lun "data_1" {
        ctl-lun "4"
        path "/dev/zvol/bmlfwbmfnasp01-zpool/data_1"
        blocksize "512"
        serial "b4969146d20401"
        device-id "iSCSI Disk      b4969146d20401                 "
        option "vendor" "FreeNAS"
        option "product" "iSCSI Disk"
        option "revision" "0123"
        option "naa" "0x6589cfc0000003d76064531dc405b977"
        option "pool-avail-threshold" "25508669764403"
        option "rpm" "1"
}

lun "db_bk_1" {
        ctl-lun "0"
        path "/dev/zvol/bmlfwbmfnasp01-zpool/db_bk_1"
        blocksize "512"
        serial "b4969146d20400"
        device-id "iSCSI Disk      b4969146d20400                 "
        option "vendor" "FreeNAS"
        option "product" "iSCSI Disk"
        option "revision" "0123"
        option "naa" "0x6589cfc000000df46582e219aef9e032"
        option "pool-avail-threshold" "25508669764403"
        option "rpm" "1"
}

## Here is the other of the two disks with duplicate serial -- this is the "old" one
lun "misc_bk_1" {
        ctl-lun "1"
        path "/dev/zvol/bmlfwbmfnasp01-zpool/misc_bk_1"
        blocksize "512"
        serial "b4969146d20401"
        device-id "iSCSI Disk      b4969146d20401                 "
        option "vendor" "FreeNAS"
        option "product" "iSCSI Disk"
        option "revision" "0123"
        option "naa" "0x6589cfc0000002281ce7b4c311cdf6f2"
        option "pool-avail-threshold" "25508669764403"
        option "rpm" "1"
}

lun "os_1" {
        ctl-lun "3"
        path "/dev/zvol/bmlfwbmfnasp01-zpool/os_1"
        blocksize "512"
        serial "b4969146d20403"
        device-id "iSCSI Disk      b4969146d20403                 "
        option "vendor" "FreeNAS"
        option "product" "iSCSI Disk"
        option "revision" "0123"
        option "naa" "0x6589cfc00000064fe57fe87af0ca57ee"
        option "pool-avail-threshold" "25508669764403"
        option "rpm" "1"
}

[snip]

 

[Dan Gratton]

Seems we just came across a similar issue. I would suggest Reporting this bug in their bug tracking system -> https://jira.ixsystems.com/secure/Dashboard.jspa (or click on Report a bug a the top of this page). I'd love to follow that thread/bug report to see what ixSystems says.

Thanks

 

[tomobott]

OK, I've resolved my issue. If you edit the extent, and manually change the Serial number to something not used (I chose b4969146d29c02) the device starts showing up in ESX. NOTE: I deleted and recreated the extent, and the duplicate serial number was recreated. All the devices had different ZVOL sizes, so it was clear I was choosing the correct device when creating the extent.

Others on our team have also seen similar behavior, if an extent was initially created incorrectly to point to a wrong device, and later updated, the serial number is not upudated. The fix in that case is to remove the extent, and recreate correctly.

 

[shpokas]

I see the same behaviour with FreeNAS-11.3-U2.1. Need to manually edit serial after iSCSI wizard. Looks like a really old bug that noone wants to fix.
https://redmine.ixsystems.com/issues/6947