SOLVED iSCSI - getting it working?

[Stilez]

This should be easy but I'm totally lost. As far as I can tell Im doing what I should, both in FreeNAS and Windows..

I get all kinds of results, but none usable.

• Sometimes the iSCSI iniiator in WIndsows tells me its connecxted, Device Manager shows an iSCSI device "Working", but the disk isnt visible in Diskpart or DiskManagement MMC.
• Or it shows up in Disk Management but as 16384 MB compared to 100GB in FreeNAS ZVOL
• Or it shows up as 100GB but when I click to initialise, it spends hours spinning and doesnt seem to do anything, and I have to reboot WIndows to free it.
• If I try to "disconnect" at the client, it also seems to just take forever and again, I reboot windows.

Every now and then it pulls a new trick. Just now, trying yet again, it's detected the target on Windows startup, the disk is sitting in "Device Manager", but so far has spent 20 minutes trying to scan for the new disk that should appear automatically, to actually initialise and use it

.​

Could someone please guide me from scratch, how to set it up? I don't need complicated, just a "proof of concept" that it works. If I can define a 100GB ZVOL, consistently detect and connect to it, format it NTFS from a single Windows 10 machine, and store files on it, that would be ample

I don't know if I screwed up any settings, related to iSCSI. I cant think of any as they are all very easy GUI and everything else works fine.

Help pleasejust to walk me through step by step, to get it basic working.

 

[Stilez]

And now a brand new failure mode. The UI says its a 100GB ZVOL. Which is correct. newly created and never used.

but on the client, diskmgmt.msc detects it as 16384 GB, and doesn't show a right context menu to initialize it, while diskpart detects it as 100GB and won't let it be partitioned.

I've deleted and recreated the ZVOL and diskmgmt.msc now recognises it but...

Except there's no sign of disk error anywhere. ZFS and server side logs don't seem to have any relevant messages and the pool has just been scrubbd and smartctl'ed, every device is fine, as is the LAN connection.

Like I said, help please? No idea whats up

 

[Samuel Tai]

First, make sure the iSCSI service is running.

Next, create a zvol. Set a convenient size, and leave all other settings at default.

Now, create the iSCSI share. The Target Global Configuration mostly sets the service base name. Leave this at the default.

The Portal is how you advertise services to initiators, i.e., iSCSI clients.

The only thing to do here is to set the IP address.

On the Initiator tab, set the client restrictions. For simplicity, you can allow all initiators.

The Target tab defines services that will be advertised on the portal to initiators.

Extents define the zvol you want to serve to initiators.

Finally, Associated Targets connect the zvol extent to the target listing in the portal.

This is pretty much all that needs to be done on the FreeNAS side.

On the Windows side, open iSCSI Initiator Properties from Administrative Tools, and go to the Discovery tab. Click the "Discover Portal" button:

Enter the same portal IP as defined on FreeNAS's Portal tab. Now, on the Targets tab, you should see the target service with the FreeNAS global base name listed. Click the "Connect" button to attach to the target.

The status should change to Connected, and the target should now appear as a disk in Disk Management.

 

[Samuel Tai]

Continuing, due to the 10 attachment limit.

After defining a partition scheme, the disk with the appropriate size should appear.

Right-click the unallocated space to define a volume and format it.

To detach this volume, first check if Task Manager is running. Exit Task Manager if it is, as Task Manager will hold onto the initiator connection, preventing disconnection.

In the iSCSI Initiator Properties, click "Disconnect" to disconnect the target session.

Acknowledge the disconnection:

And you'll see the target status changed

 

[ornias]

I can confirm the way @Samuel Tai did it, this is precisely the way i did (step by step) and should work just fine....

 

[Stilez]

@Samuel Tai - thanks! But it's not working? Instructions carefully followed.....
@ornias - thanks also, same.

It seems to work fine right up to where I initialise the disk in diskmgmt.msc, and then hangs at the spinner. There are log messages on the NAS side, but I can't see any that reflect any error in input. Screenshots below:

I set up the settings, exactly as shown...

I set up the Windows side initiator, exactly as shown...

So far so good.

I connect. The iSCSI target disk shows up as a device in Device Manager, but randomly does or doesn't in diskmgmt.msc. This time it doesn't. So I click "rescan disks" in diskmgmt.msc, and the spinner appears.

15 minutes later it's still spinning.

There's nothing in /var/logs except this all day in /messages:

Aug 4 09:37:01 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:38:07 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:39:13 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:40:19 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:41:25 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:42:31 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:43:37 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:44:43 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:45:49 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:46:59 TrueNAS-Box WARNING[3488]: Last message 'CLIENT_IPADDRESS (iqn.200' repeated 4 times, suppressed by syslog-ng on TrueNAS-Box.earlobe.site
Aug 4 09:47:35 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:48:44 TrueNAS-Box WARNING[3488]: Last message 'CLIENT_IPADDRESS (iqn.200' repeated 4 times, suppressed by syslog-ng on TrueNAS-Box.earlobe.site
Aug 4 09:49:21 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:50:30 TrueNAS-Box WARNING[3488]: Last message 'CLIENT_IPADDRESS (iqn.200' repeated 4 times, suppressed by syslog-ng on TrueNAS-Box.earlobe.site
Aug 4 12:38:36 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:39:42 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:40:48 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:41:54 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:43:00 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:44:06 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:45:12 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection

And in Windows Event Log, quite a few more helpful messages. Several of each of these:

• event 7: "The initiator could not send an iSCSI PDU"
• event 251 "Connection to the target was lost"
• event 39 "Initiator sent a task management command to reset the target"
• event 245 "Target failed to respond in time to a Task Management request"
• event 44 "The IO operation at logical block address (random block) for Disk 5 (PDO name: \Device\000000b0) was retried" - disk 05 would be the iSCSI target judging by the existing disks going up to disk 4.

A connection issue sounds plausible. But Ive never configured anything for iScsi or any but default "out of the box" firewalling on either box (they're on the same switch). So if I have a config issue, everyone else should have, too. Worth noting that I'm also using PuTTY+SSH to connect and it's never dropped out, for days on end, and I'm not seeing network dropout at either end. So a problem at network level seems unlikely.

 

[Samuel Tai]

@Samuel Tai - thanks! But it's not working? Instructions carefully followed.....
@ornias - thanks also, same.

There's nothing in /var/logs except this all day in /messages:

Aug 4 09:37:01 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:38:07 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:39:13 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:40:19 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:41:25 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:42:31 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:43:37 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:44:43 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:45:49 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:46:59 TrueNAS-Box WARNING[3488]: Last message 'CLIENT_IPADDRESS (iqn.200' repeated 4 times, suppressed by syslog-ng on TrueNAS-Box.earlobe.site
Aug 4 09:47:35 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:48:44 TrueNAS-Box WARNING[3488]: Last message 'CLIENT_IPADDRESS (iqn.200' repeated 4 times, suppressed by syslog-ng on TrueNAS-Box.earlobe.site
Aug 4 09:49:21 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 09:50:30 TrueNAS-Box WARNING[3488]: Last message 'CLIENT_IPADDRESS (iqn.200' repeated 4 times, suppressed by syslog-ng on TrueNAS-Box.earlobe.site
Aug 4 12:38:36 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:39:42 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:40:48 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:41:54 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:43:00 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:44:06 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection
Aug 4 12:45:12 TrueNAS-Box WARNING: CLIENT_IPADDRESS (iqn.2000-01.client-iscsi-address): no ping reply (NOP-Out) after 5 seconds; dropping connection

Hmm. This sounds like you never allowed iSCSI traffic through the Windows client's firewall. Have you configured the client to be on a Private network or a Public network?

And in Windows Event Log, quite a few more helpful messages. Several of each of these:

• event 7: "The initiator could not send an iSCSI PDU"
• event 251 "Connection to the target was lost"
• event 39 "Initiator sent a task management command to reset the target"
• event 245 "Target failed to respond in time to a Task Management request"
• event 44 "The IO operation at logical block address (random block) for Disk 5 (PDO name: \Device\000000b0) was retried" - disk 05 would be the iSCSI target judging by the existing disks going up to disk 4.

A connection issue sounds plausible. But Ive never configured anything for iScsi or any but default "out of the box" firewalling on either box (they're on the same switch). So if I have a config issue, everyone else should have, too. Worth noting that I'm also using PuTTY+SSH to connect and it's never dropped out, for days on end, and I'm not seeing network dropout at either end. So a problem at network level seems unlikely.

Check the client to make sure there are iSCSI permits in place in the Windows firewall advanced view in Administrative Tools.

 

[Stilez]

Found it.
I have another puzzle, but I think I've found the original major issue.

It might have been the Wiindows Firewall, but that didn't actually seem to help. So I threw tcpdump at it to check packets were flowing both ways, and that triggered a memory about some old bug where some enterprise network cards don't get on well with full Jumbo frames over 4000 bytes. My LAN uses mtu 9000 throughout. So I took mtu down to 4000 on server and client, and redid the Windows Firewall, and that fixed it.

Partly.

iSCSI now responds properly. It's prompt when I connect/disconnect, as well, or mostly so. It hasn't tried to badly hang again.

In addition, an issue I had with Samba also seems to have fixed itself..... and in hindsight that issue also looks like maybe dropped connections (dir lists didnt show at client but did show in smbd log as being crawled and presumably sent)

My remaining iSCSI problem is that the disk which does show up, shows up as read-only!?!

And I didn't accidentally check the "read only" option in defining the extent, or in the ZVOL's properties!

Putting MTU at the client back to 9K kills everything instantly. Reverting MTU to 4K fixes it all, instantly. Any clues either about whether Jumbo Frames are a "known issue" and I missed the memo, and if so how big a problem?

Meantime, this "read only" thing now looks like my remaining iSCSI issue...?

 

[Samuel Tai]

Jumbo frames over 4000 sounds like a driver issue. Read-only sounds like an incomplete firewall permit.

 

[Stilez]

Jumbo frames over 4000 sounds like a driver issue. Read-only sounds like an incomplete firewall permit.

I checked and it wasn't. But apparently "iscsi read only" is distinctive enough for Google to be ones friend (https://www.digistor.com.au/the-latest/isci-windows-server-clients-techblog)

No idea how, but the read-only attribute got set on it. Maybe during the connections, some corruption occurred? If anyone hits this in future, here's the fix:

DISKPART​

LIST DISK​

SELECT DISK <WHATEVER NUMBER>​

DETAIL DISK (AND CONFIRM iSCSI DISK AND IT SAYS "READ ONLY")​

ATTRIBUTES DISK CLEAR READONLY​

EXIT​

And I now have fully working iSCSI! Thank you both!!