SOLVED Quick Help iSCSI multiple portals/targets?

[snowmirage]

I have had 2 iSCSI "shares" setup on my FreeNAS box for over a year now and as far as I know everything has been working fine. Today I went to add a third and in the process noticed something that has me confused / baffled.

I have 3 portals setup as follows

Portal Group ID Listen Comment
1 0.0.0.0:3260 desktop-portal
2 0.0.0.0:3261 syslog-portal
3 0.0.0.0:3262 esxi-portal

all of which are associated with different targets and file extents.

#1 and #2 have been working for over a year. When I went to add the 3rd for my esxi server I could not get it to discover a device when I pointed the discovery to port 3262 for the IP of my FreeNAS box. But if I pointed it to port 3260 it discovered a device.

In troubleshooting I noticed that my syslog server and my desktop which both have working iSCSI devices setup appear to both be connecting to the FreeNAS server over port 3260! (verified in tcpdumps)

I fear that both the syslog box and my desktop are some how connected to the same iSCSI file extend. I can't imagine they would be working this whole time if the system would even allow that to happen.

If my syslog box is connecting to the correct extents wouldn't it HAVE to be using port 3261 based on my config? If all iSCSI connection go through one port I don't see how the system differentiates which traffic is for which file extent if you are not using authentication (which I am not I have them all using the same "Initiators" group)

Since its a block layer device I can't just ssh to the FreeNAS box cat out a file or look a directory to make sure each iSCSI client is connecting to the right "share".

Can anyone suggest some way I can verify they are connecting to the right file extents? I can't even go monitor the size of the files on the FreeNAS box since the file extent size is set when its created.

 

[jpaetzel]

Well, because you are using port 3261 the biggest diagnostic tool that could be used (The LUC and istgtcontrol) is a non option.

Can you attatch the output of sockstat -l and the contents of /usr/local/etc/istgt/istgt.conf please?

Generally the way you'd wire things down when not using users is via the authorized initiators.

 

[snowmirage]

Code:

[root@freenas8] /mnt/Cloud1-ZFS/iSCSI-syslog# sockstat -l
USER    COMMAND    PID  FD PROTO  LOCAL ADDRESS        FOREIGN ADDRESS
www      nginx      94013 7  tcp4  *:80                  *:*
www      nginx      94013 8  tcp6  *:80                  *:*
root    istgt      71145 12 tcp4  *:3260                *:*
root    istgt      71145 13 tcp4  *:3261                *:*
root    istgt      71145 14 tcp4  *:3262                *:*
root    winbindd  41682 18 stream /tmp/.winbindd/pipe
root    winbindd  41682 19 stream /var/db/samba/winbindd_privileged/pipe
root    winbindd  6726  18 stream /tmp/.winbindd/pipe
root    winbindd  6726  19 stream /var/db/samba/winbindd_privileged/pipe
root    winbindd  4856  18 stream /tmp/.winbindd/pipe
root    winbindd  4856  19 stream /var/db/samba/winbindd_privileged/pipe
root    mountd    4710  5  udp6  *:807                *:*
root    mountd    4710  6  tcp6  *:807                *:*
root    mountd    4710  7  udp4  *:807                *:*
root    mountd    4710  8  tcp4  *:807                *:*
root    rpcbind    4706  4  udp6  *:*                  *:*
root    rpcbind    4706  5  stream /var/run/rpcbind.sock
root    rpcbind    4706  6  udp6  *:111                *:*
root    rpcbind    4706  7  udp6  *:707                *:*
root    rpcbind    4706  8  tcp6  *:111                *:*
root    rpcbind    4706  9  udp4  *:111                *:*
root    rpcbind    4706  10 udp4  *:769                *:*
root    rpcbind    4706  11 tcp4  *:111                *:*
root    sshd      4428  4  tcp6  *:22                  *:*
root    sshd      4428  6  tcp4  *:22                  *:*
root    nginx      4238  7  tcp4  *:80                  *:*
root    nginx      4238  8  tcp6  *:80                  *:*
root    python2.7  4161  3  tcp4  127.0.0.1:9042        *:*
root    winbindd  4117  18 stream /tmp/.winbindd/pipe
root    winbindd  4117  19 stream /var/db/samba/winbindd_privileged/pipe
root    winbindd  4112  18 stream /tmp/.winbindd/pipe
root    winbindd  4112  19 stream /var/db/samba/winbindd_privileged/pipe
root    smbd      4109  28 tcp6  *:445                *:*
root    smbd      4109  29 tcp6  *:139                *:*
root    smbd      4109  30 tcp4  *:445                *:*
root    smbd      4109  31 tcp4  *:139                *:*
root    nmbd      4106  9  udp4  *:137                *:*
root    nmbd      4106  10 udp4  *:138                *:*
root    nmbd      4106  11 udp4  192.168.0.41:137      *:*
root    nmbd      4106  12 udp4  192.168.0.255:137    *:*
root    nmbd      4106  13 udp4  192.168.0.41:138      *:*
root    nmbd      4106  14 udp4  192.168.0.255:138    *:*
root    nmbd      4106  17 stream /var/run/samba/nmbd/unexpected
root    ntpd      3141  20 udp4  *:123                *:*
root    ntpd      3141  21 udp6  *:123                *:*
root    ntpd      3141  22 udp4  192.168.0.41:123      *:*
root    ntpd      3141  23 udp6  ::1:123              *:*
root    ntpd      3141  24 udp6  fe80:8::1:123        *:*
root    ntpd      3141  25 udp4  127.0.0.1:123        *:*
root    syslogd    2048  4  dgram  /var/run/log
root    syslogd    2048  5  dgram  /var/run/logpriv
root    syslogd    2048  6  udp6  *:514                *:*
root    syslogd    2048  7  udp4  *:514                *:*
root    devd      1756  4  stream /var/run/devd.pipe
[root@freenas8] /mnt/Cloud1-ZFS/iSCSI-syslog#

Code:

[root@freenas8] /mnt/Cloud1-ZFS/iSCSI-syslog# cat /usr/local/etc/istgt/istgt.conf
# Global section
[Global]
  NodeBase "iqn.2011-03.betoria.net.istgt"
  PidFile "/var/run/istgt.pid"
  AuthFile "/usr/local/etc/istgt/auth.conf"
  MediaDirectory /mnt
  Timeout 30
  NopInInterval 20
  MaxR2T 32
  DiscoveryAuthMethod None
  MaxSessions 16
  MaxConnections 8
  FirstBurstLength 65536
  MaxBurstLength 262144
  MaxRecvDataSegmentLength 262144
  MaxOutstandingR2T 16
  DefaultTime2Wait 2
  DefaultTime2Retain 60
 
[UnitControl]
 
# PortalGroup section
[PortalGroup1]
  Portal DA1 0.0.0.0:3260
[PortalGroup2]
  Portal DA1 0.0.0.0:3261
[PortalGroup3]
  Portal DA1 0.0.0.0:3262
 
# InitiatorGroup section
[InitiatorGroup1]
  InitiatorName "ALL"
  Netmask ALL
 
# LogicalUnit section
[LogicalUnit1]
  TargetName "desktop-target"
  Mapping PortalGroup1 InitiatorGroup1
  AuthMethod Auto
  UseDigest Auto
  ReadOnly No
  UnitType Disk
  UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "000acd1e1feb00"
  UnitOnline yes
  BlockLength 512
  QueueDepth 32
  LUN0 Storage /mnt/Cloud1-ZFS/iSCSI-desktop/desktop-fileextent 1TB
  LUN0 Option Serial 000acd1e1feb000
[LogicalUnit2]
  TargetName "syslog-target"
  Mapping PortalGroup2 InitiatorGroup1
  AuthMethod Auto
  UseDigest Auto
  ReadOnly No
  UnitType Disk
  UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "000acd1e1feb01"
  UnitOnline yes
  BlockLength 512
  QueueDepth 32
  LUN0 Storage /mnt/Cloud1-ZFS/iSCSI-syslog/syslog-fileextent 100GB
  LUN0 Option Serial 000acd1e1feb010
[LogicalUnit3]
  TargetName "esxi-target"
  Mapping PortalGroup3 InitiatorGroup1
  AuthMethod Auto
  UseDigest Auto
  ReadOnly No
  UnitType Disk
  UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "08606e799c4502"
  UnitOnline yes
  BlockLength 512
  QueueDepth 32
  LUN0 Storage /mnt/Cloud1-ZFS/iSCSI-esxi/esxi-fileextent 4TB
  LUN0 Option Serial 08606e799c45020
[root@freenas8] /mnt/Cloud1-ZFS/iSCSI-syslog#

If they are writing to the same file extent I still can't wrap my head around how that could be working since they should have started over lapping data by now

I appreciate the help

 

[jpaetzel]

There's no way they are connected to the same target. You'd get immediate filesystem corruption.

Try doing the ESXi extent as 2TB-1byte. That's the limit for some older versions of ESXi. It ignores larger than the limit, so in this case where the only LUN is 4TB it wouldn't see any LUNs at all.

 

[snowmirage]

I removed the 4TB file extent for the esxi target, added another with the same name that was 520GB

Code:

[root@freenas8] /mnt/Cloud1-ZFS/iSCSI-esxi# cat /usr/local/etc/istgt/istgt.conf
# Global section
[Global]
  NodeBase "iqn.2011-03.betoria.net.istgt"
  PidFile "/var/run/istgt.pid"
  AuthFile "/usr/local/etc/istgt/auth.conf"
  MediaDirectory /mnt
  Timeout 30
  NopInInterval 20
  MaxR2T 32
  DiscoveryAuthMethod None
  MaxSessions 16
  MaxConnections 8
  FirstBurstLength 65536
  MaxBurstLength 262144
  MaxRecvDataSegmentLength 262144
  MaxOutstandingR2T 16
  DefaultTime2Wait 2
  DefaultTime2Retain 60
 
[UnitControl]
 
# PortalGroup section
[PortalGroup1]
  Portal DA1 0.0.0.0:3260
[PortalGroup2]
  Portal DA1 0.0.0.0:3261
[PortalGroup3]
  Portal DA1 0.0.0.0:3262
 
# InitiatorGroup section
[InitiatorGroup1]
  InitiatorName "ALL"
  Netmask ALL
 
# LogicalUnit section
[LogicalUnit1]
  TargetName "desktop-target"
  Mapping PortalGroup1 InitiatorGroup1
  AuthMethod Auto
  UseDigest Auto
  ReadOnly No
  UnitType Disk
  UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "000acd1e1feb00"
  UnitOnline yes
  BlockLength 512
  QueueDepth 32
  LUN0 Storage /mnt/Cloud1-ZFS/iSCSI-desktop/desktop-fileextent 1TB
  LUN0 Option Serial 000acd1e1feb000
[LogicalUnit2]
  TargetName "syslog-target"
  Mapping PortalGroup2 InitiatorGroup1
  AuthMethod Auto
  UseDigest Auto
  ReadOnly No
  UnitType Disk
  UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "000acd1e1feb01"
  UnitOnline yes
  BlockLength 512
  QueueDepth 32
  LUN0 Storage /mnt/Cloud1-ZFS/iSCSI-syslog/syslog-fileextent 100GB
  LUN0 Option Serial 000acd1e1feb010
[LogicalUnit3]
  TargetName "esxi-target"
  Mapping PortalGroup3 InitiatorGroup1
  AuthMethod Auto
  UseDigest Auto
  ReadOnly No
  UnitType Disk
  UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "08606e799c4502"
  UnitOnline yes
  BlockLength 512
  QueueDepth 32
  LUN0 Storage /mnt/Cloud1-ZFS/iSCSI-esxi/esxi-fileextent 520GB
  LUN0 Option Serial 08606e799c45020
[root@freenas8] /mnt/Cloud1-ZFS/iSCSI-esxi#

Ran a scan from the esxi box still didn't see anything when trying on port 3262. I'm starting to think its esxi, running a scan on my desktop in windows for 3262 I can see the esxi-target and the drive shows up in windows as 520GB and I see it connecting on port 3262 over the network and on the FreeNAS box.

But my syslog server is till showing connection in the FreeNAS window and over the network on 3260 which is the same port the desktop is using for its share.

I guess maybe I should just backup my syslog data blow away everything but the desktop iSCSI and try to start again, I must have done something really weird here.... still can't explain what the syslog server has been writing to this whole time though... and something still seems weird with ESXi doesn't seem to want to pickup iSCSI on any ports but 3260

 

[jpaetzel]

From the clients can you see which targets they are using?

If two clients were connected to the same lun you'd get immediate filesystem corruption.

An easy test would be to md5 the extent files, do some writes, then md5 them again.

I'd use initiator groups to manage this, not portal ports. Or create several alias ips and use one ip per portal. I can imagine vmware barfing on ! 3260.

 

[snowmirage]

On the desktop the tail of the name it discovers ends with the expected "desktop-target" / "esxi-target" / "syslog-target" depending on what port I discover on.

It's been so long since I set this up I can't remember my way around open-iscsi on the syslog box. Here's the closest I've gotten to some useful information so far.

Code:

splunk@splunk:/usr/sbin$ sudo ./service open-iscsi status
*iscsid is running
*Current active iSCSI sessions:
tcp: [1] 192.168.0.41:3260,1 iqn.2011-03.betoria.net.istgt:desktop-target
splunk@splunk:/usr/sbin$

Which seems to indicate that its connecting to the desktop-target on port 3260, which my desktop also is (successfully... browsing the nearly full TB of data on there now). Unless there is some config item in open-iscsi where its connecting to a different target but I was stupid and called it "desktop-target" when I set it up last year... I'm not sure what could be going on

 

[snowmirage]

the only thing I can think of is maybe the directory I THINK is the iscsi share is actually just a local disk on the syslog server..... can't believe I could have screwed that up but I'll go dig around and see if I can verify that.... they are coincidently the same size... (100GB iSCSI syslog-target and 100GB local drive on the server (a VM in esxi))

if thats the case it could be connecting to the desktop iscsi target but not actually doing anything with it....

Ya its sounding like my stupidity is probably the most likely answer this this mess :) I'll keep digging

 

[jpaetzel]

Ok. Sounds like confusion of one sort or another. Let me know if there's anything I can do to help.

 

[snowmirage]

It looks like some how the iSCSI target that was mounted on /mnt/iscsivol (setup in fstab from /dev/sdb1) either never was actually working or changed as some point beacuse the /mnt/iscsivol is running off the local device /dev/sda1

So it appears it was querying for the iscsi connection on port 3260 but never actually connecting it.

Another note that my help someone else down the road it appears that at least in my version of ESXi (5.1) its running a firewall! Had no idea... and by default it allows port 3260 but not 3261/2 as I was trying.

found this..

http://www.yellow-bricks.com/2013/01/23/how-to-disable-esxi-firewall/

Disabled the firewall with the command he listed

Code:

esxcli network firewall set --enabled false

And now I can discover the esxi-target on port 3262. I was even able to change it back to the original size of 4TB and it works.