QLE8152 Network & FCoE Adaptor with Freenas 11.0-U6

[the2faces]

Hi everybody,

I'm user of freenas for a very long time (at least 4 years+) and I'm very happy about it.
A few weeks ago I was thinking about pumping up the performance on the network side, because I was struggling with low iscsi performance on my installed Intel gbe nics.

Thankfully I was able to order damn cheap some QLE8152 converged network adaptors, which are capable of FCoE and usuall Ethernet.

Unfortunately I'm not able to get the QLE-adaptors running reliably within my Freenas, together with two direct attached ESXi host, connected over two SFP+ 10gbe connections to the Freenas.

I only receive the following error message in the dmesg, after I stared up the iscsi service and tried to access the iscsi luns, which are presented via the 10gbe interfaces:

Code:

bridge0: Ethernet address: 02:ef:fa:2d:60:00
bridge1: Ethernet address: 02:ef:fa:2d:60:01
ql0: qls_mbx_isr: AEN [0x00008400 0x00012808 0x00000007 0x8000007e 0x00000002 0x0000000e 0x00000010 0x00000010 0x00000002]
ql0: qls_mbx_get_link_status 0x00004000 0x10000091 0x00000000 0x00000033 0x0009000c 0x03040401
ql1: qls_mbx_isr: AEN [0x00008400 0x00012808 0x00000007 0x8000007e 0x00000002 0x0000000e 0x00000010 0x00000010 0x00000002]
ql1: qls_mbx_get_link_status 0x00004000 0x10000091 0x00000000 0x00000033 0x000c000c 0x03040401
ql2: qls_mbx_isr: AEN [0x00008400 0x00012808 0x00000007 0x8000007e 0x00000002 0x0000000e 0x00000010 0x00000010 0x00000002]
ql2: qls_mbx_get_link_status 0x00004000 0x10000091 0x00000000 0x00000033 0x0009000c 0x03040401
ql3: qls_mbx_isr: AEN [0x00008400 0x00012808 0x00000007 0x8000007e 0x00000002 0x0000000e 0x00000010 0x00000010 0x00000002]
ql3: qls_mbx_get_link_status 0x00004000 0x10000091 0x00000000 0x00000033 0x000c000c 0x03040401
lagg0: link state changed to DOWN
ql0: link state changed to UP
ql1: link state changed to UP
ql2: link state changed to UP
ql3: link state changed to UP
em0: link state changed to UP
lagg0: link state changed to UP
vlan99: link state changed to UP
vlan109: link state changed to UP
igb0: link state changed to UP
ql0: promiscuous mode enabled
bridge0: link state changed to UP
ql1: promiscuous mode enabled
bridge1: link state changed to UP
ql2: promiscuous mode enabled
ql3: promiscuous mode enabled
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
WARNING: 172.19.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
ql3: qls_rx_comp: DS bit not set
WARNING: 172.19.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
ql3: qls_mbx_isr: AEN [0x00008110 0x08080505 0x00100004 0x00f00005 0x00f00005 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: qls_mbx_get_link_status 0x00004000 0x10000051 0x00000000 0x00000033 0x000e0046 0x04050501
ql3: qls_mbx_isr: AEN [0x00008110 0x080807f4 0x00100005 0x00f00006 0x00f70006 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: qls_mbx_isr: AEN [0x00008110 0x080807f4 0x00000005 0x00000006 0x00100007 0x00000000 0x00000000 0x00000000 0x00000000]
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
WARNING: 172.18.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
ql1: qls_rx_comp: DS bit not set
WARNING: 172.18.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
ql1: qls_mbx_isr: AEN [0x00008110 0x08080505 0x00100004 0x00f00005 0x00f00005 0x00000000 0x00000000 0x00000000 0x00000000]
WARNING: 172.17.0.3 (iqn.2014-02.de.esxi01.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 172.16.0.3 (iqn.2014-02.de.esxi01.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 172.17.0.3 (iqn.2014-02.de.esxi01.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 172.16.0.3 (iqn.2014-02.de.esxi01.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
ql1: qls_mbx_isr: AEN [0x00008012 0x00000051 0x00000033 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: ql1: qls_mbx_isr: AEN [0x00008012 0x00000051 0x00000033 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000]
qls_mbx_isr: AEN [0x00008110 0x08080507 0x00000004 0x00000005 0x00000005 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: qls_mbx_isr: AEN [0x00008110 0x08080707 0x00000005 0x00000006 0x00000007 0x00000000 0x00000000 0x00000000 0x00000000]
WARNING: 172.16.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 172.17.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 172.16.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
ql1: qls_mbx_isr: AEN [0x00008011 0x10000090 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000]
ql1: qls_mbx_isr: AEN [0x00008110 0x08000506 0x00100005 0x00100006 0x00e00006 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: qls_mbx_isr: AEN [0x00008011 0x10000090 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: qls_mbx_isr: AEN [0x00008110 0x08000506 0x00100006 0x00f00007 0x00f00008 0x00000000 0x00000000 0x00000000 0x00000000]
ql1: qls_mbx_isr: AEN [0x00008110 0x080807f4 0x00100006 0x00f00007 0x00f70007 0x00000000 0x00000000 0x00000000 0x00000000]
ql3: qls_mbx_isr: AEN [0x00008110 0x080807f4 0x00100007 0x00f00008 0x00f70009 0x00000000 0x00000000 0x00000000 0x00000000]
WARNING: 172.16.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection
WARNING: 172.16.0.4 (iqn.2014-02.de.esxi02.iscsi): no ping reply (NOP-Out) after 5 seconds; dropping connection

Both nics ql0 and ql1 (PCIE card 1) are bridged together, as well the two nics ql1 and ql2.

Mabe someboy has seen this issue as well, and give me some additional tips about that problem.

Thanks and best regards!

 

[Elliot Dierksen]

Both nics ql0 and ql1 (PCIE card 1) are bridged together, as well the two nics ql1 and ql2.

I think you would want to channel them together (ideally LACP), not bridge them. If you bridge them, only 1 will be active. Either that, or you need to have some way to separate traffic.

 

[the2faces]

Hi Elliot,

I guess you are right. I was able to solve my issues with the dropping connections, after changing the NICs from Qlogic QLE8152 to Chelsio n320 10G Nics. After this, I was at least able to get up one host up and running, but unfortunately the second nic wasn't active. So no ping to the iSCSI portal of the Freenas was possible. Only one host with one direct attached link to the Freenas was able to access the dedicated IP.
One thing I changed, was that not ql0 and ql1 were bridged together, but ql0 and ql2 (host one) and ql1 and ql3 (host 2). Just with the same result.

 

[Elliot Dierksen]

If the links are point to point (FreeNAS to host), try assigning a different IP to each link. I assumed you were going to a switch, but I am still not sure that bridging the NIC's together will give you what you want. As a disclaimer, I am an NFS user and not an iSCSI user. I am coming at this from a layer 2 ethernet/layer 3 IP perspective. If you are bridging all the physical interfaces together, you would need to assign the IP to some sort of bridge virtual interfaces and not to a physical one if you want to use the same IP across multiple physical interfaces. That would PROBABLY be bridge0, or that is what a quick google on the matter indicated.