"no ping reply (NOP-Out) after 5 seconds" kills iSCSI volumes under load? (v11.1)

[Elliot Dierksen]

All traffic blended together Yes it is but all the Traffic is ISCSI and vmotion.

If you want any kind of separation, you need to do separate VLAN's. Again, I can't tell you for sure that this is what was causing your problems. I can tell you for sure that doing it that way not the best practice.

 

[kdragon75]

If you want any kind of separation, you need to do separate VLAN's. Again, I can't tell you for sure that this is what was causing your problems. I can tell you for sure that doing it that way not the best practice.

VLANs and set prio. to something above 1.

 

[Dudleydogg]

VLANs and set prio. to something above 1.

I am trying to figure out how to setup Vlans on the netapp switch 1610, not very straight forward. Once I setup some Vlans I cannot get any traffic to pass. Dell switch I have all Vlans setup with no issues. Not much documentation online for old netapp switches.

 

[kdragon75]

https://library.netapp.com/ecm/ecm_get_file/ECMP1117874
Chapter 5

 

[kdragon75]

Hey I have been looking for a cheap 10gbe switch, hows the power consumption on that 1610?

 

[Dudleydogg]

Hey I have been looking for a cheap 10gbe switch, hows the power consumption on that 1610?

Its a loud switch many fans, not something you want in living room. I saw them on ebay for $300 ish, then found this one for $75 so grabbed it. Switch works fine my issue is getting it configured correctly for Freenas. not sure how power efficient it runs pretty warm.

 

[Dudleydogg]

https://library.netapp.com/ecm/ecm_get_file/ECMP1117874
Chapter 5

I found that Manual and allegedly it does show you how to setup Vlans, but I cannot make it work with VMware. I have Dell switch and I believe you have to rethink how vlans are done on this switch. in my dell switch I tag the vlan in vkernal in VMware. but on this switch that does not seem to work. I am searching internet for real world example and have yet to find anything.

 

[kdragon75]

I found that Manual and allegedly it does show you how to setup Vlans, but I cannot make it work with VMware. I have Dell switch and I believe you have to rethink how vlans are done on this switch. in my dell switch I tag the vlan in vkernal in VMware. but on this switch that does not seem to work. I am searching internet for real world example and have yet to find anything.

You might be seeing the difference between host based and port based vlans. On one side, the port will tag all incoming frames unless already tagged. On the other side, the switch accepts all vlans on a port (some times called trunking) and the host (VMware) does all the tagging. The latter is generally what you want with VMware ESXi.
I just bought one of those switchs so I'll take a look when I get mine.

 

[Dudleydogg]

You might be seeing the difference between host based and port based vlans. On one side, the port will tag all incoming frames unless already tagged. On the other side, the switch accepts all vlans on a port (some times called trunking) and the host (VMware) does all the tagging. The latter is generally what you want with VMware ESXi.
I just bought one of those switchs so I'll take a look when I get mine.

That sounds good I would appreciate a CLI example to apply to my switch. Might be speaking too soon but I disabled MTU on switch to default, also set VMware back to 1500 and have not dropped any packets in almost 12 hours. But I have seen this before so still waiting.

 

[Elliot Dierksen]

Might be speaking too soon but I disabled MTU on switch to default, also set VMware back to 1500 and have not dropped any packets in almost 12 hours.

YMMV, but my personal opinion is that the gains from an increased MTU are pretty minimal these days. Particularly when you consider all the potential issues an MTU mismatch can cause, it just isn't worth it. IMHO, of course.

 

[Johnnie Black]

YMMV, but my personal opinion is that the gains from an increased MTU are pretty minimal these days.

I believe it depends mostly on the NICs used, with the Mellanox Connect-X2 I use it maxes out at around 800MB/s with MTU@1500, with jumbo frames I get 1.1GB/s

 

[Elliot Dierksen]

I believe it depends mostly on the NICs used, with the Mellanox Connect-X2 I use it maxes out at around 800MB/s with MTU@1500, with jumbo frames I get 1.1GB/s

It sounds like it certainly makes a difference in your use case. My comment is coming from a network infrastructure perspective. I can't tell you how many times I have had to troubleshoot issues with jumbo frames. 99.9% of the time it ended up being that the server/storage side admin didn't realize that jumbo frames need to be supported and enabled on every single hop in the path, and there was something along the way that couldn't do it.

 

[Johnnie Black]

It sounds like it certainly makes a difference in your use case. My comment is coming from a network infrastructure perspective. I can't tell you how many times I have had to troubleshoot issues with jumbo frames. 99.9% of the time it ended up being that the server/storage side admin didn't realize that jumbo frames need to be supported and enabled on every single hop in the path, and there was something along the way that couldn't do it.

I agree, in this case I'm using a direct connection, so no switch involved and less chance of issues, still I do believe there's some issue with the FreeBSD driver where sometimes the NIC stops responding or slow downs a lot, this occurs much less with latest FreeNAS but it still happens once in a while, and it never happened with MTU 1500, but since it's not often and there's a significant performance difference I'm living with it for now hoping that it will eventually be 100% OK on newer releases.

 

[Dudleydogg]

I am going on 24 Hours with no more Dropped packets or Ping time outs on the Freenas iscsi. I forgot that you have to reboot for the MTU to take effect so Freenas NIC is still set to 9000, NetApp Switch is default (No MTU), ESXI vkernal and vswitch are set too 1500. To be 1500 across the board I will need to reboot the Storage. also thank you all for your opinions input and direction.

cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_

 

[kdragon75]

That sounds good I would appreciate a CLI example to apply to my switch. Might be speaking too soon but I disabled MTU on switch to default, also set VMware back to 1500 and have not dropped any packets in almost 12 hours. But I have seen this before so still waiting.

Depending on exactly what your trying to do with vlans, it looks fairly easy. I have my doing trunking and port based vlans. I still have to get mine working with host based vlans in a secure fashion.
basically, if I want a port from freenas dedicated to vlan 70 (I use 70 and 71 for iSCSI) on the switch I run the following from the privileged # mode:

Code:

(CN1610) #config

(CN1610) (Config)#interface 0/9

(CN1610) (Interface 0/9)#switchport mode access

(CN1610) (Interface 0/9)#switchport access vlan 70

(CN1610) (Interface 0/9)#exit

(CN1610) (Config)#exit

(CN1610) #write memory

This operation may take a few minutes.
Management interfaces will not be available during this time.

Are you sure you want to save? (y/n) y

Config file 'startup-config' created successfully .


Configuration Saved!

On FreeNAS, I don't do anything with vlans, I just set an IP on the interface and away I go.

 

[Dudleydogg]

If the port only has one Vlan what do I do with the rest of the traffic, I have Vmotion, Management, that needs its own Vlan? I did get this to work with one Vlan but used mode General, then on each end point I had to specify the vlan. But the port only supports one vlan when I need more than one.

 

[kdragon75]

Interface 0/X switchport mode trunk worked for me. As long as the vlans are all created in the vlans database mode.

 

[Dudleydogg]

Yes vlans are created, so switchport trunk then do I just tell each port to participate in that vlan? Then I can tag each vkernal in VMware to separate that traffic?
since the switch is dedicated to iscsi and vmotion, at first I did not see point in vlans. but others have suggested I do it that way.

 

[kdragon75]

Access mode is port based vlans (the frame gets tagged as it enters the port) trunk mode just assumes we should accept all frames tagged or otherwise. This allows hosts like vmare to do the tagging for multiple vlans on a single adapter and port.

 

[Dudleydogg]

I still cannot get this to work, But....
OK I got it working for vlan 2 but now want to set vlan on Vmotion, I set vkernals to vlan 3 and they cannot talk across the trunked vlan.
Below is working for me, Thanks for the Hints.

Set Vkernal to Vlan2

interface 0/1
description 'Esxi Host 4'
switchport mode trunk
exit

interface 0/2
description 'Esxi Host 4'
switchport mode trunk
Exit

interface 0/10
description 'Freenas iSCSI Vmotion'
switchport mode access
switchport access vlan 2

exit