Ever since I upgraded my FreeNAS to 11.3 I've noticed that my two ESXi hosts that are directly connected via QSFP are not using the 40Gb iSCSI path but are only using the fallback 10Gb connections. Before the upgrade I was using the tunables on FreeNAS to build out the bridge interface:
autobridge_bridge100=mlxen0 mlxen1
autobridge_interfaces=bridge100
cloned_interfaces=bridge100
ifconfig_mlxen0=up
ifconfig_mlxen1=up
ifconfig_bridge100=up 10.1.1.1 netmask 255.255.255.248
Since FreeNAS 11.3 has built in support from the network tab to create bridge interfaces I migrated the configuration over there. Doing any kind of network connectivity test on either end with a pcap running on either the bridge interface or the actual intefaces mlxen0/mlxen1 results in 0 packets captured (ping, open port test)
The other hosts are 10.1.1.3 and 10.1.1.2
autobridge_bridge100=mlxen0 mlxen1
autobridge_interfaces=bridge100
cloned_interfaces=bridge100
ifconfig_mlxen0=up
ifconfig_mlxen1=up
ifconfig_bridge100=up 10.1.1.1 netmask 255.255.255.248
Since FreeNAS 11.3 has built in support from the network tab to create bridge interfaces I migrated the configuration over there. Doing any kind of network connectivity test on either end with a pcap running on either the bridge interface or the actual intefaces mlxen0/mlxen1 results in 0 packets captured (ping, open port test)
Code:
root@SAN[~]# ifconfig mlxen0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ed03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> nd6 options=1<PERFORMNUD> media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>) status: active mlxen1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ed03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> nd6 options=1<PERFORMNUD> media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>) status: active bridge100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:33:ff:9c:1a:64 inet 10.1.1.1 netmask 0xfffffff8 broadcast 10.1.1.7 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: mlxen1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 5 priority 128 path cost 55 member: mlxen0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 55 root@SAN[~]# lspci | grep Mellanox 02:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] root@SAN[~]# dmesg | grep mlx mlx5en: Mellanox Ethernet driver 3.5.1 (April 2019) mlx4_core0: <mlx4_core> mem 0xdf200000-0xdf2fffff,0xdd000000-0xdd7fffff at device 0.0 on pci2 mlx4_core: Mellanox ConnectX core driver v3.5.1 (April 2019) mlx4_core: Initializing mlx4_core mlx4_core0: Unable to determine PCI device chain minimum BW mlx4_en mlx4_core0: Activating port:1 mlx4_en: mlx4_core0: Port 1: Using 8 TX rings mlxen0: link state changed to DOWN mlx4_en: mlx4_core0: Port 1: Using 8 RX rings mlx4_en: mlxen0: Using 8 TX rings mlx4_en: mlxen0: Using 8 RX rings mlx4_en: mlxen0: Initializing port mlx4_en mlx4_core0: Activating port:2 mlx4_en: mlx4_core0: Port 2: Using 8 TX rings mlxen1: link state changed to DOWN mlx4_en: mlx4_core0: Port 2: Using 8 RX rings mlx4_en: mlxen1: Using 8 TX rings mlx4_en: mlxen1: Using 8 RX rings mlx4_en: mlxen1: Initializing port mlxen0: promiscuous mode enabled mlxen1: promiscuous mode enabled mlx4_en: mlxen1: Link Down mlx4_en: mlxen1: Link Down
The other hosts are 10.1.1.3 and 10.1.1.2