Network Issues

Helheim

Cadet
Joined
Jun 27, 2020
Messages
7
I'm having random network drops on my FreeNAS and I don't really know why or where to start to figure out what is going on. I run FreeNAS-11.3-U1 in a VM under Proxmox and have it on it's own network bridge/interface tied to a Mellanox ConnectX-3. The only thing I notice is a spike in other activity on the other interface at the same time. What should I look at to figure this out? Thanks.

inerfaces.png


networkissue.png

VM Info:
vm.png


Host:
2x Intel Xeon CPU E5-2698 v3 @ 2.30GHz
128GB RAM
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Do you have the 1-port or 2-port ConnectX-3? Is this in a x8 PCI-E 3.0 slot, or is the slot 2.0?
 

Helheim

Cadet
Joined
Jun 27, 2020
Messages
7
Do you have the 1-port or 2-port ConnectX-3? Is this in a x8 PCI-E 3.0 slot, or is the slot 2.0?

2-port, x8 PCI-E 3.0

82:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Part number: MCX354A-FCBT
Engineering changes: A4
Vendor specific: PCIe Gen3 x8
 

Helheim

Cadet
Joined
Jun 27, 2020
Messages
7
Info from switch:

Ethernet49/1 is up, line protocol is up (connected)
Hardware is Ethernet, address is 001c.7379.17b0 (bia 001c.7379.17b0)
Ethernet MTU 9214 bytes , BW 40000000 kbit
Full-duplex, 40Gb/s, auto negotiation: off
Up 20 days, 21 hours, 5 minutes, 8 seconds
7 link status changes since last clear
Last clearing of "show interface" counters never
5 minutes input rate 37.2 kbps (0.0% with framing overhead), 4 packets/sec
5 minutes output rate 10.2 kbps (0.0% with framing overhead), 6 packets/sec
286109512 packets input, 184922444283 bytes
Received 38056 broadcasts, 172631 multicast
0 runts, 0 giants
0 input errors, 0 CRC, 0 alignment, 0 symbol, 78063 input discards
0 PAUSE input
683838050 packets output, 972839691456 bytes
Sent 772817 broadcasts, 5289056 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 124 output discards
0 PAUSE output
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Is the other port tied to vmbr0?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
I suspect you'll need to try reconfiguring to enable SR-IOV.
 

Helheim

Cadet
Joined
Jun 27, 2020
Messages
7
Is the other port tied to vmbr0?

No the other port isn't used on the Mellanox currently. vmbr0 is tied to one of the ports on the onboard Intel I210-AT, everything else except FreeNAS uses that. I'll look into SR-IOV, thanks.
 

Helheim

Cadet
Joined
Jun 27, 2020
Messages
7
I suspect you'll need to try reconfiguring to enable SR-IOV.

Sounds like this for passing through the entire network card instead of using a bridge on the host? Do I need to do that? Right now I'm passing through the HBA card and enabled IOMMU but didn't do anything with SR-IOV.

Thanks
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Host bridge needs to traverse the CPU. With SR-IOV, the NIC handles the "bridging" by instantiating a virtual NIC in the guest.
 

Helheim

Cadet
Joined
Jun 27, 2020
Messages
7
Host bridge needs to traverse the CPU. With SR-IOV, the NIC handles the "bridging" by instantiating a virtual NIC in the guest.

I couldn't get SR-IOV to work so I just passed through the entire card like the HBA, should that be ok to? Should I switch to a machine type of q35 and use pcie instead?

Untitled.png

lan.png

Thanks
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
See this thread. You may need to set a loader tunable hw.pci.honor_msi_blacklist=0 to get SR-IOV to work.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Shouldn't be.
 
Top