SOLVED Intel x520 connection losses in VM

[ChrisChros]

Hi,

I'm running FreeNAS as bear metal, specs in the signature. My Network infrastructure is a Unifi System with USG, US-48-500 PoE Switch and UAP-AC-Pro Access Points. In the USG I have setup several VLans and on the switch the SFP+ port for FreeNAS is setup as a Trunk.
In FreeNAS I have a CentOS 7.6 VM (VirtIO NIC) with a home Automation Server for my KNX System. The VM has a VLan Tagg (20) and VLan is handled in FreeNAS, I followed this instruction to set everything up. In parallel I have different Jails running, including the Unifi Controller on VLan1.
Since the installation of my 10 GbE NIC, I recognize in my Centos VM irregularly connection losses to the KNX IP Gateway (also VLan 20, tagged port in the switch) and also to another apcupsd server. In the Unifi Controller there is nothing reported about connection losses, also in FreeNAS itself.

For trouble shooting I switched over the parent interface for my VLan 20 from my 10 GbE Intel X520-DA2 to one of my on-board 1 GbE Intel NICs. Since this time the system is not recognizing any connection loss to the KNX system, whether to the apcupsd server.

Does anybody of you have an idea where these connection losses came from? Obviously it has something to do with the Intel NIC, but my research pointed out that it should run out of the box.

Regards Chris

 

[jgreco]

bear metal

Bear metal hehehe

Where'd you get the X520? eBay? A seller in Asia?

This leads back into what seems to be an uptick in fakes on the market.

11.3 RC1 doesn't like X520

I got myself two X520s (one for freenas, other for PC) with 10G transceivers along with a 10G switch. Fiber is the only connection between these machines. Freenas clearly has a driver, can talk with the SFP+ modules and is alive enough to assign it an IP and even start pushing some http...

www.ixsystems.com

Follow the discussion there to see if you can find easy signs of fake card.

 

[ChrisChros]

Thanks for the hint.
No the card is not from Asia, its from Germany from a privat seller, but it could be that he had the same problems with the card and thats the reason why he sold the card.
I checked my NIC according to your suggestions and it looks linke original, but I am also not an expert.


I also changed now to the second port of the card, maybe this one will work.

 

[jgreco]

Well, it's not OBVIOUSLY fake, but it does seem to be a Dell OEM card, and lacks the Yottamark that you'd have to validate a retail Intel card. Hm.

 

[KevDog]

@jgreco -- really hard on those intel cards aren't you?

 

[jgreco]

@jgreco -- really hard on those intel cards aren't you?

No. I've got a pile of them. Lately I've been putting Solarflare's in the hypervisors and pulling the X520's, but they're still awesome cards.

 

[ChrisChros]

In the morning I had again some messages in the Log-File auf my home automation server VM regarding connection problems.
Is there any else what I can do e.g. driver update on FreeNAS or is there a possibility of any firmware update on the card itself?

Which card would you suggest? Chelsio is difficult to buy in Germany, they are not so common.
What's about the Intel X710? Will this card work with FreeNAS?

 

[jgreco]

If it isn't working, it probably isn't a driver issue. The Intel driver is generally *very* good.

Basically this can go one of just a few ways IMO.

1) Find some physical problem that might affect a legit card. That could be heat; make *sure* you have air flowing over the card. These cards throw a ton of heat. They are designed for use in professionally-designed servers and rely on front-to-back airflow. Notice the orientation of the fins. Remediate with airflow and pray you haven't baked the card. That's by far the most likely physical problem with a legit card.

2) You have a high quality but yet fake card, or possibly a legit card that someone abused (knocked parts off, handled without ESD precautions, etc). Replacement is the only option for this case.

3) You have some sort of configuration issue, which can be either PEBCAK or you're trying to do something that FreeNAS doesn't understand or isn't setting up right. A different card will not help in this case. You can get a feel for if this is the case by trying to use a regular 1Gbps port in the same configuration.

 

[ChrisChros]

1) Find some physical problem that might affect a legit card. That could be heat; make *sure* you have air flowing over the card. These cards throw a ton of heat. They are designed for use in professionally-designed servers and rely on front-to-back airflow. Notice the orientation of the fins. Remediate with airflow and pray you haven't baked the card. That's by far the most likely physical problem with a legit card.

Airflow should not be the problem. I have 2 140mm fans in the front for the HDDs and 1 140mm fan in the back to have a propper air flow through the casing.

3) .... You can get a feel for if this is the case by trying to use a regular 1Gbps port in the same configuration.

This I have already checkt by using the internal 1 Gbps NIC instead of the 10 Gbps with the same configuration, I just changed the parent interface of the Vlan's.

... (knocked parts off, handled without ESD precautions, etc). Replacement is the only option for this case.

This could always be with second hand parts. In case of replacement what would be your suggestion. With which NIC you have had good experiences in the past?

 

[jgreco]

The X520 and the Chelsio T520-CR for FreeBSD/FreeNAS.

The X520 and the Solarflare for ESXi.

 

[ChrisChros]

What about the intel X710?

 

[jgreco]

Perhaps. The X710 was problematic for real-world use in certain circumstances. Since my company isn't big enough to have staff dedicated to play these kinds of years-long games of occasional network death debugging, policy is that we stick with stuff that's battle-tested. I mean, really, the X710 was launched in 2014 to all sorts of PSOD's and is still having problems in 2018?

 

[ChrisChros]

OK thanks for your opinion.

 

[ChrisChros]

@jgreco
apparently my x520 NIC has a defect. I have now got a new used card and since then I have not lost any connections within my VM.
Thanks for your support.

 

[jgreco]

@jgreco
apparently my x520 NIC has a defect. I have now got a new used card and since then I have not lost any connections within my VM.
Thanks for your support.

Good to know. Now and then used cards get sold that have had parts knocked off. The folks who strip old servers out of data centers and then break them down into components aren't always as gentle as they ought to be.