10 Gbe Nic not staying connected in peer-to-peer

[venis022]

I recently just built a freenas system and wanted to tie it together to the rest of my systems via a 10G peer-to-peer network. I have all 3 systems (freenas, win 10, win 7) on seperate subnets (100, 150, 200) and seperate IP addresses. I have gotten it to work and maintained pings between all of the systems, along with multiple file transfers. However, the 10G network shuts down and does not connect after 2 minuets on the freenas side (after working perfectly). The windows 10 to windows 7 connection remains but the freenas to win10 and freenas to win7 no longer connects. I get a "No route to host" from the freenas server, however, it was connecting and I did not touch anything prior to the message. The more interesting thing is that my windows 10 machine can still ping the server on the 10g connection (but not register the network drive) but my windows 7 pc cannot ping or register a network drive. However, I can delete the interface from freenas, and recreate it using the same subnet and IP address and it reappears for another 2 min and then returns the "No route to host" afterwards. In addition, on the interface tab itself, the 10G connections state that they are "active" inside those 2 min, and then "no carrier" afterwards, even though no connections have been pulled and the pings are still (attempting to) pinging.

Freenas v.11.1 u2

specs.
Freenas system
Supermicro x10sra-o
xeon e5-2667v3
32gb hynix ecc
12 6TB hgst hard drives - 2x 6x2 raid2z
1 chelsio t420-BT
LSI 9207-8i
Intel 320 40Gb x2 (boot)

win 10 - test
asus z270 pro
i5-6600k
8gb corsiar
samsung 840 evo 250gb
chelsio T420-BT

win 7 - main rig
Asus Hero VI
i5-4670k
24gb g.skill
samsung 850 evo 500g
4x4tb hgst drives. 2x software raid 1
chelsio T420-BT

 

[dlavigne]

Paste the output of ifconfig (from the FreeNAS system) within code tags.

 

[Chris Moore]

Please don't let my comments interfere with the troubleshooting, but I am curious about this.

chelsio T420-BT

Those cards are $600 each on Amazon:
https://www.amazon.com/Chelsio-T420-BT-2-PORT-PROFILE-ADAPTER/dp/B009AF2CSE

What made you choose such an expensive solution for this that didn't even include a switch?
There are many other ways to do this that would have been less expensive, including use of a switch, and that would likely have prevented the current problem from ever happening?

 

[venis022]

The cards were $95 each on ebay used. The intel x540-t2 was 120. simple economics on that choice, also , I read that chelsio was recommended for 10g.

I couldnt get the code tags, but I did get screenshots of various parts of the system, both when it was working and when it wasnt. The ifconfig png is all of the config while the 10g is down. The other 2 config pics show when each interface was accepting data. (on the win 7 and win 10 subnets respectively).

 

[venis022]

update:

So i did some testing today and got the 10G stable. It is now staying up, however, I can no longer connect to plex. When i check the ifconfig the epair connections are defaulting to the 10g connections... but plex is set to route out the regular gigabit connection (selected nic. vimage off. static ip). additionally, it will no longer start or stop throught the GUI and in the shell I recieve an error "No more processes" when I attempt to start or stop the service along with any other plugin's

Solved 1 problem in exchange for 2 others.