Peer to Peer 10Gbe on FreeNAS 11.1 U5

[Charles Rhoades]

I want to setup 10Gig peer to peer between my NAS and two Win10 Clients. I'm planning on using the following EN NICS, all Mellanox: MCX312A-XCBT Dual Port (NAS) & MCX311A-XCAT Single Port (Clients) with Cisco SFP-H10GB-CU5M Cables. I tried using Chelsio cards, but there isn't a Win10 Client driver that I could find, so they've been returned. Mellanox has Win10 and FreeBSD drivers.

I cannot find instructions on how to set this up in FreeNAS 11.1. My MB has 2x Intel i210 built-in ethernet ports, which I'm connecting to my home router via Cat 6A, and I access FreeNAS via the web interface from either of the two Win10 clients. FreeNAS does not list these in the Network - Interface screen, but I could add either using the Add Interface button where igb0 & igb1 are included in the drop down. In the Network - Network Summary page it indicates the following:

Name IPv4 Address IPv6 Address
igb0 192.168.1.131/24

Nameserver
192.168.1.1

Default route
192.168.1.1

My plan is to physically mount the MCX312A dual port card in the NAS, and then a MCX311A single port card in one of the Win10 clients, and connect an SFP+ cable between them...then boot up both devices. I have a pretty good idea on setting up the Win10 client, but questions about the FreeNAS setup. I don't plan on disabling the Intel i210 interfaces since they provide the FreeNAS web interface.

- do I have to now add the i210 interfaces to the Network - Interface page (or not since they work now without) or should I do that before physically installing the 10gig card, and then add the 10gig interface ports to the Network - Interface page after physical installation?
- how do I setup the IP addressing for the Mellanox ports?
- how do I setup the Mellanox ports without interfering with the web access IP addressing?
- how do I setup the Mellanox ports to use the MTU 9000 (Jumbo Frames) setting?
- do I have to add static routes for all 4 ports?
- how do I setup each client as Users for the peer to peer connections in FreeNAS?

Once I have the first connection working, repeat the process for the other Win10 client.

My plan is to use the 10gig connections for the data handling, and the 1gig connection for FreeNAS system management. The NAS is used exclusively for backup purposes...each of the clients have daily backups of over 1TB and the data handling over the 1gig network is unacceptably slow, lengthy, and can sometimes interfere with the actual backup processes.

Clear and concise instructions would be greatly appreciated, as I am really not more than knowledgeable at the User level, certainly not an expert on clients nor FreeNAS.

My configuration:
FreeNAS 11.1 U5
ASRock Rack E3C226D2I
Xeon E3-1231 v3
16GB ECC Ram
6X Seagate 4TB NAS HDD (ST4000VN000) in RAID Z2

 

[Chris Moore]

https://www.youtube.com/watch?v=p39mFz7ORco

 

[Chris Moore]

- how do I setup the Mellanox ports to use the MTU 9000 (Jumbo Frames) setting?

You don't need to do that.

 

[Charles Rhoades]

https://www.youtube.com/watch?v=p39mFz7ORco

Thanks for looking Chris...but I'm not building a switch...I'm tring peer to peer since there are only three computers, and the two clients will not be connected to each other. Just each client to the NAS.

 

[Charles Rhoades]

You don't need to do that.

And why not? The only FreeNAS video I found on YouTube used Corral...not 11.1, so the graphical interface made it look simple. But the 11.1 interface is less intuitive.

 

[Chris Moore]

And why not? The only FreeNAS video I found on YouTube used Corral...not 11.1, so the graphical interface made it look simple. But the 11.1 interface is less intuitive.

The idea of jumbo frames was developed a long time ago when CPUs were not as powerful as they are now. Jumbo frames is all about reducing CPU load, but it is a technology that is no longer needed, documentation just has not been updated. Many of the documents that talk about using jumbo frames were created in the 90s and have never been updated. Technology has moved on and it is simply not needed any more.

 

[Chris Moore]

but I'm not building a switch

He has two other videos that talk about peer to peer.
https://www.youtube.com/watch?v=MgNpI6VAAhI

https://www.youtube.com/watch?v=KHiucyRYX_w

He talks about setting the MTU, but it isn't needed. I have built the 10gig switch he talks about in video 3 and it was super simple to do. It gives me 10 ports at 1gig and 4 ports at 10gig.

 

[Charles Rhoades]

So any specifics on my FreeNAS questions above?

 

[kdragon75]

So any specifics on my FreeNAS questions above?

you will need to setup the interfaces as there own subnets and to access FreeNAS you MUST use the IP on the corresponding interface. It's simple. If the /24 does not make sense, google it.

 

[kdragon75]

DELETED

 

[Chris Moore]

So any specifics on my FreeNAS questions above?

I already linked to a video that walks you through it. What else do you need to know?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Charles Rhoades]

you will need to setup the interfaces as there own subnets and to access FreeNAS you MUST use the IP on the corresponding interface. It's simple. If the /24 does not make sense, google it.
View attachment 24545

Your diagram is exactly what I'm trying for.

 

[Charles Rhoades]

I've gotten the dual port NIC installed in the FreeNAS, and I can see the two ports in the Add Interface dropdown when trying to add. But the task errors out and I have to go into the Console and execute option 8 - Reset Configuration To Defaults, because the IP address for the Web User Interface keeps getting changed when I add the NIC.

 

[Chris Moore]

I've gotten the dual port NIC installed in the FreeNAS, and I can see the two ports in the Add Interface dropdown when trying to add. But the task errors out and I have to go into the Console and execute option 8 - Reset Configuration To Defaults, because the IP address for the Web User Interface keeps getting changed when I add the NIC.

You will probably need to configure each NIC manually from the console. Once they have been manually configured, it should not change. I had to change my IP configuration from the console when I switched from 1Gig to 10Gig, but I wasn't trying to run both at once, I completely moved to 10Gig.