10Gb peer to peer connection between 2 FreeNAS server setup query

[Ishak]

Hi,

Just joined the forum today as I am quite new[less than a year] to the FreeNAS operating system. I have been using FreeNAS to host my movies & Plex since September 2016 and I am thinking of a new project in mind. The purpose of the project is to achieve 10 Gb peer to peer connection between 2 FreeNAS servers so that when I am copying or moving files between these 2 servers, I can get blazing 10 Gb transfer rates.

Below is the specs of my home network:
Server #1:
FreeNAS-9.10.2 U6 Stable
Intel Xeon e3-1225v5 @ 3.3GHz
1 GB igb0 Interface 192.168.1.154
10 GB ix0 Interface 192.168.100.2[via X540T1 NIC PCIe]

Server #2:
FreeNAS-9.10.2 U6 Stable
Intel Xeon e3-1220v5 @ 3.0GHz
1 GB re0 Interface 192.168.1.247
10 GB ix0 Interface 192.168.100.1[via X540T1 NIC PCIe]

Main PC:
Windows 10 with on board 1 Gigabit LAN

TP-Link 5-Port Gb Desktop Switch

The PC, Server #1 & Server #2 are currently connected locally via the switch. I have recently installed the X540T1s and peer connect the servers together.
My intention is to be able to use the main PC to do 10 Gb file transfer speed between the 2 FreeNAS Servers. However, as per my test transfers turned out, I am still stuck with 1 Gb transfers. File transfers are still utiilzing the re0 and igb0 ports; there is no activity on both ix0 ports of each servers. =(

Can someone be kind enough to guide me how I should go about doing this properly, without buying a 10 Gb switch?

Is there any software tweaks on FreeNAS that allows prioritizations of the ix0 connections over the 1 Gb connections when detecting file transfers between the servers?
Do I have to get a X540T2 for my PC so that all 3 devices are on the same network segment with the same 10 Gb ports?
Are there any other options that can be considered?

Thank you in advance for any ideas and solutions offered.

Best Regards
Ishak

 

[Pezo]

I'm pretty sure you would need to initiate the transfer on one of the two servers for them to use the 10G connection, otherwise the data has to go over the main PC somehow (how would it not, one server just sees reads from the main PC, the other one just sees writes from the main PC).

 

[chris crude]

When you connect your 10GB you must give it an IP address in a different subnet than your 1GB. Once you have a proper IP, you can enter FreeNAS from the new address. Map your network drive to the new address not the old.

 

[Pezo]

Even then, when you just copy and paste files on the main PC it still has to go over the 1G connection.

 

[chris crude]

Even then, when you just copy and paste files on the main PC it still has to go over the 1G connection.

On your windows client you should have 2 network shares mapped. 1 to the 10GB port on server 1 and the other the 10GB port on server 2. You should be able to copy from one to the other at 10GB speeds. If you can't I would use iperf to check the speeds between them.

 

[Pezo]

You should be able to copy from one to the other at 10GB speeds.

How does that work? Wouldn't the data still have to flow through the Windows client?

 

[chris crude]

How does that work? Wouldn't the data still have to flow through the Windows client?

You are correct. I forgot that windows wasn't on the same network as the 10GB. You could always use the snapshot/replicate feature to backup one server to the other, or rsync, but that doesn't let you just move a single file from one to the other from windows. Could add a dual 10GB NIC to windows client and connect to both.

 

[Ishak]

Hi Pezo/Chris,

My sincere thanks for your responses.

In summary I will have 2 options, albeit their advantages and disadvantages:

Setup Option 1:
Server#1 with X540T1 <---> PC with X540T2 <---> Server#2 with X540T1 [peer to peer connection directly on the 10Gb subnet]
Advantage:
- Can copy/move files selectively between the servers, as per my heart content.
Disadvantage :
- Need to invest in a X540T2 for the PC which acts as the interface between the 2 servers
- Additional tweaking on the Windows 10 PC to discover the 10Gb subnet

Setup Option 2:
Server#1 with X540T1 <---> Server#2 with X540T1 [peer to peer connection directly on the 10Gb subnet] ]
Advantage:
- schedule a one-time rsync task to copy the dataset from one server to another using the remote host IP address field to point to the 192.168.100.x [10Gb subnet] using either the pull or push methodology.
- No need additional investment of a new X540T2 for the PC
Disadvantage:
- Copy is done at dataset level instead of individual files in the dataset
- Need to ensure that enough disk space available on target server before triggering rsync task

I hope that my understanding of the above is correct/accurate. Thanks again for your guidance.

Best Regards

 

[Pezo]

I'm not an rsync guru but I'm sure you can only copy some files using filters or something like that...
Also, you could just copy a single folder or file instead of the whole dataset, or what do you mean by dataset level?

 

[chris crude]

Hi Pezo/Chris,

My sincere thanks for your responses.

In summary I will have 2 options, albeit their advantages and disadvantages:

Setup Option 1:
Server#1 with X540T1 <---> PC with X540T2 <---> Server#2 with X540T1 [peer to peer connection directly on the 10Gb subnet]
Advantage:
- Can copy/move files selectively between the servers, as per my heart content.
Disadvantage :
- Need to invest in a X540T2 for the PC which acts as the interface between the 2 servers
- Additional tweaking on the Windows 10 PC to discover the 10Gb subnet

Setup Option 2:
Server#1 with X540T1 <---> Server#2 with X540T1 [peer to peer connection directly on the 10Gb subnet] ]
Advantage:
- schedule a one-time rsync task to copy the dataset from one server to another using the remote host IP address field to point to the 192.168.100.x [10Gb subnet] using either the pull or push methodology.
- No need additional investment of a new X540T2 for the PC
Disadvantage:
- Copy is done at dataset level instead of individual files in the dataset
- Need to ensure that enough disk space available on target server before triggering rsync task

I hope that my understanding of the above is correct/accurate. Thanks again for your guidance.

Best Regards

You have detailed the advantages/disadvantages for both options very well. The additional tweaking of Windows 10PC that i used can be found here http://www.cinevate.com/blog/confessions-of-a-10-gbe-network-newbie-part-5-smb3/
and i also set a few tunables in FreeNAS https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/page-7#post-272868

 

[Ishak]

Hi Chris,
Thanks for the tips.
Just want to update everyone that I have settled on option 1. The T2 was delayed due to USPS sitting on it for 2 weeks...
Anyways, I am getting speeds of 125MB initiating the transfer for server 1 to 2 [or vice versa], after the tweaking done on both FREENAS and WIN10.


Server #1:
FreeNAS-9.10.2 U6 Stable
Intel Xeon E3-1225v5 @ 3.3GHz
MSI C236M WORKSTATION Motherboard
Kingston Technology 32GB 2133MHz DDR4 ECC Memory
7 x WD Red 4TB on Z1
1 GB igb0 Interface 192.168.1.154
10 GB ix0 Interface 192.168.100.2[via X540T1 NIC PCIe]

Server #2:
FreeNAS-9.10.2 U6 Stable
Intel Xeon e3-1220v5 @ 3.0GHz
ASUS P10S-M WS Motherboard
Kingston Technology 32GB 2133MHz DDR4 ECC Memory
8 x WD Red 4TB on Z2
1 GB re0 Interface 192.168.1.247
10 GB ix0 Interface 192.168.100.1[via X540T1 NIC PCIe]

Since I have no SSD on both servers, I guess I have to settle with the current performance. I supposed the transfer speed is limited by the write speed of the WD Red HDD:

I have 2 SSD [Samsung 850 EVO 500GB and a Crucial MX200 250GB] lying around unused, but I doubt adding them as L2ARC for the servers would've helped much as my transfers are just copying/moving files once from one server to another another.

Perhaps defining them as 'intermediary datasets' for copy of mainly media files between the sizes of 2GB-16GB between the servers, before moving them into permanent directories for PLEX, maybe a better idea...I should be getting better write speeds...

Anyways, I appreciate all ideas shared.

Thank you & Best Regards

 

[FNoob]

Very good and informative thread, thanks to the OP and others who shed some light in here.
Just want to ask if a (third option) would work
Server 1: (dual) SFP+ ports
Server 2: (dual) SFP+ ports
PC : (dual) SFP+ ports

Server 1 connects (directly) to Server 2 by using only (1) of the SFP+ ports on each server , and then:
Both Server 1 + Server 2 connect (the remaining) SFP+ (on each server) directly to the (2) empty SFP+ on the main PC

Doable ? impossible ? Dunno
Just asking to eliminate the cost of 10GbE switch , and have flexibility of having both : rsync in a dataset level , and in file level
Thanks

 

[Johnnie Black]

Doable ?

Don't see why not as long as correctly configured, I have a different config but the principle is the same, my desktop has two dual 10GbE NICs directly connected to 4 servers, and all of them are also connected with Gbe to a switch, so all traffic between my desktop and the servers uses 10GbE, all other traffic, including server to server uses gigabit.

 

[FNoob]

@Johnnie Black
Cool, thx 4 the reply. I'll see what I can afford and either opt for triple dual NICs, or just do as what the OP did