10GB en. Fast IPerf, slow data transfers?

[Urglefloggah]

Hi all,

I’ve just built a FreeNAS box to replace my Netgear NAS.

The hardware is all new: HP Proliant ML10, 12Gbyte of compatible DDR4 ECC RAM, Intel G4400 processor and 2x WD 1Terrabye Red drives as a stripe, with another to be added as parity when it arrives.

Once set up and running satisfactory with FreeNAS 11, I added a used Mellanox ConnectX-2 card to the pci8 port, and one to my workstation running Debian 9. These are configured on a separate subnet and talk to each other fine.

The problem I have is that while running iperf gives a transfer rate of 9.85-9.89 Gbits / sec, when I transfer files over this connection I rarely achieve 150kbits/sec. After much faffing about with settings and jumbo packets, very little has changed. The only thing left I can think of is a driver problem, and I will try my Workstation this weekend with Ubuntu 16.04 with the latest driver from Mellanox (there is no official Debian 9 driver, only what’s inbuilt into the OS).

Can anyone offer any useful insights?

Thanks,
Nik.

 

[dlavigne]

How are you transferring the files? What is the output of ifconfig?

 

[Urglefloggah]

I’ve tried NFS, AFP and Samba, with AFP being slightly faster. ifconfig shows 9000 byte frames and a 10000Kb connection.

Last night I tried another copper cable and Ubuntu 16.04 on my workstation with the most recent drivers. The result was slightly faster at around 160Mb read and 125Mb write.

 

[joeschmuck]

How about small file sizes and fragmentation is causing some of the pain you are feeling?

 

[Urglefloggah]

I don't think so. Big ISO's are just the same.