ConnectX-2 - Great Write, but slow Read Speed

[Grinchy]

Hello,

I'm trying to get my new 10Gbit Network working, but while copying something from my NAS (FreeNAS 11) -> PC (Windows 10), it starts with ~400MB/s and gets slower down to ~50 - 80 MB/s in seconds. Its slower than my old gbit connection. The other way (PC > NAS) it works great, with about 450MB/s.

I'm using an ConnectX-2 on both sites with a ~30m Fiber Connection. On my Windows System the latest Driver are installed for the Card. Also, tried to exchange the cards between PC and NAS but, it's exactly the same Problem.

Tried it with "gstat -p" and PC -> NAS, Drives are about ~80-100% usage. NAS -> PC, Drives are about 10% usage.

I really don't know anymore what else I could try. Would be so great if someone could help me here.

Patrick

 

[BigDave]

My experience has shown me that two issues are generally responsible for slow speeds.
First Issue: Drives that contain a lot of data will not write data as fast as an empty drive, as they fill up,
they slow down.

If I transfer a big fat MKV file from my server to the NVME OS drive on my desktop Windows 10, I average 470 MB/s


If I transfer the same type of file from my server to a 65% full 3TB HDD
on that same desktop machine, It starts that fast, but drops to 96 MB/s

When this 3TB drive was new, the write speed was north of 125 MB/s

Second Issue: I don't know how your fiber is terminated, but the brand of transceivers will some times affect speeds and also the quality of any direct attach cables.

ConnectX-2 on both sites with a ~30m Fiber Connection

 

[BigDave]

This resource will be a good read, it's long but worth the time spent to digest.
https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/

 

[Grinchy]

My experience has shown me that two issues are generally responsible for slow speeds.
First Issue: Drives that contain a lot of data will not write data as fast as an empty drive, as they fill up,
they slow down.

I'm using a Samsung Pro NVME SDD on the Windows Side and a 7x6TB Raid Z2 on the NAS. That shouldn't be the bottleneck. Also tried it with an Samsung Evo 850 (1TB) on the NAS Side, but the read speeds had the same Problem.

Second Issue: I don't know how your fiber is terminated, but the brand of transceivers will some times affect speeds and also the quality of any direct attach cables.

I'm using A7EL-SN85-ADMA SFP+-SR-10G-300M-850NM transceivers on both sides. Tried to exchange them between PC and NAS, but the bad Read Speed remains the same.

This resource will be a good read, it's long but worth the time spent to digest.
https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/

Thank you for the link. Great Info, but unfortunately nothing that helps me with my "special" Problem :-(

 

[Johnnie Black]

Are you using 10GbE optimized tunables?

I get around 400MB/s read and 600MB/s write speed to my HDD pool with the same ConnectX-2, rest of the specs on sig, I'm using these tunables I've found on the forum somewhere:

Code:

kern.ipc.maxsockbuf=16777216
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_inc=32768
net.route.netisr_maxqlen=2048

 

[SweetAndLow]

Run some iperf tests.

 

[Grinchy]

iperf:

PC = Client, NAS = Server -> ~4,5GBit/s
PC = Server, NAS = Client -> ~ 9,3GBit/

Screenshots:
PC = Client

PC = Server

I don't know how to read this :-(

Why is there a difference of 50%?!

 

[SweetAndLow]

I'm not sure but my guess is something has a driver problem. But your network speed is still much faster than the real world speeds you are seeing. You're at the edge of my knowledge so I'm out of ideas.

 

[Grinchy]

Thanks for your answers!

I don't know why, but now I get around 500Mb/s Reading and 450Mb/s writing. Maybe the Client SSD had some performance issues.

 

[SweetAndLow]

Thanks for your answers!

I don't know why, but now I get around 500Mb/s Reading and 450Mb/s writing. Maybe the Client SSD had some performance issues.

Did the number of files or size of file change? Lots of small files transfer slower than one big file.