Copy speed is fast for a while and then drops

[mikesoultanian]

Hi,
I have a Windows box that is connected to two FreeNAS boxes via iSCSI over 10Gb ethernet. I need to copy a few TB worth of data from one FreeNAS box to the other so opened each drive and dragged the folders from one box to the other and speed (seemed) to start really high (~4Gb/s per the network resource view on the TX box) and then after 5 minutes or so drops down to ~1Gb/s. I am seeing a discrepancy between the TX and RX box:

TX box:

RX box:

So, I'm guessing the TX box isn't quite accurate?

All of these machines have dual 10Gb NICs, but I can't figure out how to get MPIO working, so I'm only seeing traffic on one NIC on each of the FreeNAS boxes - I'm kinda new to FreeNAS and MPIO so I don't even know what all is possible, but I figured it might be worth mentioning. Should I be able to get more throughput to these?

TX FreeNAS box:
Build FreeNAS-9.10.1-U4 (ec9a7d3)
Platform Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz
Memory 196466MB
System Time Thu Aug 03 12:30:52 PDT 2017
Uptime 12:30PM up 11 days, 10:37, 0 users
Load Average 1.28, 1.36, 1.29

RX FreeNAS box:
Build FreeNAS-9.10.1-U4 (ec9a7d3)
Platform Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz
Memory 65400MB
System Time Thu Aug 03 12:32:28 PDT 2017
Uptime 12:32PM up 9 days, 11 hrs, 0 users
Load Average 1.04, 2.53, 2.99

Windows box:
OS Windows Server Datacenter 2012 R2
Platform Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.0GHz (2 processors)
Memory 192GB



I hope that is helpful - if you need me to post any more information, just let me know.

Thanks in advance!!
Mike

 

[Thomas102]

Hi,

You might have ARC in action and your windows buffering data at first explaining the peak.
When buffer is full TX align to RX.

 

[mikesoultanian]

Hi Thomas,
What is ARC? And would that buffer really go for 5 minutes?

Should I expect faster throughput?

Thanks!

 

[Thomas102]

ARC is zfs cache, the reason you have 192GB Ram on FreeNAS TX.
Your windows also have 192GB ram, Such buffer will sustain 400 seconds at 4gb/s. So yes it can go for the 3 minutes on your graph.
Your throughput is equivalent to a single SATA hard disk. It is not fast.
You should check where is located the iscsi extent on RX.

 

[mikesoultanian]

Ahh.. ok, that makes sense! Thanks for the help. We had a cluster disaster and I had to move all of our VMs to our backup FreeNAS box, but I did not configure it, nor do I know what kind of drives were in there. We assumed there were SAS drives in there, but they could very well be SATA spinning disks. We'll be migrating away from it shortly!

Thanks

 

[Thomas102]

Ahh.. ok, that makes sense! Thanks for the help. We had a cluster disaster and I had to move all of our VMs to our backup FreeNAS box, but I did not configure it, nor do I know what kind of drives were in there. We assumed there were SAS drives in there, but they could very well be SATA spinning disks.
Thanks

Off course this is pure speculation but actually it can be worst. The iSCSI target might be a test one hosted on a single disk without parity protection.