SOLVED Problem copying large file onto FreeNAS

[Apollo]

Okay, I tried to copy a 30GB-file from a USB 3.0 stick (260MB/Sec read, 170MB/sec write to/from the SSD) as well as from the internal SSD, but the problem remains.

By the way: Copying a large file from one dataset to another with the Windows client (so, copying from one share to the other), works flawlessly at about 82MB/sec.

I just realized, I didn't answer an earlier question: I don't use deduplication on the dataset, I'm copying the bis image files to/from and only use the standard lz4 compression on all datasets.

82MB/sec is good assuming it doesn't take into account the sum of the read speed and write speed.
Copying from a dataset to another dataset, means Freenas is not updating the link of the files, so this is good as it will have to travel across the network.
Also doing a read and write from the same array add latency, hence lower speed.
It does however take out the loop the SSD and USB 3.0.
Do you have a spare disk you can mount in Freenas and perform a copy as you just did, from one volume to the other over the network.

 

[Blues Guy]

Why would I want to do that? We established that the problem is on the client and copying from one dataset to another works perfectly well. Why should it be any different from one volume to another?

 

[Apollo]

Why would I want to do that? We established that the problem is on the client and copying from one dataset to another works perfectly well. Why should it be any different from one volume to another?

"Established" maybe a bit premature. Adding a second drive on Freenas and having the copy performed between the two volume will guarantee maximum disk throuput. Then doing the copy from intitating the command directly under Windows will cause Freenas to loopback the data via Windows over the network. This is an extra step of course but if it works without incident, we can exclude the network being the issues and try to focus on the real issue. I doubt however we can achieve full duplex max throughput but who knows.

 

[Blues Guy]

I have an easier solution for that. I did an iperf benchmark in both directions simultaneously. If I do it in only one direction, the connection is stable at 950Mbit/sec.
In both directions, it looks like this:
[ ID] Interval Transfer Bandwidth
[256] 0.0-120.0 sec 597 MBytes 41.7 Mbits/sec
[272] 0.0-120.0 sec 597 MBytes 41.7 Mbits/sec
[264] 0.0-120.0 sec 596 MBytes 41.7 Mbits/sec
[224] 0.0-120.0 sec 601 MBytes 42.0 Mbits/sec
[240] 0.0-120.0 sec 600 MBytes 41.9 Mbits/sec
[232] 0.0-120.0 sec 601 MBytes 42.0 Mbits/sec
[208] 0.0-120.0 sec 597 MBytes 41.7 Mbits/sec
[248] 0.0-120.0 sec 597 MBytes 41.7 Mbits/sec
[216] 0.0-120.0 sec 601 MBytes 42.0 Mbits/sec
[SUM] 0.0-120.0 sec 5983 MBytes 418 Mbits/sec

[460] 0.0-120.0 sec 1181 MBytes 82.5 Mbits/sec
[512] 0.0-120.0 sec 1498 MBytes 105 Mbits/sec
[444] 0.0-120.1 sec 1313 MBytes 91.7 Mbits/sec
[360] 0.0-120.1 sec 1209 MBytes 84.4 Mbits/sec
[476] 0.0-120.1 sec 1398 MBytes 97.6 Mbits/sec
[384] 0.0-120.1 sec 1269 MBytes 88.6 Mbits/sec
[416] 0.0-120.1 sec 1247 MBytes 87.1 Mbits/sec
[400] 0.0-120.1 sec 1220 MBytes 85.2 Mbits/sec
[492] 0.0-120.1 sec 1376 MBytes 96.1 Mbits/sec
[428] 0.0-120.5 sec 1215 MBytes 84.6 Mbits/sec
[SUM] 0.0-120.5 sec 12924 MBytes 900 Mbits/sec

 

[Apollo]

Not as simple as you would think.
With Iperf, you can adjust buffer size, number of threads and if I am not mistaken doesn't necessaraly use Samba shares.
It doesn't take into account drive throughput.
So in my opinion, the test is not representative of the system's performance with Samba and drives.

You can try setting up ftp services and run something like Filezilla to perform multi stream trnasfer.
This will be a little bit closer to reality, but you will have to deal with IO performance issues related to the drives. And yet, you will not be able to identify the source of the real bottleneck.

 

[Blues Guy]

I know well, what iperf tests and what it doesn't.
Of course, it doesn't use Samba (how would it) and doesn't test the pool.

But I have no indication that the pool is the bottleneck here, since gstat never showed a disc activity above 50% and the response time is always in the one-digit milliseconds. Also, iostat showed a pool write speed well above 220MB/sec. So, the pool isn't the bottleneck. CPU isn't as well, because TOP never showed any core being more than 20% busy, so this isn't the case either. By the way, the same holds true for the client.

The next thing is switches/cabling. And for this, iperf is a very good benchmark. Also, it tests the NIC speed and stability. Since I was able to reach wire-speed for long periods of time in one direction and also stable data rates in duplex-usage, this isn't the bottleneck either.

So, throwing another one-drive vdev into the server isn't going to give me any more clues than I already have.

What's more interesting is that I have a stable data-rate in iperf, where data is not read from a drive but instead generated in memory. So it can't be a simple driver-hickup, because then the numbers should be consistent with what I see in file transfer (given that the problems occur from an external drive, thumbdrive or internal SSD and the drive activity on the client is always well below 50%). Interestingly, I also have no problems transferring data from the internal SSD to an external drive/thumbdrive. The data rates there also are far above 150MB/Sec.

So, it must be a problem when reading data from a drive and transferring it over the PCI-E-NIC. Strange behaviour, nontheless. Disabling the antivirus-program didn't help (that *could* potentially block the file transfer before it's properly scanned. Perhaps this could be a problem with very large files, which don't fit into ram. But, it's not the case). Next, I'm trying installing the NIC into another slot, maybe this one is damaged. I don't think, this will help, because in that case, the problem should also arise with iperf.

 

[Blues Guy]

Okay, I finally solved that beast. It was actually a bit embarassing. During investigations for another problem, I found out that I accidentially deduped the dataset, I back up to and that pushed the system oft the cliff. Without dedup, it works fine and stable.