Very slow local disk transfer rates on FreeNAS

[plague]

Greetings everyone!

When moving files internally in FreeNAS, I noticed that the "move" transfer rates are an appalling 1.5 MBps - 2 MBps. This explains why I experience really slow network transfer speeds as well. Would someone be kind to help me identify why I am getting such slow speeds to begin with?

OS Specs
FreeNAS-11.0-U2 (e417d8aa5)

Hardware Specs

• Motherboard make and model: Apple Mac Pro (5,1)
• CPU make and model: 6-Core: 3.33GHz Intel Xeon “Westmere” W3680 processor
• RAM quantity: 32 GB ECC DDR3 1333 MHz
• Hard drives, quantity, model numbers, and RAID configuration: Five 4TB Western Digital Red drives in RaidZ2 configuration
• SATA 2 controller

Monitoring Metrics (moving 4GB worth of data internally)

CPU stats

Disk stats

Memory stats

 

[joeschmuck]

When moving files internally in FreeNAS, I noticed that the "move" transfer rates are an appalling 1.5 MBps - 2 MBps.

I'm going to treat you like we are both terrible at communications and ask you a lot of stupid questions so that we do not make assumptions.

What does that statement above mean. When I read it I think you are moving a file via the Shell and typing mv, not on a computer over the LAN copying a file via SMB or other protocol. The two are very different. Also, specify the protocol you are using for LAN transfers and the destination device/system. And if you are using WiFi, state that too.

If you have slow speeds and you are doing the mv then you have soemthign internal causing your slowdown. The Apple Mac Pro could be the cause. You may not have proper device support for this hardware.

You can perform a dd test to find out how fast an internal transfer rate really is. Check out this link.

Post the results from the dd tests.

 

[plague]

I'm going to treat you like we are both terrible at communications and ask you a lot of stupid questions so that we do not make assumptions.

What does that statement above mean. When I read it I think you are moving a file via the Shell and typing mv, not on a computer over the LAN copying a file via SMB or other protocol. The two are very different. Also, specify the protocol you are using for LAN transfers and the destination device/system. And if you are using WiFi, state that too.

If you have slow speeds and you are doing the mv then you have soemthign internal causing your slowdown. The Apple Mac Pro could be the cause. You may not have proper device support for this hardware.

You can perform a dd test to find out how fast an internal transfer rate really is. Check out this link.

Post the results from the dd tests.

Thank you for the reply joeschmuck! My apologies for the confusion I caused! By "move" I meant cut (cmd + c) and paste (cmd + option + v) files from one folder on the NAS to another folder on the NAS. I performed that operation by accessing the NAS directories via AFP using a MacBook Pro over wifi.

I performed the data duplicator (dd) test you recommended and the following are the results:

The test reveals that I have 122 MBps of transfer rate. Since FreeNAS is installed on a SATA 3 SSD, I ran the same test with the output file located on one of the spinning NAS disks and the rate was 122 MBps too.

 

[MrToddsFriends]

There is no hard disk or other mass storage device involved in a test like

Code:

dd if=/dev/random of=/dev/null bs=1024k count=5k

A test like this was just given to show that /dev/random is very cpu intensive. /dev/random and /dev/null are special Unix devices.
https://en.wikipedia.org/wiki//dev/random
https://en.wikipedia.org/wiki/Null_device
Please read this thread again.

 

[SweetAndLow]

Don't use wifi it's always slow. Use dd to test your pool speed with if=/dev/zero and no compression for writes and of=/dev/null for reads. bs=1M is also going to give you the best numbers.

Sent from my Nexus 5X using Tapatalk

 

[plague]

There is no hard disk or other mass storage device involved in a test like

Code:

dd if=/dev/random of=/dev/null bs=1024k count=5k

A test like this was just given to show that /dev/random is very cpu intensive. /dev/random and /dev/null are special Unix devices.
https://en.wikipedia.org/wiki//dev/random
https://en.wikipedia.org/wiki/Null_device
Please read this thread again.

Got it! I read the thread again and used the values SweetAndLow recommended. The following are the results:

Code:

[root@Alexandria ~]# dd if=/dev/zero of=/mnt/Alexandria/Music/ddfile bs=1M count=10k												
10240+0 records in																												
10240+0 records out																												
10737418240 bytes transferred in 28.879363 secs (371802457 bytes/sec)															  
[root@Alexandria ~]# dd of=/dev/null if=/mnt/Alexandria/Music/ddfile bs=1M count=10k												
10240+0 records in																												
10240+0 records out																												
10737418240 bytes transferred in 1.824299 secs (5885776698 bytes/sec)																											

I will try to connect to my NAS over ethernet to get around the poor wifi performance.

 

[joeschmuck]

WiFi, I thought that was the culprit. When I see low transfer rates like that, this is what I think.

 

[SweetAndLow]

Got it! I read the thread again and used the values SweetAndLow recommended. The following are the results:

Code:

[root@Alexandria ~]# dd if=/dev/zero of=/mnt/Alexandria/Music/ddfile bs=1M count=10k												
10240+0 records in																												
10240+0 records out																												
10737418240 bytes transferred in 28.879363 secs (371802457 bytes/sec)															  
[root@Alexandria ~]# dd of=/dev/null if=/mnt/Alexandria/Music/ddfile bs=1M count=10k												
10240+0 records in																												
10240+0 records out																												
10737418240 bytes transferred in 1.824299 secs (5885776698 bytes/sec)																											

I will try to connect to my NAS over ethernet to get around the poor wifi performance.

Your write speeds look exactly what they should be. Read are a little skewed because your done was smaller than your arc but that ok.

Next you should test your network speed using iperf or iperf3 for Windows. if using WiFi your problem is wifi, it's always slow just the way it is.

Sent from my Nexus 5X using Tapatalk

 

[plague]

Your write speeds look exactly what they should be. Read are a little skewed because your done was smaller than your arc but that ok.

Next you should test your network speed using iperf or iperf3 for Windows. if using WiFi your problem is wifi, it's always slow just the way it is.

Sent from my Nexus 5X using Tapatalk

I performed the iperf test over wifi since that is the medium I have been using and the results are as follows:

NAS output:

Code:

[root@Alexandria ~]# iperf -s																									  
------------------------------------------------------------																		
Server listening on TCP port 5001																								  
TCP window size: 64.0 KByte (default)																							  
------------------------------------------------------------																		
[  4] local 192.168.86.68 port 5001 connected with 192.168.86.63 port 63227														
[ ID] Interval	   Transfer	 Bandwidth																						
[  4]  0.0-100.1 sec   833 MBytes  69.8 Mbits/sec						  

Client output:

Code:

➜ iperf -c 192.168.86.68 -p 5001 -P 1 -i 1 -f k -t 100
------------------------------------------------------------
Client connecting to 192.168.86.68, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  7] local 192.168.86.63 port 63227 connected with 192.168.86.68 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  7]  0.0- 1.0 sec  6144 KBytes  50332 Kbits/sec
[  7]  1.0- 2.0 sec  6400 KBytes  52429 Kbits/sec
[  7]  2.0- 3.0 sec  6528 KBytes  53477 Kbits/sec
[  7]  3.0- 4.0 sec  6656 KBytes  54526 Kbits/sec
[  7]  4.0- 5.0 sec  7552 KBytes  61866 Kbits/sec
[  7]  5.0- 6.0 sec  8576 KBytes  70255 Kbits/sec
[  7]  6.0- 7.0 sec  6272 KBytes  51380 Kbits/sec
[  7]  7.0- 8.0 sec  7680 KBytes  62915 Kbits/sec
[  7]  8.0- 9.0 sec  7296 KBytes  59769 Kbits/sec
[  7]  9.0-10.0 sec  6400 KBytes  52429 Kbits/sec
[  7] 10.0-11.0 sec  6016 KBytes  49283 Kbits/sec
[  7] 11.0-12.0 sec  5376 KBytes  44040 Kbits/sec
[  7] 12.0-13.0 sec  6272 KBytes  51380 Kbits/sec
[  7] 13.0-14.0 sec  7808 KBytes  63963 Kbits/sec
[  7] 14.0-15.0 sec  7552 KBytes  61866 Kbits/sec
[  7] 15.0-16.0 sec  7296 KBytes  59769 Kbits/sec
[  7] 16.0-17.0 sec  7168 KBytes  58720 Kbits/sec
[  7] 17.0-18.0 sec  8960 KBytes  73400 Kbits/sec
[  7] 18.0-19.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 19.0-20.0 sec  8704 KBytes  71303 Kbits/sec
[  7] 20.0-21.0 sec  8320 KBytes  68157 Kbits/sec
[  7] 21.0-22.0 sec  8832 KBytes  72352 Kbits/sec
[  7] 22.0-23.0 sec  9344 KBytes  76546 Kbits/sec
[  7] 23.0-24.0 sec  8576 KBytes  70255 Kbits/sec
[  7] 24.0-25.0 sec  6656 KBytes  54526 Kbits/sec
[  7] 25.0-26.0 sec  8704 KBytes  71303 Kbits/sec
[  7] 26.0-27.0 sec  6400 KBytes  52429 Kbits/sec
[  7] 27.0-28.0 sec  8704 KBytes  71303 Kbits/sec
[  7] 28.0-29.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 29.0-30.0 sec  8192 KBytes  67109 Kbits/sec
[  7] 30.0-31.0 sec  8192 KBytes  67109 Kbits/sec
[  7] 31.0-32.0 sec  9984 KBytes  81789 Kbits/sec
[  7] 32.0-33.0 sec  9472 KBytes  77595 Kbits/sec
[  7] 33.0-34.0 sec  9728 KBytes  79692 Kbits/sec
[  7] 34.0-35.0 sec  8832 KBytes  72352 Kbits/sec
[  7] 35.0-36.0 sec  9856 KBytes  80740 Kbits/sec
[  7] 36.0-37.0 sec  10880 KBytes  89129 Kbits/sec
[  7] 37.0-38.0 sec  9856 KBytes  80740 Kbits/sec
[  7] 38.0-39.0 sec  9600 KBytes  78643 Kbits/sec
[  7] 39.0-40.0 sec  10752 KBytes  88080 Kbits/sec
[  7] 40.0-41.0 sec  10752 KBytes  88080 Kbits/sec
[  7] 41.0-42.0 sec  9344 KBytes  76546 Kbits/sec
[  7] 42.0-43.0 sec  9856 KBytes  80740 Kbits/sec
[  7] 43.0-44.0 sec  9472 KBytes  77595 Kbits/sec
[  7] 44.0-45.0 sec  9984 KBytes  81789 Kbits/sec
[  7] 45.0-46.0 sec  10240 KBytes  83886 Kbits/sec
[  7] 46.0-47.0 sec  9984 KBytes  81789 Kbits/sec
[  7] 47.0-48.0 sec  9472 KBytes  77595 Kbits/sec
[  7] 48.0-49.0 sec  9472 KBytes  77595 Kbits/sec
[  7] 49.0-50.0 sec  9600 KBytes  78643 Kbits/sec
[  7] 50.0-51.0 sec  9856 KBytes  80740 Kbits/sec
[  7] 51.0-52.0 sec  9856 KBytes  80740 Kbits/sec
[  7] 52.0-53.0 sec  9600 KBytes  78643 Kbits/sec
[  7] 53.0-54.0 sec  7680 KBytes  62915 Kbits/sec
[  7] 54.0-55.0 sec  9728 KBytes  79692 Kbits/sec
[  7] 55.0-56.0 sec  10880 KBytes  89129 Kbits/sec
[  7] 56.0-57.0 sec  10624 KBytes  87032 Kbits/sec
[  7] 57.0-58.0 sec  11264 KBytes  92275 Kbits/sec
[  7] 58.0-59.0 sec  10496 KBytes  85983 Kbits/sec
[  7] 59.0-60.0 sec  4608 KBytes  37749 Kbits/sec
[  7] 60.0-61.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 61.0-62.0 sec  8448 KBytes  69206 Kbits/sec
[  7] 62.0-63.0 sec  7424 KBytes  60817 Kbits/sec
[  7] 63.0-64.0 sec  9216 KBytes  75497 Kbits/sec
[  7] 64.0-65.0 sec  7552 KBytes  61866 Kbits/sec
[  7] 65.0-66.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 66.0-67.0 sec  8960 KBytes  73400 Kbits/sec
[  7] 67.0-68.0 sec  9600 KBytes  78643 Kbits/sec
[  7] 68.0-69.0 sec  8576 KBytes  70255 Kbits/sec
[  7] 69.0-70.0 sec  9472 KBytes  77595 Kbits/sec
[  7] 70.0-71.0 sec  9088 KBytes  74449 Kbits/sec
[  7] 71.0-72.0 sec  7680 KBytes  62915 Kbits/sec
[  7] 72.0-73.0 sec  7936 KBytes  65012 Kbits/sec
[  7] 73.0-74.0 sec  7168 KBytes  58720 Kbits/sec
[  7] 74.0-75.0 sec  8832 KBytes  72352 Kbits/sec
[  7] 75.0-76.0 sec  7680 KBytes  62915 Kbits/sec
[  7] 76.0-77.0 sec  9216 KBytes  75497 Kbits/sec
[  7] 77.0-78.0 sec  8320 KBytes  68157 Kbits/sec
[  7] 78.0-79.0 sec  9216 KBytes  75497 Kbits/sec
[  7] 79.0-80.0 sec  8192 KBytes  67109 Kbits/sec
[  7] 80.0-81.0 sec  9728 KBytes  79692 Kbits/sec
[  7] 81.0-82.0 sec  7680 KBytes  62915 Kbits/sec
[  7] 82.0-83.0 sec  7936 KBytes  65012 Kbits/sec
[  7] 83.0-84.0 sec  6016 KBytes  49283 Kbits/sec
[  7] 84.0-85.0 sec  7424 KBytes  60817 Kbits/sec
[  7] 85.0-86.0 sec  7680 KBytes  62915 Kbits/sec
[  7] 86.0-87.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 87.0-88.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 88.0-89.0 sec  8064 KBytes  66060 Kbits/sec
[  7] 89.0-90.0 sec  7808 KBytes  63963 Kbits/sec
[  7] 90.0-91.0 sec  7040 KBytes  57672 Kbits/sec
[  7] 91.0-92.0 sec  7936 KBytes  65012 Kbits/sec
[  7] 92.0-93.0 sec  7552 KBytes  61866 Kbits/sec
[  7] 93.0-94.0 sec  10880 KBytes  89129 Kbits/sec
[  7] 94.0-95.0 sec  8704 KBytes  71303 Kbits/sec
[  7] 95.0-96.0 sec  9856 KBytes  80740 Kbits/sec
[  7] 96.0-97.0 sec  8448 KBytes  69206 Kbits/sec
[  7] 97.0-98.0 sec  9088 KBytes  74449 Kbits/sec
[  7] 98.0-99.0 sec  10496 KBytes  85983 Kbits/sec
[  7] 99.0-100.0 sec  11136 KBytes  91226 Kbits/sec
[  7]  0.0-100.0 sec  852864 KBytes  69857 Kbits/sec

I will get a thunderbolt to ethernet adapter and redo the test over wired connection. I will share the results again!

 

[SweetAndLow]

No need to share or test if you don't want to. Wifi is slow, problem solved. Don't ever expect high speed through wifi

Sent from my Nexus 5X using Tapatalk

 

[Stux]

Next you should test your network speed using iperf or iperf3 for Windows. if using WiFi your problem is wifi, it's always slow just the way it is.

Wifi is slower than ethernet in general, but the problem is that FreeNAS has very poor networking defaults for connections over WiFi.

So, maybe if people didn't just parrot "wifi is crap" in relation to these posts, we might be able to deal with the real problem, which is poor defaults in FreeNAS.

To the OP, once you determine that FreeNAS <-> client wifi performance is the problem, try the suggestions in this thread:
https://forums.freenas.org/index.ph...ly-to-freenas-sanity-check.43811/#post-292119

Meanwhile, if you're using AFP/SMB and you play with vfs_fruit then you might be able to get server-side moves working as well.

 

[plague]

Wifi is slower than ethernet in general, but the problem is that FreeNAS has very poor networking defaults for connections over WiFi.

So, maybe if people didn't just parrot "wifi is crap" in relation to these posts, we might be able to deal with the real problem, which is poor defaults in FreeNAS.

To the OP, once you determine that FreeNAS <-> client wifi performance is the problem, try the suggestions in this thread:
https://forums.freenas.org/index.ph...ly-to-freenas-sanity-check.43811/#post-292119

Meanwhile, if you're using AFP/SMB and you play with vfs_fruit then you might be able to get server-side moves working as well.

I reran the `ipperf` test with gigabit ethernet connection and I get 941 Mbps as opposed to 69.8 Mbps I got over wifi last week. Thanks Stux! I will follow the link you shared!