SOLVED Can Someone Interpret This For Me?

[joeschmuck]

With respect to the transfer rates, try FreeNAS 9.2.1.6. Unfortunately I cannot speak to AFP, I can't test that at my house since I don't have any apple computers. The dd speed test seams a little slow at 108MB/s. I'm off to work yet again, I should be able to get back in a few hours with some recommendations. I'd like to know what you set up the ECC Check speed for in the BIOS, although I doubt it would have any real affect on the transfer speed issue you are having. Are you encrypting your drives? Is your heatsink on good? Have you run any stability products to ensure your CPU and RAM work properly? (CPU test for 2 hours max, RAM test for 24+ hours, more is better in the RAM test) Right now I'm more concerned with your CPU. And I hate saying this but adding an Intel NIC could save the day if your CPU is an issue.

 

[joeschmuck]

Okay, so getting back to benchmarking... Here is the link to my testing http://forums.freenas.org/index.php?threads/intel-nic-vs-realtek-nic-performance-testing.10325/ and if you could conduct test 1 and post the results of each test.

Other things to look at is your Ethernet cable(s). This is why when testing you FreeNAS for throughput issues we recommend a single direct Ethernet cable between your NAS and the computer you are using. We also make an assumption that the computer you are using can saturate the Gb connection as well. You will have to ensure that is true or you could be chasing a problem which doesn't really exist in the FreeNAS system.

Other things you could look at is the CPU usage during a transfer... are you using a lot? I doubt it but you never know. And of course you have your jails turned off as well, correct? It's no fun chasing down a problem which was self-induced like leaving a jail open which uses the network connection and thus taking away some of the bandwidth.

Just keep an open mind and think like a troubleshooter and do not assume anything at all.

And I think your file sharing sounds fine with respect to the MiniDLNA jails you have established.

 

[NiceTry]

The 108MB/s is internal speed of my computer, not the FreeNAS. I have a Mac Mini with internal 2.5" laptop drives. Straight disc performance, no striping, etc. FreeNAS RAIDZ1 does internal transfer rates of 1.56GB/s.

ECC is set to "Good." I've not tried to adjust any clocking parameters. No encryption, just lz4 compression. A dd test across my network, Computer to the FreeNAS (/dev/zero on computer ->tmp.dat on FreeNAS), yielded a steady 100MB/s which pretty much uses 800Mb/s of the available network bandwidth.

BTW, is there a way to confirm FreeNAS is using all 4 cores? Using Top, the biggest load I've seen is 0.6 which is consistent with the reporting graph of CPU on FreeNAS. "Platform" under System Information lists "AMD FX(tm)-4130 Quad-Core Processor."

All performance numbers are with the jail enabled.

 

[joeschmuck]

I'm a bit confused, please post the output of your dd test for the FreeNAS machine, 1.56GB/s seams a bit fast, mine is about 315MB/s in the first test and 375MB/s for the second test.

EDIT: Please put into code brackets all the dd stuff you type in and the screen output so i can see exactly what you are doing.

The ECC setting on Good should be fine.

I don't follow you on the results of doing a dd over your network connection. you said you got a steady 100MB/s which sounds like things are working fine for a 1Gb/sec connection.

And lastly the cores issue, yea, FreeNAS is not a well threaded products but it will make use of the cores you have. You will find out that your CPU will not be under heavy load 99% of the time. Actually I don't think I've ever seen a high load except for one version of FreeNAS which has a Samba issue which sucked the CPU time away to 100%. If you transcode video using something like Plex, you will see some load but it's still minor.

 

[cyberjock]

Your pool has compression enabled. dd writes zeros if you used /dev/zero and that will ALWAYS give impossibly high results....

So yeah, test results meant nothing at all.

 

[joeschmuck]

Arg, I don't run compression, never felt the need to, that and I use my NAS primarily for backups and zip files, PDFs, etc... which are already compressed or do not generally benefit from compression. I have very little data which could benefit from compression.

Thanks Cyberjock for jumping in and pointing that out.

 

[cyberjock]

Yeah, anyone that gets impossibly high numbers is using compression. Since 9.2.0 or 9.2.1 (I forget which) has made lz4 the default on pool creation (and the WebGUI doesn't even hint at it during the creation process) most users don't even know.

 

[NiceTry]

A little slower this time. Jail enabled.

[root@ServerA /mnt/MS0/SERVERFILES0]# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 73.164668 secs (1467568776 bytes/sec)
[root@ServerA /mnt/MS0/SERVERFILES0]#

Yes, 100MB/s over the net. And yes, it's working near capacity. A real file transfer from my not-too-fast HD's would peak at 100MB/s and settle in at between 40 and 90MB/s. That includes all the overhead on both ends.

cyberjock: I get it. All zeroes compress down to nothing. So all it is testing is the processor speed doing the compression and a little bit of transfer. Can I turn off compression for the dataset without damaging anything?

But, I think it's working about as good as it is going to get. Do you guys agree?

 

[joeschmuck]

The only thing I don't like is you reporting the 40-90MB/s speeds for a single large file transfer. I'm curious what is causing the large fluctuation. Otherwise Ithink it's working fine.

Cyberjock will have to answer the compression issue, unless it's in the manual somewhere (I haven't looked).

 

[NiceTry]

Yes, I'd like to understand that as well. All this started when the fluctuations in speed dropped transfers to near zero when a jail was present. No jail, fluctuations in the 40-90MB/s range, average of 65-75MB/s. With jail, starts high and ground down to a few MB/s. It appears the 8GB RAM has addressed the jail-no jail problem.

Disk Speed Test results. See attached .jpg. I don't know the details of what is actually being tested or how. I have selected the highest stress option for the test although the results don't seem to change much low-to-high stress.

I will do a dd of a real file from the computer to FreeNAS, with compression as is, as a reference.

 

[cyberjock]

Yep...


zfs set compression=off <poolname>

when done doing your test you can reset it back to default with...

zfs set compression=lz4 <poolname>

 

[solarisguy]

@NiceTry, as cyberjock mentioned, when creating a ZFS volume/pool the FreeNAS GUI does not give a choice to have compression or not. However, immediately after volume/pool creation you can adjust Compression level: using icon at the bottom named Edit ZFS Options. In most cases, you only want to toggle between Off and lz4.

For real speed testing, create a compressed file that is at least 150% of your RAM size, 200% is better. (Concatenate some video files. If you do not have any grab some from YouTube. You can use one file many times.)

 

[NiceTry]

Latest test:

Code:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 569.347947 secs (188591498 bytes/sec)
[root@ServerA /mnt/MS0/SERVERFILES0]#

Is this more consistent with SATA I drive performance? My 11.74GB H.264 video file averaged about 60MB/s with compression turned off.

solarisguy: My test file is 11.74GB, almost 150% of the 8GB ram. I've adjusted the configuration so the pool is off and the datasets inherit using EDIT ZFS Op. In the above test, I turned off compression using the GUIs' edit feature.

Maybe it's time to upgrade my drives.

 

[solarisguy]

With very large video files whether you have lz4 compression enabled or not plays no role, since lz4 algorithm becomes pass-through if it is unable to compress the beginning of the file.

Take a look at the post http://forums.freenas.org/index.php?threads/zfs-volume-shrinking.22062/#post-130316 ! The user has >12TB of data and lz4 compressed less than 1%.

There very many factors influencing performance. Still more art, than science ;) If testing internal speeds, you really want to have another (second, third etc.) controller and a very fast SSD. Then in the shell you can copy back and forth files larger than your RAM and have meaningful results. SSD is not a must, a single very fast hard drive would be enough too.

 

[joeschmuck]

Not sure why your drive speed is so slow but your drives could be causing the issue. I know Cyberjock would typically jump in here and tell you that the Realtek NIC is causing some of your slowdown and while it would be a true statement, the slowdown should be very minor given your CPU.

What model are your drives? Also could you output the results of

Code:

zpool status

and wrap it in code brackets as it retains the formatting of the data. If you don't know what that looks like, I'm going to adjust your earlier posting to make that change and you can look at it by hitting "edit" for your posting.

 

[NiceTry]

Not sure why your drive speed is so slow but your drives could be causing the issue. I know Cyberjock would typically jump in here and tell you that the Realtek NIC is causing some of your slowdown and while it would be a true statement, the slowdown should be very minor given your CPU.

What model are your drives? Also could you output the results of

Code:

zpool status

and wrap it in code brackets as it retains the formatting of the data. If you don't know what that looks like, I'm going to adjust your earlier posting to make that change and you can look at it by hitting "edit" for your posting.

Code:

[root@ServerA ~]# zpool status
pool: MS0
state: ONLINE
scan: scrub repaired 0 in 1h34m with 0 errors on Thu Jun 26 19:34:58 2014
config:

NAME STATE READ WRITE CKSUM
MS0 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/5b1086b2-f916-11e3-a2db-002421ecc4e5 ONLINE 0 0 0
gptid/5b60aca9-f916-11e3-a2db-002421ecc4e5 ONLINE 0 0 0
gptid/5bbcd93b-f916-11e3-a2db-002421ecc4e5 ONLINE 0 0 0
gptid/62f93973-fa18-11e3-bbc9-002421ecc4e5 ONLINE 0 0 0

errors: No known data errors #

3 of the drives are Seagate Barracuda 7200.12 ST31000528AS 3.0Gb/s. The fourth is a Hitachi, HGST Deskstar 7K1000.C HDS721010CLA332. I had four Seagates but one was reporting bad, unrepairable sectors so I replaced it with the Hitachi. All are 1TB and SATA II.

Do you think the internal disk speed is slow (188MB/s)? With the history of jail interference, should I disable the jail and run the internal test again?

Edit: I turned off the minidlna plugin, stopped the jail, edited the jail configuration to not start on reboot and rebooted the system. dd internal transfer from /dev/zero yielded the same 185-ishMB/s result. Compression is off.

 

[solarisguy]

Possibly you are being hit by having RAID-Z1 with four drives, instead of three or five

 

[joeschmuck]

185MB should be able to saturate a 1Gb connection and you do have periods where you are almost saturating it. I'd say you should be able to get about 80MB/sec constantly. On my system I am getting from 105 to 125MB/sec reading and writing a 4GB mkv video file and I don't see why you can't get that either. Now 60MB/sec isn't bad either but it makes me feel like you have a much slower CPU than you have stated if this is all you can get out of it.

My suggestions are:
1) A direct Ethernet connection between the NAS and PC. Tell us the results of that.
2) Upgrade to FreeNAS 9.2.1.6 as the CIFS default setup is better (correct).
3) Replace the Ethernet cable.
4) If running through a router, is QOS turned on? I have a switch that has QOS that I can't turn off and is dependent on which port you use. This is why I recommend a direct connection.
5) Are you sure your PC can maintain a high throughput for the duration with some other device? It could be the issue.
6) Have you done a CPU stress test and validated the CPU model and speed? It's possible you bought a counterfeit part. It's a big thing now days. At least I hope that isn't the case.

Also, the jails should have no impact on your throughput, none at all unless you have some intensive task running and sucking up all the CPU cycles.

 

[NiceTry]

185MB should be able to saturate a 1Gb connection and you do have periods where you are almost saturating it. I'd say you should be able to get about 80MB/sec constantly. On my system I am getting from 105 to 125MB/sec reading and writing a 4GB mkv video file and I don't see why you can't get that either. Now 60MB/sec isn't bad either but it makes me feel like you have a much slower CPU than you have stated if this is all you can get out of it.

My suggestions are:
1) A direct Ethernet connection between the NAS and PC. Tell us the results of that.
2) Upgrade to FreeNAS 9.2.1.6 as the CIFS default setup is better (correct).
3) Replace the Ethernet cable.
4) If running through a router, is QOS turned on? I have a switch that has QOS that I can't turn off and is dependent on which port you use. This is why I recommend a direct connection.
5) Are you sure your PC can maintain a high throughput for the duration with some other device? It could be the issue.
6) Have you done a CPU stress test and validated the CPU model and speed? It's possible you bought a counterfeit part. It's a big thing now days. At least I hope that isn't the case.

Also, the jails should have no impact on your throughput, none at all unless you have some intensive task running and sucking up all the CPU cycles.

Code:

[root@ServerA /mnt/MS0/SERVERFILES0]# dd if=/dev/zero of=tmp.dat bs=2048k count=15k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 160.770165 secs (200362143 bytes/sec)
[root@ServerA /mnt/MS0/SERVERFILES0]#

I turned off atime in the pool and the internal speed increased to over 200MB/s.

The dd test on my computer only yielded 108MB/s for the large file transfer to the 2.5" HD's. If that is a true average, caching, buffering and transfer synchronization could explain the seemingly erratic transfer rates. I don't have another computer that I can stress the ethernet connection with. I'll try dd over the net from /dev/zero to FreeNAS. That should circumvent my computer's hard drive.

I too have seen bursts of over 100MB/s on my network. I don't see anything about QOS options in the setup of my router.

The info I have on the CPU comes from the BIOS setup page. It shows all the right stuff, at least as reported to the BIOS setup. How do you validate the model and speed? What stress test do you have in mind?

The performance quirks I'm seeing (variation in speed) are similar to that exhibited by my previous Intel-based FreeNAS build. The system elements that are common between the setups are the RAIDZ1 drives, and my desk computer. Different routers have been used in the past with the same variability. Now, new motherboard, AMD vs. Intel CPU, 8GB vs. 4GB RAM.

I'll have to round up some CAT5 cables to do the direct connect test.

Is the GUI update to 9.2.1.6 okay?

Thanks for the suggestions. It may take me a day or so to get these tests accomplished.

 

[solarisguy]

The speed people are talking about is a measure of a continuous transfer. Otherwise you depend on IOPS capability of your system components.