Looking to improve 70MB/s read speed over SMB/CIFS

[palmboy5]

I do think the CPU isn't the bottleneck here, and believe when you say your slower i3 than my i3 can push dual NICs. I mean, none of the outputs of top I have later in this post show the AMD CPU being even half used.

I don't see my 2125 model i3 in your benchmark link...? Not that it matters.

1. Oh 8.3.0 is released now, cool. I'll get to that at a leisurely-ASAP. :P I doubt it has any changes that would result in a performance improvement for me.
2. In my defense, the 8x2TB RAIDZ2 upgrade (from 4x2TB RAIDZ1) was a recent change and ok I'll consider getting some 8GB sticks as the motherboard only has two slots.
3. Highest iperf I saw was... 926 Mbit/s, but more typically around 890 Mbit/s. Also, the Windows port of iperf is horrible... the port receives alright, but sends at about 315 Mbit/s. I ran a VM of Ubuntu and used that to get actual benchmarks of the desktop and server NICs.
4. No thanks. If anything, the i3 temporary upgrade was more of an "overclock" than I would have been able to do with that AMD CPU/motherboard anyway. And throughput didn't improve.

Running iperf comes after upgrading RAM? Ordered in terms of cost and amount of time needed? :P

No compression, no deduplication (was deduplication even available before 8.3.0?). 4K sector checkbox was checked, but even if it wasn't... my dd speeds are well above gigabit ethernet and 4K or not wouldn't really be the factor here.
As for trying with consistent drives, I didn't use the SAS controller but here's something:
MSI H61M-P31 SATA II Ports - 3x WD30EZRX RAIDZ1 - i3 2125 3.3GHz

Code:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
124336268 bytes/sec
dd of=/dev/zero if=tmp.dat bs=2048k count=50k
122613320 bytes/sec

In CrystalDiskMark, this setup got 62 MB/s read and 106 MB/s write.


Here are some top outputs for different situations using the AMD setup specified in the OP, freshly rebooted:

3 minutes into dd if=/dev/zero of=tmp.dat bs=2048k count=50k:

Code:

last pid:  3343;  load averages:  0.81,  0.51,  0.29    up 0+00:11:21  18:29:12
51 processes:  1 running, 50 sleeping
CPU:  0.4% user,  0.0% nice, 26.1% system,  2.3% interrupt, 71.2% idle
Mem: 102M Active, 39M Inact, 5163M Wired, 816K Cache, 140M Buf, 2095M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3286 root          1  49    0  9788K  3060K tx->tx  0   0:19  8.69% dd

199022291 bytes/sec - what happened here?? I rebooted to try to fix this but it just refuses to even hit 200MB/s.

3 minutes into dd of=/dev/zero if=tmp.dat bs=2048k count=50k:

Code:

last pid:  3573;  load averages:  1.03,  0.89,  0.61    up 0+00:21:52  18:39:43
51 processes:  2 running, 49 sleeping
CPU:  0.0% user,  0.0% nice, 35.3% system,  4.3% interrupt, 60.4% idle
Mem: 103M Active, 39M Inact, 5207M Wired, 892K Cache, 142M Buf, 2051M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3517 root          1  59    0  9788K  3060K RUN     1   0:40 21.58% dd

448543526 bytes/sec

CrystalDiskMark
During Seq Read:

Code:

last pid:  3647;  load averages:  0.51,  0.80,  0.63    up 0+00:24:32  18:42:23
50 processes:  1 running, 49 sleeping
CPU:  1.1% user,  0.0% nice,  7.0% system,  1.1% interrupt, 90.8% idle
Mem: 101M Active, 39M Inact, 5186M Wired, 892K Cache, 143M Buf, 2073M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  49    0 50092K 10396K select  1   0:07  9.08% smbd

During Seq Write:

Code:

last pid:  3664;  load averages:  0.46,  0.74,  0.61    up 0+00:25:20  18:43:11
50 processes:  1 running, 49 sleeping
CPU:  1.5% user,  0.0% nice, 24.8% system, 11.1% interrupt, 62.7% idle
Mem: 101M Active, 39M Inact, 5186M Wired, 892K Cache, 143M Buf, 2073M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  62    0 50092K 10396K sbwait  0   0:16 24.76% smbd

During 512K Read:

Code:

last pid:  3693;  load averages:  0.17,  0.55,  0.55    up 0+00:27:06  18:44:57
50 processes:  1 running, 49 sleeping
CPU:  0.4% user,  0.0% nice,  4.5% system,  1.3% interrupt, 93.8% idle
Mem: 101M Active, 39M Inact, 5188M Wired, 892K Cache, 143M Buf, 2071M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  48    0 50092K 10468K select  0   0:24 10.06% smbd

During 512K Write:

Code:

last pid:  3722;  load averages:  0.53,  0.61,  0.57    up 0+00:27:24  18:45:15
50 processes:  1 running, 49 sleeping
CPU:  2.3% user,  0.0% nice, 28.0% system,  8.1% interrupt, 61.7% idle
Mem: 101M Active, 39M Inact, 5186M Wired, 892K Cache, 143M Buf, 2074M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  63    0 50092K 10468K sbwait  1   0:28 20.36% smbd

During 4K Read:

Code:

last pid:  3739;  load averages:  0.27,  0.53,  0.54    up 0+00:28:10  18:46:01
50 processes:  1 running, 49 sleeping
CPU: 13.2% user,  0.0% nice, 10.7% system,  1.5% interrupt, 74.6% idle
Mem: 101M Active, 39M Inact, 5184M Wired, 892K Cache, 143M Buf, 2075M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  51    0 50092K 10468K select  0   0:35 16.36% smbd

During 4K Write:

Code:

last pid:  3742;  load averages:  0.65,  0.60,  0.57    up 0+00:28:34  18:46:25
50 processes:  2 running, 48 sleeping
CPU:  0.4% user,  0.0% nice, 20.5% system,  3.4% interrupt, 75.8% idle
Mem: 101M Active, 39M Inact, 5069M Wired, 892K Cache, 143M Buf, 2191M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  50    0 50092K 10468K RUN     0   0:38 11.96% smbd

During 4K QD32 Read:

Code:

last pid:  3761;  load averages:  0.57,  0.58,  0.56    up 0+00:29:22  18:47:13
50 processes:  2 running, 48 sleeping
CPU: 12.0% user,  0.0% nice, 37.5% system,  6.4% interrupt, 44.1% idle
Mem: 101M Active, 39M Inact, 5054M Wired, 892K Cache, 143M Buf, 2206M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1 110    0 50092K 10468K CPU1    1   0:53 67.19% smbd

During 4K QD32 Write:

Code:

last pid:  3791;  load averages:  0.49,  0.56,  0.56    up 0+00:29:46  18:47:37
50 processes:  2 running, 48 sleeping
CPU:  1.9% user,  0.0% nice, 31.1% system,  2.6% interrupt, 64.4% idle
Mem: 101M Active, 39M Inact, 5132M Wired, 892K Cache, 143M Buf, 2127M Free
Swap: 8191M Total, 8191M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 3226 root          1  54    0 50092K 10488K RUN     0   1:03 25.29% smbd

 

[mattlach]

I built a system with an i3-530(it's one of the first gen i3s), your i3 should smoke my i3. I can max out both Gb NICs with my i3 simultaneously. That is, I can push/pull over 100MB/sec from both NICs at the same time(total of over 200MB/sec) with that machine. It has 16GB of RAM and kicks butt.

If you check out http://www.cpubenchmark.net/mid_range_cpus.html and compare my i3 to your AMD, my i3 has roughly 20% higher marks. I never take 1 benchmark to be a comparison of a CPU, nor do I assume that their arbitrary values are linear or logarithmic.

For you, I'd do the following to improve performance. Do take note that alot of people would be thrilled to get 70MB/sec. I got my speeds with only a small handful of tweaks to Samba, most of which are default now in FreeNAS 8.3.

Here's what I would try doing and the exact order I'd do them in based on cost and amount of time needed to implement the change.

1. Upgrade to FreeNAS 8.3 if you haven't. DO NOT upgrade the zpool to v28(have to do it manually from the CLI.. there is no option in the GUI to do this). Test speeds with autotune on and off, reboot between each change. Also disable the power management option in the FreeNAS GUI to make sure it's not making the CPU sleep too much. If you have no data on the zpool that you don't have elsewhere, you could try upgrading the zpool, but I wouldn't normally expect it to help performance.
2. Add more RAM to the server. 8GB of RAM for 12TB of zpool space is kind of stingy. You typically want 6GB minimum, with a thumbrule of 1GB of RAM per TB of data. 16GB seems to be a pretty sweet spot even for arrays up to 30-40TB. If I were building a server with more than 20TB of data I'd look at going to 24-32GB of RAM. As the guide I wrote and is stickied in the noobie section, if you aren't maxing out your RAM you shouldn't be complaining about stability and/or performance. RAM is one place where you shouldn't consider skimping.
3. Run iperf tests. If those tests aren't giving you over 900Mb/sec then something is wrong with your network topology. Find it and fix it. You have Intel NICs(Good boy!) so unless they are fake your NICs shouldn't be the problem. I typically get 950-980Mb/sec when I test my setup, but anything over 900Mb/sec is good.
4. Overclock your CPU slightly. Do not try to "unlock" more cores or do a significant overclock. Do a small overclock(5-10%) and see if performance jumps a lot. If there is a significant improvement, then I'd get a new CPU. But don't dismiss a CPU upgrade if performance doesn't jump significantly.
5. If all of this stuff turns up nothing, I'd do a CPU upgrade.

Some other food for thought:

-If you have compression on don't complain about your performance. Compression hurt performance, plain and simple. Same for deduplication. Of course, if you have deduplication on and you are having a problem requiring a post in this forum, you are a certified nutjob and I will stop replying to this thread ;)
-Check your BIOS settings. See if everything makes sense. Alternately, you can try disabling some of those power saving features and see if it affects performance. My last AMD computer was a POS because of the power savings features. (long story I won't go into but it's the reason why I'll never buy AMD again)
-I noticed you have advanced format and standard format hard drives. Did you check that box for the 4k sector size? You should have. If you didn't you can only fix it by destroying the zpool and recreating it. Also, it's not recommended that you mix the two formats. There have been mixed results with mixing formats. Maybe try a zpool with only 1 format type for testing?


If ALL of this stuff fails, you get to do one of two things.

1. Start tweaking FreeNAS to see if you can eek out more performance. There's some people that have posted comments of stuff in the forum that worked for them, but there's no "dummies guide" to tweaking unfortunately.
2. Buy more powerful hardware. By more powerful I mean use an i3+ ;). I can personally vouch that even 1st gen i3s make FreeNAS a smoking fast server while also being fairly low power. They kick butt AND take names. I can't vouch for how AMDs should perform, or yours in particular, so someone else would have to pipe up as to how their server performs for comparison with your CPU.

I assume you are running in a RAIDZ2 configuration. If you are running a RAIDZ1 you are not using the optimal hard drive configuration(5,7,9 drives) for a zpool.

Good suggestions! I was also going to suggest upgrading to FreeNAS 8.3.0, some people have reported miraculous speed increases with that upgrade.

How come you recommend against upgrading the Zpool? Is this a risky proposition, or is there something wrong with ZFS v28?

 

[mattlach]

1. Oh 8.3.0 is released now, cool. I'll get to that at a leisurely-ASAP. :P I doubt it has any changes that would result in a performance improvement for me.

I wouldn't be too sure about that. People have been reporting performance gains across the board with 8.3.0, probably due to a combination of a newer kernel and better drivers in combination with the disabling of Nagles algorithm, which apparently is supposed to help LAN speeds at the cost of hurting WAN speeds somewhat.

 

[cyberjock]

Highest iperf I saw was... 926 Mbit/s, but more typically around 890 Mbit/s. Also, the Windows port of iperf is horrible... the port receives alright, but sends at about 315 Mbit/s. I ran a VM of Ubuntu and used that to get actual benchmarks of the desktop and server NICs.

You may be the only person on the planet to say that the windows port of iperf is horrible. iperf has virtually been an industry standard for testing network bandwidth for years. Even my Android phone has iperf on it for testing wifi throughput.

The reason why I mentioned doing a slight overclock is because the i3 is obviously faster. But it also wasn't a significant performance increase. As new as that CPU is the hardware may not be supported very well with FreeBSD 8.3. Sometimes having "too new" of hardware can be a problem.

The reason why I chose upgrading the RAM over iperf tests is that since you have Intel NICs(which is 90% of the solution for people wanting more performance if CPU/RAM isn't horribly limited) I thought it was much more likely to be a RAM problem than a network problem. But since you had problems with the windows iperf I don't know. I've used the Windows version of iperf for years, even on Pentium 4s back when Intel was far from a performance leader. Iperf still rocked then.

I'm really not sure what the point of all of those pastes are. ZFS has advanced read and write caches, so performance at any one moment can appear to be very high and CPU usage can appear to be very low even if its CPU bottlenecked. That's why FreeNAS tests recommend doing a DD of something like 100GB (to force the cache to flush to disk) and to get a good average of data throughput and potentially CPU usage. Your uptime numbers are sometimes less than 30 seconds apart, so I really don't see much of a value in the data collected.

One guy did a 100GB DD and was upset because he got something like 1.8GB/sec and couldn't understand why his performance was so low over Samba despite CPU usage. He had compression on and the output of /dev/zero compresses REALLY well.

There's alot more to CPU performance than just looking at a percentage through TOP. IRQs can kill performance too. I've seen that Intel CPUs are typically more forgiving performance-wise with IRQs than AMD(of course, with each redesign of the CPU that could change too). IRQs will kill the actual performance, but TOP won't really "see" the killed performance.

I think the iperf thing should be investigated more thoroughly. The fact that you were only able to get 315Mb/sec is an indicator of something wrong. It could be that the NICs aren't the problem but are an indication of the problem. I'm not sure what I'd do but I'd probably get a 3rd computer and start doing iperfs back and forth between the 3 machines, maybe try a spare network switch or different cabling to see if that matters. I've never seen an iperf test get such a low value on Gb NICs. Even crappy Realtek NICs score at least 600Mb/sec.

 

[mattlach]

You may be the only person on the planet to say that the windows port of iperf is horrible. iperf has virtually been an industry standard for testing network bandwidth for years. Even my Android phone has iperf on it for testing wifi throughput.

Agreed. This almost suggests that there may be a problem on the Windows install, rather than an iPerf problem.

The reason why I chose upgrading the RAM over iperf tests is that since you have Intel NICs(which is 90% of the solution for people wanting more performance if CPU/RAM isn't horribly limited) I thought it was much more likely to be a RAM problem than a network problem.

Agreed, Intel NIC's are the way to go. Only issue with them is that every now and then a counterfit NIC will show up, that isn't what it claims to be, but that;s usually on the higher end multi-port NIC's rather than on the cheaper ones (as it diesn't necessarily pay off to counterfit cheaper parts).

If you want to be on the safe side, do an LSPCI from console to confirm its actually reported as an Intel part. This won't guarantee it (some counterfits actually use lower binned parts out of spec, that report themselves properly, but still don't work to their best, but it least its a simple check you can do).

I have found that BroadCom NetExtreme NIC's are also pretty solid, reliable and perform well.

One guy did a 100GB DD and was upset because he got something like 1.8GB/sec and couldn't understand why his performance was so low over Samba despite CPU usage. He had compression on and the output of /dev/zero compresses REALLY well.

HAHAHAHAH! Compressing a ton of zeroes as a performance test! :D

I think the iperf thing should be investigated more thoroughly. The fact that you were only able to get 315Mb/sec is an indicator of something wrong. It could be that the NICs aren't the problem but are an indication of the problem.

Maybe try swapping out patch cables too? Are you using Cat6 cables or Cat5/Cat5e cables? What kind of Switch are you using? Some are great, others are notorious for failing (like Netgear's GS line)

 

[palmboy5]

Sorry, I wasn't paying attention to the window size iperf was choosing as sender on the different platforms (32.5 KByte on FreeNAS, 8 KByte on Windows, 16 KByte on Ubuntu). However, setting the window size manually still showed Ubuntu in a VM being able to better utilize the host's NIC than the host could with the Windows port.

I don't know whats up with the Ubuntu version but its stated window size is always twice that of my requested size.

Code:

vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 64K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 44882 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.07 GBytes   917 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 32K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 64.0 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 44881 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.05 GBytes   902 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 16K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 49498 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   945 MBytes   793 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 8K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 40783 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   849 MBytes   712 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 4K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 8.00 KByte (WARNING: requested 4.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 40784 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   770 MBytes   646 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 2K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 4.00 KByte (WARNING: requested 2.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 40785 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   762 MBytes   639 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 1K
WARNING: TCP window size set to 1024 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 2.00 KByte (WARNING: requested 1.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.118 port 40786 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  36.5 MBytes  30.3 Mbits/sec

Code:

D:\Junk>iperf -c 192.168.2.111 -w 64K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 64970 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec  1.07 GBytes   923 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 32K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 32.0 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 64969 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec  1.00 GBytes   860 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 16K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 16.0 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 64968 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   658 MBytes   552 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 8K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 8.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 65049 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   390 MBytes   327 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 4K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 4.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 64971 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   397 MBytes   333 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 2K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 2.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 64972 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   397 MBytes   332 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 1K
WARNING: TCP window size set to 1024 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 1.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 64973 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   394 MBytes   330 Mbits/sec

CPU usage was asked for at least for the dd stuff, I was just trying to be complete by also getting top values for when using CrystalDiskMark.

EDIT: Is what I get from the QR code sticker on the back of the NIC trustworthy proof of legitimacy?

 

[palmboy5]

The switch is a TRENDnet TEG-S8g, 50ft CAT6 goes to the server, while I guess about a 15ft CAT5 goes to the desktop. I'll try replacing the CAT5 cable but seeing as how I'm getting major throughput differences just by what platform I'm running iperf on, I'd be surprised if a CAT5e or CAT6 cable actually solved the problem. As a sidenote, I no longer trust what category the cables say they are, after finding out a CAT5e cable that always caused fallback to 100 Mbit/s didn't even have twisted pair wiring when I cut it open.

 

[mattlach]

The switch is a TRENDnet TEG-S8g, 50ft CAT6 goes to the server, while I guess about a 15ft CAT5 goes to the desktop. I'll try replacing the CAT5 cable but seeing as how I'm getting major throughput differences just by what platform I'm running iperf on, I'd be surprised if a CAT5e or CAT6 cable actually solved the problem. As a sidenote, I no longer trust what category the cables say they are, after finding out a CAT5e cable that always caused fallback to 100 Mbit/s didn't even have twisted pair wiring when I cut it open.

I've very rarely had ethernet cables fail on me, but when they have, I've had them fail in very strange ways.

You are probably right, but I figured it would be a cheap thing to try just in case, before you start getting into the expensive stuff.

 

[mattlach]

Well, I am a little at a loss here, but another thing that might be worth trying:

When you configured the volume, did you set it up as "force 4k sectors"?

I'm fairly certain those WD Greens you are using are 4k drives, and sometimes FreeNAS fails to detect this on its own, which can cause some real performance issues.

 

[cyberjock]

Sorry, I wasn't paying attention to the window size iperf was choosing as sender on the different platforms (32.5 KByte on FreeNAS, 8 KByte on Windows, 16 KByte on Ubuntu). However, setting the window size manually still showed Ubuntu in a VM being able to better utilize the host's NIC than the host could with the Windows port.

And you're SURE it's not a driver problem with the NIC on Windows? See, that's the problem with troubleshooting and diagnosis. You sometimes have to dig REALLY deep to get a solid answer on why something is how it is. For instance, if you had some POS Realtek NIC you could run all sorts of tests, spend hours on various experiments and be upset that you can't fix the poor performance. Or, you could save your time and just buy an Intel NIC, which works.

And what about the guy that had an Intel PCI NIC and upgraded to a PCIe NIC? Suddenly his bottleneck(PCI bus) went away with the upgrade, but what did he think was wrong? He threw away the PCI NIC because he thought it was broken! You seem pretty knowledgable, but what stuff do you not know with regards to the problem. That's the problem with complex stuff. It's easy to pin the problem on a component you removed and throw it away. How will you ever learn what was wrong without the knowledge and solid evidence(and what is solid evidence anyway?) of what is wrong?

Is what I get from the QR code sticker on the back of the NIC trustworthy proof of legitimacy?

Those QR stickers are easy to fabricate. If you REALLY want to know, you can get the serial number of the card, preferably via software and not a sticker/stamp on the card, and call Intel to ask.

 

[mattlach]

Those QR stickers are easy to fabricate. If you REALLY want to know, you can get the serial number of the card, preferably via software and not a sticker/stamp on the card, and call Intel to ask.

Agreed.

I really don't think you need to worry too much on the ~$30 single Intel NIC cards.

The ones that I have heard of being counterfeit are the higher end models, which can have some significant cost associated with them.

 

[palmboy5]

Well, I am a little at a loss here, but another thing that might be worth trying:

When you configured the volume, did you set it up as "force 4k sectors"?

I'm fairly certain those WD Greens you are using are 4k drives, and sometimes FreeNAS fails to detect this on its own, which can cause some real performance issues.

4K was forced, and I'm glad that FreeNAS made it so easy to force, as opposed to manually doing so when I used FreeBSD. FreeNAS WILL fail to detect because those WD Greens (the EARS anyway) are 4K drives that report themselves as 512B for "backwards compatibility" with ancient software like Windows XP.

I'm not too worried about the NICs being fake. They were ordered from Newegg and while Newegg did have an incident regarding some uber fake Core i7's, I still trust them in general.

I think I just went with whatever Intel NIC driver got bundled with Windows 7 SP1. "If it ain't broke, don't fix it" attitude, but I guess I'll consider it broken now and update tonight.

So what's my to-do list :P

Upgrade to FreeNAS 8.3.0
"Upgrade" ethernet cable going to desktop
Update Intel NIC driver

later:
Upgrade server RAM to 16GB

Am I missing anything?

 

[cyberjock]

I remember that Newegg shipped "fake" CPUs. I wonder if they ever caught the scammers.

Anyway, I think that's about it. At least for "easy" stuff.

 

[palmboy5]

NIC driver has been updated from a 2009 version to a 2012 version (using Windows Update), no change has been noticed.
lol, now I got reminded that the FreeNAS installation to flash drive also uses MiB/s, http://i.imgur.com/ds2OR.png
Desktop now connects to the switch with a CAT6 cable.

dd with 8.3.0 on the AMD system

Code:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
320171420 bytes/sec
dd of=/dev/zero if=tmp.dat bs=2048k count=50k
428192505 bytes/sec

I see the write speed got a nice little boost.

As for iperf on the server using Windows and Ubuntu VM... Windows still loses handily.

Code:

D:\Junk>iperf -c 192.168.2.111 -w 64K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50229 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec  1.05 GBytes   904 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 32K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 32.0 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50235 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec  1.03 GBytes   886 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 16K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 16.0 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50242 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   761 MBytes   638 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 8K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 8.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50247 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   469 MBytes   393 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 4K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 4.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50252 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   478 MBytes   401 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 2K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 2.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50259 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   481 MBytes   403 Mbits/sec

D:\Junk>iperf -c 192.168.2.111 -w 1K
WARNING: TCP window size set to 1024 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 1.00 KByte
------------------------------------------------------------
[156] local 192.168.2.139 port 50264 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[156]  0.0-10.0 sec   479 MBytes   401 Mbits/sec

Code:

vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 64K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35598 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.06 GBytes   915 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 32K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 64.0 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35599 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.04 GBytes   889 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 16K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35600 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   942 MBytes   790 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 8K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35601 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   846 MBytes   709 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 4K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 8.00 KByte (WARNING: requested 4.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35602 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   796 MBytes   668 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 2K
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 4.00 KByte (WARNING: requested 2.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35603 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   769 MBytes   645 Mbits/sec
vcn64ultra@ubuntu:~$ iperf -c 192.168.2.111 -w 1K
WARNING: TCP window size set to 1024 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to 192.168.2.111, TCP port 5001
TCP window size: 2.00 KByte (WARNING: requested 1.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.123 port 35604 connected with 192.168.2.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   740 MBytes   621 Mbits/sec

CrystalDiskMark got 66-75MB/s read, 101-106MB/s write

Strangely enough, I can read/copy a large file to the desktop SSD at about 96MB/s now, which pretty much qualifies as a problem-solved. Now I'm still wondering why other operations, such as CrystalDiskMark and restoring a VM that's stored on the server are still maxing out around the 70MB/s mark.

 

[cyberjock]

Hold on a second. You were running iperf from Windows and an Ubuntu VM? Can you explain this? Do you mean running Windows as an iperf server(or client) and an ubuntu VM on the same machine as the opposite of the Windows machine?

 

[palmboy5]

iperf client -> Windows 7 -> NIC -> Switch -> NIC -> FreeNAS -> iperf server
iperf client -> Ubuntu 11.04 -> VMWare Player -> Windows 7 -> NIC -> Switch -> NIC -> FreeNAS -> iperf server

Ubuntu in a VM is still faster. In later tests I didn't bother sending in the opposite direction because... lazy I guess.

I did once try from Windows 7 (host) to Ubuntu (VM) and it was a little faster with the smaller window sizes, but no where near the 2.x Gbit/s I got sending from Ubuntu VM to a second Ubuntu VM on the same host. :P

 

[palmboy5]

On the receiving direction, Windows fares much better.

Server to Desktop (Windows 7)

Code:

[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 64k
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 65.0 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 37214 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.07 GBytes   920 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 32k
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 32.5 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 39103 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.05 GBytes   906 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 16k
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 17.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 52870 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   700 MBytes   587 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 8k
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 8.48 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 65464 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   425 MBytes   356 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 4k
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 4.24 KByte (WARNING: requested 4.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 45122 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   168 MBytes   141 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 2k
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 2.83 KByte (WARNING: requested 2.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 31323 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   194 MBytes   163 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.139 -w 1k
WARNING: TCP window size set to 1024 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to 192.168.2.139, TCP port 5001
TCP window size: 1.00 KByte
------------------------------------------------------------
[  3] local 192.168.2.111 port 46175 connected with 192.168.2.139 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-25.6 sec   128 KBytes  40.9 Kbits/sec

Server to Desktop to VM (Ubuntu 11.04)

Code:

[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 64k
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 65.0 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 19719 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   817 MBytes   685 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 32k
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 32.5 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 19720 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   852 MBytes   715 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 16k
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 17.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 19721 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   524 MBytes   440 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 8k
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 8.48 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 25582 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   304 MBytes   255 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 4k
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 4.24 KByte (WARNING: requested 4.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 20467 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   163 MBytes   137 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 2k
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 2.83 KByte (WARNING: requested 2.00 KByte)
------------------------------------------------------------
[  3] local 192.168.2.111 port 51349 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   151 MBytes   127 Mbits/sec
[vcn64ultra@callisto-1 /]$ iperf -c 192.168.2.123 -w 1k
WARNING: TCP window size set to 1024 bytes. A small window size
will give poor performance. See the Iperf documentation.
------------------------------------------------------------
Client connecting to 192.168.2.123, TCP port 5001
TCP window size: 1.00 KByte
------------------------------------------------------------
[  3] local 192.168.2.111 port 23209 connected with 192.168.2.123 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-13.2 sec   384 KBytes   238 Kbits/sec

Is such a throughput with these window sizes normal?

 

[cyberjock]

Is such a throughput with these window sizes normal?

No idea.


But using a VM for the test kind of goes against the whole purpose of the test. iperf was designed to use almost no CPU power to generate the traffic to ensure that your CPU isn't affecting your scores. By using a VM you are forcing the virtualizing software to receive the data, then pipe that data to the VM. That will require some CPU power. Will it be enough to affect results? I don't know. Using a virtual machine could potentially be preventing the NIC from using its onboard packet checksums for sending and/or receiving(may improve performance). Even if the virtualizing software doesn't significantly increase CPU usage, that small amount of delay could have detrimental affects on the test.

But before you start acting on these results I'd try running the tests on real hardware.

 

[palmboy5]

I know the VM is at a disadvantage, that's why the fact that the VM is faster than the host when sending to the server is such an embarrassing thing to happen that I called the Windows port of iperf horrible.

 

[cyberjock]

I just gotta say this.. your shit is AFU.

Anyway, I thought about this earlier and wasn't at the computer. But is it possible that your limitation is the read speeds of the hard drive you are copying data from? Are you using an SSD to copy to the server? If not, I'd give one a try. Or use a ramdrive(imdisk is a windows program that is free). To be honest, and I'm sure I said this earlier in the thread, 70MB/sec is more than alot of people get.