Performance issue with 10Gbps network

[pclausen]

I added the tunable and it unfortunately didn't make a difference. I ran dd (after turning off compression) and got the following results:

Code:

[root@argon] ~# dd if=/dev/zero of=/mnt/v1/misc/foofile bs=1048576
load: 5.68  cmd: dd 26184 [running] 11.02r 0.01u 8.02s 39% 2624k
17339252736 bytes transferred in 10.982776 secs (1578767743 bytes/sec)
load: 4.23  cmd: dd 26184 [dmu_tx_delay] 95.26r 0.14u 47.34s 25% 2624k
133439684608 bytes transferred in 95.222746 secs (1401342537 bytes/sec)
load: 3.66  cmd: dd 26184 [dmu_tx_delay] 515.73r 0.63u 232.75s 23% 2624k
694745563136 bytes transferred in 515.693642 secs (1347205989 bytes/sec)
load: 3.84  cmd: dd 26184 [dmu_tx_delay] 976.99r 1.12u 428.79s 17% 2624k
1277145645056 bytes transferred in 976.949585 secs (1307278968 bytes/sec)
load: 6.31  cmd: dd 26184 [running] 1498.13r 1.69u 644.36s 23% 2624k
1914715504640 bytes transferred in 1498.091036 secs (1278103572 bytes/sec)
6055083900928 bytes transferred in 5115.036750 secs (1183781114 bytes/sec)

So I start out at around 1.58 Gbps and then after about 1.5 hours, performance drops to about 1.18 Gbps. So it does seem that my system is not able to write to the pool any after than what my real life copy tests are showing.

While dd was running, I looked at the 1st disk in vdev0 and the last disk in vdev4 and they were both showing about the same I/O (as were the other 48 disks in the pool):

CPU and Load stats:

 

[jgreco]

I added the tunable and it unfortunately didn't make a difference.

Did you reboot?

6055083900928 bytes transferred in 5115.036750 secs (1183781114 bytes/sec)

So I start out at around 1.58 Gbps and then after about 1.5 hours, performance drops to about 1.18 Gbps.

No, that's bytes per second. That translates to around 8-9Gbps. But it's quite possible for that to be reduced by half, maybe even more, once you start trying to access from remote.

I have a feeling that the relatively low clock speed on your CPU may be hurting you somewhat. That part is rated for 2GHz but turbos up to 3GHz. The thing is that when you're dealing with very high speeds on a NAS, the latency is one of the bottlenecks, at least when you're working singlethreaded. I am very fond of the E5-1650 V3, which carries a very good price-to-performance ratio AND delivers the biggest bang for buck in the E5 lineup. The CPU base is 3.5GHz with turbo to 3.8; with six cores, a NAS will never use several of the cores so you're very likely to be hitting that magic 3.8.

In my mind that could maybe make a 20-30% difference in write speed. However, the interactions in these things are not always obvious or trivial. In any case, the 1.5Gbps-to-the-server, 7Gbps-from-the-server you described several messages back is not particularly unusual behaviour for ZFS, which can be better at reading than writing, but I'd expect that a larger pool such as yours would be capable of better numbers than that, at least on the write side of things.

 

[pclausen]

Oops, you're right, I forgot to multiply by 8 to get bits. So at least the system should be able to write at close to 10 Gbps speed. You're right about my single thread performance being lackluster to say the least.

Yep, I did reboot. Just to double check, the type should be "loader" correct?

Just for grins, I ran my iperf test again and for this:

Code:

[root@argon] ~# iperf -s -p 5001 -w 1024k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[  4] local 10.0.1.50 port 5001 connected with 10.0.1.53 port 62501
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  3.89 GBytes  3.33 Gbits/sec
^C[root@argon] ~# iperf -c 10.0.1.53 -p 5001 -w 1024k
------------------------------------------------------------
Client connecting to 10.0.1.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 10.0.1.50 port 15313 connected with 10.0.1.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  5.83 GBytes  5.01 Gbits/sec

I still think something is up with my UniFi switch, but I love it for many of it's other features since I run a bunch of UniFi access points and a AirMAX bridge down to my shop. I'm still pushing Ubiquiti for a fix as I have demonstrated better performance when I connect my workstation directly to my FreeNAS server.

I had a E5-1620 V3 before the going to the 14 core E5-2683. The switch to the 14 core was driven by my transcode needs to serve media to many clients at once. I wasn't sure adding just 2 more cores (1650) would meet my needs.

I'm going to build myself a Skylake workstation for Christmas with a Samsung 950 Pro M.2 SSD and my spare X520 NIC. That will allow me to perform some workstation to workstation tests to see if I can narrow down exactly what's going on here.

Hopefully in a few years the soon to be released Samsung 4TB SSDs will come down in price enough that I can replace all my spinners with them, and upgrade to some 12 Gbps HBAs, and trade in my Supermicro 846 monsters for a single Supermicro 417 chassis with SAS3 backplanes. Then there won't be issues saturating a 10 Gbps network, but by then, 100 Gbps will be all the rage. It's a vicious cycle. :D

 

[jgreco]

Could definitely be getting hurt by the switch. Latency is king with 10Gbps.

As for the 1620 vs 1650, the 1620 is a 3.5-peaks-at-3.6 part whereas the 1650 is a 3.5-peaks-at-3.8 part. It isn't really about the cores. For NAS use, I haven't really seen FreeNAS even using four cores unless you've got a slow CPU and lots of clients. It's just that little extra bump from 3.6 to 3.8, coupled with the fact that on the 1650 you're likely to be *getting* that.

But if you've got other reasons such as transcoding to go the 26xx route, that's fine. It's just that all the 26xx's are pretty miserable by comparison when you look at raw clock, which seems to be the biggest kick for FreeNAS itself.

As for 100Gbps? Don't hold your breath. As I point out in the 10G primer, we rapidly went from 10Mbps to 10Gbps standards at a clip of about one order of magnitude every three years. But it's been a HUGE amount of time to see 10Gbps commodity level gear coming out. We're just now getting hints of that, with some of the new chipsets supporting 10GbaseT ... there's an argument to be made that 1Gbps is really fast enough for so many purposes, that we're just not that likely to see 10Gbps be very popular even in another year or two. And a hundred is WAY out there. Even I'm only up to 40Gbps.

 

[pclausen]

Interesting note on the 1650 being a 3.5-peaks-at-3.8 part. Didn't realize that. My 2683 is actually a 2.3GHz unit (all cores are locked at 2.3) and I don't think any of the cores can turbo to a higher clock rate. Don't really have away to test that when running FreeNAS, but for my transcoding needs, it definitely took care of the issue I was having.

Yeah, I know 100Gbps is way off for home use, but we started deploying 100 Gbps DWDM rings at work back in March of this year. We serve a lot of data centers in the Mid-Atlantic region. Prior to my current role, I managed a smallish data center with a bunch of Lefthand storage that was displaced by 3PAR over time. Today we're running the 3PAR stuff in 2 data centers for DR.

 

[jgreco]

That's not what Intel ARK says.

http://ark.intel.com/products/81055/Intel-Xeon-Processor-E5-2683-v3-35M-Cache-2_00-GHz

But you've got the part there in hand, and ARK is occasionally fulla crap.

 

[pclausen]

I was going by what these told me about my specific (ES) CPU:

Note how CPUZ is reporting 2.4 GHz and Intel 2.3 GHz?

So maybe the cores can "turbo" to 3 GHz but I don't think FreeNAS has a tool that can report that to me when the system is under load. I can say that temps stay nice and low, having never exceeded 65 Celsius under load.

 

[jgreco]

Ah, an engineering sample, all bets are off. Intel releases lots of those especially to server OEM's in order to help them develop their products pre-launch, and they are often tuned differently.

 

[Ericloewe]

I can totally see engineering samples being clocked lower than final products, pending silicon verification.

That's what happened with early Xeon-D review boards. Clocks were 200ish MHz lower than final poroducts.

 

[jgreco]

I can totally see engineering samples being clocked lower than final products, pending silicon verification.

That's what happened with early Xeon-D review boards. Clocks were 200ish MHz lower than final poroducts.

It isn't clear that's the case here. It looks to me like the final revision had a lower clock but a better turbo, maybe.

 

[pclausen]

I figured I'd provide an update on this issue.

I reconfigured my primary server (the one with the SuperMicro X10 mobo and 14 core CPU) to run Windows 10 64-bit. I added an Areca ARC-1882IX-16 to it and set up 2 RAID 6 arrays. One made up of 12 x 6TB WD REDs and one with 12 x 4TB Seagate ST4000s.

I then stood up a fresh FreeNAS server consisting of a SuperMicro X8 mobo with a pair of quad core L5630 CPUs and 48 GB of RAM. I paired that with 60 2TB drives configured in 12 raidZ1 vdevs with 5 members in each. This was not a good long term config for data safety, but I was going for performance to see if I could get anywhere near 10 Gpbs performance with FreeNAS.

The test was done with a direct server to server 10G connection between the windows server and the FreeNAS server.

I was very disappointed with the result:

No CPU starvation:

Network utilization from the FreeNAS perspective:

And the write speed to each of the 60 disks:

So about 4 MB/sec per disk. Pretty sad...

So I blew out FreeNAS from the X8 server, and installed Windows 10 64 bit and dropped in a LSI 9266 RAID card. I then configured a RAID 60 array with 5 12 disk spans.

I was finally able to achieve saturation over my 10G network:

So with the exception of adding the LSI RAID card in place of a pair of LSI 2008 based HBAs, the hardware and network configuration was exactly the same in these tests.

So even with 12 "stripes", I don't seem to be able to coax FreeNAS into delivering anything past 2 Gbps.

Maybe CIFS is the bottleneck?

 

[Pheran]

Something is definitely not right here since my simple 8 disk FreeNAS box with a low-end quad-core Xeon can do 4 Gbps over CIFS.

 

[bartnl]

I think CIFS is problematic and seems to be operating single threaded. I run Freenas on a Supermicro X10SRA-F with E5-1620 v3 CPU and 64GB of RAM and it will do almost exactly the same speed as your system. I get 185MB/s on CIFS writes. Tried changing network hardware (Chelsio T420-CR and Intel X540-T2) but nothing helped.
After that I tried FTP and Freenas will do 400 MB/s to 500MB/s on exactly the same setup as with the CIFS transfers, client in both cases Windows 10 pro. Difference being that FTP wil do multiple transfers concurrently.
Also if I run Windows on the fileserver it will do close to line speed on file transfers.
I tried several FreeBSD "tuning parameters" on CIFS but none of them is really working.
On the positive side the 185MB/s I get is stable. On the negative side the slow performance is the reason I'm not using the system for it's intended Freenas use.

 

[Magnus Eriksson]

I am just a newbie but right now Im copying about 2TB movies from my local PC running win 10 pro. The share on FreeNAS is CIFS and my average throughput on network is around 970 mbit/s on copper utp5e with link speed 1gbps.

No optimization done or anything, mostly just default.

 

[bartnl]

Yeah, it seems to get problematic only at speeds above 1 gbit/s. The 185MB/s I get equals the 1500 mbit/s as reported by pclausen in the graph posted at 2:44AM. Had I not had the 10Gbit network I would not have noticed a problem with cifs speed ;)

 

[Magnus Eriksson]

Ok then I have a 4gbps FC that Should be mounted later on. Is there problem over FC as well?

 

[bartnl]

My Chelsio 420-CR is 10Gb FC. Some people seem to have no problem and get better speeds and I don't think it is network card related as it will do better speeds through FTP with Freenas as FTP server.

 

[titan_rw]

Strange. I've been really happy with 10 Gb performance with freenas. I get ~900 MB/sec (70-80% of 10 Gb in windows task manager) copying large (200 GB) files over CIFS.

I've got 3 vdevs of 6 drives in z2. Both ends are Intel 540's (copper). 64 gigs of ram with an e5-1650v2. Pool can write at 1.8 GB/sec or something.

This is reading off of two sata ssd's in raid 0 on my windows machine. 900 MB/sec is close to what I'd expect out of them anyway, as they probably top out at 1,000.

 

[bartnl]

I would be very happy with your speeds

I'm using Windows 10 64 bit client with above mentioned Freenas setup with 12 drives in z2. For my setup I'm using a Zyxel XS1920-10Gb switch with currently copper Intel 540 in the Windows system and fiber Chelsio 420 in the Freenas NAS. I started out with Intel 540 in the Freenas also.
What motherboard are you using for your Freenas system, what Windows cliënt and are you using a switch?

 

[titan_rw]

MB is a Supermicro X9SRH-7TF. Windows 7 x64 for the client. Direct connect cable, no switch. It's not even cat 6. I'm pretty sure it's just 5e. It's not terribly long though, probably 50-60 feet.