Performance bottleneck

[silvergoat]

I just built a NAS server throughout the week and finally got everything ready for testing. Here is the setup:

Desktop PC (current gen equipment) + Intel Gigabit CT NIC + SSD drive on Intel SATA 3 + older Velociraptor drive on Intel SATA 2 on Windows 7

Server (Asus P5KPL-AM EPU + Celeron E3300 + 4GB RAM + PCI-e Marvell 9128 SATA3 Controller + 4 Velociraptor 500GB drives in ZFS stripe w/ 4K forced + Intel Gigabit CT NIC)

Router = D-Link dir-615 flashed with DD-WRT 10/100

Switch = TP-Link SG1005-D 10/100/1000

My PC and servers are wired to the switch, which is wired to the router. I'm not sure if this bypasses the router's slower 10/100 speeds entirely, but I hope so.

When I first got everything hooked up, I was running everything through the router and I was getting capped at 12MB/s, once I set it up through the switch, I went up to about 60MB/s on some, and as low as 30MB/s on other transfers. SSD to server had the highest and most consistent speeds of 60, and the Raptor had equal speed with dips down into the 30MB/s range. However, from the server to my computer, the speed only averaged around 30MB/s, which is what I'm confused about. I have since set the MTU on the server to 9000, and enabled Jumbo packets of 9014 on my PC, but it's still not where I thought speeds would be. The transfers now start a little slower, and end about 10MB/s higher on average, but that's still not near the theoretical max (only about 35%).

I've come to the conclusion that it is either the router somehow having an effect on the computers that are not directly wired to it, but may seek a path from the router, or that the potentially slower Marvell drive controller on my server may be causing the slower speeds. Unfortunately, the server does not have any native SATA3 ports to test, and I was unable to configure a direct connection between my computer and my server to see if the router or switch were at fault. Any ideas on what step to take next with the equipment I have?

Edit: I forgot to indicate the use for this server: It is intended to be a media server with standard definition content, and 1-2 clients with a maximum of 3 at the same time. I will eventually have HD content on the server, which is why I want to make sure my throughput can support a couple streams of that size.

 

[jgreco]

The router is not involved in the current scenario you describe.

You can eliminate both your router and your switch as sources of problems by getting rid of them for testing purposes; connect the FreeNAS to a client directly and just go at it. Run iperf on things while the switch is out of the mix. Put it back in the mix and see if there's a noticeable difference. If you cannot get your server to connect directly to a client, you have a serious problem - that's supposed to Just Magically Work under the gigE specification (auto MDI/MDI-X) so that's worth investigating further.

I think most people will jump to the very likely conclusion that 4GB of RAM is disabling prefetching and that's hurting you, but I'll also note that you have an intermediate level CPU and that is an indication that you need to be somewhat more aggressive about tuning. Some users get very good speeds out of smaller platforms than yours, but usually there's some work involved.

I've spent a fair amount of time messing with performance for iSCSI on FreeNAS, but a lot of the lessons are general in nature: the biggest one is that the time to fix your pool is when it's empty and you can make changes. Is it responsive and quick doing writes? Is it responsive and quick during a scrub? A scrub with writes and reads? Try doing tests directly on the server. You will not be able to get speeds faster than that from a client, after all...

 

[louisk]

I would suggest you skip the jumbo frames. Messing around with layer 2 is generally bad news unless you know what you're doing.

I would consider 100MB close enough to max, and you're at 60%. Have you tried doing any benchmarks on the nas itself to see if the bottleneck is actually the network?

 

[silvergoat]

.....yea, I'm going to switch the jumbo thing back to 1500

I did forget to mention one very important thing- I used TeraCopy to get my transfer speeds, so I'm not sure that it is a good enough standard of measure. I have not tried any benchmarks because this is my first time working with A.) a server B.) Unix. It took me a while just to figure out how to change the MTU on the server, and I haven't really started to mess with the other settings within the GUI, let alone the CLI. I wanted to make sure the fundamentals were solid because the first transfer (on 10/100 router before the addition of the 10/100/1000 switch) took me 2 days (~600GB) and I wanted to make sure that I could mitigate the slower transfer speeds as much as possible before continuing.

Thanks for your response louisk- I did have one very important question that I wanted to know the answer to before worrying about the potential of this setup. Does the 10/100 router slow down my computers even though they are connected directly to the unmanaged 10/100/1000 switch? I did also want to determine why the download is half the speed of the upload, since the download is what I'll be relying on once I start streaming.....I wouldn't mind it so much if it were the other way around.

 

[silvergoat]

The router is not involved in the current scenario you describe.

You can eliminate both your router and your switch as sources of problems by getting rid of them for testing purposes; connect the FreeNAS to a client directly and just go at it. Run iperf on things while the switch is out of the mix. Put it back in the mix and see if there's a noticeable difference. If you cannot get your server to connect directly to a client, you have a serious problem - that's supposed to Just Magically Work under the gigE specification (auto MDI/MDI-X) so that's worth investigating further.

I think most people will jump to the very likely conclusion that 4GB of RAM is disabling prefetching and that's hurting you, but I'll also note that you have an intermediate level CPU and that is an indication that you need to be somewhat more aggressive about tuning. Some users get very good speeds out of smaller platforms than yours, but usually there's some work involved.

I've spent a fair amount of time messing with performance for iSCSI on FreeNAS, but a lot of the lessons are general in nature: the biggest one is that the time to fix your pool is when it's empty and you can make changes. Is it responsive and quick doing writes? Is it responsive and quick during a scrub? A scrub with writes and reads? Try doing tests directly on the server. You will not be able to get speeds faster than that from a client, after all...

Thanks for the detailed response! I only searched for the server from the client (W7) because the CLI was/is intimidating. I did not consider trying the other way around....I will give it a try in the next day or two once I figure out how to test from the server directly to the client. I was hesitant to use such a low-grade system, but I had read a lot on the internet that led me to believe it wasn't a big deal for such a small implementation. I am not fully satisfied with the current build, but I had to use what I had with the budget and parts I had. My goal was to put all media on one server, but computer case size and hard drive costs limited me and I figured the best way to do this was to eventually build another server to hold my DVD collection, and to use this one for my smaller TV DVD sets I have.

Thanks again for the critique.

 

[louisk]

I have a router that is 10/100 and I can still achieve transfer speeds of ~850Mbit with the system in my signature. The desktop I'm using is a Mac Pro with a soft-raid Seagate 2T mirror. I'm using LACP (so I can't ever go faster than the theoretical 1Gbit) between any 2 hosts.

 

[silvergoat]

I have a router that is 10/100 and I can still achieve transfer speeds of ~850Mbit with the system in my signature. The desktop I'm using is a Mac Pro with a soft-raid Seagate 2T mirror. I'm using LACP (so I can't ever go faster than the theoretical 1Gbit) between any 2 hosts.

That's great news because I like my router and didn't want to change it out. I almost went to BB today to get a new one, but I'm definitely glad I didn't. I did not plan on setting up a dual NIC server anytime soon, but it is something I considered today......hopefully unnecessary.

 

[Daisuke]

I think most people will jump to the very likely conclusion that 4GB of RAM is disabling prefetching and that's hurting you...

The memory increase and an SSD for cache will do miracles. Read my build story, check my signature. :)
I still struggle with the loss of data speed I used to have, while running FreeNAS 8.0.2 release.

 

[silvergoat]

I think I may have stumbled on the problem, but I would have to get your opinions on the likelihood and extent to which it would be detrimental to performance. I have compression set at gzip=6 (IIRC) and of course I have the mediocre CPU. Would that be enough to slow read/write speeds, and would it account for the significantly lower read speeds?

Also, as I was going to adjust my MTU back down (I was going to experiment with 4088 Jumbo rather than the 9014), but I noticed that the MTU was back to a default of 1500 after the last shutdown. How do I make changes permanent (until manually set otherwise)?

 

[cyberjock]

You should leave the MTU at 1500 unless you understand what oyu are doing. Alot of hardware doesn't work well with MTU above 1500.

Yes, compression can single handedly kill read and write performance.

 

[louisk]

MTU applies to the entire Layer2 domain, not just a pair of hosts. You don't have MTU settings on a per-host basis. If you change MTU, your host will expect to be able to use that MTU to talk to any host on the network, and it can cause quite a bit of troubleshooting to figure out why so many packets are being dropped on your network.

 

[silvergoat]

You should leave the MTU at 1500 unless you understand what oyu are doing. Alot of hardware doesn't work well with MTU above 1500.

Yes, compression can single handedly kill read and write performance.

That's good to know- now I can start trying to fix it the right way. Saw the link for the guide- I still have tons of stuff to read through, but I'll be checking it out. Hopefully there will be some answers to some of the other weird issues that have come up......and something about setting up permissions.

 

[silvergoat]

MTU applies to the entire Layer2 domain, not just a pair of hosts. You don't have MTU settings on a per-host basis. If you change MTU, your host will expect to be able to use that MTU to talk to any host on the network, and it can cause quite a bit of troubleshooting to figure out why so many packets are being dropped on your network.

I can see what you mean, but I don't do networking at all- this is my first try ever. I have absolutely no network connections of any importance other than to the internet, and now the media server. The rest of the outlets are spares of which the only one that sees any use is an xbox. However, I do not stream to or from it- just xbox live. That being said, both NIC cards are back at 1500 because it would be pointless to try the jumbo packets if the server can't even hold the setting.

-deleted-

Nevermind that question- I'm guessing that's some of the wackiness you were referring to, in which you would have to switch the entire network, router and all, to the desired size, and hope everything just works with each other.

 

[jgreco]

You don't have MTU settings on a per-host basis.

That's not correct. You can arrange MTU settings on a per-host basis, by arranging one in the routing table. "route add <dest.ip.add.ress> <dest.ip.add.ress> -mtu <foo>" You can see the host's MTU by doing "route get <dest.ip.add.ress>".

Sadly, in order for this to work in an ethernet broadcast domain, with the way most FreeBSD device drivers work, you have to set the interface MTU's to 9000 on all hosts you want to work that way, then on those same hosts add "-mtu 1500" routes for EVERY HOST that isn't MTU 9000. I haven't seen anyone do this in quite some time, but if you had a network where you only had like one device that was 1500, it's *possible* to make it work.

Very stupid though. Don't do it.

 

[louisk]

OK, you're correct. Its technically possible, but its excessive work, and ripe for lots of extra debugging when things don't work (and since they already don't work, this should be a warning).

 

[jgreco]

It is probably apparent to most people, but we've gone from 10 megabit ethernet to 10 gigabit ethernet while largely maintaining the same MTU. 9K or 16K MTU's are at best an order of magnitude larger than 1500 byte. This has largely been because it makes maintaining compatibility with older gear much easier; how would you go attaching a 10Mbps, 1500-byte-MTU-only device to a gigabit switch? At the same time, so much related stuff has been built that would probably not be immediately compatible with larger MTU's.

Interesting additional reading by people who are maybe unrealistically optimistic, but they've done some actual research along with a lot of theorizing.

 

[silvergoat]

On a final note:

I just did the transfer test on the computer while watching the reporting screen in the FreeNAS web interface. Inactive memory dropped down to less than 37 Mb!! It is likely my bottleneck because CPU didn't even hit 20%.

Looks like I'm going to have to change the whole system up to allow for the highest amount of RAM I can get because this board is very limited. I'm glad I caught this before I bought an upgraded CPU.

Thanks again.

 

[cyberjock]

I can't log into any FreeNAS servers right now, but the RAM should be 99% filled. That's to maximize its use and hopefully system performance. Why would you not use it if you have it? Windows doesn't use all of its RAM because its stupid. FreeBSD and FreeNAS are better than that. :p

From the manual:

The best way to get the most out of your FreeNAS® system is to install as much RAM as possible. If your RAM is limited, consider using UFS until you can afford better hardware. ZFS typically requires a minimum of 8 GB of RAM in order to provide good performance. The more RAM, the better the performance, and the FreeNAS® Forums provide anecdotal evidence from users on how much performance is gained by adding more RAM. For systems with large disk capacity (greater than 6 TB), a general rule of thumb is 1 GB of RAM for every 1TB of storage. This post describes how RAM is used by ZFS.

NOTE: by default, ZFS disables pre-fetching (caching) for systems containing less than 4 GB of usable RAM. Not using pre-fetching can really slow down performance. 4 GB of usable RAM is not the same thing as 4 GB of installed RAM as the operating system resides in RAM. This means that the practical pre-fetching threshold is 6 GB, or 8 GB of installed RAM. You can still use ZFS with less RAM, but performance will be effected.

If you plan to use ZFS deduplication, a general rule of thumb is 5 GB RAM per TB of storage to be deduplicated.

If you use Active Directory with FreeNAS®, add an additional 2 GB of RAM for winbind's internal cache.

Then, from my guide for noobs (sticky in the FreeNAS 4 Noobs section.. ):

For maximum performance, more RAM is always better. Using less than 6GB is generally not ideal if performance and system reliability is a concern and ZFS is used. Too many people try to get by with less RAM. The most effective way to increase system performance is RAM. There is no such thing as too much RAM. Maximum allowed motherboard RAM is recommended.

Then again later in my guide:

Manually optimizing the system settings should not normally be necessary as long as basic system requirements are met. In particular plenty of RAM for ZFS users.

Then again later in my guide:

Have I mentioned you can never have too much RAM?

One or two users have seen that switching to NAS4Free with less than 6GB of RAM has given them faster servers while using ZFS. NAS4Free doesn't have all of the features FreeNAS does, but if you are unable to upgrade to more RAM I'd recommend you check out NAS4Free. I've never used it so i'm just mentioning it for your benefit.

 

[silvergoat]

I was limited by the motherboard. Max 4GB RAM. I was trying not to spend money since I had a couple of computers and spare parts. The server works, but honestly I'm the kind of person that will never be happy with it so I should just leave it as is until I have the cash for a new board, case, hard drives, CPU, RAM......it will have to be a new computer to do this. I'll take a look at NAS4Free since I don't need features at my level of understanding.