Performance tuning iSCSI - system review

[sfcredfox]

Greetings,

Questions/Objectives of post:
(These are the questions I'm hoping to get answered in the wordiness below)

I would like to get some ideas of things I can tune/control within my current system to both test and optimize the performance of iSCSI to it's highest obtainable levels with the hardware I have.

-Should/could I adjust settings on FreeNAS' iSCSI target such as the queue depth or other settings to optimize performance?
-Can system RAM be used somehow as a cache for iSCSI like it would with ZFS?
-Has anyone gotten volume manager to see disks behind a P400/P800 controller to format for UFS?
-Given the constraint of my hardware, are there any better ways to setup FreeNAS for ESX hosts?
-Can I use any DD (disk performance tools in FreeNAS) with no UFS volumes?

Current Infrastructure:

FreeNAS Platform:
HP DL 380 G5
(x2) Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
16GB ECC
P400 (512MB w/BBC) with 8x72GB 10K (RAID5)
P800 (512MB w/BBC) with 10x146GB 10K in SAS enclosure (RAID5)
HP MSA70 SAS attached drive enclosure
dual Broadcom 5709

FreeNAS 9.2.1.7-Release 64-bit

Network Infrastructure:
Cisco SG200-26 (SMB Gigabit managed switch)
Purchased/pre-made Cat5e cabling (should be no layer 1 issues)

Other systems (VMware platforms):
(x2) HP DL 360 G5
E5410@2.33Ghz/32GB
BC5709
ESXi 5.5U1

Background:
While I am fairly comfortable with VMware / SANs / FC / etc, I don't feel like I have the master's degree in iSCSI and FreeBSD that I have seen some of the experts here demonstrate. I have poured over some of the forum posts regarding iSCSI trying to get some answers to my questions, instead I feel like a high school kid sitting in a college physics class.

I fully expect to draw fire from some of the resident experts (jgreco and cyberjock come to mind first) for both not using the best hardware configurations for disk, but I'll take the abuse to get some further guidance and how to move forward in a productive direction.

Disk/Storage
I fully understand that ZFS is usually the recommended way to setup storage for obvious performance benefits and going with software RAID. Unfortunately, the HP P400&P800 controllers do not at all support JBOD, and everyone says trying to do logical drives for individual disks is a terrible idea. The P800 is the only controller I have to SAS connect the external disk enclosure. So, best I can tell, hardware RAID is all I have. I have chosen RAID 5 on both arrays (internal 8 disks, and external 10 disks). I have tried to do the best with what I have and those are excellent cards, just not for FreeNAS. As for hardware RAID selection, I need the capacity of RAID5 versus doing RAID 10 for performance.

FreeNAS 8/9 (FreeBSD) apparently hates those controllers (unless I'm doing something wrong) because I can't see the disks in volume manager. I can't figure out how to format them using UFS to do fileshares/iSCSI files/etc. I have read a few posts similar, but they were all talking about ZFS which I am not using. So, I have setup the iSCSI targets using device extents. Each of the Arrays is an extent assigned to it's own target (two targets, one for each RAID 5 array). Is there any way to use those arrays in a UFS volume?

I would also like to know if I can do any disk performance tests with no volumes formatted for ZFS/UFS? I have read many posts that describe using DD, but they create files for writing/reading. I obviously don't have any volumes since the arrays are being used as devices extents, and I can't find any write-ups about doing that (probably because it can't be done?) So any other ways to do a local performance analysis?

CPU/Memory
This system has a huge amount of free CPU and memory I wish I could better utilize. When doing mass virtual machine startup/shutdowns, I can tap out the 1GB link, but the CPU and memory are barely utilized. Is there any way to use the system's RAM as a massive write/read cache for the iSCSI target process? I can't find any information about that either.

Networking
Since I can barely spell FreeBSD (which is not linux!-won't make that mistake and be slain), I'm not sure if I need to worry about getting more specialized drivers for the Broadcom NICs? They support jumbo frames and TOE, but since FreeBSD only supports offloading a few of the aspects like checksum, etc, are there other specific things I should be doing for the iSCSI interface to increase it's performance? Are there any step-by-step walk-throughs for setting up the supported offloads? Are they even worth it?

I read some write-ups that were a little over my head regarding multipathing in FreeNAS/FreBSD. Whats the best way to accomplish that? Maybe an example of using two nics for my scenario (Two NICs, two iSCSI targets, each with only one LUN/Extent?)

I have tried using Jumbo Frames, and it seems to degrade performance, even though I enabled it on all involved hardware (ESX hosts/virtual networks, switch, FreeNAS) and tested using non-fragmented pings, which worked all the way up to 8792. Any thoughts?

iSCSI Targets

I have read some very good forum posts and articles on iSCSI tuning and feel like the stupid kid in class, so does anyone have some very specific examples of changes I should try? I have thought about changing the queue depth from 32 to 64 to ensure that's not being tapped out and telling all the initiators to shutup. (I guess ESX changes initiator queue back down to something very low like 2 and works its way back up slowly)

What have other people changed about their target settings and what positive effects did it have?

I know this reads like a book, so it you actually read this far, thanks for any suggestions you can give me. I didn't include a metric-crap-ton of performance data both because I didn't know what would be of most use or how to collect it. I'm sure someone will let me know if you want to know specific performance data.

Thanks!

 

[cyberjock]

Well, our abuse is for good reason. You're doing things that hurt performance (and basically admitting that you know better) and then expect help with undoing it. To me, it seems that you haven't really internalized the problem and are looking for "solutions that don't involve money". The reality is that there really aren't any. If there were we'd be telling people to do that instead of spending money. We're not here to bleed people dry. We're here to help to the maximum extent reasonable. Sometimes that means spending $1000, sometimes that means spending $50 on ebay.

The bigger problem is that the second people talk about doing iSCSI(or NFS) for VMs you are either spending far more than you *think* you are going to spend when you first decide to go with FreeNAS or you are going to have a slow as molasses set of VMs. Few people are willing to drop down money and go with 64 or 96GB of RAM (oh, but they are willing to ask for advice and spend countless hours tinkering). VMs are the nastiest performance killers you can get. So your choices for getting good performance is slim to none, and you need some meaty hardware to deal. Yes, that means spending $1k+ on RAM, getting rid of those god forsaken RAID controllers, etc.

You want to go UFS, good luck. UFS is already dead to FreeNAS and so what you install now will never see an update.

What you need to internalize is that all that advice we keep giving people, you are part of that group. If you don't want the advice, don't create a thread asking for even more because any advice we're going to give is going to include comments like "you didn't listen to us before.. why would you start now?" and you won't like it one bit.

iSCSI and other tuning assumes you have resources that can be tuned. Sorry, but 16GB of RAM is not sufficient resources to tune. Every choice you make with tuning is a "this-for-that" teeter-totter. If you have no resources you aren't riding the teeter-totter. You're playing by yourself and you are always on the losing side.

If I were you I'd up your RAM to at least 64GB of RAM and follow the other "best practices" you're ignoring for reasons I don't understand and *then* and only then see what problems you have. I've got 32GB of RAM in my system and running 2 VMs makes me wanna puke (which is why I don't run them). 16GB is like a homeless guy getting upset because you haven't bought him a nice big house for him. Well, duh. Gotta work for it!

Edit: I debated all night whether to even reply. If you don't want my replies let me know. When dealing with threads like this I always have to ask myself "you didn't listen to all my advice that's already here.. why would you listen now?"

 

[jgreco]

FAR too lazy this fine Friday to edit appropriately. Sorry.

Greetings,

Questions/Objectives of post:
(These are the questions I'm hoping to get answered in the wordiness below)

I would like to get some ideas of things I can tune/control within my current system to both test and optimize the performance of iSCSI to it's highest obtainable levels with the hardware I have.

-Should/could I adjust settings on FreeNAS' iSCSI target such as the queue depth or other settings to optimize performance?

If you are able to max out your gigE, how do you expect it to go faster?

-Can system RAM be used somehow as a cache for iSCSI like it would with ZFS?

Not within the FreeNAS framework.

-Has anyone gotten volume manager to see disks behind a P400/P800 controller to format for UFS?
-Given the constraint of my hardware, are there any better ways to setup FreeNAS for ESX hosts?

I don't think there are good options, no.

ZFS is a bad idea on that hardware because the disks present as a single volume (and if they don't then there are more complications). So your P400 and P800 might each show up as a single disk. In theory you can run ZFS on top of each of those and could even have them as a striped pool. BUT! If ZFS notices any bitrot, it won't have any way to reconstruct the data, and will instead lose you that data.

-Can I use any DD (disk performance tools in FreeNAS) with no UFS volumes?

You can always do dd reads on raw devices. Use "camcontrol devlist" to help you identify them.

I fully expect to draw fire from some of the resident experts (jgreco and cyberjock come to mind first) for both not using the best hardware configurations for disk, but I'll take the abuse to get some further guidance and how to move forward in a productive direction.

Not so much fire as sadness for your situation.

I fully understand that ZFS is usually the recommended way to setup storage for obvious performance benefits and going with software RAID. Unfortunately, the HP P400&P800 controllers do not at all support JBOD, and everyone says trying to do logical drives for individual disks is a terrible idea.

And that advice is pretty much correct. Best suggestion is to see if you can swap out the RAID controllers for an HBA. You only need a single HBA and we could help you ascertain what would be correct for cabling, since the mix of internal and external is a bit ugly.

I would also like to know if I can do any disk performance tests with no volumes formatted for ZFS/UFS? I have read many posts that describe using DD, but they create files for writing/reading. I obviously don't have any volumes since the arrays are being used as devices extents, and I can't find any write-ups about doing that (probably because it can't be done?) So any other ways to do a local performance analysis?

Your arrays may not be exporting any volumes to the operating system. I don't recall the specifics on the P400 and P800 controllers. If they're LSI-based there's a great LSI sticky in the hardware forum that has some info on setting up MFI. Otherwise search through "dmesg | more" output from the CLI to see what - if anything - is appearing to the OS.

This system has a huge amount of free CPU and memory I wish I could better utilize. When doing mass virtual machine startup/shutdowns, I can tap out the 1GB link, but the CPU and memory are barely utilized. Is there any way to use the system's RAM as a massive write/read cache for the iSCSI target process? I can't find any information about that either.

Under ZFS the system memory would be used as ARC, essentially read cache. 16GB isn't actually all that much in the ZFS scheme but it'd be well-used.

Since I can barely spell FreeBSD (which is not linux!-won't make that mistake and be slain), I'm not sure if I need to worry about getting more specialized drivers for the Broadcom NICs? They support jumbo frames and TOE, but since FreeBSD only supports offloading a few of the aspects like checksum, etc, are there other specific things I should be doing for the iSCSI interface to increase it's performance? Are there any step-by-step walk-throughs for setting up the supported offloads? Are they even worth it?

Broadcom has had problems with FreeBSD drivers, and basically FreeBSD supports everything that can be reliably supported with the available documentation. That's my recollection. I stopped buying bge's a bunch of years ago due to some serious problems. If you're getting gigE speeds then don't bother. Jumbo was created back when the packet processing overhead for gigE was a larger percentage of a CPU's resources. It is there to reduce the number of packets and increase the overall throughput. But if you get gigE speeds without jumbo then who cares.

I read some write-ups that were a little over my head regarding multipathing in FreeNAS/FreBSD. Whats the best way to accomplish that? Maybe an example of using two nics for my scenario (Two NICs, two iSCSI targets, each with only one LUN/Extent?)

Two subnets. One interface on each subnet. It works but I haven't done it for a while and I'm on the train so I'm not going to try to go into detail.

 

[sfcredfox]

jgreco,
I hugely appreciate you taking some time to answer my questions, hugely helpful.

Storage

Your arrays may not be exporting any volumes to the operating system. I don't recall the specifics on the P400 and P800 controllers. If they're LSI-based there's a great LSI sticky in the hardware forum that has some info on setting up MFI. Otherwise search through "dmesg | more" output from the CLI to see what - if anything - is appearing to the OS.

Yeah, device iSCSI extend it working correctly, but not for making volumes, must just be a 'hates these RAID controllers' thing.

And that advice is pretty much correct. Best suggestion is to see if you can swap out the RAID controllers for an HBA. You only need a single HBA and we could help you ascertain what would be correct for cabling, since the mix of internal and external is a bit ugly.

ZFS is a bad idea on that hardware because the disks present as a single volume (and if they don't then there are more complications). So your P400 and P800 might each show up as a single disk. In theory you can run ZFS on top of each of those and could even have them as a striped pool. BUT! If ZFS notices any bitrot, it won't have any way to reconstruct the data, and will instead lose you that data.

...Not so much fire as sadness for your situation....
...I don't think there are good options, no. ...

I was under the impression (wrongly) that even though I wasn't using an HBA/ZFS and would loose out on some sexy and amazing benefits, I thought there would be more tuning or things I could do without it. Your response about doing DD on raw device finally got me googling in the right direction (instead of strickly volume based exmaples) and found this: 'http://unix.stackexchange.com/quest...-raw-device-and-so-fast-on-filesystem-usb-key'

So, I think what I learned from all of this regarding storage is:

You either take a very basic, maybe terrible, 'what you see is what you get' system using strictly hardware components, or you use the benefits of ZFS/RAM and then have the ability to get peak performance, and some ability to tune performance based on what you want to do with RAM/SSDs, etc?

...And with that said, it's not FreeNAS' fault if it does not perform awesome (which I never implied or would imply) when using only hardware, now it seems like to get even acceptable performance, you must take full advantage of ZFS/RAM/etc, or not do it at all?

I didn't originally understand hardware only would be so limited (P400/P800 controllers don't suck either, I guess just not good for a FreeBSD/FreeNAS setup). I was just testing it out with the hardware I already have available and learning what needs to be done before I go and buy components.

You can always do dd reads on raw devices. Use "camcontrol devlist" to help you identify them.

This was hugely helpful, I was able to finally figure this out after reading your post and knew what portions of the MAN page to look at.

I was finally able to get some storage performance data:
dd if=/dev/zero of=/dev/da1 bs=1M count=50000
50000+0 records in
50000+0 records out
52428800000 bytes transferred in 185.224997 secs (283054668 bytes/sec)

Now I was able to compare it to this:
http://forums.freenas.org/index.php...ne-with-ibm-serveraid-1015.15025/#post-137591

The other internal array was about that same.

HUGE difference.
Are there any other test I should run to confirm how terrible that is?
I assume that the 1M block is a decent choice, and that the 50000 is a sequential read/write?
Has anyone posted examples of doing random R/W?

I guess option A (hardware only and sucky) doesn't even come close to option B, which means I have to replace P400/P800 and learn how to SAS connect and MSA70.


Networking

If you are able to max out your gigE, how do you expect it to go faster?

Why I was asking about multi-pathing below:

Two subnets. One interface on each subnet. It works but I haven't done it for a while and I'm on the train so I'm not going to try to go into detail.

I'll research it more now since it's possible and a general idea what is the 'right' way to do it. I'm sure there are already some treads about this...

Broadcom has had problems with FreeBSD drivers, and basically FreeBSD supports everything that can be reliably supported with the available documentation. That's my recollection. I stopped buying bge's a bunch of years ago due to some serious problems. If you're getting gigE speeds then don't bother. Jumbo was created back when the packet processing overhead for gigE was a larger percentage of a CPU's resources. It is there to reduce the number of packets and increase the overall throughput. But if you get gigE speeds without jumbo then who cares.

I didn't realize that FreeBSD/FreeNAS hated this so much. Unlucky for me the HP DL/G5 platforms come with these. Boo.

Overall
Seems like the only option to improve from this point (since there is no tuning with hardware only) is to replace RAID for HBA, as mentioned and
to try and bring RAM up to 64GB+.

Cyberjock, totally understand your point of view. Not intentionally setting the system up in a way that FreeNAS hates, that's just what I had available to start with. Also, it's not that I am not willing to spend money, just needed to see what I could get with what I had (which was not great for FreeNAS), and then choose to spend money on the right things, rather than just buy a bunch of stuff without knowing. Who knows, there might be another solution that doesn't mind the hardware I already have like server 2012, just wanted to see what my options were. I am really interested in how an awesome FreeNAS solution compares to a different SAN solution of similar hardware.