Some disks not showing as Multipaths

[Bjarke Emborg Kragelund]

We have 2 systems as below :

Dell PE R720 - 256 GB Memory
2 X LSI 9207-8e
1 X LSI 9207-8i
2 X DNS-1640 JBOD ( LSI SAS-2 chipset on backplane )
48 X WD Xe 900 GB

On 9.2.1.5 we had FW on the LSI on stage 19, and BSD driver 19. I saw exactly 2 disks showing as disks rather than the expected 48 Multipath-disks.

Tried a few other LSI FW stage 17 and stage 18. No effect.

Then tried downgrading the driver to the FreeNAS built-in / stage 16. Et voila. All my disks were now Mulipath. No problem.

On my second system - which I see as identical - I again have the same problem. Even with the built-in FreeNAS driver (Stage 16) - which I thought was the solution.

Having 2 disks showing as disks with no Multipath.

Both systems have current BIOS / FW / Microcode on all HW.

Both systems are running FreeNAS 9.2.1.7.

Any ideas how I can troubleshoot further.

My gut is telling me something with the internal timing of the LSI HW istself, but can not elaborate. The setting on the HBA's should both be vanilla / default. But have not double checked yet.

Best regards
Bjarke Emborg Kragelund

 

[cyberjock]

First, is there anything different about the boxes? You said this:

On my second system - which I see as identical - I again have the same problem. Even with the built-in FreeNAS driver (Stage 16) - which I thought was the solution.

I see that as either you aren't sure they are identical, aren't confident they are identical, or the differences you consider to be so small that it shouldn't matter. Can you clarify for me what you know of that is different, if anything?

Second, phase 16 drivers are what come with FreeNAS. So unless you are compiling FreeNAS with your own newer driver, your statement about the BS driver being on phase 19 is inaccurate. The fact that 17, 18, and 19 didn't work tends to confirm my suspicions that you aren't accounting for the complexity involved with throwing your own driver into FreeNAS.

Third, multipath is something that is almost never used in the world of FreeNAS. So you're going to find a very small subset of users that have any experience with multipath with FreeNAS. I'll try to help you as best as I can. But you are already over the head of the the vast majority of users in this forum just because you are doing multipath.

To be honest if you are looking for multipath (which generally means high reliability type of setup) you were probably better off buying a TrueNAS box. iX's support is there when you need it and turning to a forum or IRC when you have a problem is not exactly good for "high reliability" situations. We've had a few companies that went this route thinking they were saving money by doing it themselves, but if something goes wrong and you call iX you aren't going to get support the same day. You'll have to be put on a contract, it'll only be for so many hours, etc. Not something you want to deal with if things go horribly wrong. Some of these companies had significant monetary losses because they had no support structure when things go wrong. I know of at least one that ended up out of business because of the lost data. They made some serious mistakes on their end that resulted in data loss, but that doesn't make the data loss any less unpleasant. Please keep this in mind when deciding on your server.

 

[Bjarke Emborg Kragelund]

Thx cyberjock,

I am sure that the BIOS settings on the heads them selves are identical. But have not checked the inside of the HBA's - I expected them to be default settings from when leaving the factory, but am not 100% as I have not checked this.

I will check this ASAP on all the HBA's.

This is the BSD driver stage 19 we have used together with a stage 19 FW on the LSI HBA's : http://www.lsi.com/downloads/Public...Files/SAS_SATA_6G_P19/Free_BSD_Driver_P19.zip

Right now the FW / Driver is : 19 / FreeNAS built-in 16. We have been on FW 17, 18, with the same effect. FW stage 19 is a lot let chatty in the console. I can not meassure performancewise diferences.

And you could be right about that we are better off with a real TrueNAS. We have however plunged our selves into the deep end. We are building our new storage on FreeNAS - two identical boxes - which will be agressively replicated to a third box. Both boxes are able to run with full load.

 

[cyberjock]

Yeah, you can't grab that driver and use it. Use the one FreeNAS comes with, period. Trying to add your own drivers without recompiling is just asking for trouble, and is downright stupid for a system that is used for work. The thumbrule for FreeNAS is that if you are having to do any kind of file editing for FreeNAS you are doing a major no-no.

So I'd reinstall FreeNAS on your USB stick to make sure you are back with stock software and reflash your cards to phase 16 IT mode firmware.

Aside from that, I don't really know what to say. When you are this early in and making a mistake like you are making (no offense) I really cringe inside because I have to wonder what stuff you may be doing that you shouldn't. For example, if you've done any kind of serious reading on the forums you'd know that:

1. FreeNAS 9.2.1.x uses p16 drivers.
2. Your firmware should be p16 so that it matches the driver.
3. You should reflash it to IT mode.

The fact that you messed up #1 and #2, which for all intents and purposes is "basic level knowledge for using FreeNAS" I'm inclined to think that there's a lot of other basic knowledge you may not know and you should probably slow down and do much more reading (or get a contractor involved with FreeNAS experience).

All that aside, once you've handled the hardware side of things, installing and using FreeNAS should "just work". The fact it isn't leaves so many doors unopened I don't even know where to direct you to start looking at your problems more closely. :(

 

[Bjarke Emborg Kragelund]

Hi cyberjock,

I believe I have read the documentation you are refering to. Am well aware of all my mistakes - they were done for a reason though.

Ad 1) Super well aware.
Ad 2) We did try the FW stage 16 on our LSI 9207 HBA's, but they generated a lot af chat in the console. On another thread we were encouraged to flash to a newer build. Which did reduce the chattyness. Will go back to FW Stage 16 to have look.
Ad 3) Wouldn't have it any other way.

It takes _some_ tweaking to make FreeNAS "just work" as a NFSv3 datastore for vSphere. It is useless out-of-the-box. It has nothing to do with FreeNAS itself though. The vanilla ZFS-settings and TCP-settings is just poor for that kind of access-profile. So we will just have to disagree on that one.

Will try the FW stage 16 / FreeNAS driver, and doublecheck the HBA settings on all adapters.

Brgds.
Bjarke Emborg Kragelund

 

[cyberjock]

It takes _some_ tweaking to make FreeNAS "just work" as a NFSv3 datastore for vSphere. It is useless out-of-the-box. It has nothing to do with FreeNAS itself though. The vanilla ZFS-settings and TCP-settings is just poor for that kind of access-profile. So we will just have to disagree on that one.

Work, or perform? Because it should absolutely "just work". I know this as I've set up plenty of vsphere datastores with NFSv3 (and a few with NFSv4 enabled). Saying it doesn't perform is definitely a debate neither of us will win. I know I can easily get 300MB/sec+ over 10Gb with no weaking as that is what many TrueNAS users get.

Anyway, try fixing the firmware and drivers to match. Got a link to the thread that said you should upgrade past v16? That doesn't sound like something that has been said in this forum.

 

[Bjarke Emborg Kragelund]

Thx for all the input.

Not able to downgrade the LSI HBA's on the FreeNAS itself. Will have to wait until I get onsite and boot on another OS.

I would say work. But then again I have only seen it on 4 systems. That is not nearly representative enough. All systems had in excess of 100 seconds latency on VMs. That is perhaps "working", but the BSOD in Windows makes me write not work. When in fact your could say that it is a performance related "issue" - and I would agree.

Allthough you have NFSv4 enabled on your FreeNAS, I would have sworn that ESXi does not support / use that yet?

As I understand only NFSv3 TCP is supported on the current version 5.5.

We had an unresponsive system in the begining - the FW upgrade was a way of troubleshooting in the begining. Recommandations to upgrade FW was not from this forum. As I do not have the errors on screen I can not supply you with a link. It was a BSD-based / ZFS-forum though.

 

[cyberjock]

If you were dealing with 100 seconds of latency on VMs then the server was not given appropriate hardware for the task. This isn't a problem with FreeNAS but more of a problem with expecting too much with the hardware used. I've seen VMs with 30-40 seconds of latency and it's always a matter of not properly sizing the system to handle the workload being expected of it. So I'd argue that the server 'worked' but was overloaded for the expected load. This is a common problem, especially for people new to ZFS, and the only fix is to get yourself learned up or get someone to handle designing your server. My own home server can handle one or two VMs without it sucking terribly. Since I didn't build my server to handle VM workload that's to be expected. But over my 10Gb LAN I can easily do over 500MB/sec.

ESXi doesn't support NFSv4 yet as far as I know. I'm not a pro at ESXi so there may be some hidden trick to enable it or it may actually support it and I don't know it. That's why I said "with NFSv4 enabled" since you can enable it on the FreeNAS side but have to fallback to NFSv3 because of ESXi's limitations.

 

[Bjarke Emborg Kragelund]

Do not agree. Throuhput =! latency. Responsiveness is what makes or brakes a good vSphere datastore. Raw throughput is of less importance. Balancing the two is the real art of ZFS. IMHO.

On same HW after tweaking ZFS, latency drops to below 10 ms for 25 runing VMs - which is our watermark - before it was impossible to power on 5 VMs which all were peaking at latencies on _seconds_ rather than _ms_.

 

[jgreco]

It takes _some_ tweaking to make FreeNAS "just work" as a NFSv3 datastore for vSphere. It is useless out-of-the-box. It has nothing to do with FreeNAS itself though. The vanilla ZFS-settings and TCP-settings is just poor for that kind of access-profile. So we will just have to disagree on that one.

Works fine here pretty much "out-of-the-box", but admittedly the environment here is well-designed and appropriate to the task, plus I'm not actually doing anything stressy most of the time.

Conspicuously missing from your system description is the pool design and how many IOPS you're pushing at the filer. Mistakes in engineering design typically translate to much unhappiness in production.

It is unclear why you failed to test and burn in the hardware prior to putting it into production. Problems like "my disks aren't all multipath" are properly diagnosed in the workshop as solving that class of problem may well be hardware in nature, a bad backplane or other similar issue.

So basically we shouldn't even be having this discussion ... you've optimistically assumed everything's going to work right and that's turned out not to be the case. The correct fix here is to disconnect the filer from the virtualization environment and go back to hardware validation and burn-in. Once you get a week's worth of heavy duty stress testing to work without faults, then look at what an appropriate pool design and configuration would be for ESXi and NFS.

 

[cyberjock]

Do not agree. Throuhput =! latency. Responsiveness is what makes or brakes a good vSphere datastore. Raw throughput is of less importance. Balancing the two is the real art of ZFS. IMHO.

Did you even read my post? I said I can only run one or two VMs but get 500MB/sec. It should have been clear from that statement alone that throughput =! latency. Do you really think I'm not fully aware of this very simple fact and that in almost 13k posts and over 2 years on this forum I'd be that naive to such a simple reality?

On same HW after tweaking ZFS, latency drops to below 10 ms for 25 runing VMs - which is our watermark - before it was impossible to power on 5 VMs which all were peaking at latencies on _seconds_ rather than _ms_.

And I bet what you did was probably far from recommended, but whatever. Your data and your server. ;)

I do need to leave this thread though. Just as jgreco said, I don't need to get into more frustrating arguments with someone today.

Good luck to you!

 

[Bjarke Emborg Kragelund]

Again. Do not agree. Albeit I have only 4-6 boxes to show for.

The boxes are not in real production as of yet.

The two disks not showing as multipath disks are in FreeNAS only. OI, SmartOS et. al are showing all 48 spindels as having two paths. Also CentOS - which I use to flash the HW - shows 48 multipath disk.

I mistakenly gathered that going to the FreeNAS built-in driver would be the trick - as that seemed to work on the 1 head. It was not. Hence I am looking at differences between the two boxes.

ZFS-Pool is made of 23 mirrored vdevs.
2 x spares
4 x SLOG ( sTec 16 GB )
1 x PCIe 800 GB L2ARC.

About 2500 OPS at the moment. But have seen peaks at ~250k OPS when pushed. Most of those OPS are from the ARC though. Nevertheless... the HW _can_ deliver.

During the early days when testing the spindels, they all performed according to spec. That is the, pool could read and write pretty much exactly what were expected.

During vMotion of 10 concurretn VMs, we have seen writes to the pool of around 400 MB/sec. That is not considering compression, which is about 1.8 for our testing VMs. So in excess of 600 MB/sec on the wire. That is easy 2 times better than in our curretnt production environment, which is about twice as expensive.

 

[Bjarke Emborg Kragelund]

@cyberjock, trying to learn.

 

[jgreco]

So your drives are attaching via mps and not mfi? I just checked and I don't have any dualported stuff online right now...

Anyways at 2500 IOPS you may well be near or actually exhausting the pool's capacity. A 3.5" 10K drive should give at least 100 but not more than 200 IOPS per spindle, but mirroring two reduces that slightly and 23 sets gives you probably 2000-4000 IOPS for random small write operations hitting the pool.

But I think that sorting out the connectivity issue needs to happen first..