SOLVED FreeNAS 9.3 FC Fibre Channel Target Mode DAS & SAN

[mav@]

I would suggest to try updating FreeNAS to one of latest nightly builds. ispfw(4) module there got much newer firmware for all cards generations, especially for 24xx and 25xx, while isp(4) driver got many fixes/updates, including some, fixing alike 'INIT FIRMWARE' errors when running on newer firmware. It all should be in the next regular software update, but it will probably take few weeks.

 

[severusx]

Ok, I will try one of the nightlies. Doing a little research I see that 25xx isn't listed as supported by FreeBSD 9.3R but I'm not sure how that correlates to FreeNAS 9.3.1 exactly. I also bought a 2462 off eBay but I'd really rather use the newer card if I can.

Still baffled as to how I got it working the first time. I took the HBA back to the desktop I was testing with and reinstalled using the original ISO I used to no avail. I then updated and tried again and it still didn't work. I'm also going to try the next stable build after the one I tested with to see if there is something there.

 

[mav@]

24xx and 25xx series are actually not so much different from the driver perspective to tell that some of them is supported better then another now. Even newest 26xx 16Gbps cards, that are more different, took me only couple weeks to add basic support in FC mode, which by the way is also present in the nightlies. So, if your problem won't go away after updating to latest nightly, I would probably like to look closer on it.

 

[severusx]

I updated to FreeNAS-9.3-Nightlies-201512181501 and I'm still seeing the same issue. Based on some of your previous posts, I have the following Tunables:

ctl_load="YES"
hint.isp.0.role=0
hint.isp.1.role=1

And a postinit command of "ctladm port -o on -t fc"

I didn't include the ispfw_load="YES" as I see it is already in loader.conf and there is no reason to put the ctladm command in as a tunable.

No matter what I get "no_light" from my HBA on the switch, but if I go into the HBA bios and scan for LUNs it will light up the switch port.

dmesg output from last attempt (I had left the postinit script out so I could compare the boot messages to the ctlamd output):

[root@freenas] ~# ctladm port -o on -t fc
Front End Ports enabled
[root@freenas] ~# dmesg | grep isp
isp0: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xe100-0xe1ff mem 0xf7e84000-0xf7
isp0: Chan 0 setting role to 0x0
isp1: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xe000-0xe0ff mem 0xf7e80000-0xf7
isp1: Chan 0 setting role to 0x0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0
isp0: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xe100-0xe1ff mem 0xf7e84000-0xf7
isp0: Chan 0 setting role to 0x0
isp1: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xe000-0xe0ff mem 0xf7e80000-0xf7
isp1: Chan 0 setting role to 0x0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0
ctlfe_onoffline: isp0 current WWNN 0x20000024ff346c70
ctlfe_onoffline: isp0 current WWPN 0x21000024ff346c70
isp0: Mailbox Command 'INIT FIRMWARE' failed (COMMAND PARAMETER ERROR)
isp0: isp_reinit: cannot init card
ctlfe_onoffline: SIM isp0 (path id 0) target enable failed with status 0x4
ctlfe_onoffline: isp1 current WWNN 0x20000024ff346c71
ctlfe_onoffline: isp1 current WWPN 0x21000024ff346c71
isp1: Mailbox Command 'INIT FIRMWARE' failed (COMMAND PARAMETER ERROR)
isp1: isp_reinit: cannot init card
ctlfe_onoffline: SIM isp1 (path id 1) target enable failed with status 0x4
[root@freenas] ~#

 

[mav@]

Hmm. That is odd. We should increase log verbosity to get more info. Please try setting hint.isp.0.debug tunable to the value 0xff7f and send me full dmesg output. It should be quite large and noisy, but hopefully informative.

 

[severusx]

[root@freenas] ~# dmesg
(ctl1:isp0:0:-1:-1): Put FREE INOT, (seq id 0x0) count now 310
isp0: isp_action code 336

##This repeats over and over until the count reaches 1024. Omitted for brevity##

(ctl1:isp0:0:-1:-1): Put FREE INOT, (seq id 0x0) count now 1024
isp0: isp_action code 18
ctlfe_onoffline: isp0 current WWNN 0x20000024ff346c70
ctlfe_onoffline: isp0 current WWPN 0x21000024ff346c70
isp0: isp_action code 19
(noperiph:isp0:0:-1:-1): changing role on from 0 to 1
isp0: Setting ICB Node 0x20000024ff346c70 Port 0x21000024ff346c70
isp0: isp_fibre_init_2400: atioq 0000000001be8000
isp0: isp_fibre_init_2400: fwopt1 0x6836 fwopt2 0x1020 fwopt3 0xc028
isp0: isp_fibre_init_2400: rqst 0000000001bc0000 rsp 0000000001bd0000
isp0: isp_fibre_init_2400:
isp0: 0x00000000: 01 00 00 00 00 08 ff ff 00 08 00 00 21 00 00 24
isp0: 0x00000010: ff 34 6c 70 20 00 00 24 ff 34 6c 70 00 00 00 00
isp0: 0x00000020: 03 00 00 00 00 04 00 04 00 00 00 00 00 00 bc 01
isp0: 0x00000030: 00 00 00 00 00 00 bd 01 00 00 00 00 00 00 00 00
isp0: 0x00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
isp0: 0x00000050: 00 80 be 01 00 00 00 00 00 00 1e 00 36 68 00 00
isp0: 0x00000060: 20 10 00 00 28 c0 00 00 00 00 00 00 00 00 00 00
isp0: 0x00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
isp0: INIT F/W from 0000000003455000
isp0: Mailbox Command 'INIT FIRMWARE' failed (COMMAND PARAMETER ERROR)
isp0: isp_reinit: cannot init card
ctlfe_onoffline: SIM isp0 (path id 0) target enable failed with status 0x4
ctlfe_onoffline: isp1 current WWNN 0x20000024ff346c71
ctlfe_onoffline: isp1 current WWPN 0x21000024ff346c71
isp1: Mailbox Command 'INIT FIRMWARE' failed (COMMAND PARAMETER ERROR)
isp1: isp_reinit: cannot init card
ctlfe_onoffline: SIM isp1 (path id 1) target enable failed with status 0x4
[root@freenas] ~#

 

[mav@]

Hmm. I wanted to avoid some of messages, but instead avoided some useful. Please try once more but with 0xffff tunable value.

 

[severusx]

I'll run it as soon as I get back in front of a PC. Is there a way to make the dmesg log buffer bigger or output to file so you can see the whole boot log?

 

[severusx]

Here is the output. the "isp0: isp_action code 336" repeats over and over from count 317 to 1024. I started at the last one so the log wasn't 3 pages.


isp0: isp_action code 336
(ctl1:isp0:0:-1:-1): Put FREE INOT, (seq id 0x0) count now 1024
isp0: isp_action code 18
ctlfe_onoffline: isp0 current WWNN 0x20000024ff346c70
ctlfe_onoffline: isp0 current WWPN 0x21000024ff346c70
isp0: isp_action code 19
(noperiph:isp0:0:-1:-1): changing role on from 0 to 1
isp0: Mailbox Command 'SET FIRMWARE OPTIONS'
isp0: IN mbox 0 = 0x0038
isp0: IN mbox 1 = 0x0001
isp0: IN mbox 2 = 0x0000
isp0: IN mbox 3 = 0x0000
isp0: RISC2HOST ISR 0x40000010
isp0: RISC2HOST ISR 0x40008010
isp0: OUT mbox 0 = 0x4000
isp0: Setting ICB Node 0x20000024ff346c70 Port 0x21000024ff346c70
isp0: isp_fibre_init_2400: atioq 0000000001be8000
isp0: isp_fibre_init_2400: fwopt1 0x6836 fwopt2 0x1020 fwopt3 0xc028
isp0: isp_fibre_init_2400: rqst 0000000001bc0000 rsp 0000000001bd0000
isp0: isp_fibre_init_2400:
isp0: 0x00000000: 01 00 00 00 00 08 ff ff 00 08 00 00 21 00 00 24
isp0: 0x00000010: ff 34 6c 70 20 00 00 24 ff 34 6c 70 00 00 00 00
isp0: 0x00000020: 03 00 00 00 00 04 00 04 00 00 00 00 00 00 bc 01
isp0: 0x00000030: 00 00 00 00 00 00 bd 01 00 00 00 00 00 00 00 00
isp0: 0x00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
isp0: 0x00000050: 00 80 be 01 00 00 00 00 00 00 1e 00 36 68 00 00
isp0: 0x00000060: 20 10 00 00 28 c0 00 00 00 00 00 00 00 00 00 00
isp0: 0x00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
isp0: INIT F/W from 0000000003455000
isp0: Mailbox Command 'INIT FIRMWARE'
isp0: IN mbox 0 = 0x0060
isp0: IN mbox 1 = 0x0000
isp0: IN mbox 2 = 0x0345
isp0: IN mbox 3 = 0x5000
isp0: IN mbox 6 = 0x0000
isp0: IN mbox 7 = 0x0000
isp0: RISC2HOST ISR 0x40000010
isp0: RISC2HOST ISR 0x40000010
isp0: RISC2HOST ISR 0x40000010
isp0: RISC2HOST ISR 0x40060011
isp0: RISC2HOST ISR 0x40068011
isp0: OUT mbox 0 = 0x4006
isp0: OUT mbox 1 = 0x0000
isp0: OUT mbox 2 = 0x0003
isp0: OUT mbox 3 = 0x5000
isp0: Mailbox Command 'INIT FIRMWARE' failed (COMMAND PARAMETER ERROR)
isp0: isp_reinit: cannot init card
ctlfe_onoffline: SIM isp0 (path id 0) target enable failed with status 0x4
isp0: RISC2HOST ISR 0x40060011
isp0: RISC2HOST ISR 0x40060011
ctlfe_onoffline: isp1 current WWNN 0x20000024ff346c71
ctlfe_onoffline: isp1 current WWPN 0x21000024ff346c71
isp1: Mailbox Command 'INIT FIRMWARE' failed (COMMAND PARAMETER ERROR)
isp1: isp_reinit: cannot init card
ctlfe_onoffline: SIM isp1 (path id 1) target enable failed with status 0x4

 

[severusx]

mav@, any chance to look at this? I'm trying to wrap up my build before the holiday. :)

 

[mav@]

I am looking on it. Unfortunately firmware does not report what exactly it does not like, so I have to guess. My best guess at this point is that there is a bug in the driver around NVRAM link speed settings. I am now pushing the patch that should fix that in tomorrow nightly (please try it). Until that you may try to go to card's BIOS and change link speed settings there from 8Gbps to Auto.

 

[severusx]

Good news, setting the speed back to auto lit the port up as desired and I can see the target with my vSphere cluster. I am going to reinstall to the latest STABLE train and test further as this is going to be a production system and I would rather not run on nightlies. I will report back with an update and the final config I am using.

 

[mav@]

Could you at least test this issue on next nightly build before reverting to STABLE, as compensation for my time? Thanks.

 

[severusx]

Absolutely, I have 2 of the same model HBA so I will test your nightly build with my POC system and build the production system with the stable build. I really really appreciate your time and assistance. I will have the test system available to me for quite some time, so if you need testing with further builds let me know.

 

[severusx]

Here is my full setup for the STABLE train (I will test the new nightly with my test system):

Set my QLogic QLE2562 HBA to Speed AUTO and disabled the boot BIOS.

Installed FreeNAS-9.3-STABLE-201512121950

Tunables:
Variable: ctl_load Value: YES Type: Loader
Variable: hint.isp.0.role Value: 0 (zero) Type: Loader
Variable: hint.isp.1.role Value: 0 (zero) Type: Loader

Tasks - Init/Shutdown Scripts:
Type: command Command: ctladm port -o on -t -fc When: postinit

After setting these and then setting up the iSCSI settings (Step 5) per Aran Kaspars guide at the start of this post, I am able to see my LUN with vSphere.

A BIG thanks to mav@ for your help here and also to aran for all the work on this. I will post back after I test the nightly with the speed set to 8.

 

[severusx]

mav@,
I am happy to report that the latest nightlies do work with the speed hard-set at 8. While I will be remaining on the Stable train for my productions system, if you need me to test additional changes with my test box let me know. Thanks again for your help.

 

[fips]

Hey Guys,

I followed the how to in this thread, but I have my read performance stuck at 200MB/s @ 4Gb FC.
Can you FC experts check my post where I described my situation?

https://forums.freenas.org/index.ph...e-is-fine-read-stuck-at-200mb-s-4gb-fc.40307/

Thanks

Steve

 

[VSXi-13]

Hey, how did everything go? Judging by the silence I'm - hoping - that it all worked great. Haha.

Over a year later... everything went swimmingly well.

 

[LearnLearnLearn]

First I've not seen a notice of this thread for a long long time. No idea why.

Mine has also been operational non stop since getting this setup completed. I love not being tied into proprietary hardware anymore.

My only complaint is that I have to stop the vmware vms in order to back them up.
I've never found enough information to be able to automate this. It seems there is a way of doing it using freenas but everything I've read/tried has lead to nothing so I back them up manually.

I never checked but I can probably use my OnStor storage managers to consolidate and aggregate all of the storage and simply have snapshots.

 

[Ryan Blakelsee]

Hello,

I am trying to understand if and how it's possible to use FreeNAS with this FC configuration to provide storage to multiple Proxmox PVE hosts (or any host for the sake of discussion) without worrying about data corruption.

My configuration is as follow:

FreeNAS Setup:
1. Dell PowerEdge 2950 Server, Qlogic QLE2462 (qty 2)
2. Dell MD1000
3. Two zvolumes, mapped to two separate iSCIS extents.

Host setup:
1. HP Server with Qlogic QLE2462.

I have connected server (A) with point-to-point, followed the instructions at the top of this post, and I have working FC storage to my host from FreeNAS. Great! Now I want to setup server (B), same configuration, provide storage via FreeNAS / FC. This is where I am confused.

Server A and Server B both see the same iSCSI target when I go into the QLogic bios settings. Also, I notice that to the QLogic HBA sees each extent as a LUN. I am unsure of how to provide LUN1 to Server (A) and LUN2 to Server (B) when it's all visible to both.

Doesn't that create a risk of file corruption, even if I only mount one LUN (i.e. LUN1) to one physical host (i.e. Server A.) ..? My end goal is to be able to setup separate zvolumes for each server, all of the same FreeNAS box.

Side question to that last statement: assuming there is a way to accomplish all this, would i be better carving out separate zvolumes for each physical host, or having one large zvolume avail (somehow, if it's safe from corruption) to all physical hosts?

Thank you very much in advance!