FreeNAS only detecting 2 drives of Smart Array P420i

[bigphil]

looked to see if I could disable the P420i in BIOS, but I was not able to find an option to disable it. I

It's there...under PCI devices on the main menu in the BIOS you can disable the P420i.

 

[drabadue]

It's there...under PCI devices on the main menu in the BIOS you can disable the P420i.

@bigphil -

I'm on FreeNAS 11.1, with the p420i firmware on 8.32. I can set the p420i to HBA mode and get all the drives to show up in FreeNAS just fine, so long as I leave the FBWC installed. When I remove the FBWC, FreeNAS only shows 2 drives.

My last post was moreso talking about using a non-HP LSI sas card and it's effects on fan speed. The fans do run a bit louder with the flashed PERC H310's, but it's not crazy loud, which confirms what you had mentioned in an earlier post, that the fans are no where near 100% when using non-HP hardware.

At the end of my post, I was mentioning that I may go back to the p420i and buy a p420 to hook up both of my 8 bay cages up, so the fans would run slower, and so I would get the HDD indicators back.

I am perfectly happy with the way the p420i works in HBA mode with FreeNAS.

Also, thanks for the heads up on where the p420i is in bios. I need to poke around some more and see if I can find that.

 

[drabadue]

Something else I noticed, I am having trouble reading S.M.A.R.T. info on *most* of the drives in FreeNAS.

In the CLI, I can run "smartctl -c /dev/da4" followed by "smartctl -a /dev/da4" and looking at the info, I see:

SMART support is: Unavailable - device lacks SMART capability.

I find this interesting, because I can boot into CentOS 7 on a USB stick on the same server (dl380p g8), and run the same commands and get the results of the short SMART test.

The weird thing is, SMART does work on drive da2. But none of the others.

I am going to pop my IT flashed Perc H310 back in and see if I can get the SMART commands to work.

 

[kdragon75]

99% of RAID cards that do "passthrough" or "jbod" mode still hide critical information (i.e. SMART) from the system and should NOT be used unless that support being flashed to IT mode firmware and have be flashed.

 

[drabadue]

99% of RAID cards that do "passthrough" or "jbod" mode still hide critical information (i.e. SMART) from the system and should NOT be used unless that support being flashed to IT mode firmware and have be flashed.

@kdragon75 - with my H310 card installed, FreeNAS can run smart tests on all the drives. So I'll agree that FreeNAS plays nice with a true 'IT' mode card, rather than a raid card set in HBA mode. I'm just a smidge upset that CentOS 7 can run the smart commands through the p420i no problem, but FreeNAS can't.

 

[kdragon75]

Perhaps its supporting a special driver/module for the card. In any case, A proper HBA can be had for under $50 and will resolve your issues as well as provide better/more consistent performance.

 

[drabadue]

Perhaps its supporting a special driver/module for the card. In any case, A proper HBA can be had for under $50 and will resolve your issues as well as provide better/more consistent performance.

@kdragon75

I've got two Dell Perc H310's (LSI 9211-8i) cards that are flashed to IT mode. And they do work as they should. I was just hoping I could get the onboard card to work as it should, since it says it's supported in FreeBSD, and so the fans would run super slow with all original HP hardware. No biggie. Just pointing out my experience.

 

[bigphil]

Something else I noticed, I am having trouble reading S.M.A.R.T. info on *most* of the drives in FreeNAS.

In the CLI, I can run "smartctl -c /dev/da4" followed by "smartctl -a /dev/da4" and looking at the info, I see:

SMART support is: Unavailable - device lacks SMART capability.

I find this interesting, because I can boot into CentOS 7 on a USB stick on the same server (dl380p g8), and run the same commands and get the results of the short SMART test.

The weird thing is, SMART does work on drive da2. But none of the others.

I am going to pop my IT flashed Perc H310 back in and see if I can get the SMART commands to work.

Have a look at this thread. It seems possible to get the smart data when behind a Smart Array raid controller like the P420i. Essentially using the -d option on smartctl and it connects to a specific device on the controller and from there can read the other drives behind it. Look at the -d option on the smartctl man page as there is a specific usage for cciss drivers. Again...the main problem is that the driver FreeNAS uses for HP Smart Array controllers. It's old. Pretty much every other OS moved away from the ciss driver to the hpsa driver a long time ago. There doesn't seem to be one for FreeBSD.

 

[bigphil]

I take back anything good I said about the P420i...it's got issues ;-) If you don't have a cache module installed your queue depth will only be 32 (in HBA mode...not tested with RAID mode). With a cache module (I've tested with 512MB and 1GB modules) it'll jump up to 1024. Needless to say, without the module its basically useless for any performance. That's all fine and dandy, but the real issue seems to be that in HBA mode it doesn't seem to play nice with some SATA SSD's. You can see the device just fine but creating file systems may be an issue in ESXi. I've changed the physical block size from 4Kn to 512n on an Intel S3700 and the same problem with both sizes. Many people have had the same experience (just one example) and nobody has found a fix except to move to a different HBA. I know you've already moved on to the H310's, but for any future user searches...I recommend NOT using a P420i in HBA mode...garbage.

 

[Elliot Dierksen]

I have absolutely no idea why I allowed myself to waste a ton of time on this, but I did. Perhaps it is because it hurts my pride to lose to equipment, but I do have an answer to a question that no longer matters to me. :) None of the version of HP SSA that I could find will let you mess with the actual settings in the controller (A P822 in my case), but you can do it with hpssacli. HP has little to no support for FreeBSD, so I installed Ubuntu Server 16.04 on one of my test boxes and installed the hpssacli-2.40-13.0_amd64.deb package from https://downloads.linux.hpe.com/sdr/repo/mcp/pool/non-free/, and Viola! The command inside hpssacli is

Code:

controller slot=X modify hbamode=on

and that is it. I think there is an abbreviation for controller (cntrl or something like that), but I don't feel like swapping the disks back to Ubuntu to find out. OMFG! I looked for this for years, and now I can find it because I don't need it any longer. Hopefully it helps someone else. I will say from may past experience using a CISS controller in RAID mode (not doing it any longer, don't roast me!), it is pretty good about sending alerts when a drive is going bad. I am still much happier with the LSI 9207. Sigh...

 

[titusc]

What about using the LSI 9300-8i HBA? I can find these at $99 online so appear to be a good deal if it can solve the problems described here.
I found a source selling DL360 G8 for $119.95 USD so am very interested in this.
DL360 G8 $119.95 USD
Xeon E5-2630L v2 $35.25 USD
LSI 9300-8i HBA $90 USD
Total excl memory and hard disks = $245.20 USD

Very tempting! The only concern I have with 1U server is the noise of the fans.....

 

[Elliot Dierksen]

What about using the LSI 9300-8i HBA?

I am sure that would be fine. That is 12G SAS HBA which is probably more than you need. I believe the internal SAS expander in that server is only 6G. A 9207-8i would be cheaper, and I can personally attest to those working working. I strongly suspect the 9300-8i would work, but I don't have any personal experience with it. I think you might also need different cables to hook to the SAS expander, but I don't know for sure.

 

[titusc]

I am sure that would be fine. That is 12G SAS HBA which is probably more than you need. I believe the internal SAS expander in that server is only 6G. A 9207-8i would be cheaper, and I can personally attest to those working working. I strongly suspect the 9300-8i would work, but I don't have any personal experience with it. I think you might also need different cables to hook to the SAS expander, but I don't know for sure.

Yes you are right. I am now looking for some of these cables to convert to the internal 6Gbps internal expanders.... Oh well.

 

[Ralms]

Thanks to everyone for your recommendations! Looks like I'm just going to get a HBA then.

I've heard some horror stories about the fans ramping up to 100% when non-HP cards are used, so that's why I'm looking at the H220. Can you confirm that this does not happen with the LSI 9207?

@kdragon75

I've got two Dell Perc H310's (LSI 9211-8i) cards that are flashed to IT mode. And they do work as they should. I was just hoping I could get the onboard card to work as it should, since it says it's supported in FreeBSD, and so the fans would run super slow with all original HP hardware. No biggie. Just pointing out my experience.

I'm running FreeNAS on an HP DL360p with a HP P420i fine with 4 Sata drives currently. ( 4 x Seagate NAS 3TB)
SMART is fully working and I can see both attribute details as well as run tests using smartctl.
Funny part, the controller being in HBA mode gives a notice while booting saying "Hardware raid disabled on NVRam" or something like that, and even then It will be still monitoring the drives, as I had one drive starting to fail and the caddy started blinking orange, I though that was cool.

For future people that might be on the same situation:
I changed it to HBA mode using HP tools repository. Altough a lot of people in the Internet say that you need the HP Service Pack ISO, you don't.
The downside was that had to install CentOS on a spare harddrive to be sure everything would work fine.
This is what I've done:

• The Repo used was: http://downloads.linux.hpe.com/SDR/repo/spp-gen8/
• Following this steps to install the tools: https://www.centos.org/forums/viewtopic.php?t=55506
  I had to install the repositories manually as explained and only installed the packages hpssacli, hpssa and hp-health.
  I've skipped everything else, I don't even think I needed the hp-health.
• I used this blog for guidance on the commands: https://www.stephenrlang.com/2017/03/using-hpamcli-for-hp-servers/
  With no Arrays created on the controller, I've ran: "hpssacli controller slot=0 modify hbamode=on forced"
  

I've have very little experience with FreeNAS, but the fact that both the controller and FreeNAS detected my WD Red 3TB dying at the same time due to high relocation sector count give me confidence it's working well.

Hope this helps.

 

[jgreco]

I have absolutely no idea why I allowed myself to waste a ton of time on this, but I did. Perhaps it is because it hurts my pride to lose to equipment, but I do have an answer to a question that no longer matters to me. :) None of the version of HP SSA that I could find will let you mess with the actual settings in the controller (A P822 in my case), but you can do it with hpssacli. HP has little to no support for FreeBSD, so I installed Ubuntu Server 16.04 on one of my test boxes and installed the hpssacli-2.40-13.0_amd64.deb package from https://downloads.linux.hpe.com/sdr/repo/mcp/pool/non-free/, and Viola! The command inside hpssacli is

Code:

controller slot=X modify hbamode=on

and that is it. I think there is an abbreviation for controller (cntrl or something like that), but I don't feel like swapping the disks back to Ubuntu to find out. OMFG! I looked for this for years, and now I can find it because I don't need it any longer. Hopefully it helps someone else. I will say from may past experience using a CISS controller in RAID mode (not doing it any longer, don't roast me!), it is pretty good about sending alerts when a drive is going bad. I am still much happier with the LSI 9207. Sigh...

The real problem here would be if this was still running through the CISS driver. I don't know the specifics of the more modern HP controllers, but I find it unlikely that they actually work in a true HBA mode that doesn't still involve the CISS driver, which is known to be quirky. The thing that we know works really well is the LSI HBA driver (the non-mfi drivers, mps/mpt).

Can you make the drives accessible on your FreeNAS? Absolutely, without a doubt, and there are even ways to get SMART data, just as there are with the 3Ware or Areca cards. The problem is that you're really out there on your own with a poorly tested and unsupported card, doing highly demanding I/O tasks, and there's a much higher chance of instability or issues.

 

[Ralms]

The real problem here would be if this was still running through the CISS driver. I don't know the specifics of the more modern HP controllers, but I find it unlikely that they actually work in a true HBA mode that doesn't still involve the CISS driver, which is known to be quirky. The thing that we know works really well is the LSI HBA driver (the non-mfi drivers, mps/mpt).

Can you make the drives accessible on your FreeNAS? Absolutely, without a doubt, and there are even ways to get SMART data, just as there are with the 3Ware or Areca cards. The problem is that you're really out there on your own with a poorly tested and unsupported card, doing highly demanding I/O tasks, and there's a much higher chance of instability or issues.

How can I check what driver is being used?

 

[Ralms]

The real problem here would be if this was still running through the CISS driver. I don't know the specifics of the more modern HP controllers, but I find it unlikely that they actually work in a true HBA mode that doesn't still involve the CISS driver, which is known to be quirky. The thing that we know works really well is the LSI HBA driver (the non-mfi drivers, mps/mpt).

Can you make the drives accessible on your FreeNAS? Absolutely, without a doubt, and there are even ways to get SMART data, just as there are with the 3Ware or Areca cards. The problem is that you're really out there on your own with a poorly tested and unsupported card, doing highly demanding I/O tasks, and there's a much higher chance of instability or issues.

Well, it seems that's the case with the P420i

Code:

da1 at ciss0 bus 32 scbus1 target 1 lun 0
da1: <ATA ST3000VN000-1H41 SC43> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number XXXXXXX
da1: 135.168MB/s transfers
da1: 2861588MB (5860533168 512 byte sectors)

da0 at ciss0 bus 32 scbus1 target 0 lun 0
da0: <ATA ST3000VN000-1HJ1 SC60> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number XXXXXXX
da0: 135.168MB/s transfers
da0: 2861588MB (5860533168 512 byte sectors)

da2 at ciss0 bus 32 scbus1 target 2 lun 0
da2: <ATA ST3000VN000-1HJ1 SC60> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number XXXXXXX
da2: 135.168MB/s transfers
da2: 2861588MB (5860533168 512 byte sectors)

da3 at ciss0 bus 32 scbus1 target 3 lun 0
da3: <ATA ST3000VN000-1HJ1 SC60> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number XXXXXXX
da3: 135.168MB/s transfers

da4 at umass-sim0 bus 0 scbus6 target 0 lun 0
da4: <SanDisk Ultra USB 3.0 1.00> Removable Direct Access SPC-4 SCSI device
da4: Serial Number XXXXXXX
da4: 40.000MB/s transfersda3: 2861588MB (5860533168 512 byte sectors)
da4: 29327MB (60062500 512 byte sectors)
da4: quirks=0x2<NO_6_BYTE>

da5 at umass-sim1 bus 1 scbus7 target 0 lun 0
da5: <SanDisk Ultra USB 3.0 1.00> Removable Direct Access SPC-4 SCSI device
da5: Serial Number XXXXXXX
da5: 40.000MB/s transfers
da5: 29327MB (60062500 512 byte sectors)
da5: quirks=0x2<NO_6_BYTE>

 

[jgreco]

Use dmesg and determine what driver is in use. You're looking for sets of lines like

Code:

da3 at mps0 bus 0 scbus3 target 59 lun 0
da3: <ATA WDC WD80EMAZ-00W 0A83> Fixed Direct Access SPC-4 SCSI device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 7630885MB (15628053168 512 byte sectors)

That first line says that da3 is hooked up to "mps", which is the driver for an LSI HBA controller, and that's known to be fine. Additionally, it isn't showing up as any sort of virtual disk device, which is double good.

If your system's been up for awhile, a copy of the boot-time dmesg output is in /var/run/dmesg.boot

This isn't as good:

Code:

da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 18432MB (37748736 512 byte sectors)
da0: quirks=0x140<RETRY_BUSY,STRICT_UNMAP>

It's a virtual disk device of some sort and would need more evaluation to determine if it's trustworthy (in this case it is, because I know what's going on there.)

This is bad:

Code:

ciss0: <HP Smart Array 6i> port 0x5000-0x50ff mem 0xf7ef0000-0xf7ef1fff,0xf7e80000-0xf7ebffff irq 24 at device 4.0 on pci2
ciss0: [ITHREAD]
da0 at ciss0 bus 0 scbus0 target 0 lun 0
da0: <COMPAQ RAID 0 OK> Fixed Direct Access SCSI-5 device
da0: 135.168MB/s transfers
da0: Command Queueing enabled
da0: 1907697MB (3906963632 512 byte sectors: 255H 32S/T 65535C)
da0: quirks=0x1<NO_SYNC_CACHE>

It means that the disk is a virtual disk on ciss0.

Both of the latter strongly imply that there's crap between FreeNAS and the hard drives that could potentially cause problems. The middle one isn't actually a problem because it's known to be the FreeNAS boot environment as provided by the hypervisor, which is backed up by RAID1 storage. That'll be fine. The ciss one, well, let's just say it's dodgy at best.

 

[jgreco]

Well, it seems that's the case with the P420i

Code:

da1 at ciss0 bus 32 scbus1 target 1 lun 0
da1: <ATA ST3000VN000-1H41 SC43> Fixed Direct Access SPC-4 SCSI device

Well, that's kind of tragic. The ciss driver has historically been flaky especially when it comes to problematic disks.

The good news is that as long as you're able to get SMART data off of them, some portion of the issues that we have with RAID cards might not be a problem for you.

The thing that would be interesting to examine, in this case, is whether or not this is truly HBA passthru or if the RAID card is still trying to do caching and other speed optimizations. I don't have time to discuss that today, sorry. I suppose it'd also be really interesting is if it supported hot-swap and error reporting correctly. You don't happen to have a small pile of variously-failing SATA disks hanging around, do you?

I'd love for it to work correctly. Having suffered through the CISS trainwreck years ago, I'm doubtful. Perhaps that's unfair.

 

[Ralms]

Well, that's kind of tragic. The ciss driver has historically been flaky especially when it comes to problematic disks.

The good news is that as long as you're able to get SMART data off of them, some portion of the issues that we have with RAID cards might not be a problem for you.

The thing that would be interesting to examine, in this case, is whether or not this is truly HBA passthru or if the RAID card is still trying to do caching and other speed optimizations. I don't have time to discuss that today, sorry. I suppose it'd also be really interesting is if it supported hot-swap and error reporting correctly. You don't happen to have a small pile of variously-failing SATA disks hanging around, do you? :)

I'd love for it to work correctly. Having suffered through the CISS trainwreck years ago, I'm doubtful. Perhaps that's unfair.

Well, I currently have a WD RED 3TB that is currently failing, started when I was copying stuff to FreeNAS, very high rellocated sector count (1920 currently lol).
I have a couple of very old dying drives also xD
I would be ok with examining these stuff with you, being an advanced Windows user I would for sure learn something in the process in FreeBSD.

I've done some more research and what I found is fairly alarming for HP users and FreeNAS (FreeBSD) in the near future.
The CCISS driver has been essentially deprecated by HP for a good while now, many years in fact.
Hp as moved on to a new driver called HPSA which from what the explain did a big change in terms of how it works:

Code:

The hpsa driver is a SCSI driver, while its predecessor was a block-layer driver. Hpsa combines traditional HP Smart Array
driver and controller technology with the large body of Linux community development on the SCSI storage layer. Hpsa
retains Smart Array reliability but enables these devices to benefit from the continuing development of the Linux SCSI
layer. This results in better error handling, enhanced device management, and quick integration with standard system
tools and utilities that rely on the SCSI subsystem. These benefits do not affect data or array configuration compatibility.
Hpsa maintains compatibility with pre-existing storage configurations and data from systems using the cciss driver.

A great whitepaper from HP can be found here:
https://h50146.www5.hpe.com/product...nstream/support/whitepaper/pdfs/c02677069.pdf

They say the driver is open-source, now I wonder where is it and if it can be adapted to work on FreeBSD xD
However this big dependency on the Linux SCSI storage layer makes me feel that we will have little luck.

I've been really enjoying HP hardware and they have some really affordable solutions on Ebay which you don't find on other brands, as such didn't really want to change to some old LSI stuff.