I Have 2 SSDs in a System but I see them Listed as both syspd devices and ada

[Brian Wilson]

Here is a list of all the disks. The ones of interest are mfisyspd20 and mfisyspd21, and then also ada0 and ada1. So I truncated the output:

Code:

[root@u90c10nas004] ~# geom disk list
Geom name: mfisyspd0
Providers:
1. Name: mfisyspd0
  Mediasize: 600127266816 (559G)
  Sectorsize: 512
  Mode: r0w0e0
  descr: (null)
  ident: (null)
  rotationrate: unknown
  fwsectors: 63
  fwheads: 255

Truncated........

Code:

Geom name: mfisyspd20
Providers:
1. Name: mfisyspd20
  Mediasize: 480103981056 (447G)
  Sectorsize: 512
  Mode: r0w0e0
  descr: (null)
  ident: (null)
  rotationrate: unknown
  fwsectors: 63
  fwheads: 255

Geom name: mfisyspd21
Providers:
1. Name: mfisyspd21
  Mediasize: 480103981056 (447G)
  Sectorsize: 512
  Mode: r0w0e0
  descr: (null)
  ident: (null)
  rotationrate: unknown
  fwsectors: 63
  fwheads: 255

Geom name: mfid0
Providers:
1. Name: mfid0
  Mediasize: 598999040000 (558G)
  Sectorsize: 512
  Mode: r1w1e2
  descr: (null)
  ident: (null)
  rotationrate: unknown
  fwsectors: 63
  fwheads: 255

Geom name: ada0
Providers:
1. Name: ada0
  Mediasize: 480103981056 (447G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  descr: INTEL SSDSC2BB480G6K
  lunid: 55cd2e404c72d584
  ident: PHWA613503N2480FGN
  rotationrate: 0
  fwsectors: 63
  fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
  Mediasize: 480103981056 (447G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  descr: INTEL SSDSC2BB480G6K
  lunid: 55cd2e404c72d02e
  ident: PHWA61340498480FGN
  rotationrate: 0
  fwsectors: 63
  fwheads: 16

These are exposed through the RAID Controller as JBODs. Wondering why i am seeing them listed twice.

Its making things confusing when going to setup the ZIL device for a ZFS pool since it shows that I Have 4 drives when i know for a fact that there are only 2 of these.

http://screencast.com/t/P2eCBvca

TIA for any help or suggestions on this.

Brian

 

[Mirfster]

These are exposed through the RAID Controller as JBODs

What Make/Model Raid Controller? Some (like most) may not truly be presenting the disk(s) in true JBOD fashion....

 

[DrKK]

Almost certainly that's the case. RAID controllers, versus proper HBA's, are very frequently hugely problematic. You start saying words like ZIL on top of it, and I definitely start sweating.

I hope Brian knows what he's doing---RAID controllers, "JBOD" or not, are big no-nos, most of the time.

 

[depasseg]

That is yet another issue with using "JBOD" on a RAID controller. You are seeing both the hardware devices and then the configured JBOD devices sitting on top. Good luck.
https://www.freebsd.org/cgi/man.cgi?query=mfi&sektion=4

/dev/mfi means you are using HW RAID (even if it's "only" as a JBOD).

 

[Brian Wilson]

Thanks for the replies,

I hope Brian knows what he's doing---RAID controllers, "JBOD" or not, are big no-nos, most of the time.

I admittedly am just getting deeper into this sort of thing so I may not entirely know what i am doing.




Here's some more details
The Storage Controller in Question:

Code:

C240-FCH2024V01X /chassis # show storageadapter detail
PCI Slot SLOT-HBA:
  Health: Good
  Controller Status: Optimal
  ROC Temperature: 58 degrees C
  Product Name: Cisco 12G SAS Modular Raid Controller
  Serial Number: AAAAAAAAAA
  Firmware Package Build: 24.9.1-0011
  Product ID: LSI Logic
  Battery Status: BBU Not Supported
  NVRAM Size: 32 KB
  Memory Size: 0 MB
  Flash Memory Size: 32 MB
  Cache Memory Size: 0 MB
  Boot Drive: 0
  Boot Drive is PD: false
  TTY Log Status: Not Downloaded
  Controller is Secured: 0

I have JBOD Mode Enabled in the Cards Setup:

Code:

C240-FCH2024V01X /chassis/storageadapter # show settings
PCI Slot SLOT-HBA:
  Info Valid: Yes
  Enable JBOD Mode: true
  Info Invalid Cause:
  Predictive Fail Poll Interval: 300 sec
  Rebuild Rate: 30 %
  Patrol Read Rate: 30 %
  Consistency Check Rate: 30 %
  Reconstruction Rate: 30 %
  Cache Flush Interval: 4 sec
  Max Drives to Spin Up at Once: 4
  Delay Among Spinup Groups: 6 sec
  Physical Drive Coercion Mode: 1 GB
  Cluster Mode: false
  Battery Warning: false
  ECC Bucket Leak Rate: 1440 min
  Expose Enclosure Devices: true
  Maintain PD Fail History: false
  Enable Copyback on SMART: true
  Enable Copyback to SSD on SMART Error: true
  Native Command Queuing: enabled
  Enable Spin Down of Unconfigured Drives: true
  Enable SSD Patrol Read: false
  AutoEnhancedImport: true
C240-FCH2024V01X /chassis/storageadapter #

I have the Disks marked as JBODs with the exception of 2 (disks 3 and 4) which i am using for the Install and Boot Drive setup in a Raid1 HW Array.

Code:

C240-FCH2024V01X /chassis/storageadapter # show physical-drive
Physical Drive Number Controller Health  Status  Boot Drive Manufacturer  Model  Predictive Failure Count Drive Firmware Coerced Size  Type  Block Size Link Speed Locator LED FDE Capable FDE Enabled FDE Secured FDE Locked
--------------------- ---------- -------------- ---------------------- ---------- -------------- -------------- ------------------------ -------------- -------------- ----- ---------- ---------- ---------- ---------- ---------- ---------- ----------
1  SLOT-HBA  Good  JBOD  false  ATA  INTEL SSDSC... 0  CS01  456809 MB  SSD  512  6.0 Gb/s  false  0  0  0  0
2  SLOT-HBA  Good  JBOD  false  ATA  INTEL SSDSC... 0  CS01  456809 MB  SSD  512  6.0 Gb/s  false  0  0  0  0
3  SLOT-HBA  Good  Online  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
4  SLOT-HBA  Good  Online  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
5  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
6  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
7  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
8  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
9  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
10  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
11  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
12  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
13  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
14  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
15  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
16  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
17  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
18  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
19  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
20  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
21  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
22  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
23  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0
24  SLOT-HBA  Good  JBOD  false  SEAGATE  ST600MP0005  0  N002  571250 MB  HDD  512  12.0 Gb/s  false  0  0  0  0

thanks again
Brian

 

[Mirfster]

I have the Disks marked as JBODs with the exception of 2 (disks 3 and 4) which i am using for the Install and Boot Drive setup in a Raid1 HW Array.

While I have not looked at that card's specs, you don't want to do a Hardware Raid for the Boot Drives either. Just select them both during the install and FreeNAS will mirror them for you.

Maybe it is an issue where you are mixing some drives as JBOD and some not on the controller? Personally, I prefer to use an actual HBA and forget the Hardware Raid altogether. Even if it does do JBOD, you are not going to get any benefit from the BBU or Cache on that card (that I am aware of) while it is in that mode.

I know it is difficult to understand not needing to use Hardware to Raid the Boot Drives, but trust me you want FreeNAS to do that instead and steer clear of Hardware Raid cards altogether...

 

[Brian Wilson]

Thanks for suggestion i was already thinking the same thing so i was trying it out before i saw your reply.

Removed the Hardware Virtual Disk so that all the disks on the Controller are now in JBOD and booted up the install CD to see what i saw.

First Screen Cap still sees the ada disks, on the right there is the verification of config that changed for 3/4 to jbod mode with no HW Raid setup. Second Screen Cap shows those same disks exposed as mfisyspd22 and 23 as well

Oh and for good Measure I booted up an Installer for 9.10 Since everything thus far has been on Latest Stable 9.10.1

Same thing on it as well...



Yeah i am getting used to the idea of SW Raid over HW Raid for sure and see now that these controllers don't give any benefit over a true passthrough HBA card, but we do have a good handful of these systems already so was trying to find the best way to provision them with FreeNAS.


I Am wondering if i just use the syspd22,23 for the SLOG device mirror and ignore the ada devices if there will be any downside just can't understand why these 2 disks are doing this but the spindle disks are only exposed as syspd, a weird anomaly for sure.

 

[depasseg]

ZFS on HW RAID (including JBOD mode) is a bad idea. Look into flashing that controller to IT mode. It looks like it's an LSI 9300-8e (based on the Cisco 12G SAS description), but you should confirm. http://www.avagotech.com/products/server-storage/host-bus-adapters/sas-9300-8e#downloads

 

[Brian Wilson]

Thanks to all who offered advice I do want to leave the answer as to why the disks where exposed twice for anyone else who comes along with the same issue.

The CiscoUCS240 has a built in PCH Controller that was also exposing the 2 SSDs. By Disabling this you are left with only the disks being exposed through the SLOT-HBA Connected Controller, the Cisco 12G SAS Modular Raid Controller, in this case.
Howto do this : http://www.cisco.com/c/en/us/td/doc...cs/c/hw/C240M4/install/C240M4/raid.html#92191


Also to note on that page there are other options for the 240 Model that may suit FreeNAS Better as they are not RAID Cards but rather Passthrough HBA Models:
http://www.cisco.com/c/en/us/td/doc...cs/c/hw/C240M4/install/C240M4/raid.html#98288


Thanks All
Brian

 

[Mirfster]

Glad you got it sorted. :)

 

[Stux]

Thanks to all who offered advice I do want to leave the answer as to why the disks where exposed twice for anyone else who comes along with the same issue.

The CiscoUCS240 has a built in PCH Controller that was also exposing the 2 SSDs. By Disabling this you are left with only the disks being exposed through the SLOT-HBA Connected Controller, the Cisco 12G SAS Modular Raid Controller, in this case.
Howto do this : http://www.cisco.com/c/en/us/td/doc...cs/c/hw/C240M4/install/C240M4/raid.html#92191


Also to note on that page there are other options for the 240 Model that may suit FreeNAS Better as they are not RAID Cards but rather Passthrough HBA Models:
http://www.cisco.com/c/en/us/td/doc...cs/c/hw/C240M4/install/C240M4/raid.html#98288


Thanks All
Brian

You should check SMART works on the drives hosted by that card