Dell R510 - One drive bay ignored by FreeNAS 11.1-U4

[doubleyousee]

I have a new (to me) Dell R510 with 8 3.5" drive bays. and FreeNAS 11.1-U4

(I should mention though I am a full-time IT professional, I work with MS Servers & workstations primarily. This is my first time using FreeNAS, and it's a hobby project in my basement, so I am wading out into the unknown...)

Currently I have bays 0-3 populated with 4 10TB drives, all setup in a RAIDZ, everything is working fine now for about a month.

Last week I tried to populate bays 4-7 with some 4TB drives I have.
3 of the 4 bays are detecting a drive but bay 5 is not. I can see on the console output in FreeNAS (on a connected monitor) that each drive is detected and some details are reported on-screen as I hotswap them in or out. Except bay 5! I have tried all 4 drives in bay 5 and FreeNAS will not acknowledge any of them. (Not on the web console either, I might add)

The R510 has a H200 RAID controller and if I head into the BIOS or the RAID card configuration during bootup the drive in Bay 5 detects just fine and in all accounts appears to be working normally. Once I am booted into FreeNAS however it doesn't appear to be acknowledged by the OS.

I have a feeling there is something I can be doing at the shell in FreeNAS to investigate this further and narrow down why FreeNAS appears to be ignoring bay 5 but I haven't the first idea where to start.

Any pointers would be appreciated.
If I am missing relevant details please let me know!

 

[Elliot Dierksen]

If it is still in the build stage, can you swap the drives around? When you do that, does the problem follow the drive or follow the bay? Also, it might now show up in the GUI but do you see it in the system boot logs? You can check that using dmesg -a | more from the CLI. When you are doing things like that, it would probably be good to take a page from @Chris Moore's book and label the drives with all or the trailing part of the serial number so you can be sure which one is missing. The serial number should be visible in the boot messages as well as the storage section of the GUI.

 

[Elliot Dierksen]

The R510 has a H200 RAID controlle

Can you wipe the disk in question from the RAID controller BIOS? Perhaps that drive has some configuration from the RAID controller written to it that is causing the controller to not make the drive visible to FreeNAS. ZFS is at the core of FreeNAS, so your best bet is to let ZFS handle all the aspects of RAID configuration. If you have a system with a RAID controller (as I do), you have to configure the drives as JBOD so they pass through to the OS.

 

[melloa]

you have to configure the drives as JBOD so they pass through to the OS.

I was about to ask if his controller was in IT mode :)

 

[HoneyBadger]

I was about to ask if his controller was in IT mode :)

If it's the "integrated" or "mini" H200 that lives in the dedicated storage slot of the R510, it likely can't be reflashed to LSI firmware. But yes, a full IT firmware should be used in this case, as the IR firmware only adds the potential for weirdness like we're seeing now.

A PERC H200 will by default pass through unconfigured drives, but if there is a "leftover" configuration somewhere in the BIOS or RAID settings of the card it may be doing something silly like trying to build a virtual disk.

@doubleyousee - when you installed your first four disks, did you configure anything in the PERC's BIOS/RAID settings, or did you leave them unconfigured? Hopefully the latter.

 

[doubleyousee]

Thank you all for replying! I will try to clarify all the questions below:

If it is still in the build stage, can you swap the drives around? When you do that, does the problem follow the drive or follow the bay? Also, it might now show up in the GUI but do you see it in the system boot logs? You can check that using dmesg -a | more from the CLI. When you are doing things like that, it would probably be good to take a page from @Chris Moore's book and label the drives with all or the trailing part of the serial number so you can be sure which one is missing. The serial number should be visible in the boot messages as well as the storage section of the GUI.

I have swapped the drives around (between bays 4 and 7) and the problem is always with bay 5. All 4 drives are detected fine if I put them in one of the other bays.
I can see the serial numbers in in the BIOS but not in the GUI or anywhere in the output of dmesg -a | more

Can you wipe the disk in question from the RAID controller BIOS? Perhaps that drive has some configuration from the RAID controller written to it that is causing the controller to not make the drive visible to FreeNAS. ZFS is at the core of FreeNAS, so your best bet is to let ZFS handle all the aspects of RAID configuration. If you have a system with a RAID controller (as I do), you have to configure the drives as JBOD so they pass through to the OS.

Ya, I did extensive reading about this before I went down the FreeNAS rabbit hole. I purchased a used server, I got to be picky about the raid controller and I read all about letting FreeNAS do all the RAID work. I was initially trying to get the card into IT mode when I started this project but I ran across a post that claimed that when left "unconfigured" the controller will pass the drives directly to the OS. So I tried that with my first bank of drives (bays 0-3) and it has worked flawlessly.

Three of the four bays in the second set work as well but for some reason bay 5 is not talking to freeNAS. I would happily "wipe" the disk but, as the problem follows the the bay, I thought I should "wipe" the bay. That is, look for any configuration leftover in the controller/BIOS, but I can't find anything! In the BIOS all the drives are reported and there is no configuration unique to bay 5 that I can find to "remove".

If it's the "integrated" or "mini" H200 that lives in the dedicated storage slot of the R510, it likely can't be reflashed to LSI firmware. But yes, a full IT firmware should be used in this case, as the IR firmware only adds the potential for weirdness like we're seeing now.

A PERC H200 will by default pass through unconfigured drives, but if there is a "leftover" configuration somewhere in the BIOS or RAID settings of the card it may be doing something silly like trying to build a virtual disk.

@doubleyousee - when you installed your first four disks, did you configure anything in the PERC's BIOS/RAID settings, or did you leave them unconfigured? Hopefully the latter.

According to the invoice when I ordered the server it's an H200 PERC Controller 6 Gbps (RAID 0,1,10) but I can't find any configuration "leftover" to remove.

I didn't configure anything in the BIOS/RAID controller for the first array. :/

As for the question of IT mode, I don't recall the exact part of the process that hung me up but I am sure I at least updated the firmware on the H200 to the latest available. I don't think I was able to get it into IT mode because of some impediment at the time... needing a longer cable, having to move the card to flash it... I can't remember, I read SOOO much on the topic at the time and tried to cobble together what was relevant to my particular hardware situation. Then I came across that post that said to leave the drives unconfigured and the controller would pass them through (a la JBOD) and so I tried that and it worked!

If the prevailing advice is indeed to flash the controller to IT mode, should I be worried about my existing 30TB RAIDZ1? I would very much like to keep that intact and if there is any potential for loss I would almost prefer to just stick a spacer in bay 5 and treat it like Chernobyl ;)

 

[Elliot Dierksen]

I have swapped the drives around (between bays 4 and 7) and the problem is always with bay 5. All 4 drives are detected fine if I put them in one of the other bays.

I am at a loss to explain how, but it does sound like the problem follows the bay. This is reaching for straws, but is there perhaps something in the RAID controller that has this bay marked as a hot spare or something like that? The only other thing I can think to try is using a different drive controller connected to the backplane and see if that works. My gut leans towards it being something in the RAID controller, but I can't define it in any tangible kind of way.

 

[Chris Moore]

If I am missing relevant details please let me know!

There are a few different hardware configurations for that model server. What number of front drive bays do you have?

For the next question, you may need to go to the basement and rip the lid off so you can look inside.
Does your H200 controller look like this:


Or like this:

I should mention though I am a full-time IT professional, I work with MS Servers & workstations primarily.

I am sure that you were reading until your eyes glazed over but I have some additional reading assignments for your free time:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[doubleyousee]

There are a few different hardware configurations for that model server. What number of front drive bays do you have?

For the next question, you may need to go to the basement and rip the lid off so you can look inside.
Does your H200 controller look like this:
View attachment 24847

Or like this:

I am sure that you were reading until your eyes glazed over but I have some additional reading assignments for your free time:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

OK, so ya, I did read a lot before I jumped into this project but now that I have read through both your links I clearly didn't read from the "beginning of the book"
Upon reflection, I'm not sure I have ever really jumped into a new topic in my IT life by starting at the "beginning", I suppose I probably should have but where is the fun in that!?

Alright, onto my server config. It is an 8 bay configuration, all in the front, all 3.5" bays.
The controller is the second of the two you posted, with the single silver heat sink.

 

[Chris Moore]

The controller is the second of the two you posted, with the single silver heat sink.

Are you setup to where you can SSH in on the server?
At the command prompt, issue the command: sas2flash -listall
That should give you an output like this:

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

  Adapter Selected is a LSI SAS: SAS2008(B2)

Num  Ctlr  FW Ver  NVDATA  x86-BIOS  PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)  16.00.00.00  10.00.00.06  No Image  00:01:00:00

  Finished Processing Commands Successfully.
  Exiting SAS2Flash.

Or, an error, but it will tell us something. Your adapter type may be different.
If you only have one SAS controller, the 'Num' column will have a 0 in it, but I have seen systems with as many as 4 SAS controllers, so you need to know which one you are querying for the next step. The command should be sas2flash -c 0 -list and that should give you something like this:

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		Adapter Selected is a LSI SAS: SAS2308_2(D1)

		Controller Number			  : 0
		Controller					 : SAS2308_2(D1)
		PCI Address					: 00:03:00:00
		SAS Address					: 500605b-0-09ef-7220
		NVDATA Version (Default)	   : 14.01.00.06
		NVDATA Version (Persistent)	: 14.01.00.06
		Firmware Product ID			: 0x2214 (IT)
		Firmware Version			   : 20.00.07.00
		NVDATA Vendor				  : LSI
		NVDATA Product ID			  : SAS9207-8i
		BIOS Version				   : N/A
		UEFI BSD Version			   : N/A
		FCODE Version				  : N/A
		Board Name					 : SAS9207-8i
		Board Assembly				 : H3-25412-00J
		Board Tracer Number			: SV45308383

		Finished Processing Commands Successfully.
		Exiting SAS2Flash.

If you will post up your results on these, it will give us some more insight into what might be happening.

 

[Chris Moore]

OK, so ya, I did read a lot before I jumped into this project but now that I have read through both your links I clearly didn't read from the "beginning of the book"

I understand. I jumped into the middle of it myself and made mistakes along the way. We learn from our mistakes and it makes us better.

 

[IQless]

30TB RAIDZ1

Do you plan on keeping this as a Raidz1? or are you going to migrate to RaidZ2 or RaidZ3?
With 10TB drives it might get tricky when drives start to fail :\

 

[doubleyousee]

I need to preface this reply with 2 things:

1. THANK YOU! I don't have it fixed yet but the responsiveness of this forum/community is enough to lift the cloud of frustration that has been following me around for days!
2. I am leaving for a week of holiday, camping in the woods, away from all of these spinning drives.
   So I will not be able to respond until this time next week. However I am looking forward to some homework when I get back :)

Are you setup to where you can SSH in on the server?
At the command prompt, issue the command: sas2flash -listall
That should give you an output like this:

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

  Adapter Selected is a LSI SAS: SAS2008(B2)

Num  Ctlr  FW Ver  NVDATA  x86-BIOS  PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)  16.00.00.00  10.00.00.06  No Image  00:01:00:00

  Finished Processing Commands Successfully.
  Exiting SAS2Flash.

Or, an error, but it will tell us something. Your adapter type may be different.
If you only have one SAS controller, the 'Num' column will have a 0 in it, but I have seen systems with as many as 4 SAS controllers, so you need to know which one you are querying for the next step. The command should be sas2flash -c 0 -list and that should give you something like this:

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		Adapter Selected is a LSI SAS: SAS2308_2(D1)

		Controller Number			  : 0
		Controller					 : SAS2308_2(D1)
		PCI Address					: 00:03:00:00
		SAS Address					: 500605b-0-09ef-7220
		NVDATA Version (Default)	   : 14.01.00.06
		NVDATA Version (Persistent)	: 14.01.00.06
		Firmware Product ID			: 0x2214 (IT)
		Firmware Version			   : 20.00.07.00
		NVDATA Vendor				  : LSI
		NVDATA Product ID			  : SAS9207-8i
		BIOS Version				   : N/A
		UEFI BSD Version			   : N/A
		FCODE Version				  : N/A
		Board Name					 : SAS9207-8i
		Board Assembly				 : H3-25412-00J
		Board Tracer Number			: SV45308383

		Finished Processing Commands Successfully.
		Exiting SAS2Flash.

If you will post up your results on these, it will give us some more insight into what might be happening.

Here is the output of the sas2flash -listall

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		No LSI SAS adapters found! Limited Command Set Available!
		ERROR: Command Not allowed without an adapter!
		ERROR: Couldn't Create Command -listall
		Exiting Program.

Do you plan on keeping this as a Raidz1? or are you going to migrate to RaidZ2 or RaidZ3?
With 10TB drives it might get tricky when drives start to fail :\

Oy! ya :(

Ignorance is bliss (until it isn't) and after reading the links from Chris, I am seriously considering adding some more 10TB drives and trying to migrate to RAIDZ2 at least.
The drives are not full by any stretch and so I should do it sooner than later, but the cost is prohibitive for a "hobby" project...

 

[Chris Moore]

I am leaving for a week of holiday, camping in the woods, away from all of these spinning drives.
So I will not be able to respond until this time next week. However I am looking forward to some homework when I get back

Well, I hope you have a great vacation. It is always nice to get away from the grind.
The controller didn't show because it is still using the Dell firmware. That means we will need to use a different program called megarec.exe which only runs from DOS (MSDOS or FreeDOS) and we can look into it more once you are back.
Just post here again when you are ready to tinker with this. I will do some investigation into this. I would like to try to verify that it isn't some configuration issue. Since you said the drive shows up in the controller BIOS, I am making the leap that the connector and cable are good and it is just some funny behavior from the controller.

 

[teppers]

Well, I hope you have a great vacation. It is always nice to get away from the grind.
The controller didn't show because it is still using the Dell firmware. That means we will need to use a different program called megarec.exe which only runs from DOS (MSDOS or FreeDOS) and we can look into it more once you are back.
Just post here again when you are ready to tinker with this. I will do some investigation into this. I would like to try to verify that it isn't some configuration issue. Since you said the drive shows up in the controller BIOS, I am making the leap that the connector and cable are good and it is just some funny behavior from the controller.

I wanted to chime in and add a data point. I'm having the exact same issue with FreeNAS 11.1 U5 on a Dell PowerEdge R715 with an H200 controller. The installer doesn't even see all of the disks. It sees 5 of the 6 bays, it just doesn't see bay 5. However, I just ran the 11.0 U3 installer on it and it sees all 6 disks so it would seem to be something to do with 11.1.

sas2flash -listall

Code:

LSI Corporation SAS2 Flash Utility											 
Version 16.00.00.00 (2013.03.01)												
Copyright (c) 2008-2013 LSI Corporation. All rights reserved					
																			   
	   Adapter Selected is a LSI SAS: SAS2008(B2)							 
																			   
Num   Ctlr			FW Ver		NVDATA		x86-BIOS		 PCI Addr	 
----------------------------------------------------------------------------	
																			   
0  SAS2008(B2)	 07.15.08.00	07.00.00.19	07.11.10.00	 00:05:00:00	
																			   
	   Finished Processing Commands Successfully.							 
	   Exiting SAS2Flash. 

 

[Chris Moore]

You should file a bug report.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[teppers]

You should file a bug report.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Submitted - https://redmine.ixsystems.com/issues/40396

 

[jgreco]

According to the invoice when I ordered the server it's an H200 PERC Controller 6 Gbps (RAID 0,1,10) but I can't find any configuration "leftover" to remove.

I didn't configure anything in the BIOS/RAID controller for the first array. :/

As for the question of IT mode, I don't recall the exact part of the process that hung me up but I am sure I at least updated the firmware on the H200 to the latest available. I don't think I was able to get it into IT mode because of some impediment at the time... needing a longer cable, having to move the card to flash it... I can't remember, I read SOOO much on the topic at the time and tried to cobble together what was relevant to my particular hardware situation. Then I came across that post that said to leave the drives unconfigured and the controller would pass them through (a la JBOD) and so I tried that and it worked!

If the prevailing advice is indeed to flash the controller to IT mode, should I be worried about my existing 30TB RAIDZ1? I would very much like to keep that intact and if there is any potential for loss I would almost prefer to just stick a spacer in bay 5 and treat it like Chernobyl ;)

It is known to do odd things if you do not crossflash it to LSI IT mode. Note that I said *LSI* IT mode. Not Dell IT mode. The process is annoying and a bit finicky - I just finished doing a bunch a month ago - but definitely worth it.

As long as the drives are identifying in the OS (when you do a "camcontrol devlist") as the correct model numbers and not as virtual disks, your data should be safe, go ahead and flash to IT. I do personally prefer to unplug the drives while flashing and testing, but I'm paranoid. The main risk is that you brick the card and cannot figure out how to recover it, which leaves your data inaccessible. These things are pretty hard to permanently toast (trust me, I've tried) and someone can probably help if you manage that tough feat.

 

[pro lamer]

@doubleyousee
Just a wild guess of one who doesn't know that it's impossible ;) (but it may be impossible anyway 'cos I am a noob :P and to be serious: I don't know that chassis/those bays at all)
- is it possible to shuffle with cabling to check if it is a cable problem?

 

[jgreco]

@doubleyousee
Just a wild guess of one who doesn't know that it's impossible ;) (but it may be impossible anyway 'cos I am a noob :p and to be serious: I don't know that chassis/those bays at all)
- is it possible to shuffle with cabling to check if it is a cable problem?

The 8 bay chassis does not use a SAS expander so that's a reasonable thing to try. The 12 bay chassis does, so it's more complicated there.