Zpool state unknown after reboot. I'm stumped!

[michaelrcolton]

Hi everyone,

First let me say I really appreciate any help you can offer and I’ll provide any additional information you need. I apologize if this is posted in the wrong place or I’ve otherwise made any forum faux pax.

I've got a home file server that's been running FreeNAS for at least a year now without any serious problems. I decided to add some additional hard drives, which required the addition of a SATA PCI-Express card. Now, every zpool I try to create with these drives on this card appears to work, but is in an “unknown state” when I reboot.

The system information is as follows:
Running on FreeNAS-11-STABLE (I update whenever it tells me there is an update)
Motherboard is an Asus P5N-D
CPU is an Intel Core 2 Quad Q8400
It has 10GB of DDR3 (2x4GB and 2x1GB) non ECC, don’t be mad.
Up until now it had 4x2TB SATA drives connected to the motherboard SATA ports
2 are WD Enterprise Drives (Yellows?)
2 are WD Reds
The drives are set up as a pair of mirrors, one yellow and one red in each mirror and those two pairs are in a single zpool that acts as the main file storage for the house.
There is a single 160GB PATA drive that is used for storage on a security camera and is non-critical but works fine.
PSU is about 450-500 Watts, I forget. Seems to run fine even with 9 drives connected.
Boot drives are a mirrored pair of 16GB flash drives.

Other than the occasional hiccup, performance has been totally satisfactory.

I’m trying to add two more 2TB drives (HGSTs with TLER turned on). I also have an additional 2x2TB drives (WD Red Pros) that I am borrowing in case I need temporary storage while I am shuffling things around. I only have those for a few more days. They should also allow me to rule out the HGSTs being the problem.

So I added a Syba SY-PEX40008 (based on the SIL3124, which is supported by FreeBSD) I thought I had done my homework and found a card that was specifically known to work well with FreeNAS, subsequent searching makes me wonder if I actually got that right.

I have been making new, temporary zpools with these new drives (usually as mirrored or striped pairs, but I’ve also tried a RAIDZ2) to make sure everything is working before I make any changes to may main pool.

No matter what I’ve tried so far, they always up as “state unknown” after a reboot.

I have tried exporting and reimporting the test pools, they always fail, with a message to check zpool status, which reports something like an I/O error (I’ll need to do it again to get the actual message)

I have tried wiping the drives between some attempts. I also tried the full zero wipe once (using the command line, so I could wipe two drives at once (which, btw, crashed the web server, but I could still be managed by the console on the actual machine))

I have read that fake raid cards will overwrite the last sector or more of the drive, so I tried partitioning myself, with additional space at the end, and manually creating the zpool from the command line and then importing that into the FreeNAS GUI, it works up to the GUI import step, that fails.

I found what I believe to be the current firmware for the Syba card. If I am reading it right, there is a dumb firmware (just connects the sata ports to the computer) and a RAID firmware. Most of the tests were done with the raid firmware before I installed the update (that was version 6.3.18 of the raid firmware). I tried with the drives unconfigured in the BIOS (FreeNAS could see them just fine, but the reboot issues persisted) and in a “Concatenated” mode, which, I think, is their word for JBOD. This time the problem was a little different (I can’t remember how) but it still didn’t work.

Then I tried updating the card (thankfully it is of the vintage where this is possible, apparently newer cards can’t be updated) to the dumb firmware (6.6.00, the newest dumb firmware I have found, the raid firmware goes up to to 6.6.03). It no longer shows anything during boot up (the raid firmware has a BIOS setup screen) and FreeNAS still sees the drives. But the reboot problem persists.

I tried only using two of the ports on the Syba card, but the issues persists.

I am wary to connect the new drives to the motherboard ports because I don’t want to do anything to the main pool. If it’s safe to take that pool offline and hook in the new drives, I could at least eliminate the Syba card as an issue, or prove that it is one.

I have found other people that seems to be using the same SATA card (or at least other cards based on the same SIL3124) who aren’t complaining of issues.

I have found at least one person that had almost the exact same problem, but he wasn’t able to figure it out, he ended up buying a new motherboard. I would prefer not to do that unless I have to.

Once this starts working, I’m either going to RAIDZ2 or 3xMirror the 6 drives. I was leaning toward RAIDZ2, but this article ( http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ ) makes me hesitate.

I have the extra 2TB drives as a place to store the data during the change (if it’s the mirror then I don’t need to do that, but if I switch to RAIDZ2, then I’ll have to transfer the data, then delete and reconfigure the zpool) I only have those extra 2tb drives for a little longer so I am trying to get this done. Also it would be nice to move on with my life. I have been fighting this for two weeks!

 

[SweetAndLow]

You can't use that sata card thing. They do not play nice with zfs. Get a hba from the supported FreeNAS list.

 

[Jailer]

So I added a Syba SY-PEX40008 (based on the SIL3124, which is supported by FreeBSD) I thought I had done my homework and found a card that was specifically known to work well with FreeNAS, subsequent searching makes me wonder if I actually got that right.

I think you've pretty much answered your own question on whether or not the SATA card you have will work. @joeschmuck has used a different 4 port card with success if you still want to try something relatively inexpensive to add some more drives.

https://www.amazon.com/IO-Crest-SI-PEX40062-Controller-Green/dp/B00AZ9T41M

You might want to consider another power supply as well, 9 drives is an awful lot for a 500 watt power supply.

 

[Ericloewe]

First let me say I really appreciate any help you can offer and I’ll provide any additional information you need. I apologize if this is posted in the wrong place or I’ve otherwise made any forum faux pax.

Don't worry, we realize that the experience can be somewhat overwhelming. We just appreciate good faith efforts to make things easier for us.

As for your problem, I'm afraid that the best option might be to sell the card you have now and buy an LSI HBA.

 

[joeschmuck]

@Jailer Thanks for bringing me into this thread.

I too did a lot of research before I purchased a SATA add-on card and ensured it met a few conserns of mine.
1) Chipset would be compatible for at least FreeNAS 11 (it was the next future version at the time) and I hoped it would retain compatability for a few more versions of course.
2) It was capable of supporting full 6Gbit data transfers for all four SATA ports at the same time. I think it's close to perfect. I needed to use a two PCI-E lane card, the single lane cards would definately not do it even if they would be fine for normal home use of FreeNAS, it was only a few bucks more for the faster throughput card.
3) It was by a reputable company that gets good reviews.

Like you I took a gamble when I purchased the SATA card, I had no idea if it would actually work or not until someone had tested it specifically in the application I was using, FreeNAS. Call me lucky that I had absolutely no issues what so ever. I have recommended this card to many people and have never heard a single complaint about it's operation. However, I cannot say that it would fix your problem, you could have something else going on so this does warrant further troubleshooting to pin down the issue and also keep you from spending money needlessly if possible.

I am wary to connect the new drives to the motherboard ports because I don’t want to do anything to the main pool. If it’s safe to take that pool offline and hook in the new drives, I could at least eliminate the Syba card as an issue, or prove that it is one.

Yes, it is safe to remove your current pool from your motherboard SATA ports and then connect up your new drives and test out your system. If you want to take that extra step of safety, remove your current boot devices and install a single USB Flash Drive, install FreeNAS 11 to it and call your computer a Test Rig until you have figured out what is wrong.

Unfortunately you went and added your hard drives (or some of them) to a Software RAID. This in itself could cause issues as well. You should ensure that the controller cards are not looking for a RAID setup and I'd erase the first and last 10,000 LBA on the hard drive. I pulled 10,000 out of my rear, you just need to ensure they are not marked for RAID. Zeroing out an entire hard drive takes way too long.

Ensure your SATA card RAID is disabled.

You should post the output of dmesg which will give us some details about the system and some error info, maybe we will get lucky and see something obvious.

Are you able to check SMART data on drives connected to this SATA card? If yes, please post the output of smartctl -a /dev/xxx (where xxx = drive).

Maybe the others here can think of any other checks to make so we can see what is exactly going on.

I did a very little bit of research on this card and here is what the manufacturers site says:

[xtable=skin1]
{tbody}
{tr}
{td}Q: How do I install this controller in non-raid mode.{/td}
{/tr}
{tr}
{td}A: You can't install this card in non-RAID mode. The BIOS flash currently on the card is slated for RAID operations. Any attempt to install non-RAID drivers for the card will result in failure or the card and/or its operations. You will need to install the RAID drivers for the card, and then use the card in its default JBOD mode.{/td}
{/tr}
{/tbody}
[/xtable]
So I'm not sure how accurate this statement is but if you cannot disable the card looking for a RAID configured hard drive then I don't see how this card will properly work. And this is an old card, PCI-E 1.0 ???

Speaking of PCI-E 1.0, while your MB should be able to accept it, you should ensure your PCI-E buss speed for the slot you are using is actually running at the correct speed. If your MB isn't recognizing the card properly then this could be an issue. I'm not saying it is the issue, just something to look at.

Good luck and Happy Troubleshooting!