Some disks detected as Multipath, some not.

[Grendell]

System info:
Single Opteron on a Supermicro board, 12GB ECC Registered RAM, 6 750GB WD Green drives, 1 640GB WD Black, Asus DVD-RW, 8GB USB drive on motherboard header, Highpoint (I know, I bought it before I had ever heard of FreeNAS...) Rocket Raid 4320.
The green drives are attached to the 4320, the Black and the DVD-RW are attached to SATA ports on the system board.

For whatever reason when I try to set the RR 4320 to non-RAID mode it does not appear to work and I cannot see the green drives in FreeNAS. When I reboot and check the RR 4320 BIOS it shows the card set to RAID mode again (reboot or cold boot, same result). I created 6 JBODs with the green drives and then I could see them in FreeNAS. This is where the weirdness begins: half of the drives show up as multipath devices and half show up as normal drives in the FreeNAS GUI.

If I open a shell...

Code:

[root@freenas ~]# camcontrol devlist
<HPT DISK 0-0 4.00> at scbus0 target 0 lun 0 (da0,pass0)
<HPT DISK 0-1 4.00> at scbus0 target 1 lun 0 (da1,pass1)
<HPT DISK 0-2 4.00> at scbus0 target 2 lun 0 (da2,pass2)
<HPT DISK 0-3 4.00> at scbus0 target 3 lun 0 (da3,pass3)
<HPT DISK 0-4 4.00> at scbus0 target 4 lun 0 (da4,pass4)
<HPT DISK 0-5 4.00> at scbus0 target 5 lun 0 (da5,pass5)
<HPT RCM DEVICE 4.00> at scbus0 target 32 lun 0 (pass6)
<WDC WD6401AALS-00J7B1 05.00K05> at scbus1 target 0 lun 0 (ada0,pass7)
<ASUS DRW-24B1ST 1.04> at scbus6 target 0 lun 0 (cd0,pass8)
<PNY USB 2.0 FD 1100> at scbus8 target 0 lun 0 (pass9,da6)

In the FreeNAS GUI I see, under View Multipaths:

Code:

Name                  Status                      LUN ID
 
multipath/disk3      DEGRADED          ...
 
  da5                    ACTIVE                  00193c0000000000
 
multipath/disk1      DEGRADED          ...
 
  da1                    ACTIVE                  00193c0000000000
 
multipath/disk2      DEGRADED          ...
 
  da2                    ACTIVE                00193c0000000000

and, under View Disks:

Code:

Name  Serial                Disk Size
 
ada0    WD-WMAT...    640.1 GB
 
da0                                750.0 GB
 
da3                                750.0 GB
 
da4                                750.0 GB

I know almost nothing about multipath, but after some searching I found a thread on the forums here where someone had un-natural multipath devices showing up in FreeNAS. The problem appeared to be solved by using gmultipath destroy <disk#> then gpart recover <driveID>. This is a test system so I thought I'd give it a shot. gmultipath destroy disk3 removed the listing in View Multipaths. gpart recover da5 (or anything else reasonable) accomplished nothing. If anyone wants the exact error message I'd be happy to do it again. I could not find any way to make the drive show up in FreeNAS again until I rebooted the machine and it appeared as, once again: multipath/disk3 da5. :bangheadonwall:

If it makes any difference I was able to create a volume with two of the multipath drives.

I have no idea why this is happening, including why my (not) very expensive RAID card will not switch to non-RAID mode and stay that way. (I also tried mixed mode which also appeared to accomplish nothing at all.)

I did not create all of the JBODs at the same time. I created two initially when I first booted the machine, then created the other four later. They were created in the order they appear in the RR4320 BIOS, though. That is to say that I started at the top of the list and worked my way down. Why it should choose da1, da2, and da5 to call multipath devices I cannot imagine.

Any help is greatly appreciated.

 

[Grendell]

Oh, and just for you, cyberjock, before I put any data on this system I'll be buying a Blueray drive to do backups with. ;)

 

[cyberjock]

I have a RocketRAID 4320. Do your data a favor and give up on it right now. http://forums.freenas.org/index.php?threads/highpoint-controller-info.8217/

You will have zero SMART monitoring, zero SMART testing, and disk failure will not work properly. So yeah.. might as well just set yourself on fire if you want to use it with FreeNAS.

Not to mention the fact that you are having to create the JBODs means you will NOT be able to take the drives to another controller later. You'll be forced to use the 4320 or equivalent forever.

As for why multipath is behaving this way, my guess is something to do with the controller. Aside from that I can't tell you much more, and it's really not worth anyone's time to investigate since there are so many other ways this is a "fail" that it's not worth the effort.

 

[cyberjock]

Doh, you posted just before I did.

Blu-Ray doesn't make a particularly good backup, especially when dealing with TBs of data. At 50GB per disk, you'd need 10 disks just to backup 500GB. Not to mention the fact that for that cost in Blu-Ray media + the drive you could have bought a single 2TB or 3TB drive, setup ZFS replication and had time-based backups via ZFS snapshots.

 

[Grendell]

I have a RocketRAID 4320. Do your data a favor and give up on it right now. http://forums.freenas.org/index.php?threads/highpoint-controller-info.8217/

You will have zero SMART monitoring, zero SMART testing, and disk failure will not work properly. So yeah.. might as well just set yourself on fire if you want to use it with FreeNAS.

Yes, I read that thread but thought it might be usable since I'd find out pretty fast if a disk failed since it's in my bedroom...
If I could set it to non-RAID mode. That is really weird...

Not to mention the fact that you are having to create the JBODs means you will NOT be able to take the drives to another controller later. You'll be forced to use the 4320 or equivalent forever.

That could be a problem.

As for why multipath is behaving this way, my guess is something to do with the controller. Aside from that I can't tell you much more, and it's really not worth anyone's time to investigate since there are so many other ways this is a "fail" that it's not worth the effort.

It would only be a Fail if I had put data on there. As I see it, so far, it is a complete success: I'm learning and I haven't lost anything (except maybe the money I threw away on that Highpoint controller...)

So do you have any suggestions for what I might look at to figure this out on my own? Things have changed quite a bit since I printed out all 488 pages of the FreeBSD handbook...

 

[Grendell]

Doh, you posted just before I did.

Blu-Ray doesn't make a particularly good backup, especially when dealing with TBs of data. At 50GB per disk, you'd need 10 disks just to backup 500GB. Not to mention the fact that for that cost in Blu-Ray media + the drive you could have bought a single 2TB or 3TB drive, setup ZFS replication and had time-based backups via ZFS snapshots.

I've read enough of your posts to know what you were expecting. I expect you have good reasons to expect it, too.
In most instances you're correct, but the only things that I would have a real problem if I lost would be the family photos. Yeah, my wife would kill me...

 

[cyberjock]

Not a clue where to point you at further. RAID causes so many problems with ZFS there's about 50 reasons why I said "Hardware RAID + ZFS = fail". Then I usually leave that person to decide how to proceed.

For someone that's going to FreeNAS for the data integrity and performance characteristics someone wants, it's very very counterproductive to then turn around and choose to add on Hardware RAID. There's so many reasons why it's stupid that I just don't even explain myself anymore. It's a "proceed at your own risk" type of thing. I'd figure that the warnings in the manual, my noobie guide, and the sheer number of users that have lost data should be enough to make you decide it's a big big mistake to mix RAID and ZFS.

Even in non-RAID mode you fixed nothing. Still have no SMART monitoring, SMART diagnostics, and reporting of failed disks because the controller doesn't let you.

With that I bid you adieu. I can't even fathom why anyone would even consider go further with that controller(or any RAID controller) if you actually care about your data. It's smarter to throw that Highpoint away and buy an M1015 then to continue on the path you are taking. But, you could always ebay the Highpoint to offset the cost of the M1015. That's what I did with some of my older controllers.

 

[Grendell]

With that I bid you adieu. I can't even fathom why anyone would even consider go further with that controller(or any RAID controller) if you actually care about your data. It's smarter to throw that Highpoint away and buy an M1015 then to continue on the path you are taking. But, you could always ebay the Highpoint to offset the cost of the M1015. That's what I did with some of my older controllers.

So are you still using that Areca? If so, how is it working for you? Do you have the battery for yours? Do you think it is necessary if the card isn't going to be used for RAID? That card appears to be around $150-$200 on ebay. I think I paid more than that for that RR 4320... five years ago.

It seems strange that the 4320 is still being sold new and the Areca 1280ML isn't.

 

[cyberjock]

Nope.. bailed on that card too.. the whole "Hardware RAID + ZFS = fail" thing. I almost lost my pool over that controller.

 

[Grendell]

So you're running the M1015, cyberjock? You're running it in a VM, right?

So... around 7-8PM my FreeNAS box was doing... something. I really have no idea what and can't figure out how to find out. When I got to the GUI around 9PM it showed a lot of system activity including the network around then. I cannot imagine why. I wasn't using it and no one else on my LAN would have been.
I have no cron jobs set up, there are no SMART jobs set up, there is really nothing set up on it yet. I don't even have any volumes set up. What was it doing?? The console shows nothing of any use. Except that it appears to have rebooted. On its own. At around 11AM. When I looked at the uptime it appears to have rebooted itself about once every day. For no apparent reason. The last time it rebooted it changed another of the drives to multipath.

There doesn't seem to be any kind of access to the system logs except in the console from the GUI.

 

[cyberjock]

Not a clue what it was doing. That would be dependent on your settings for FreeNAS. I could easily give lots of things it might have been doing since you can schedule things to happen whenever you want from the WebGUI.

As for the rebooting I have no clue. There's so many things that could be causing a reboot I can't even give a speculation unless you have more information than an uptime. Error messages or something would be useful.

 

[Grendell]

Not a clue what it was doing. That would be dependent on your settings for FreeNAS. I could easily give lots of things it might have been doing since you can schedule things to happen whenever you want from the WebGUI.

That's the thing: I hadn't scheduled it to do anything at all. It's only been installed for a few days and I was being pretty cautious about changing anything until I had an idea about what it would do. It seems very strange that it should have been doing anything.

As for the rebooting I have no clue. There's so many things that could be causing a reboot I can't even give a speculation unless you have more information than an uptime. Error messages or something would be useful.

Yes, I thoroughly agree: Error messages would be useful. If I knew where to look for them I might be able to find some. ;-)

On the plus side the 4320 is running in non-RAID mode now. But FreeNAS is still sort of "rotating" my drives through being multipath devices at every reboot.

 

[cyberjock]

I can't believe you are even talking to me while still using the 4320. I've already given plenty of reasons why you can never use that controller.

Well, good luck. I guess some people just are determined to learn the hard way.

 

[Grendell]

Do you know that responses like that are why people don't run Linux/BSDs?

Wow, and from a forum admin, no less. So I can expect this thread to be locked if not deleted entirely, I suppose.

Yes, I am running forbidden hardware. I am also learning about FreeNAS before I decide to make a commitment to it. One of the things I was interested in learning was what sort of reception I would get on the forums. I seem to have my answer to that one, don't I?

Frankly, I don't care how many times you have heard the same question. Rudeness from an administrator of a help forum? I always thought administrators were supposed to prevent flame wars, not initiate them. You have a consulting business? Remarkable. Well, you didn't say it was a successful consulting business, did you?

So now I have seen what the FreeNAS community is about. What a shame, it looks like a fairly usable product.

And, yes, I can use that controller, while I am learning about FreeNAS.

 

[cyberjock]

You know why you get "responses" like what you get on Linux/BSD? Because most people aren't in those forums to validate bad decisions. Someone asks a question, someone gives an answer. That answer should be a frank answer as to what you should do.

Yes, admins should be preventing flame wars. We should also be preventing the dissemination of incorrect information. At some point, my time is worth something to me. We've *thorougly* documented how crappy RAID controllers are. The fact that you still insist on using one tells me that you're what we call an 'askhole'. It's a real word. Feel free to Google it if you've never heard of it.

Frankly, even for learning FreeNAS, it's still a bad choice. It won't work properly with inappropriate hardware, won't provide expected performance, and generally is a poor way to "evaluate" FreeNAS for a particular scenario.

 

[Grendell]

Can you direct me to some of this thorough documentation because everything I've read about them so far, including your thread on Highpoint controllers in general said, in essence, "Usable, but not recommended at all."

So far, it hasn't caused me any problems with learning how disk space is allocated in FreeNAS and some of the rather odd conventions used in FreeNAS. Things that I couldn't see at all without installing it. I think I'll just connect the drives directly to the board and see what it thinks of that for now.

 

[cyberjock]

doc.freenas.org

There's literally dozens and dozens of threads on this topic too.