Areca card uses single LUN for all pass-through drives

[Jamie Gruener]

Okay, that's not exactly true because each drive has a unique SCSI ID. But they all share the same LUN. We have 3 Areca cards, two 1882XI cards, and an 1882LP. All exhibit the same behavior. We've set them to JBOD, but we still have to export a single disk as a "pass-through" disk to make it available to the OS (FreeNAS, Windows, Linux, ESXi, whatever).

When we do, the Logical Unit id value reported by SMART (or in Linux) is always the same for all disks from a given vendor. Below is the results of smartctl from a Seagate. Other Seagate drives will all have the same Logical Unit id. This value is the same across all three Areca cards. (Western Digital drives come up with a different id; haven't tested if the WD value is common among our 3 Areca cards, but I assume it is.)

This causes a couple of problems for us. First, FreeNAS will sometimes report these drives as Multi-path, which isn't true. We can delete the multi-path info easily enough. Also, when a drive is pulled and then re-inserted, it's name changes. Maybe that's to be expected for drives that haven't been made part of a vdev or zpool, but it seems weird to me.

But, once the drive is made part of a vdev or zpool, it seems to behave normally. If I pull a drive and then re-insert I do have to "remove" it from the volume and the re-add it to restore fault tolerance/resilver, I can't just say "it's back!", but maybe that's to be expected, too.

Is there any other possible downside to this weirdness? I'm considering using this card for a production environment.


Here are smartctl results. Note the Logical Unit id value.

[root@folio ~]# smartctl -i /dev/da0​

smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p3 amd64] (local build)​

Copyright (C) 2002-13, Bruce Allen, Christrian Franke, www.smarmontools.org​

​

=== START OF INFORMATION SECTION ===​

Vendor: Seagate​

Product: ST4000NM0033-9ZM​

Revision: R001​

User Capacity: 4,000,787,030,016 bytes [4.00 TB]​

Logical block size: 512 bytes​

Rotation Rate: 10000 rpm​

Logical Unit id: 0x001b4d2000000000​

Serial number: Z1Z0AHE1​

Device type: disk​

Transport protocol: Fiber channel (FCP-2)​

Local Time is: Mon Mar 17 11:36:15 2014 PDT​

SMART support is: Available - device has SMART capability.​

SMART support is: Enabled​

Temperature Warning: Disabled or Not Supported​

Thanks,

--Jamie

 

[dlavigne]

I asked a dev and he suggested the following:

Can you have them post the disk, raidset, and volume configuration from arccli?
I think you can tell it to create new targets instead of LUNs on a single target, but this person is also trying to bypass the RAID function and should just replace their card with an HBA if that is what they want.

 

[cyberjock]

I just dealt with a client that used an 1880ix-24 on his FreeNAS box. It killed his pool. I'm not sure for your exact model, but for his model the only solution was a new controller. Those Areca's, while great hardware RAID controllers, are very poorly suited for ZFS. The Areca also doesn't pass through SMART properly, so he could run no SMART tests, SMART monitoring was broken, and disks that failed while attached to the controller wasn't being properly passed to the OS for handling. Your SMART data, if that's all that was provided from your command, is also broken in the same way.

In short, using that controller, if it's like the one I dealt with last week, is not recommended in the slightest. He's currently waiting for the new controller to arrive in the mail.. and is hoping he won't have this problem again. He's also glad he still has his job as there was no backup(yikes!), so the company lost like 12TB of data.

If you give the manual a read, check out my noobie guide, read the stickies that discuss controllers, or search around at people that used RAID controllers, it's something that isn't recommended and has cost a lot of people a lot of data.

 

[Jamie Gruener]

Thanks for the input!

The arrcli tool is not installed on this box. Based on this discussion, I'm not inclined to bother installing/investigating further.

Yes, that's the exact output of smartctl.

We're being very cautious because we're going to be putting a lot of data on this box (north of 60TB, most likely), hence the concern about the Areca card and the problems with the LUN and SMART info. Another issue, possibly related, is that we don't get serial number info in the GUI. We wouldn't worry so much except that's the easiest way to identify which slot the drive is in the external 24 drive chassis. And we want to be able to do that so we pull the right drive in the case of drive failure.

Anyway, I'm happy to abandon that Areca card. What do we get instead? The lauded IBM M1015 is an internal only card. Looking at the currently available LSI cards it seems the drivers are all for FreeBSD 9.1 and we're running FreeNAS-9.2.1.3-BETA-e3f3e20-x64(beta to see if it would fix the serial number issue--it didn't) which is based on FreeBSD 9.2, right?

Our external chassis is a Norco DS-24ER with an Expander#1:Areca ARC-8026-4.01.150115. We plan to add another Norco (we already have it from another project) as soon as this one gets connected and is up and running.

 

[cyberjock]

To be honest, an M1015 is the best card, and if you want a rock solid system you can get something like https://www.pc-pitstop.com/sas_cables_adapters/AD8788-1.asp and be very happy with it.

I own and have used them before without problems. They take up space in your case, so that might not be ideal. I always go with the two port version though since I always want 1 socket free for future expansion. Here's the link for 2 port and 4 port versions:

https://www.pc-pitstop.com/sas_cables_adapters/

 

[cyberjock]

Holy crap.. that's the expander I used for that adapter.. haha. Been trying to sell it on ebay, but people don't appreciate how awesome this expander is. :(

In my setup I used an internal Adaptec controller, ran it to the converter, then from the converter to the expander box. 16 drive RAID6. ;)

It is based on FreeBSD 9.2(uname -a confirms for you). I'd upgrade to 9.2.1.4-beta. It's in testing, but we've tried very hard to get rid of all the problems people have had with 9.2.1.x, so 9.2.1.4-beta is probably the best version to use for testing and whatnot.

Don't worry about drivers. You have no chance of adding drivers yourself. It's not trivial, and its one of those questions that you have to ask how, you can't do it.

If you go with basically any LSI controller it will work out of the box. Seriously.. M1015 is the best. It is limited to 32 disks per controller. So if you want more than 32 disks on the controller you'll need multiple controllers, multiple expanders, etc.

 

[ser_rhaegar]

If you go with basically any LSI controller it will work out of the box.

The new 9300 models lack driver support currently. Though there is a bug report entered for this: https://bugs.freenas.org/issues/4779

 

[Jamie Gruener]

Hmm...

32 might not be enough, and I might not have enough slots. Also, my client would prefer to have an under-warranty solution, So, I'm trying to buy a new card and am looking at the LSI 9200 series, something like a LSI SAS 9205-8e, LSI SAS 9200-8e, or maybe a LSI SAS 9206-16e. I can work with multiple expanders, so if that solves the problem, I can live with the M1015. Your post suggests that 32 is the disk limit, regardless of the expander situation.

How can I confirm that the a specific (or appropriate) driver is in a given build of FreeNAS?

 

[cyberjock]

How can I confirm that the a specific (or appropriate) driver is in a given build of FreeNAS?

Find someone that has validated it works on FreeNAS or buy one and try it. Now you see why people usually take our recommendations because we've probably proven it works already.

If you want more than 32 drives you *can* go with multiple SATA controllers. To be honest, if you want a system that big though, you should probably just call iXsystems and get a quote or doing consultation with me. If you are about to spend the kind of money it sounds like there's *plenty* of ways to have a system that "works"(notice the quotes) and then later on your pool just fails and you have no clue why. We see it here regularly, and you probably don't want to be the guy that buys $5000 worth of hardware only to find out you have to spend another $2000 because you bought all the wrong stuff. Yes, I see it regularly.

 

[Jamie Gruener]

Here's our challenge. We already have nearly all of the hardware. We started this process with Windows 2008 R2 and got burned by, of all things, a faulty SAS cable. That means we already have the sever (Dell R510 w/64GB RAM), two external drive chassis each with a SAS expander, SAS cables, and some RAID cards. Oh, and a couple of dozen disks. 24 of the 2TB variety and 13 4TB ones. Oh, and it's already racked, etc., etc..

Partly because of the specific way in which we were burned (unstable system led to NTFS corruption which lead to data loss), we're looking for something more fault tolerant. FreeNAS is very attractive to us for RAIDZ3, among other reasons, but we're not so excited about FreeNAS that we're willing to ditch all that other hardware.

Our server has only 3 PCI slots, so while we could put two M1015 cards in there with an int/ext adapter, we'd be limited to 64 drives which is a lot more limiting than we originally planned on. It seems that there has to be a quality, under-warranty, HBA out there that we know FreeNAS supports that will run, say, 128+ drives.

I'm happy to pay for some consulting if that's the route to go. I'm really trying to not be "that guy". Honest!

And I'm a little surprised that there aren't more folks building large FreeNAS systems. Maybe I was taken in by the marketing, but it seemed that FreeNAS was designed to support this kind of third-party hardware collection. :)

 

[cyberjock]

Well, when someone wants to build a really large servers and they see some of the problems they have a change of heart. They see that going extremely large is much much more expensive than 2 smaller servers that are 1/2 the storage space each but 1/5 the price. Others go with a consultant(or have a fulltime staff that is a storage guru), or go with a fully built system for iX. Especially when companies are going with a large server they'd rather call up a company like iX and let iX deal with all the logistics than do it in-house. They also want someone's neck to ring if things go wrong. Everyone likes to make the vendor fix it if things go wrong! So in the forums only a few users will spend the necessary money for hardware that is far outside the common user. For most users, the standard 4U/24disk cases are the biggest home users are going to go with. I have one such server for myself. :) I'm one of the top 10 largest I've seen around here among people that chat around here.

FreeNAS(and ZFS) can do some amazing things. The questions really revolve around what's reasonable for the hardware given, what's reasonable for the budget, and what's reasonable for data security. Then you get to choose any two.

I'm not sure if I'm just misunderstanding you or if I'm right, but one controller isn't the most recommended setup. With one controller you're potentially going to end up with a controller that is limited in it's throughput, the PCIe connector's throughput, etc. You really do want more than 1 controller for performance reasons. Also, there seems to be conflicting information on how many disks the M1015 can handle. If you are really convinced you need that many disks for expansion there are options available as you mentioned. There's quite a few LSI controllers that can work. They are several times more expensive than an M1015(sometimes many times more expensive too). But if you are sure you'll use the space it's definitely an option. For 128+ disks I'd be trying to minimize loading by maximizing bandwidth wherever possible. Very large pools are a beast of their own in terms of how to do maintenance and how to do it properly. They have special needs that have to be addressed that aren't a problem with small servers. Most users in the forums probably aren't aware of these limitations because the problems aren't evident until you scale up MUCH larger than us "measly" home users.

While I'm thinking about it under warranty is somewhat a joke to be honest. For example, you can buy the M1015 on ebay for about $100. I've only seen a single card fail from eBay from all of the users I've seen buy M1015s on ebay. But, if you buy it new.. it's about $300(muuuuuuuch more in some countries). Most of the stuff on eBay will come from some server that was taken meticulous care of for 1-2 years and is off-contract so was sold by the pallet to some reseller who parted it out for profit. But hey, if warranty really is what they want and they are okay with paying that much more for it then definitely go with it. It's all about being happy with your server right?

 

[ser_rhaegar]

You can find more LSI controller options at:
http://www.lsi.com/products/host-bus-adapters/pages/default.aspx#tab/product-family-tab-2

Keep in mind that the 12Gb/s (9300 series) cards do not currently have driver support in FN. The 9200 series should be good though. Some of the 9200 cards can support 512 devices. Just keep in mind the bus limitation as CJ mentioned.

 

[Jamie Gruener]

Understood on all counts.

I'm going to try an LSI 9207-8e card. The throughput issue should not be significant for us; much of this data is cold storage (though important!) and we do not anticipate lots of multiple user concurrent access. If we can get wireline speed for a single user, it'll be sufficient. But that's what the testing is for, right?

Many thanks for everyone's input.

Last note about the under-warranty issue. Completely agree that under-warranty is not of great value for these <$500 single-purpose components, but there is great value in knowing that what we're buying is, in fact, what we're buying. Normally I'm not so scared of eBay, but the last thing this project needs is a mis-labeled or fake card. Overly cautious? Yup. And I'm being paid to be paranoid.

 

[cyberjock]

Yeah.. buying something "under-warranty" doesn't protect you from mislabeled or fake anything. In fact, there's PLENTY of evidence of people buying things like "Samsung TVs" and when it fails they call Samsung for support. Guess what? No valid serial number.. no warranty. Even Newegg and Amazon have been involved in these kinds of things. Just check out this one from 2010.

http://www.hardocp.com/article/2010/03/05/newegg_selling_fake_intel_cpus

Just do a search for "Amazon fake" and you'll find tons of examples of stuff Amazon has sold that was later determined to be fake.

So I'm sorry, but I don't see how buying something that's allegedly "under warranty" is going to guarantee what you bought is real. ;)

 

[Jamie Gruener]

FWIW, we ended up buying an LSI 9207-8e card. Works great! We did run into an issue where Seagate reports some weird SMART data such that some values are always out of range, unless you're using the Seagate tool to read them. For now we know to just ignore the errors for that value. Also, we'll likely be switching to Western Digital enterprise class drives moving forward. Better UBE rates and similar prices.

 

[tmueko]

I want to "upgrade" an Open-E server with 12 Harddisks and 24 Slots to FreeNAS.
I added all Disks as pass-through (areca-cli disk create drv=X)and did a "camcontrol rescan".

da1 at arcmsr0 bus 0 scbus0 target 0 lun 0
da2 at arcmsr0 bus 0 scbus0 target 0 lun 1
da3 at arcmsr0 bus 0 scbus0 target 0 lun 2
da4 at arcmsr0 bus 0 scbus0 target 0 lun 3
...
da11 at arcmsr0 bus 0 scbus0 target 1 lun 2

Smart is looking strange (with the fist 4TB 10000RPM Fibre Harddisk ;)

[root@freenas] ~# smartctl -x /dev/da1
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: WDC
Product: WD40EFRX-68WT0N0
Revision: R001
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
Rotation Rate: 10000 rpm
Logical Unit id: 0x001b4d207a2c5ee0
Serial number: WD-WCC4EKFV1YLX
Device type: disk
Transport protocol: Fibre channel (FCP-2)
Local Time is: Tue Dec 2 18:38:00 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported
Read Cache is: Enabled
Writeback Cache is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature: 30 C
Drive Trip Temperature: 25 C

Manufactured in week 30 of year 2002
Specified cycle count over device lifetime: 4278190080
Accumulated start-stop cycles: 256
Elements in grown defect list: 0

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 0.000 0
write: 0 0 0 0 0 0.000 0

Non-medium error count: 0

Device does not support Self Test logging
Device does not support Background scan results logging
scsiPrintSasPhy Log Sense Failed [unsupported field in scsi command]


I updated the Firmware first:

[root@freenas] ~# areca-cli sys info
The System Information
===========================================
Main Processor : 800MHz
CPU ICache Size : 32KB
CPU DCache Size : 32KB
CPU SCache Size : 1024KB
System Memory : 1024MB/1333MHz/ECC
Firmware Version : V1.52 2014-11-11
BOOT ROM Version : V1.51 2013-01-23
Serial Number : Y243CADHAR811053
Controller Name : ARC-1882
Current IP Address : 192.168.1.100
===========================================

GuiErrMsg<0x00>: Success.


Did they changed something or will there be Problems?

 

[tmueko]

OK, no smart tests with smartctl but with areca-cli
[root@freenas] ~# smartctl -t short /dev/da1
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

Short offline self test failed [unsupported scsi opcode]

[root@freenas] ~# areca-cli disk smart drv=09
S.M.A.R.T Information For Drive[#09]
# Attribute Items Flag Value Worst Thres Raw State
==============================================================================
1 Raw Read Error Rate 0x2f 100 253 51 0 OK
3 Spin Up Time 0x27 189 189 21 7550 OK
4 Start/Stop Count 0x32 100 100 0 6 OK
5 Reallocated Sector Count 0x33 200 200 140 0 OK
7 Seek Error Rate 0x2e 100 253 0 0 OK
9 Power-on Hours Count 0x32 100 100 0 24 OK
10 Spin Retry Count 0x32 100 253 0 0 OK
11 Calibration Retry Count 0x32 100 253 0 0 OK
12 Device Power Cycle Count 0x32 100 100 0 6 OK
192 Power-off Retract Count 0x32 200 200 0 5 OK
193 Load Cycle Count 0x32 200 200 0 5 OK
194 Temperature 0x22 124 119 0 28 OK
196 Reallocation Event Count 0x32 200 200 0 0 OK
197 Current Pending Sector Count 0x32 200 200 0 0 OK
198 Off-line Scan Uncorrectable 0x30 100 253 0 0 OK
199 Ultra DMA CRC Error Count 0x32 200 200 0 0 OK
200 Write Error Rate 0x08 100 253 0 0 OK
===============================================================================
GuiErrMsg<0x00>: Success.

[root@freenas] ~# areca-cli disk sttest drv=09 mode=short
Message: Drive#09 ==> Starting execute offline operation ( Short self-test )
GuiErrMsg<0x00>: Success.

[root@freenas] ~# areca-cli disk info drv=09

Drive Information
===============================================================
Device Type : SATA(5001B4D70F90501C)
Device Location : Enclosure#2 SLOT 01
Model Name : WDC WD40EFRX-68WT0N0
Serial Number : WD-WCC4EK2CUF8D
Firmware Rev. : 82.00A82
Disk Capacity : 4000.8GB
Device State : NORMAL
Timeout Count : 0
Media Error Count : 0
Device Temperature : 28 C
SMART Read Error Rate : 100(51)
SMART Spinup Time : 189(21)
SMART Reallocation Count : 200(140)
SMART Seek Error Rate : 100(0)
SMART Spinup Retries : 100(0)
SMART Calibration Retries : 100(0)
===============================================================
GuiErrMsg<0x00>: Success.