Building new TrueNAS server with Broadcom SAS3016 PCIe card

[talekksna]

Hi,
I am new in TrueNAS community, but I am not new in server/storage world. I have on my disposal new Supermicro AOC-S3816L-L16iT, AOC-S3216L-L16iT, AOC-S3616L-L16iT cards, which works in IT/HBA mode.
Can I use any or all of these controllers, or they are too new and/or not supported?
I am planning to order one of those cards.
At the moment, for testing purpose I build TrueNAS core on Supermicro X10DRi motherboard with RAID controler AOC-S3108L-H8iR that I configure in JBOD mode, and it is working without any problems. Disks are visible to the TrueNAS with all details and SMART capabilities.

I read general posts that are talking about RAID JBOD cases, and drawbacks with this kind of configuration, but those articles are year or more old, so I am wondering is something changed when SAS3108 in JBOD mode is in question?

If there is more articles about experience with SAS3108 and TrueNAS, please, be kind and point me to those articles.

Thank you very much.

Đorđe Talović

 

[jgreco]

I have on my disposal new Supermicro AOC-S3816L-L16iT, AOC-S3216L-L16iT, AOC-S3616L-L16iT cards, which works in IT/HBA mode.
Can I use any or all of these controllers, or they are too new and/or not supported?

Any of them that run the IT firmware and show up under the MPR driver should be fine. The list of supported cards that work under MPR is a moving target and has grown over time. Check the "man mpr" manual page for your version of FreeNAS/TrueNAS to see if it's supported on whatever you are running. For example, for FreeNAS 11.3, the supported controller list includes

Code:

     •   Broadcom Ltd./Avago Tech (LSI) SAS 3004 (4 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3008 (8 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3108 (8 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3216 (16 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3224 (24 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3316 (16 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3324 (24 Port SAS)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3408 (8 Port SAS/PCIe)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3416 (16 Port SAS/PCIe)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3508 (8 Port SAS/PCIe)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3516 (16 Port SAS/PCIe)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3616 (16 Port SAS/PCIe)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3708 (8 Port SAS/PCIe)
     •   Broadcom Ltd./Avago Tech (LSI) SAS 3716 (16 Port SAS/PCIe)

but not your 3816.

but those articles are year or more old, so I am wondering is something changed when SAS3108 in JBOD mode is in question?

No. Why would you think that? The hardware is the same as it was a year or more ago. The firmware is the same as it was a year or more ago. The fundamental issues are the same as they were a year or more ago. Why would you discount it just because the information was posted when the problems were known?

f there is more articles about experience with SAS3108 and TrueNAS, please, be kind and point me to those articles.

Yes, here you go. This is the definitive article on the topic:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[talekksna]

Bravo!! Thank you very much on such short time to help a new guy in TrueNAS community.
Than you for info on SAS3216, that I can gat fastest in storage.
Regarding SAS3108, yes, you are right, there is no reason to be different that it was year ago, but, I just ask.
Article is great insight into TrueNAS controller world! Great!

Anyway, thank you very much for help.

Cheers.

 

[Ericloewe]

I was looking into the SAS3816 myself, since Supermicro is now shipping it in their SuperStorage lineup. There's a mailing list post somewhere where a guy from Broadcom says the 38xx models are supported on FreeBSD (thus implying that the driver comments are out of date). The driver does have some defines for the 3816, but that's no guarantee that all the required bits are present.

@talekksna, did you ever test with a 3816?

 

[jgreco]

There's a mailing list post somewhere where a guy from Broadcom says the 38xx models are supported on FreeBSD (thus implying that the driver comments are out of date). The driver does have some defines for the 3816, but that's no guarantee that all the required bits are present.

I've gotten the strong impression that the firmware side of the LSI's is relatively stagnant, and that they're primarily doing opportunistic updates for faster PCIe, different specific configurations, newer embedded CPU's, etc. I would not be shocked if it was fully supported just because it looks substantially like all their other cards.

 

[Ericloewe]

That's my impression as well, but LSI/Broadcom/Avago is the company that thought that "you should pair firmware n with driver n+1" was a sensible approach.
In the meantime, searching for "9500" instead of "3816" got me a few more hits on the forum. Unfortunately for me, my name is plastered on most of them (I hate it when that happens!). Fortunately for me, @morganL has a post from last year where he mentions that iX was looking to deploy more 9500 in the future. Any news that you can share with us, @morganL?

 

[sretalla]

Interestingly, it seems that list from man mpr isn't necessarily matching the label on the box...

for my card:

Code:

# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Adapter Selected is a Avago SAS: SAS3224(A1)

    Controller Number              : 0
    Controller                     : SAS3224(A1)
    PCI Address                    : 00:03:00:00
    SAS Address                    : 500062b-2-0299-0c80
    NVDATA Version (Default)       : 10.00.00.03
    NVDATA Version (Persistent)    : 10.00.00.03
    Firmware Product ID            : 0x2228 (IT)
    Firmware Version               : 16.00.11.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9305-24i
    BIOS Version                   : 08.27.00.00
    UEFI BSD Version               : 13.00.00.00
    FCODE Version                  : N/A
    Board Name                     : SAS9305-24i
    Board Assembly                 : 03-25699-02004
    Board Tracer Number            : SP63808066

I note that what I bought was a SAS9305-24i... not on the infamous list... but the number that's interesting isn't the "board name", it's the Controller : SAS3224(A1)

Seems I'm on the list after all... maybe you are too.

 

[morganL]

That's my impression as well, but LSI/Broadcom/Avago is the company that thought that "you should pair firmware n with driver n+1" was a sensible approach.
In the meantime, searching for "9500" instead of "3816" got me a few more hits on the forum. Unfortunately for me, my name is plastered on most of them (I hate it when that happens!). Fortunately for me, @morganL has a post from last year where he mentions that iX was looking to deploy more 9500 in the future. Any news that you can share with us, @morganL?

We have tested and qualified some 9500s HBAs with specific HDDs and SSDs.

They do behave a little differently and there were a couple of unexpected drive incompatibilities. Nothing major and it wasn't clear whether it was HBA, OS, driver or the device that caused the minor issues. If you are going to use new HBAs, try to use relatively modern devices would be my recommendation.