HP Apollo 4510 - 68 disk FreeNAS

[fidomuh]

Hi guys,

I have acquired a 68 disk server and intend to use it for long-term backups off-site.
We already have 68 6TB drives for it and 2 x 2.5" SSDs for FreeNAS root. The server has 512GB RAM and I have inserted 2 PCIe SSDs for L2ARC and ZIL (probably overkill, but I already had it on hand).

So, there is a major issue. The server has 2 x HP SmartArray P440 raid cards that support HBA mode, but it's not "true" HBA / Passthrough and FreeNAS is not happy. Hotplug doesn't work, so essentially not usable.

I need to buy new HBA cards and have looked at Broadcom Megaraid and Highpoint RocketRaid 3740A.
It looks like the backplane is connected via 4 x SFF SAS 8087, so I'm guessing theres a SAS extender as well.

Do any of you have any recommendations on HBAs? Or any experience with HP servers like this?

For raids I was thinking of running 6 Raidz3 with 11 drives in each and using the spares as global hotspares. It's either that or 4 x 17 RaidZ3, but that won't align correctly, so I'm guessing that's not recommended.

 

[Elliot Dierksen]

Dude, don't go there! Get an supported controller like an LSI 9207-8i. That will work with the SAS expander in the back plane. If you don't use a supported controller, you are going to subject yourself to a bunch of pain and suffering. I have seen that movie, and I have the t-shirt. Not good....

 

[fidomuh]

Yeah, I am exploring options and haven't ordered anything yet. The 9207-8i is not enough though. I need 4 x SFF8087 ports to fill the backplane, so I was looking at the 9300-16i, which looks like it's a pure HBA.
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-16i

And the 9207-8i is a 6Gbps controller, wouldn't it make more sense to put in a 12Gbps controller now that I'm buying something anyway?
The entire box has a retail price of >80.000$ atm, so a 1.000$ on a HBA is fine :D

 

[Elliot Dierksen]

Yeah, I am exploring options and haven't ordered anything yet. The 9207-8i is not enough though.

How many cables are coming from the backplane to the controller you have? If it is just two, then the 9207-8i is fine with the sas expander that must be built into the backplane. With an expander, that controller will support up to (I think) 256 drives. I have a 9207-8i in my system that is driving 19 disks with a sas expander.

 

[HoneyBadger]

The server has 512GB RAM and I have inserted 2 PCIe SSDs for L2ARC and ZIL (probably overkill, but I already had it on hand).

As my sig says, there's no kill like overkill. What are the two PCIe SSDs, so that we can ensure you're using the right tools for the job?

And the 9207-8i is a 6Gbps controller, wouldn't it make more sense to put in a 12Gbps controller now that I'm buying something anyway?

Since you've got a 12Gbps backplane and you're feeding 68 disks - yes, absolutely buy a 12Gbps SAS3 controller for this. But bear in mind that inserting a non-HP card into an HP server may very well cause the fans to ramp up to 100% as a "protective measure." If this server is in a datacentre, that's not an issue; but just so you know.

For raids I was thinking of running 6 Raidz3 with 11 drives in each and using the spares as global hotspares. It's either that or 4 x 17 RaidZ3, but that won't align correctly, so I'm guessing that's not recommended.

17 drives in a Z3 is a little too wide of a vdev for my (and probably other's) liking. "Alignment" isn't a huge issue with compression on by default (keep it on, it's good)

6x11 Z3 is a nice number, and will give you roughly 250GB usable after the overhead of Z3 parity and "slop space" according to this calculator:
https://wintelguy.com/zfs-calc.pl

Follow @Elliot Dierksen 's lead about matching the current cable count - if there's two "unused" ports on your backplane, they may well be intended to connect to another backplane internally (or be routed out the back via an optional adapter) - check the manual/Quickspecs for the hardware to confirm.

 

[fidomuh]

Thanks for the input.

There are 4 cables connecting to the backplane, 2 from each P440 currently, so I'm thinking I'll need to stick to the 4 cables.
Would the 9300-16i be a good fit then? It's in stock at my local supplier :)

The PCIe SSDs are Samsung Evo 970 Pro (m.2 in a 4x PCIe adapter. Cheap StarCom but I've used them before and benchmarked them with no issues).

I'll watch out for the fanspeeds. It'll go in a datacenter off-site, but I'm guessing we'll get complaints if the fans run steady at max speed :D

I will go for 6x11 Z3 then and keep 2 as global hotspares.

 

[Johnnie Black]

There are 4 cables connecting to the backplane, 2 from each P440 currently, so I'm thinking I'll need to stick to the 4 cables.

You'll need to check the manual but very likely that's for redundancy, and if that's the case you might not need, or it might not even work, to connect the same HBA to the 4 ports.

 

[fidomuh]

Hmm, so it would be better to either opt for 2 x HBA 8i or simply just plug in only 2 ports?
I couldn't find anything in the manual stating either way, but it seems like the current raid cards are redundant... I just tried pulling one card out and it still registers all 68 disks..

 

[Johnnie Black]

Got this from a quick google search, don't know if this is your exact model, but it should give an idea on the possible connection options:

https://support.hpe.com/hpsc/doc/pu...287871&docId=emr_na-c04787658&docLocale=en_US

 

[fidomuh]

Oh wow, great :)
So 1 HBA with dual 4-lane SFF connected to 1A and 2A.

Thanks.

 

[fidomuh]

Hi guys,

Just wanted to post an update after receiving the LSI 9300 8i HBA.
Installed in with SFF 8643 -> 8087 connected in Port 1A and 2A.

No fan noise from the server, no warnings in bios, etc.
Booted FreeNAS without doing anything and all disks show up, hot plug works, etc.

I am still testing but so far performance is better (probably due to 12Gbps controller vs 6Gbps) and the only thing I'm really missing, is FreeNAS webui showing me which physical slot each drive is in.
I can tell from dmesg that the information is there (and correct) but I can't see it in the webui :D

But thanks for all the help guys.

 

[HoneyBadger]

the only thing I'm really missing, is FreeNAS webui showing me which physical slot each drive is in.

getencstat -v /dev/ses0 should give you a listing to start with, that you can use to map daX to bay.

setobjstat /dev/ses0 HEXELEMENT 0x80 0x00 0x02 0x00 should light up the indicator LED on the bay, eg:

setobjstat /dev/ses0 0x7 0x80 0x00 0x02 0x00

And the following line should turn it off

setobjstat /dev/ses0 0x7 0x80 0x00 0x00 0x00

 

[Elliot Dierksen]

setobjstat /dev/ses0 0x7 0x80 0x00 0x02 0x00

And the following line should turn it off

setobjstat /dev/ses0 0x7 0x80 0x00 0x02 0x00

I don't know if I am missing something, but both those lines are the same. Is that command just a toggle? Is there another command that would let you see the current status of the LED?

 

[HoneyBadger]

I don't know if I am missing something, but both those lines are the same. Is that command just a toggle? Is there another command that would let you see the current status of the LED?

Whoops. Copied the wrong line from .history for the second one - the third field should be 0x00

As far as how to get the LED status, I think you can getobjstat in a similar fashion. Let me check and get back to you. Edit: getobjstat isn't available by default in FreeNAS, it seems. I guess the method to check current LED status would be "go look" :P

 

[Elliot Dierksen]

Edit: getobjstat isn't available by default in FreeNAS, it seems. I guess the method to check current LED status would be "go look" :p

Yes, but "go look" doesn't have anywhere near the same level of nerd coolness! FYI, thanks. That was totally cool! The setobjstat commands work properly on my Cisco C240 M3S with an LSI 9207-8i.