Which Server for Dell Xyratex Compellent HB-1235 Disk Array?

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Hi All,

we've been running FreeNAS on Dell T710 server (setup with great help from this forum) for about two years.
Now we need a second server (for remote office).
There are some populated HB-1235 arrays available locally, we're considering getting one of those and connecting it to a server we're also yet to acquire.
We have already purchased a SAS9201-16e card to connect the external array.
The servers we're considering are Dell R510/R710/R520/R720, HP ProLiant D380 G7/G8 or IBM 3065 M3/M4.

When we installed FreeNAS on Dell T710, the server's storage slot was a nuisance. After we flashed H200 to LSI IT firmware the server wouldn't boot from the card in storage slot. We had to move the card to the back of the case and buy longer SAS cables. Luckily, the motherboard also had a couple of bootable SATA ports, so we installed SSD boot drive into the DVD cage.

This time around we can leave the card that comes with the server (usually Perc700/710, or P410i, or M5110 ) in place. I know it's not recommended for ZFS storage arrays. My question is, can any of these cards be configured to work adequately with FreeNAS boot disks? So that we can use the server storage cage for boot drives.
And if not, which of the abovementioned servers have bootable SATA on the motherboard and convenient place to mount boot SSD? If we can't use the server's front cage to boot we'd probably lean towards a 3.5" front cage which we could use for future storage expansion.

Also, a noob question, do we need two 8088 cables to connect SAS9201 to HB-1235 or just one (assuming that the disk array it has built-in SAS expander)?

Any advice will be greatly appreciated.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
I'm running an HB-1235, it's a pleasingly dumb box of disks. If yours has one controller, it's set up for SATA. If it has a pair, it'll expect SAS disks and make an annoying beep noise forever if you use any SATA disks, and also put an orange 'fault' light on every SATA caddy. You can "fix" the beep by opening up the little ear on the left and removing the ribbon cable. I know, I know... :D:)

You need one SFF-8088 cable between host and diskshelf. By default all three sockets on each controller unit are set up two-way, so any socket should work. I'm using the top-right one for what that's worth. The cable locks when it's in far enough.

The 9201 only talks IT mode if I recall correctly, but if you also include the BIOS on the 9201 you can boot off a disk/mirror in the HD-1235. I've chosen not to do that, preferring to boot off internal.

Picking a server - I don't know about the exact models, but some Dells will have an open SAS port by the RAID SAS port(s) that is hooked up to ports 1-4 of the motherboard SATA controller, giving raw disk access. They'll be mentioned on the label inside the lid. If any of yours do, great! Move/swap out one of the internal RAID SAS cables to that socket and use the front disk slots for your bootup mirror. The Dell 3.5" caddies have screwholes suitable for SSDs.

I'd recommend pulling the internal RAID cards out entirely, particularly DELL miniPERCs (the ones in a dedicated PERC slot) just to get them out the way - plus fair chance of bricking your whole Dell if you try and flash them with non-Dell firmware, so don't accidentally do that.

FreeNAS seems pretty irregular on how it matches /dev/daXX devices to physical disk slots, seems to just be on response time at boot, so I'd suggest to label the disk serial number on each caddy so you can hot-pull dead drives succesfully.
 
Last edited:

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
If you don't find a model with an internal SAS port, it certainly isn't the end of the world to use the server's RAID for your boot array, just build a mirror in the RAID utility and install FreeNAS on that. It's not ideal, but shouldn't cause any actual issues in practice.
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Thank you Jaimie. Once we get a server (most likely IBM X3550 M4), I'll report what we find inside :)
Great to know that only one cable is required.
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
The x3550 M4 has arrived.
some Dells will have an open SAS port by the RAID SAS port(s) that is hooked up to ports 1-4 of the motherboard SATA controller, giving raw disk access.
The IBM has such port too, but when I move the SFF-8087 cable from the M5110 to the motherboard connector, the motherboard SATA controller cannot see the drives. It's not just a BIOS issue either: I booted Linux from USB and the x79 SATA controller is enabled and is talking to the DVD drive. I'm starting to think that the SATA controller cannot talk to SATA drives via the SAS backplane. Would breakout 8087 to 4xSATA cables work?
https://www.ebay.com.au/itm/1M-Inte...-Pin-F-Forward-Breakout-Cable-AU/201762851706

If you don't find a model with an internal SAS port, it certainly isn't the end of the world to use the server's RAID for your boot array, just build a mirror in the RAID utility and install FreeNAS on that. It's not ideal, but shouldn't cause any actual issues in practice.
I don't think we even need a raid mirror, the M5110 is already in JBOD mode.
How bad is SAS2208 on FreeNAS if used for boot only? Will boot scrub or smart reporting be compromised?
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
You'll lose SMART, which is why it's worth setting up a boot mirror if you can't hook up the boot drive(s) to a direct SATA port. You will lose ZFS's healing, but keep scrubbing. That's perfectly enough for the boot drive since its data is disposable, reimaging it and reloading the config from your previously-saved backup is sufficient for repairs.

Disappointing about the internal port. That breakout cable will be sufficient, the potential issue being supplying power to the drives if they're not in the slot loading sockets. What exact model IBM did you end up with, perhaps someone knows the magic incantation to get it to work?
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
What exact model IBM did you end up with,
x3550 M4
perhaps someone knows the magic incantation to get it to work?
I hope so!
the potential issue being supplying power to the drives if they're not in the slot loading sockets.
There is power connector that would go to the second backplane, but it's a ten pin Molex, and I doubt there are ready made converters to SATA power...
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
And if so, will it be the same if we talk to the drive via M5110 using the mrsas driver?
Interesting question. Probably, but not definitely. Booting from SAS is a bit of a pain anyway, so you'd want to avoid that.
 
Joined
Dec 2, 2015
Messages
730
Ouch! Put in two Sandisk SSD Plus (detected by fine by M5110), downloaded FreeNAS11.2-1 installation ISO, and then found this:
https://redmine.ixsystems.com/issues/35065
Is it correct that the problem still exist? And if so, will it be the same if we talk to the drive via M5110 using the mrsas driver?
Also thinking for getting a 12.7" SSD caddy to put one of the boot SSDs in the optical drive slot:
https://www.ebay.com.au/itm/JEYI-No...ay-for-2nd-Hard-Drive-Caddy-Tray/221849065483
I bought a San Disk SSD Plus in late 2016 to be the boot device on my main server. I have never had a problem with it - I'm currently running FN 11.2-U2.1. There must be another variable at play which determines if the problem is present or not. Maybe it depends on firmware version of the SSD, or maybe they have changed some other relevant detail.

Bottom line: while some people have experienced problems with this SSD, there is no guarantee that every user will have a problem.
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Booting from SAS is a bit of a pain anyway, so you'd want to avoid that.
Already feeling the pain :mad:. Tried installing today. Replaced one of the Sandisks with a Kingston that was available. Installation went through w/o errors, but the boot hangs when it gets to MegaRaid controller. Not sure if that's because I chose BIOS boot instead of UEFI during installation, or something else is broken. Can't be the Sandisk problem, because I'm trying to boot from the Kingston drive.
The SSD caddy for the ODD slot is in the post. Will be booting from it when it arrives. The plan is to have the Kingston SSD connected directly to SATA controller via the caddy as a main boot drive, and a Sandisk connected via M5110 as its mirror.
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
Booted!:p
The MegaRaid was hanging when there were two drives in the backplane. I took the Kingston out, and the server boots OK from Sandisk. Go figure. Once booted, Kingston can be hotplugged and shows up in System->Boot Environments->Boot Pool Status.
I temporarily connected Kingston via an external USB dock, until the ODD bay caddy arrives. FreeNAS picks it up as a boot mirror from there too.

No checksum errors so far.

The 9201-16e didn't see the Xyratex originally, but came good after flashing P20 firmware as per instructions here:
http://www.45drives.com/wiki/index.php?title=How_do_I_update_my_LSI_9201_firmware_in_FreeNAS?
Didn't even need a reboot.

You'll lose SMART
I can actually run smartctl on the drives connected to the M5110. I ran short smart test today. Does it mean my smart is working, or there is something else ZFS needs to work properly?

One more question. I cannot see disk sizes in Storage->Disks. Any reason for this?
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
One more question. I cannot see disk sizes in Storage->Disks. Any reason for this?
I have an answer. Created a storage Pool. The disk sizes appeared. Destroyed the pool. They disappeared again. So they won't show unless you have at least one zpool.
The brand new Kingston UV400 we have is defective. That was the reason for MegaRaid hanging on reboot.
The ODD drive caddy has arrived. Kingston wouldn't boot from it, and produced hundreds of checksum errors when system booted from the Sandisk via Megaraid.
Sandisk SSD boots from either ODD caddy or MegaRaid. Two of them are now in the server. No checksum errors, but both connect at 150Mbps. Which is ok for the boot drives, but I'd like to have full speed on the other three available SAS slots...

The HB-1235 is behaving well. Doesn't beep, as long as both power supplies are plugged in and no SATA drives connected :)
Currently running badblocks on Seagate ES.2 SAS drives that the disk array was populated with.
 

2nd-in-charge

Explorer
Joined
Jan 10, 2017
Messages
94
The brand new Kingston UV400 we have is defective. That was the reason for MegaRaid hanging on reboot.
Or maybe not. The Kingston drive works properly if hot-plugged after the system is booted. Passed badblocks and all smart tests.
And MegaRaid stopped hanging after I updated the firmware to version 23.34.0-0023 from IBM. The only odd behavior by the time I pulled the Kingston drive out of the server for good was that the activity light was always on.
No checksum errors, but both connect at 150Mbps. Which is ok for the boot drives, but I'd like to have full speed on the other three available SAS slots...
And they do have full speed. Both SSDs report 6Gbps via smartctl, and the MegaRaid card reports 6Gbps link speed via MegaCli -PDList. Both drives read speeds are also well above the 150Mbps reported in dmesg (about 330MBps for Sandisk and 400MBps for Kingston). Probably a harmless bug in mrsas driver.

The drive in optical slot is indeed throttled to 150MBps, and the SATA0-3 connector on the motherboard is not seeing any drives. I'm convinced it's disabled by the server's firmware. So the M5110 will stay to drive the second boot drive and the SLOG.
 
Top