SuperMicro CSE-826 X9DRH 12x 3.5" (LFF)

Ben Lavery

Cadet
Joined
Nov 8, 2016
Messages
6
Hi all,

For a few years I've been meaning to build a FreeNAS box, each time I get some cash together something more important comes along…Earlier this year I sold some equipment and have a little over £500 to spend on such a box.

My use cases are:
  • Centralised home storage
  • Backup of a FreeBSD Linode (via zfs send/recv)
  • Some home services (www/nextcloud, www/gitea, etc)
  • Potentially a couple of bhyve VMs for port testing/occasional Linux testing
  • I'm not looking at doing video encoding/transcoding or much heavy lifting.
Some considertions:
  • I have an existing 4 HDD (SATA) RAIDz that I'm looking toad to the system
    • I'd like to convert this into something more manageable, probably a few sets of mirrors
  • I have a number of other SATA HDDs that I'm planning on using
  • I've got a bunch of 16GB PC3L-12800R DIMMs that I saved from a skip, so I'd like to be able to reuse some of them
I posted on Twitter that it's been so long since I did a build I have very little idea what I need to look for, and I got a response:

Allan Jude said:
@forquare Find the european version of http://UnixSurplus.com and get some off-lease SuperMicro gear with 8-12 bays and an LSI HBA controller (9207, 9211, etc). Then you should be good.
I've found a UK based site called Bargain Hardware, and I can put together a server within my budget:

Processor (CPU): 2x Intel E5-2600 V1/V2 Series Processors (Socket LGA 2011)
Memory (RAM): Up to 512GB of Registered (RDIMM) ECC DDR3 RAM
Hard Disk Controller: SATA 3.0 6Gb/s with RAID 0, 1, 5, 1
Hard Disk Configuration: 12 x Large Form Factor (3.5")
Hard Disk Drive: 12 x 3.5" SAS/SATA Drive
Graphics: Matrox G200eW Video Controller
Optical Drive: N/A
Network Ports: Intel i350 Quad Port Gigabit Ethernet
Power: 900W Hot-Swappable PSU
Expansion Slots: 6 (x8) PCI-E 3.0 slots
Interface Ports: 2 x SATA 3.0 ports (6Gb/s), 4 x SATA 2.0 ports (3Gb/s), 8 x SAS2 ports (6Gb/s), 4 x RJ45 Gigabit Ethernet LAN ports, 1 x RJ45 Dedicated IPMI LAN port, 9 x USB 2.0 ports total (4 rear + 4 via header + 1 Type A), 1 x VGA port
Remote Management: Nuvoton WPCM450 Controller
Bezel: N/A
Rack Rails: Configurable

My concerns and questions about the setup are:
  1. The motherboard detailed (X9DRH) doesn't seem to be specific, and I'm not sure if there are denominations of this board that will or won't work well
  2. Will the hard disk controller work with FreeNAS? It doesn't seem to specify what the controller is, so I'm having a hard time cross-referencing with the HCL
  3. The specs state that it does multiple RAID levels, for ZFS I know I don't want any RAID controller and I'm not sure how to tell if I will be able to turn this off (or potentially replace it?)
  4. There are 8xSAS2 ports…IIRC, I should be able to install SATA drives into these ports? I'm interested in the potential to upgrade to some SAS drives in the future, but right now I've got SATA
  5. When it comes to a CPU, I was looking at something like the Intel Xeon E5-2630L V2 - 6-Core 2.40Ghz, which I think should be supported by bhyve?

Many thanks for any pointers, answers, and past relationships with Bargain Hardware.

Ben
 
Joined
Oct 18, 2018
Messages
969
Hard Disk Controller: SATA 3.0 6Gb/s with RAID 0, 1, 5, 1
  1. Will the hard disk controller work with FreeNAS? It doesn't seem to specify what the controller is, so I'm having a hard time cross-referencing with the HCL
  2. The specs state that it does multiple RAID levels, for ZFS I know I don't want any RAID controller and I'm not sure how to tell if I will be able to turn this off (or potentially replace it?)
Based on the above information it is basically impossible to tell. Generally folks stick with HBAs and not RAID controllers. When a RAID controller is used it is generally flashed to IT mode to completely disable any RAID functionality, essentially making it an HBA.

Adaptec 71605 SAS/SATA 6Gb/s PCIe Gen3 RAID Kit
I think this may be the card in question, I took that from the link you posted. The Adaptec 71605 is the important bit there. It looks like this card may have a driver for FreeBSD, but I couldn't easily tell if it could be flashed to IT mode. If I were you I'd research whether this card can be flashed to IT mode and whether it works in FreeBSD as used by FreeNAS. If the answer to either isn't a definitive "yes" I'd suggest you replace it with a relatively cheap, used, and known-to-work HBA.

Hard Disk Drive: 12 x 3.5" SAS/SATA Drive
The drives they sell seem to be all 7.2k, 10k, or even 15k rpm drives. I'd avoid those and go with 5.2k 5.9k rpm drives to reduce the heat.

The motherboard detailed (X9DRH) doesn't seem to be specific, and I'm not sure if there are denominations of this board that will or won't work well
I am skeptical of that website. There are a lot of little things about it which have my bs detectors on guard.

They list the board as an X9DRH with 6x PCIe expansion slots. I can't find a single X9DRH board with fewer than 7 expansion slots. Some places they specify 6x (x8) PCIe slots; why make no mention of the x16 slot? Every single X9DRH has an x16 slot that I could find.

It also lists as having an i350 quad port NIC, but I don't see any X9DRH boards with a quad port i350, they seem to be either dual 10GBase-T via an X540 or dual 1G ports via an i350.

I'd bet it isn't the X9DRH-7F, otherwise the onboard Broadcom SAS controller would be listed.
Likewise, I doubt it is the X9DRH-iTF otherwise the 10GBase-T via the X540NIC would be listed.
The X9DRH-7TF has 10GBase-T AND the Broadcom SAS controller, neither of which are listed.

So, we are left with the X9DRH-iF or X9DRH-iF-NV. The latter supports NV-DIMMs, which I see no mention of in the listing. My money would be on the X9DRH-iF. But then, they are trying to sell you a server product; why didn't they mention it up front? And why are they wrong about the number of 1G ports?

Here is another consideration; what is the BIOS version? Earlier versions won't support all cpus such as the V2 cpus.

I have no experience with that site so I can only go off of what I'm reading right now. Perhaps I'm misreading the site or not looking hard enough for other X9DRH boards. As it stands now though I'd be extremely suspicious of this site and make them prove to me they can be trusted. What happens if you email them? Do they give you exact model numbers? Or do they give you the run around or some other vague information?

If you truly can't find appropriate parts elsewhere within your budget and you feel compelled to buy from this site I'd ask around and see if you can get a trusted, strong rec for them and still press them for specific details.

Anyway, I'm being a grump, I know. Either I'm bad at reading specs, bad at searching for boards, too easily annoyed by poorly stated specs from a retailer, or they need to get their act together. I half hope someone replies saying "Um, it is clearly {exact board model number}". :)

As for your build; unless you're certain you need all that CPU and memory maybe you can get by with a single-socket x9 or x10 board? How much total storage are you aiming for and how frequently will you be using VMs and other ram hungry activities beyond base FreeNAS operation? I only ask because it may help you stay in your budget more if you get a single-socket machine.
 

Ben Lavery

Cadet
Joined
Nov 8, 2016
Messages
6
Thanks very much for your reply, PhiloEpisteme!
I'm glad of your skepticism, I don't really want to buy something that isn't right :)

In terms of reputability, they seem to have a positive eBay score (99.7% positive feedback on 2,049 ratings over the past 12 months), but having put some feelers out I don't personally know anyone that has dealt with them.

As for your build; unless you're certain you need all that CPU and memory maybe you can get by with a single-socket x9 or x10 board? How much total storage are you aiming for and how frequently will you be using VMs and other ram hungry activities beyond base FreeNAS operation? I only ask because it may help you stay in your budget more if you get a single-socket machine.

I think I can probably get by with a smaller CPU, I currently spin up a FreeBSD-CURRENT VM maybe once a month or so mainly to bump the version of www/gohugo. I'd like to move this off of my aging desktop (2011 iMac) and onto something else since poudriere builds of lang/go can take a while and bogs down my desktop.
My (probably poor) rationale for the CPUs here were I was hoping that by going bigger I'd capture the correct requirements for bhyve using an older CPU, but I am naïve about such things! Also, getting support for the RAM I already have felt like a win, but may not be of course!

Storage right now is 16TB, broken down into an existing RAIDz (4x3TB) plus 4x1TB spare drives. IIRC there is ~6TB of data on the existing RAIDz, but as mentioned I'd like to make that more flexible by converting it into mirrors - for that I know I'm lacking in some storage since I'll need to move the data off of the RAIDz in order to break it and rebuild it.
The goal is for an 8 drive system, minimum. 12 drive bays give me some flexibility to use smaller drives (cheaper to replace, less time resilvering).
When the RAIDz was initially created, I was just storing some media on it. Today my wife has a need to store photos (mainly RAW) and related files, I'm not exactly sure about the growth rate of that, she is also not exactly sure - I just get moaned at that she's running out of space...

With the about of bays in mind, the machine in question was also interesting. As long as I can get a similar form factor, a single-socket system would be probably be more ideal!

I will keep looking, and I might also see if I can email the aforementioned company to see if I can get any specifics.

Many thanks once again!
Ben
 
Joined
Oct 18, 2018
Messages
969
My (probably poor) rationale for the CPUs here were I was hoping that by going bigger I'd capture the correct requirements for bhyve using an older CPU, but I am naïve about such things!
I think this really depends on what you plan to do. If you're only doing occasional testing etc I don't think you'll need a ton of power.

Storage right now is 16TB, broken down into an existing RAIDz (4x3TB) plus 4x1TB spare drives.
Can you clarify this? Which vdev layout are you using? RAIDZ could refer to RAIDZ1, RAIDZ2, or RAIDZ3. Also, are those spares just spare drives you have sitting outside your case? What are they spares for? At 1TB they can't be used as hot spares/replacements for your 3TB disks.

I'd like to make that more flexible by converting it into mirrors - for that I know I'm lacking in some storage since I'll need to move the data off of the RAIDz in order to break it and rebuild it.
Correct. Depending on the chassis you get this may be trivial. If all of the drives can fit in the chassis simultaneously you're all set.

The goal is for an 8 drive system, minimum. 12 drive bays give me some flexibility to use smaller drives (cheaper to replace, less time resilvering).
Not a bad idea to have a spare bay or two to burn in new drives etc. When considering drive size compare the cost/TB vs how much storage you need vs the vdev layout you want. Strips of mirrors require more disks for equivalent storage as compared to many-disk RAIDZ1|2|3 vdevs.

Today my wife has a need to store photos (mainly RAW) and related files, I'm not exactly sure about the growth rate of that, she is also not exactly sure - I just get moaned at that she's running out of space...
Keep in mind that you don't want to go beyond 80% use if possible to keep performance reasonable. And you never want to get 100% full or even close, that can cause all sorts of access headaches including issues mounting your pool. So, for example, if you were to build a RAIDZ2 array from 7 2TB disks you'd have 12TB of disks, roughly 10TB of actual usable space, and goal of not using more than 8TB following the 80% rule.

Also, have you considered backups? Keep in mind that no amount of redundancy in a FreeNAS box is a replacement for a backup. If your server catches fire etc you're out of luck. Generally you want at least 1 backup, if not 2, and at least 1 backup off-site. This, of course, depends on your budget and how irreplaceable the data is.
 

Ben Lavery

Cadet
Joined
Nov 8, 2016
Messages
6
Thanks for coming back to me again!

I think this really depends on what you plan to do. If you're only doing occasional testing etc I don't think you'll need a ton of power.

That's fine then, as long as I can virtualise (needs to be so I can run FreeBSD -CURRENT), I don't need huge amounts of horse power.

Which vdev layout are you using? RAIDZ could refer to RAIDZ1, RAIDZ2, or RAIDZ3. Also, are those spares just spare drives you have sitting outside your case? What are they spares for? At 1TB they can't be used as hot spares/replacements for your 3TB disks.

IIRC, it is a RAIDz1. I'm 99% sure it is, but I don't currently have a machine I can use to test that theory...
The 1TB drives are just spare drives, I understand I can't use them as spares for the 3TB drives. I was thinking I could use them, along with some other drives to copy the current data off of the RAIDz1 so I can break it into mirrors.

Not a bad idea to have a spare bay or two to burn in new drives etc. When considering drive size compare the cost/TB vs how much storage you need vs the vdev layout you want. Strips of mirrors require more disks for equivalent storage as compared to many-disk RAIDZ1|2|3 vdevs.

My rationale for mirrors is that they are (generally) a smaller unit to deal with. Increasing storage is a case of upping the capacity of drives in the vdev, so three drives in a mirror (2+1HS) is going to be cheaper than 4+ drives in a RAIDz, though the overall cost/TB will be greater.

Keep in mind that you don't want to go beyond 80% use if possible to keep performance reasonable. And you never want to get 100% full or even close, that can cause all sorts of access headaches including issues mounting your pool. So, for example, if you were to build a RAIDZ2 array from 7 2TB disks you'd have 12TB of disks, roughly 10TB of actual usable space, and goal of not using more than 8TB following the 80% rule.

Been there, done that :) When I did my Solaris Internals course we did this to deliberately see how the system reacted. I've also seen some people let their systems get into this state "in the wild"...

Also, have you considered backups? Keep in mind that no amount of redundancy in a FreeNAS box is a replacement for a backup. If your server catches fire etc you're out of luck. Generally you want at least 1 backup, if not 2, and at least 1 backup off-site. This, of course, depends on your budget and how irreplaceable the data is.

Yes...I have considered it. At this time my budget won't stretch to having a second system.
The short term goal is for consolidation of a number of hard drives and CDs that are scattered across various places in the house. External hard drives have already died, and CDs are getting more and more difficult to read thanks to optical drives going out of fashion (my USB DVD drive has recently died and been replaced).
Consolidation + redundancy offers better protection than I currently have.

A short to mid-term goal would be to properly look at the data and pick out what is important and irreplaceable then look into either either standalone offline hard drives, or more likely some form of cloud service.

A longer term goal would be to mock a similar system to have at a relations house. What I've had "The Conversation" before, my mother in law has been willing to host such a system so long as it was too big or loud.
 
Top