Finally building a FreeNAS box

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
I'm been planning on building a FreeNAS box for almost 10 years. I bought a Norco RPC-3216 3U rackmount server case to start the project back in November 2009. It's been sitting collecting dust (a lot!) as well as a triple redundant power supply I shoved in it way back then. It's got an old Asus mobo (P5K?) in it and some drives that I haven't pulled out yet. I probably fired it up and played with it at one time (It's been 10 years!) but it's time to get serious now.

I've been running a Drobo Pro (walking really as it has notorious slow read/write speeds) for ten years and it has served me well all this time. I'm just REALLY sick of its slow speed and I'm already pushing it over supported capacity so I figure I'll build a nice new (sic) box that can handle a lot of storage and move my Plex server and torrent downloader over to the new NAS out of a VM running on my main Dell PowerEdge R710 that is also due for a refresh in the future.

I'm going to move 25-30 TB on to the new NAS as soon as it's built. (I'm sure it will take weeks.) So I figure 5x10 TB drives is a good immediate start and I can expand more later (12 bays available!). I'll re-purpose the Drobo to (slow) backup storage.

I've read the HCL, but it's out of date. I've read some of the forums and see many references to SuperMicro boards. I'm more of an Asus fan and would prefer to go that route. So before I hit the purchase button on Amazon or eBay, I figure I should hit up the FreeNAS community for advice before I push those little orange boxes in ignorance.

This is my current thinking:
MOBO
- with dual gigabit NICs though it would be nice to have 10 GB...but there's always the add-on card upgrade path for the future. Currently I'm thinking of the Asus WS C246 Pro but I'm not set on it and up for suggestions.

CPU - I'd prefer 9th gen, but I guess 9th gen Xeon's aren't out yet. So 8th gen it is. I need QSV to lighten the load on Plex, I'm thinking the E-2144G. It's 3.6Ghz, 4 cores, 8 threads, and sucks up 71 watts. I'd prefer to go 65W, but from what I can see that would only get me a 4 core 4 thread with the E-2104G. My server closet already sucks up an estimated $300+/mo in juice! (My math could be wrong.)

RAM - 16 GB to start or 32 GB ECC DDR4. I just plan to buy a brand I trust at the best price point. (Lifetime warranty is a big plus!)

HDDs - I'm ready to push the orange button on 5 Seagate IronWolf 10TB NAS or 5 WD Red 10TB NAS, whichever is cheaper at the time of purchase. I'd be fine with the Enterprise versions, but just don't want to drop the extra ~$30@.

BOOT - The mobo will have unused SATA ports since I'll be using an HBA, so I'll most likely throw in a small SSD for boot up; possibly a M.2 if it's available on there. I have a spare SSD someplace or I'll upgrade some other computer with a bigger one and use it's old one.

HBA - Here's where I don't have a clue yet. I've got a Adaptec ASR-5805-512MB in the case now but inclined not to use it since it's RAID and only 3 Gb/s. I could use it temporarily and swap in something less mature in the future... but I have fears of swapping controllers! My ears are all open on suggestions here.

Now let the discussion/comments/suggestions being!

... and TIA!
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I've read the HCL, but it's out of date.
You should read the Hardware Recommendations Guide. You could say it's somewhat out of date because it doesn't yet mention Coffee Lake. I've actually had the Coffee Lake bits written for a few months now, but I just cannot gather the patience to even begin to piece together the Xeon Scalable line.
Still, all the principles apply and all X11 hardware is still more than viable.
If you want to know more about the Coffee Lake stuff, you can check out the Quick Hardware Guide, which I have updated (by saying nope to Xeon Scalable - if anyone wants Xeon Scalable, they can pay iX or figure it out on their own). You can also check out the X11 Coffee Lake board overview.


Currently I'm thinking of the Asus WS C246 Pro but I'm not set on it and up for suggestions.
Doesn't seem like a great option. As per above, the X11SCH-F is probably what you should be aiming for.

CPU - I'd prefer 9th gen, but I guess 9th gen Xeon's aren't out yet. So 8th gen it is. I need QSV to lighten the load on Plex, I'm thinking the E-2144G. It's 3.6Ghz, 4 cores, 8 threads, and sucks up 71 watts. I'd prefer to go 65W, but from what I can see that would only get me a 4 core 4 thread with the E-2104G. My server closet already sucks up an estimated $300+/mo in juice! (My math could be wrong.)
I'm not sure the video encoding stuff is working yet. There was some talk about it recently, but I didn't follow it closely. Definitely research before committing to it.

(Lifetime warranty is a big plus!)
I'm not sure it's worth anything beyond a warm fuzzy feeling.

BOOT - The mobo will have unused SATA ports since I'll be using an HBA, so I'll most likely throw in a small SSD for boot up; possibly a M.2 if it's available on there. I have a spare SSD someplace or I'll upgrade some other computer with a bigger one and use it's old one.
I'm partial to NVMe M.2 because it doesn't use SATA ports. Since you'll be using an HBA, SATA M.2 could be interesting, but the Supermicro X11SCH-F doesn't have the socket wired for SATA, unfortunately.

HBA - Here's where I don't have a clue yet. I've got a Adaptec ASR-5805-512MB
Definitely not. Hardware RAID is very bad. Adaptec is atrocious. Very bad times atrocious is a huge nope. Guaranteed pain and almost certain impossibility of any serious operation.
There's really only one answer here and it's LSI SAS HBAs, specifically using the SAS2008, SAS2308 or SAS3008 controllers.
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
Thanks.

HBA - I just ordered a LSI 9201-16i LSI00244 .

MOBO - RE: X11SCH-F - I'd prefer something that takes 8 DIMMs and 4 PCI-E. The extra DIMM slots would allow me to populate 4 now, and 4 later. I'll be using one PCI-E for the LSI board which leaves one free that might get a 10Gb Ethernet card in the future (or maybe a 2.5 or 5GB). I'd like to have an extra one for some unknown unplanned use in the future.

CPU - E-2144G ordered

BOOT - SOD doesn't work right? I can't remember if I read it was a h/w or s/w problem. So I'm thinking $22 for a https://www.amazon.com/SanDisk-128GB-Ultra-Flash-Drive/dp/B07855LJ99/ sounds good. Holding on order until I hear feedback.

DRIVES - Watching prices and will order the WD 10TB Reds sometime soon.

RAM - on hold until I pick a mobo
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,462
SOD doesn't work right?
SOD? If you mean a DOM, that works just fine, as does an SSD. As does a USB stick, for that matter, but those tend to have a high failure rate; as a result, we tend to recommend SSDs or DOMs.
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
DOM - yes. Thanks. 16GB is the "more room" requirement. Most company's minimum requirements leave a lot (a little?) to be desired. Is bigger better? 32GB? 64GB? 128GB? What's the real world user recommendation?

Any other Supermicro X11 mobo rec's?
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
If I do RAIDz(1) can I change it to RAIDz2 on the fly later?

Actually I'll probably never do it as this is mostly a media server with a few planned containers on it, nothing critical on it. Main server VM backups and user data are backed up to a NETGEAR ReadyNAS 314 and/or NETGEAR ReadyNAS Pro 4 (ready for retirement) and will more than likely get moved to the drobo when when it's re-purposed.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
Nope. Once a vdev has been created, you can't change the type without backing up the data and starting over.

RAIDz1 isn't recommended for drive sizes over 2TB. Even though you'll have a backup (which is GREAT), do you really want to restore 30GB, if two disks were to fail?

Six disks in RAIDz2 is one of the sweet configurations.

If I do RAIDz(1) can I change it to RAIDz2 on the fly later?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
DOM - yes. Thanks. 16GB is the "more room" requirement. Most company's minimum requirements leave a lot (a little?) to be desired. Is bigger better? 32GB? 64GB? 128GB? What's the real world user recommendation?
The real-world recommendation is get a small SSD - so 120/128 GB. Realistically, 32 GB would be pretty good, but SSDs are generally less of a pain than USB flash or DOMs (neat as DOMs are, although an M.2 SSD does the same job of keeping things neat , but more reliably and at a lower cost).

If you have an old SSD lying around not in use that isn't near death, that would also work well.
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
Thanks. I'll have plenty of free SATA ports no matter what board I buy, so SSD is fine with me and I'm sure I've got one or two lying around.

Question: Do plug-ins and Docker containers land on the boot drive by default? If so, I'd lean towards a bigger SSD.
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,924
Nope, nothing lands on the boot drive. You can put the system on it only (but it's normally on the pool where it is automatically backed up daily).
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
Thanks. Hmmmmmm. So if I throw in a 128 or 256 (more likely) GB SSD, can it be partitioned and used for something else or does the excess just become unused space?
 

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,924
Not unless you want to hack something.

Search for a post by @Stux which speaks to putting swap on the boot drive to get a flavor.
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
BOOT - Since the any extra space on the boot drive seems to be a complete waste and boot speed isn't important, it makes more sense to me to go back to the a small USB stick stuck into the system. I just ordered a SanDisk 64GB Ultra Fit USB 3.1 Flash Drive - SDCZ430-064G-G46 for $9. How can I go wrong? If it dies, no big deal I can have it backed up and image another of tens of normal size USB flash drives either temporarily or permanently that I have lying around.

MOBO - I just ordered a X11SCL-F. I feel more comfortable knowing that I could throw in 2x 10 GB NICs in the future then I care about the SATA ports. I'll plug in 1-2 SSDs soon, basically a temp processing and/or caching drive. That might be expanded to 3 total in the future, but I doubt it. (LSI 9201-16i LSI00244 will be using one of the available slots for now.)

HDDS and RAM still to go. ...and probably a good CPU fan.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
If you go that route, get another SanDisk and mirror it with the first one.

If it fails, pull the bad one out and boot off the good one.

We don't recommend imaging boot devices. Do a fresh install and restore a copy of your configuration file.

How can I go wrong? If it dies, no big deal I can have it backed up and image another
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
RAM - ordered, used Hynix 16GB DDR4 PC4-2133P 2Rx4 ECC Registered 288Pin DIMM HMA42GR7AFR4N-TF 198 from eBay

HDDs - I screwed up. Last week WD RED 10TB drives could easily be found for $219, now they're $250+... in holding pattern watching prices/availability.

M.2 - considering ordering Crucial MX500 1TB 3D NAND SATA M.2 Type 2280SS Internal SSD - CT1000MX500SSD4 to use as tier 1 temp storage download drive. No mirror, no backup, just temp storage for any processing of media files before moving them to main storage. Thoughts?
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
You can mirror a USB drive?
You can mirror just about anything larger than a floppy that the OS recognizes as a block device. ZFS doesn't discriminate.

M.2 - considering ordering Crucial MX500 1TB 3D NAND SATA M.2 Type 2280SS Internal SSD - CT1000MX500SSD4 to use as tier 1 temp storage download drive. No mirror, no backup, just temp storage for any processing of media files before moving them to main storage. Thoughts?
It's valid. Only you can really say if it's worth it.
 

bcurran3

Dabbler
Joined
Jul 13, 2019
Messages
13
USB MIRRORING - Very cool!

M.2 - went with a Samsung 970 EVO 1TB - NVMe PCIe M.2 2280 SSD (MZ-V7E1T0BW). Worth it for me to have fast IOPS for downloads and transcoding video on fast temp storage without affecting performance of those old fashioned spinny things used for more permanent streaming storage. I almost went with the 2TB version and then thought, "Overkill!" as it's supposed to be temp/processing storage only. I'm hoping one day in the future I don't have to eat my word, but in 5 years time a 4TB M.2 SSD will probably be cheaper than the 1TB I'm getting now. Easy upgrade if needed.

CPU FAN - Just ordered a Noctua NH-L12S 70mm Low-Profile CPU Cooler with Quiet 120mm PWM Fan. I really wanted one that blows backwards, but didn't really see anything <=100mm that I know would fit with no problems in the 2U case; though I'm guestimating I could go about 110mm. (I haven't cleaned up the case or put the mobo in yet to do a real measurement.) If I don't like it, it's Amazon, easy returns. Hello Kohl's.

HDDs - still bumping my head against the wall trying to find good pricing on WD Red 10TB NAS Hard Drive - 5400 RPM Class, SATA 6 Gb/s, 256 MB Cache, 3.5" - WD100EFAX $249. Might just say effit and go with Western Digital Gold Enterprise Hard Drive 12TB from eBay for $299 which is a great price considering Amazon sells them for $479 . Downside is quantity limited to 2 (per week?)... weighing in the worth ATM.
 
Last edited:

Velcade

Contributor
Joined
Mar 28, 2019
Messages
108
Might just say effit and go with Western Digital Gold Enterprise Hard Drive 12TB from eBay for $299 which is a great price considering Amazon sells them for 9.

Just make sure they're retail and not OEM drives. I got burned once; bought some WD Gold drives from Amazon and they were OEM drives (no warranty) and they had bad sectors. Big waste of money. Amazon also didn't list them as OEM drives. I was pretty bummed.
 
Top