Low-capacity, expandable, responsive and small(ish) FreeNAS build in the planning.

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
Well, that subject line captures what I am looking to build at a high level so let me drill down on this. In this post I am hoping to verify a few assumptions I have made and also looking for suggestions for motherboard and CPU.

I am coming to terms with FreeNAS and hardware recommendations but I'm not quite there yet. For obvious reasons, I'd like to make some decisions by Black Friday! Normally I wouldn't mind ploughing a fair bit of money (relative, I know!) into this. But I am also building my main PC, for the first time in 7 years. As such - and with my NAS objectives in mind - I am looking to build a NAS server that doesn't break the bank initially but I can expand later. Thankfully, I suspect my initial needs play into that quite well, particularly in terms of disks.

So, with all that said and in no particular order...

Low-capacity, expandable
My initial needs are very low, in terms of capacity, compared to most people here. 1Tb of available storage would be perfectly OK for quite some time. So, my mind is thinking "Double that, for 2GB of usable storage and that should be good for a while".

But I do, of course, want redundancy and would like to be able to tolerate two drive failures. So I believe that implies RAIDZ2. In practical terms, would I be correct in saying I'd need 4 x 2TB disks, which would give me 4Gb usable?

Responsive
I would like to edit video directly from over a share and, just cut to the chase, go with 10Gb. I accept that this will have implications for my cost, but eager to understand my options. I may revisit this. By the way, if go this route I will also purchase a motherboard for my PC with 10Gb support.

I don't plan on running any significant workload on the NAS (no Plex, etc), with the possible exception of NextCloud. So I'm thinking initially of just 8GB RAM, since it's just serving a small amount of data. And I assume the CPU doesn't need to be particularly beefy (though I'd rather overdo it initially than have to upgrade later).

As I understand this, my needs suggest a SLOG is overkill, so don't plan anything there. But maybe I have misunderstood that, in relation to video editing.

Small(ish)
This is less important but I would like a relatively small build and quiet form factor. As a baseline, an maximum of 6 drives (eventually) would be OK, though having the option to add an external enclosure would be kinda nice, I guess.

So....

My recent research has proven useful for my PC build and I have most of the components decided there (AMD x570 build). But I am just starting out on my research for the NAS.

I've read quite a bit here but, respectfully, some of the information is fragmented and (occasionally) opposite ends of the spectrum (debate is good!). So I'd like to think initially about a suitable CPU and motherboard and then build around that. I'd appreciate any specific pointers for those two, based on the above.

Thank you.
 

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
A couple of other quick notes...

I will likely go with ECC memory and don't have any problem going straight to 16Gb if there's a valid reason to do so. The 8Gb was simply based off the small amount of data, but certainly don't want to cut corners if that creates problems.
 
Joined
Aug 8, 2019
Messages
21
For your needs, a Xeon-D build sounds pretty solid. Xeon-D is a nice, power-efficient platform that also happens to have built-in 10GBase-T support. Plus, it's available in a mITX form-factor, which is good for your size constraints. Here's a build I'd recommend based on your requirements:
  • The Supermicro X10SDV-4C-TLN2F is a mITX Xeon-D board that should work fine for your purposes. It'll host six SATA devices and the cheapest NVMe boot drive you can find (no need for speed, you're just trying to keep it from sharing one of the SATA ports). The downside of this setup is that you can't run a SLOG in the M.2 slot later, but that's a tradeoff you're making for a smaller enclosure. These run for around $350 USD right now, which, while not super cheap, is a relative bargain for a motherboard, CPU, and 10GBase-T NIC. The CPU is soldered to the board here, but it should provide enough juice for what you need and then some.
  • That Supermicro board can hold up to 64 GB of unregistered ECC across 4 slots. I'd grab a single 16 GB stick and throw it in there. More RAM gives you more ARC, which is the single best thing you can do for your performance. Having room to expand later is great, and those 16 GB UDIMMs are around $40 right now. No reason to skimp.
  • Fractal Design Node 304: This is a common small form factor mITX NAS case. It'll fit your board, a standard ATX power supply, and 6 3.5" drives. Should be perfect for your needs.
  • I'd consider going with bigger drives initially just because, as you said, there are some good deals to be had for black Friday. If you're willing to buy external drives and shuck them (remove the drive from its enclosure) Best Buy in the U.S. has a deal where you can get a 12 TB white-labeled WD Red for something like $180 plus taxes. You've got a shorter warranty, but you're looking at savings of something like 50% over dedicated internal drives.
It's also worth thinking about pool topology, though. I'd generally say that if you're going to be starting out with a RAIDZ2 array, you're going to want to fill out all 6 bays to begin with. You can't expand a RAIDZ2 vDev after you create it, so if you go with your current four-drive plan you're going to end up with a four-disk RAIDZ2 vDev and a two-disk mirror if you eventually add those two other disks. That's going to lead to you having less usable space than if you had gone with six disks in RAIDZ2 from the beginning and a sort of red-headed stepchild of a pool.

However, it's also worth thinking about just having a pool of mirrors. I've been persuaded by this article that's generally what you want when you're designing a pool for future expansion, redundancy, and performance (as you are). In your situation, then, I'd begin with two 4 TB disks (or larger, if you shuck) and mirror those. You'll have 4 TB of usable space, and either disk can fail and be replaced. Then if you need more space, add another two disks, mirror those, and add them to your pool. Rinse and repeat. It's just an easier situation all around, and gives you a lot more flexibility as to when and how you upgrade--if you want to upgrade your storage on the RAIDZ2 pool you have to replace six drives to see the space, whereas on the mirrored pool you can either add two new disk or replace two existing ones. This is a somewhat nuanced discussion, though, so I'd be interested in seeing what other folks have to say.

One last thing: an expense folks sometimes don't think about with 10GBE is the cost of the switching hardware. Put simply, all the cheap stuff is SFP+ (fiber optic, power efficient, sort of a pain in the neck for wiring a home, field termination, etc.) and all the convenient stuff (10GBase-T) is expensive. Budget for something like $200-300 in additional cost for a 10GBase-T switch so you can actually use the NICs in your machines.
 

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
What a frickin awesome response!!! Really appreciate the time you spent putting this together. I am thoroughly enjoying researching all this before I make my informed decision :) Thoughtful and insightful responses like this increase that still further!

Anyway, I have some more reading to do and will let you know if I have any further questions.

Thank you again.
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
SFP+ and DAC you wont need a switch for one PC. For two PCs put a dual port card in FreeNAS box. For distances longer than 5M use fibre
www.FS.com is your best bet for DAC or transceiver/fibre. NICs is likely ebay. Intel is usually a solid choice, watch out for fake Chinese cards.

You don't mention a budget.

For directly editing on a share you should think about all SSD or two pools. Scratch editing on SSD and storage with HDD.
Most editors would put the scratch SSD in their workstation and have a sync task to the storage pool. That way you could use a local NVMe sync to HDD on FreeNAS.

For your listed needs now you could go all SSD for a reasonable cost. 4x 1TB RaidZ2, get higher TBW drives for editing.
  • Samsung 860 Pro 1TB $300 each
  • Intel 3520 960GB (ebay) $150-250 each (have to carefully choose seller, best to get new drives)
Possibly:
  • x2 SSD Mirror pool for editing
  • X4 HDD RaidZ2 Pool for storage
Also:
  • Backups
  • UPS
 
Last edited:

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
Thank you for the comments, @Jessep . I don't really have a particular budget in mind, since I have a long history of setting them and just ignoring them :) Seriously, I am still in the research phase and will be driven by what I need and/or want, rather than what would be a somewhat arbitrary guess up front. In fact, here's a good example of that, just 48 hours after I posted my question.

So, I've been revisiting this whole thing! Let me step back by saying I am already planning to purchase the remaining components for my desktop machine in Black Friday. That's PC 1 covered!

However, I have also been seriously researching two other needs I have - the aforementioned NAS server and a Blue Iris machine. Given that all three of these needs (PC, FreeNAS and Blue Iris) are new purchases and I'm OK having a bit of a splurge in a couple of weeks, it crossed my mind to see if I could unify any of these with virtualization. My research got me to the point that I do think that's an option (I know some folks disagree with this, but I've seen plenty of folks who are doing it fine, with some forethought).

However, despite enjoying that research and learning a lot, I've decided to simplify and just do the following:
  • New PC (as above)
  • Dedicated FreeNAS machine
  • Dedicated Blue Iris machine
And...that changes a lot :) The FreeNAS and Blue Iris requirements are 24 x 7 and obviously can be managed and used remotely. So for the first time, I am seriously considering tackling both of these with a small, home server rack and going with rack mounted device for both of these. Funnily enough I worked in rack-mounted server environments for many a year, but always on the software side, not hardware. As such this is a whole new world to me and I officially started my research about 1 hour ago :)

All that said, let me think again about that budget question. I have some of the parts for my PC already (case, power supply, etc). I will likely spend about another $900 in a couple of weeks (though hopefully less, given Black Friday).

If I could get all of the above set up for around $2,000 I'd be happy, which means about $1,100. I have NO idea if that is feasible, but I have a few things in my favor, at least initially...
  • The Blue Iris machine doesn't need to be very beefy (relatively low cost CPU and 8Gb would probably be OK initially)
  • I plan to offload archived clips from Blue Iris to the NAS device (Blue Iris manages "new" and "archived" folders and the latter would be in the NAS), That means I can get away with very little storage on the Blue Iris machine itself (even 256Gb would be plenty)
  • As a reminder, my NAS needs are really quite small initially. I am thinking through the comments by @reasonsandreasons about how to use my physical drives (mirroring may be OK) but even then won't need more than 2TB free space (though I will want to be able to expand easily)
  • I would hope to be able to install a single UPS that would serve both the FreeNAS and Blue Iris machines. We have a whole house generator that kicks in after 30 seconds, so that's all I have to fill
Anyway, I ramble on and my mind is going in all directions. As a baseline, I am researching something like a cheeky little 6U rack and see where that research leads.

I am open to any and all suggestions.

Thank you again!
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I have one of the D series here from Supermicro (see the sig) and agree that it's the bee's knees.

For a single user, the much less expensive D-1508 based board is likely a better choice due to the higher clockspeed of the CPU (which is beneficial for SMB and AFP).The D series really shines when it comes to these flex-ATX boards from SuperMicro. The smaller form factors are more problematic as the D series can do so much re: PCIe lanes yet the mini-ITX size limitations limit the amount of stuff that can be fitted around the CPU to take advantage of those PCIe lanes.

The Flex-ATX board I use allows for two PCIe 3.0x8 connectors, a mSATA (good for L2ARC), a m.2 PCIe 3.0 x4 (excellent spot for a SLOG), two bus-powered SATA slots (good for mirrored SATADOMs boot volumes), 18 additional SATA ports (16 via a LSI HBA), two SFP+ ports, etc. This board is essentially future-proof for the next decade or so. My only mistake was likely choosing the D-1537 over the D-1508; for my SOHO use, the less expensive board likely would perform better and cost less.

If you are sticking to mini-ITX, then I'd consider a C3xxx-based board since many of them are set up for NAS use with lots of SATA connectors. They use even less power than the D-series, though not much less, and your spinning drives will likely eclipse the CPU easily. For example, my system uses about 80-90 Watts on average, with spikes to 100 when the array is really getting hammered. The CPU is but a fraction of that.

You also want to be very clear about what an array running FreeNAS can and cannot do. Even with traditional HDDs and a single-VDEV, large files can transfer at hundreds of MB/s, provided your computer, etc. can handle it via SFP+, fiber, or whatever. However, should your workflow include a lot of small files, then things will likely slow down significantly, regardless of the hardware your throw at it. Depending on the use case, a SLOG, L2ARC, etc. may help but there is a limit to what spinning hard drives can do.

Thus, consider carefully your drive layout options as you embark, and ideally test rigs under your use-case conditions. The many different disk layouts (mirrored vs. Zx) or you could consider the use of multiple VDEVs to boost IOPS performance. However, as you already know, the more drives spin, the happier the electric company will be with you.

Last but not least, if high-speed is needed, I would consider the use of a very fast SSD-based scratch disk/array either inside the computer or an external enclosure via Thunderbolt 3 (which is basically PCIe 3.0x4). Then back up your work at your leisure to the NAS after hours. That would allow you to combine excellent responsiveness while you're working with a reliable Z2- or whatever NAS for long-term, error-free storage, at a potentially lower cost.
 

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
I haven't had quite so much in a while :) I'm an old-timer in terms of technology (Commodore Pet with memory doubled to 16k was my first computer!) but this is the first time I've built anything that has a server (hardware) context. It's been cool and I am super appreciative of the great responses to date. So, to my latest thinking...

MOTHERBOARDS, ETC
@Constantin , I am very interested in the D-1508. I came into this, with my desktop hardware mentality, just assuming I'd buy the CPU and the motherboard separately, so wasn't looking to spend quite that much on the motherboard alone. But since this comes with the CPU that obviously changed the dynamics. The fact that this has dual 10Gb ports is even better. At this point, that's my prime candidate.

That said, I am slightly confused by something @reasonsandreasons said, which - if I read that bullet correctly - implied that the X10SDV-4C-TLN2F is available for around $350. Was that used? I say that because a quick search suggest that it retails for around $550. Did I misunderstand?

By the way, I am now thinking generally of a budget of around $400 for the motherboard, assuming it has a CPU and 10Gb support. That seems very reasonable to me.

ZFS CONFIGURATION
In terms of FreeNAS I have been doing some more thinking about my pool and there have been some excellent points raised about RAIDz vs. mirroring. With some extra reading I have done, I am now considering 2 vdevs, each mirrored 4TB. So, that would mean 4 x 4TB drives, for a total free space of 8Gb. I'm good with that data efficiency.

It will take me quite some time to push the boundaries of that and, when I do, I understand I can simply add another couple of drives (for an extra mirrored vdev). So, if I add a couple of 6TB drives (mirrored) later, that would take me to 14TB in total. And this all seems very straightforward in terms of dealing with bad disks - I understand I just replace the bad disk and the resilvering from it's mirrored disk would be much quicker than if using RAIDz.

The biggest risk here is the loss of one drive and then, before I have replaced it, the loss of it's mirror. Since I understand the pool stripes across the two vdevs that means I will have lost 100% of my data in that regard.

Am I correct in all that?

BACKUP
I will need to have something solid in place to get my data offsite (the house burning down scenario). I am less sure about my immediate plans here, so would welcome advice.

I know there are lots of cloud choices. My requirements are simple (!) though not necessarily easy.
  • Totally automated. I know me - ANY form of manual process just won't happen (backing up to physical drives, for example).
  • Encryption at rest. Not sure of the full, end-to-end implications yet but that is on my radar.
  • Quick recovery / restore. I want to be able to get hold of any one file quickly, but also want to at least understand the implications of a completely lost NAS at home (stolen, house burned down, etc)
  • Snapshots. I want to be able to keep historical versions of certain (not all) my files. I understand this is a core capability of ZFS, so hoping this isn't too tricky.
  • Cost. Just want to keep it managable - no hard thoughts on this yet.
The journey continues... :)

Thank you again, all.
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
Mirrors will require a good regular backup (daily) as you will only have 1 drive redundancy on vdevs. If both drives drop off a single vdev you lose the pool. I would strongly suggest significant testing time to learn your system and look for points of failure before committing your data.
  • Nothing cloud will have fast recovery beyond single files.
  • Tape is your best bet, however the setup cost is steep. Drives can be several $k though tapes are cheap.
  • Next best is build (2) freenas systems and replicate between them. The second can be VERY low end as it purely backup. Celeron or even consumer gear. The hope is you will never need it, like car or life insurance.
 

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
Thank you, @Jessep. Yes, I'm going to be very cautious about all this and already have multiple backups of this data in non-NAS environments. I have a background in enterprise (software) so, aside from learning the hardware implications for this build and specific choices for FreeNAS configuration, I won't be putting my data at risk at any point :)
 
Joined
Aug 8, 2019
Messages
21
Oh, sorry about that! The MSRP on the X10SDV-4C-TLN2F is $550 or so last time I checked, but you can regularly get that board for $359 on eBay. If you can tolerate a bigger case, though, I'd say that a flexATX board like Constantin suggested is definitely better for long-term expandability and capacity. You can throw it in a Fractal Node 804 and get up to eight internal hard drives. Going up past mITX means you sacrifice some size, but considering in a server build you're actually likely to use the additional PCIe slots, the larger form factor feels like less of a waste.

On the ZFS point, you're right about the redundancy of the whole pool. ZFS is redundant on the vDev level, but not on the pool level. If any individual vDev fails your whole pool is hosed. That's part of the reason why, even with redundancy, it's important to have backups. You could write a book about which pool layout best protects against failure, but I tend to think that if you're designing your pool topology as your last line of defense you're going about it a bit backward. Make regular backups, and then use the layout that provides the best combination of fault tolerance, performance, expandability, and convenience for your needs.

As for those backups, here's what I've learned about a few of the more popular options:
  • Tape is the cheapest per terabyte, but it's also has high up-front costs (think thousands for a recent tape drive) and requires a lot of tape-swapping or an even more expensive tape library if you're dealing with a large backup. Good for enterprise, less good for time and cost-constrained home users.
  • Cloud storage is extremely convenient if you have fast upload speeds and affordable for smaller backups, but costs mount as you start wanting to store more data. Backblaze B2 is the cheapest option (it's also well-regarded and integrated with FreeNAS), and even there if you want to store 10 TB of data it'll cost you $50 a month, on top of $100 to restore the backup. They do have an option to ship you a hard drive with your data on it, though, which is helpful if you need to restore and don't want to wait for a multi-day download
  • Building a second FreeNAS system and storing it off-site is the option that makes the most sense for larger backups. For the cost of 12-18 months of cloud backup, you can build a low-end box with lots of storage and use it as a backup target. I'd caution against using non-server gear here, though, if only because it being off-site makes the IPMI interface that much more valuable.
 
Last edited:

markwill

Dabbler
Joined
Nov 12, 2019
Messages
35
Gotcha, @reasonsandreasons, on the eBay thing. That looks like a very good option and may well be where I end up.

Regarding the back-to-front research :), I am researching cloud-based backup in parallel, which is why I haven't focused on it here. But, as you say, it's very important. In fact, for my own needs, if I can't settle on a suitable offsite backup strategy soon I am not move to NAS at all. Through that lens, it's a primary factor for me.

I won't accept a manual backup of any nature because..I know me :) Backup to a FreeNAS server is something I might be interested down the road, but not for now.

As such, automated cloud backup is my focus. I mention this because I just wanted to highlight that I am looking at this holistically. In short, a home-only (no offsite backup) solution is not a solution I will accept.

I have opened a separate thread here and somewhat disappointed to learn that, until FreeNAS 12 or later, there's no clear way to retain encryption over the wire and at rest on a cloud service. I'd be interested in any thoughts you have on that, by the way.

I did have a look at Backblaze a month or so back and was generally impressed, at least at first glance. I may take another look as I decide on cloud backup.

As to the mirrored backup, I perhaps haven't explained myself very well (though hopefully what I wrote clarifies things). On it's own, I wouldn't consider mirrored vdevs to be adequate for my needs. It is only with an effective, well-planned and cloud backup strategy that I consider the inherent risks acceptable.

Now, that motherboard is looking very tempting!

Thank you again.
 
Top