Hardware refresh time, would appreciate thoughts and suggestions

[ranrotx]

Hi everyone,

Long time lurker, first time poster here. First off, thanks in advance for taking the time to read this. I have been using FreeNAS now for about 3 years--my original (and current build) has evolved over time and is based on a Supermicro A1SAi-2750F with 32GB RAM. It is currently hosting 1 zpool (4 x 2TB WD Red drives in a RAIDZ2 configuration) for file shares for my home network (home drives, photo archive, and Time Machine backups). There is a second zpool for jails (2 x 120GB SSDs mirrored). I had high hopes for jails and Docker but have come to the conclusion that I’d prefer instead to have the flexibility of running VMs outside of FreeNAS.

The current hardware is going on 3+ years old, so I’m wanting to refresh it before I start to experience failures. My plan is to virtualize FreeNAS under ESXi by passing through the controller for the zpool disks. I drew out a rudimentary diagram of what this would look like in practice.

I’ve got a rough framework for what I want to do, but I have not settled entirely on a processor/motherboard combination. And whatever I choose will possibly drive the need for additional controllers for the other disks (since at least controller will be passed-through to the FreeNAS VM running on ESXi).

I like the power consumption of my current Atom board, but it leaves a few things to be desired. The first is that it only takes the small form-factor SODIMMS (laptop memory basically). The challenge is finding these modules in an ECC configuration. Whatever board I buy should accept regular form-factor memory. The other issue is it’s currently in a Fractal Design Node 304 case (very cramped). The new build will be in something like a Fractal Design Define R5.

So far, here is what I am considering:

• Another Atom-based board
• Low-power Intel I3
• Low-power Xeon

In a nutshell, the design criteria is:

• Power efficiency
• Ability to handle household file sharing and run some other utility VMs (Unifi controller, Plex, ELK stack, AWS Storage Gateway perhaps)
• Allow pass-through of the SAS/SATA controller to the FreeNAS VM running under ESXi
• Nice to have: Ability to do some kind of mirroring for the VMFS volumes for the VMs. However, my plan is for my VMs to be ephemeral to where they can be reproduced with scripts. All data needing to be persisted will live on ZFS volumes shared from FreeNAS.

Does anyone have recommendations in terms of motherboards and processors? I'm partial to Supermicro, but having trouble deciding on what processor to go with (which will dictate the MB) that is low-power but still has good performance. Also depending on the motherboard, I may need to get a second SATA/SAS controller since I'll be passing one through to FreeNAS but will still need something to boot ESXi from as well as to host the VMFS volumes. In cases like this is it best to use the additional controller to pass-thru to FreeNAS or the onboard?

Thanks in advance!

 

[DrKK]

Ability to handle household file sharing and run some other utility VMs (Unifi controller, Plex, ELK stack, AWS Storage Gateway perhaps)

Virtualizing FreeNAS is always something it is better not to do, whenever that's an option, in my view anyway. Certainly, most of the things in your parentheses above can run just fine (better than fine, really) in jails. Any chance you can forego virtualization?

 

[ranrotx]

Virtualizing FreeNAS is always something it is better not to do, whenever that's an option, in my view anyway. Certainly, most of the things in your parentheses above can run just fine (better than fine, really) in jails. Any chance you can forego virtualization?

My main reason for wanting to virtualize FreeNAS is that I find FreeBSD jails to be limiting. I'm mainly a Linux person myself, and every time I have to do something inside of a jail (install software, etc.) I always run into quirks. Either the FreeBSD package isn't up to date or something else. Jails just really present more hassle than if I were to just run a Linux VM.

I've tried Linux under Virtualbox and bhyve on FreeNAS but it didn't seem like it was ready for prime time for me. And don't even get me started on vlan tagging. I tried to pass through a tagged VLAN to get one of my jails on a private VLAN on my network and I couldn't make it work for the life of me.

I figure I can mitigate a lot of the risks around virtualization by passing through the disk controllers to the FreeNAS VM so that FreeNAS has direct access to the disks. As long as FreeNAS can write directly to the disks without anything in between, what other concerns are there around virtualizing FreeNAS?

 

[Chris Moore]

We have had some really good discussions on configurations to support ESXi lately. You might find some pearls of wisdom here:

There is some great advice in this thread and references to other good threads:
https://forums.freenas.org/index.php?threads/freenas-for-iscsi-vmware-esxi-storage.61651/

This user was also asking about iSCSI for virtualization, but they are trying to do it on a tight budget.
https://forums.freenas.org/index.php?threads/backing-to-freenas-help-with-topology.61659

Then this one is one of my favorites because he goes into so much detail with photos and documentation. @Stux is very knowledgeable.
Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[diskdiddler]

Hi everyone,

Long time lurker, first time poster here. First off, thanks in advance for taking the time to read this. I have been using FreeNAS now for about 3 years--my original (and current build) has evolved over time and is based on a Supermicro A1SAi-2750F with 32GB RAM. It is currently hosting 1 zpool (4 x 2TB WD Red drives in a RAIDZ2 configuration) for file shares for my home network (home drives, photo archive, and Time Machine backups). There is a second zpool for jails (2 x 120GB SSDs mirrored). I had high hopes for jails and Docker but have come to the conclusion that I’d prefer instead to have the flexibility of running VMs outside of FreeNAS.

The current hardware is going on 3+ years old, so I’m wanting to refresh it before I start to experience failures. My plan is to virtualize FreeNAS under ESXi by passing through the controller for the zpool disks. I drew out a rudimentary diagram of what this would look like in practice.

I’ve got a rough framework for what I want to do, but I have not settled entirely on a processor/motherboard combination. And whatever I choose will possibly drive the need for additional controllers for the other disks (since at least controller will be passed-through to the FreeNAS VM running on ESXi).

I like the power consumption of my current Atom board, but it leaves a few things to be desired. The first is that it only takes the small form-factor SODIMMS (laptop memory basically). The challenge is finding these modules in an ECC configuration. Whatever board I buy should accept regular form-factor memory. The other issue is it’s currently in a Fractal Design Node 304 case (very cramped). The new build will be in something like a Fractal Design Define R5.

So far, here is what I am considering:

• Another Atom-based board
• Low-power Intel I3
• Low-power Xeon

In a nutshell, the design criteria is:

• Power efficiency
• Ability to handle household file sharing and run some other utility VMs (Unifi controller, Plex, ELK stack, AWS Storage Gateway perhaps)
• Allow pass-through of the SAS/SATA controller to the FreeNAS VM running under ESXi
• Nice to have: Ability to do some kind of mirroring for the VMFS volumes for the VMs. However, my plan is for my VMs to be ephemeral to where they can be reproduced with scripts. All data needing to be persisted will live on ZFS volumes shared from FreeNAS.

Does anyone have recommendations in terms of motherboards and processors? I'm partial to Supermicro, but having trouble deciding on what processor to go with (which will dictate the MB) that is low-power but still has good performance. Also depending on the motherboard, I may need to get a second SATA/SAS controller since I'll be passing one through to FreeNAS but will still need something to boot ESXi from as well as to host the VMFS volumes. In cases like this is it best to use the additional controller to pass-thru to FreeNAS or the onboard?

Thanks in advance!

Your old hardware is better than mine, argh

I'll give you fifty bucks :)

 

[DrKK]

I'll give you fifty bucks

"Does this proposition entail my dressing up as Little Bo Peep?"

--Fletch

 

[diskdiddler]

"Does this proposition entail my dressing up as Little Bo Peep?"

--Fletch

If it gets me that hardware for $50, I'll strip out of the bo peep costume too.

 

[ranrotx]

Thanks everyone for the suggestions and reading material! The build report from @Stux is really detailed and I will likely be putting that to use.

I'm still on the fence about whether to go with another Atom embedded board or to go with something similar but a single-chip embedded Xeon. I'm mindful of the power consumption but my fear if I go with the Atom is later I may wish I had built something with more processing power.

I could try to re-purpose my existing hardware by virtualizing on it (I would maybe need to buy an additional PCIe NVMe card or maybe a SATA M2 (I have to check to see if my motherboard has this port). However, that does seem risky and I'll probably end up building a new box. This way, I'll still have a copy of my data on the existing hardware that I can keep around long enough until I'm confident in the new build/architecture.

 

[loch_nas]

I think power consumption is totally overrated (for home use) and is not very important as long as you don't go crazy with high performance hardware.
In Europe everyone wants to save on small amounts of W for a very high cost. And I just don't understand that.

I would get a Denverton Mainboard if it serves the right features and space is a concern. But first see what performance and features you really need. After that you'll see which combination of CPU and mainboard delivers what you need.

 

[tvsjr]

Keep in mind that, for any system with a reasonable number of drives, the power consumption of the drives will dominate the overall system consumption.

 

[Chris Moore]

Keep in mind that, for any system with a reasonable number of drives, the power consumption of the drives will dominate the overall system consumption.

Absolutely true. My primary home NAS has 16 disks that draw about 8 watts each, or around 128 watts. The total draw (according to the UPS) is 215 watts. That means the entire rest of the system is only taking around 87 watts. That includes the System Board, RAM, CPU, SAS controller, two SAS expander backplanes, and all the fans.
For another data point, I have a server at work with 60 WD Red Pro drives in it and it draws somewhere between (I don't have the exact number handy) 600 and 700 watts, which is mostly the drives.

 

[wblock]

Use a normal SSD as a boot device rather than a SATADOM unless physical mounting space is extremely cramped.