ESXi/FreeNAS Build check/suggestions.

[ebzero89]

Hello everyone,

I'm looking to build a fairly stable base installation that is a bit on the quiet side as I'm in an apartment, but can be moved to a bigger/louder case whenever I have room for a rack or actually have a closet or some other place to put it that's out of the way. Additionally, I'm initially building this with 8, 10TB disks in mind, in a single vdev Z2 configuration, but would like the ability to expand to more disks once I have the space for a 4U case that could hold them all. Eventually I'd like to have a total of 3 8 disk vdevs in Z2. As far as backups go my level of hoarding data is still manageable so, a cloud service will work fine till I can build a mirror offsite somewhere. Edit: The main purpose of the storage is for media and file storage.

My general plan since space is at a premium but money isn't necessarily, is to have this box be a general "lab" box that's running ESXi, with a couple of (possibly mirrored) SSDs for datastore storage. As VMs on there I'd like to have the following:

• FreeNAS VM, with an HBA passed through
• A plex VM that can do hardware transcoding
  • This will require another PCI-E slot and a GTX 1050 at minimum for full-chroma 4k transcoding
• A services VM
  • Monitoring services
  • DNS services
  • Other services as I'd like to set them up
• A general development VM for preparing for some work related automation projects and for my own learning.

So first things first, is there some catastrophic reason I shouldn't go with the above plan? If so, what can I do to mitigate to reach my general goal? If I have to break it out I'm willing to, but would generally prefer not to.

With the above in mind I have selected the following parts and gotten general prices. For reference, all the prices I'm using are based around cursory Ebay searches so generally all parts will be used:

• Case - Fractal Design Define R5 or R6 ~200USD right now
  • I'm willing to go with a different full or mid tower case as long as it has room for at minimum 10 total drives (8 spinning, 2 SSDs)
  • Due to possible heat concerns having the ability to put a lot of fans in would be nice
• CPU - Xeon E5-2670v3 ~170-200 USD
  • 12 cores should be plenty for what I'm planning but if anyone has better suggestions at a decent price point please let me know
  • Will require a cooler for sure, any suggestions there?
• RAM - 64 GB total DDR4 ~500 USD
  • Should be more than enough RAM based on the guidelines provided in the documentation I could find, might be a little on the low side for the other VMs depending on use.
• HDDs/SSDs
  • I already have these (thanks black friday)
• HBA - LSI 9211-8i ~20-50 USD
  • Is there any reason to go with a newer one of these cards? Is there a more preferred option in general for HBAs now?
  • Will be flashed once I have it in hand.
• Video Card - GTX 1050 ~75 USD
• Mobo - Supermicro X10SRL-F ~250 USD
  • Chosen mostly for the number of PCI-E slots available and for the number of RAM slots.
• PSU - ???
  • I'm still rather unsure about the power requirements for several of the parts, my best guess comes to around 750 watts or so based on other build's calculations with comparable parts.

So that's pretty much it, this is my first "homelab" style build so I'm sure there's something I'm not accounting for or could be doing better. I'm pretty much open to any suggestions. If this isn't the right place for this please let me know and I'll delete. Thanks for taking the time to read this!

 

[kdragon75]

Are you planning on using Free NAS as a datastor for ESXi? Are you planning on running nested ESXi host for a proper vSphere lab? How will tour VMs be specd? I need the details of your plan before I can say it's sane. Also keep in mind that at least ESXi datastor must be on a different hba/raid controller than FreeNAS.

 

[Dan Tudora]

hello
you a perfect candidate for http://www.napp-it.org/doc/downloads/napp-in-one.pdf

it's, I think a more to read but .....

 

[ebzero89]

Are you planning on using Free NAS as a datastor for ESXi? Are you planning on running nested ESXi host for a proper vSphere lab? How will tour VMs be specd? I need the details of your plan before I can say it's sane. Also keep in mind that at least ESXi datastor must be on a different hba/raid controller than FreeNAS.

I don't plan to use FreeNAS as a datastore for VMs. Ideally, I wouldn't need too very much space per VM so a 256GB SSD datastore should suit me fine. I do not plan to run nested ESXi, however the development VM will likely use quite a few containers that are spun up and deleted relatively frequently.

The VMs will be specced like so:

• FreeNAS - 2 core 16 GB RAM
• Plex - 2 core 8 GB RAM
• Services - 2 core 8 GB RAM
• Dev - 6 core 32 GB RAM.

This is pretty fluid, so I'm not super tied to any of these specs and can always add more RAM. The main issue is that CPUs become prohibitively expensive after the 12c mark so that's the real limiting factor here.

 

[kdragon75]

This is pretty fluid, so I'm not super tied to any of these specs and can always add more RAM. The main issue is that CPUs become prohibitively expensive after the 12c mark so that's the real limiting factor here.

At a glance, it seems ok. If that motherboard supports real raid, you will be able to mirror the SSDs for the datastore. I think you may find that while 6 cores is acceptable for the dev VM, everything will be more responsive with only 4 cores assigned to it. Oh thats 12 physical cores... You're fine there.

 

[kdragon75]

hello
you a perfect candidate for http://www.napp-it.org/doc/downloads/napp-in-one.pdf

it's, I think a more to read but .....

I don't think this is what they are looking for. Please don't push other products on the forums unless you can point out specific reasons FreeNAS is not a good fit. This is a FreeNAS forum, not STH.

 

[ebzero89]

At a glance, it seems ok. If that motherboard supports real raid, you will be able to mirror the SSDs for the datastore. I think you may find that while 6 cores is acceptable for the dev VM, everything will be more responsive with only 4 cores assigned to it. Oh thats 12 physical cores... You're fine there.

Ok cool. Another question would be whether or not it's worthwhile to bite the bullet and go for a newer CPU from a later generation with a similar number of cores. Is there some reason I'd want to spend the extra cash? Another option I had been considering was a Threadripper based machine, but as I understand it, there are still some outstanding issues with compatibility using ESXi and this machine, especially for hardware passthrough.

 

[Ericloewe]

hello
you a perfect candidate for http://www.napp-it.org/doc/downloads/napp-in-one.pdf

it's, I think a more to read but .....

That's a fairly unimpressive manual...

Please follow the advice you were given above:

I don't think this is what they are looking for. Please don't push other products on the forums unless you can point out specific reasons FreeNAS is not a good fit. This is a FreeNAS forum, not STH.

 

[Dan Tudora]

I don't think this is what they are looking for. Please don't push other products on the forums unless you can point out specific reasons FreeNAS is not a good fit. This is a FreeNAS forum, not STH.

TNX kdragon75
my english is bad, I now
I do not want to make a "competition", but I make some test and I just play with other "solution". It's not bad.
I follow many posts of yours and ericloewe too.
no offence
but is a OVA solution
and for newbies is a good start point to learn
I use FreeNAS at work and at test system for my "heart"
do not blame for "the try". maybe I read too much :D
TNX for your works and Merry Cristhmass

 

[ebzero89]

If there are no major issues with my plan as laid out here, I'll go ahead and start looking for deals on parts. I appreciate you guys taking the time to look this over. Thanks!

 

[Bozon]

I don't plan to use FreeNAS as a datastore for VMs. Ideally, I wouldn't need too very much space per VM so a 256GB SSD datastore should suit me fine. I do not plan to run nested ESXi, however the development VM will likely use quite a few containers that are spun up and deleted relatively frequently.

The VMs will be specced like so:

• FreeNAS - 2 core 16 GB RAM
• ...

I'm a newbie, so I hesitate to give much advice, but I know I read somewhere in the manual, or forum that FreeNAS doesn't play well on VM's. Did some of the other text in the post preclude this from being an issue. I wanted to make sure I understood, the issues, or the nonissues.

Thanks.

 

[ebzero89]

I'm a newbie, so I hesitate to give much advice, but I know I read somewhere in the manual, or forum that FreeNAS doesn't play well on VM's. Did some of the other text in the post preclude this from being an issue. I wanted to make sure I understood, the issues, or the nonissues.

Thanks.

As I understand it there is even official documentation (found here) that says this is now supported. My general understanding of the situation is that there will be issues if I use it as a datastore and precautions would be needed in that case. Otherwise, passing through an HBA allows it to be just as though I was running FreeNAS on a separate physical machine. If my understanding is incorrect please let me know.

 

[Ericloewe]

As I understand it there is even official documentation (found here) that says this is now supported. My general understanding of the situation is that there will be issues if I use it as a datastore and precautions would be needed in that case. Otherwise, passing through an HBA allows it to be just as though I was running FreeNAS on a separate physical machine. If my understanding is incorrect please let me know.

Basically, running FreeNAS in a VM is best left to those who already understand what is involved in such an endeavor. A "if you have to ask..." scenario, if you will, at least when it comes to the basics.

VMs give clueless users a ton of new ways to shoot their feet off, hence the advice.

 

[ebzero89]

Basically, running FreeNAS in a VM is best left to those who already understand what is involved in such an endeavor. A "if you have to ask..." scenario, if you will, at least when it comes to the basics.

VMs give clueless users a ton of new ways to shoot their feet off, hence the advice.

I completely understand the hesitancy to recommend it. It seems like it's a huge task to understand FreeNAS as a whole (any ZFS concepts in general really) so I can see why you'd want to avoid muddying the waters with virtualization when people come here for advice/troubleshooting. I'm really familiar with virtualization (VMware more so than others) and non-FreeNAS storage concepts due to my daily work, and since this is just for home use, the worst thing that can happen is I have to restore from backup (always remember to keep good, tested, backups!). That being said I don't intend to use it as a datastore for the VMs because that's just asking for big trouble. I just really wanted to make sure that my part selections were adequate for my general needs or if I was missing something major that would be a deal breaker for my plans.

 

[Ericloewe]

As long as you have enough RAM, and CPU cores, along with a HBA that you can pass through, you should be set, hardware-wise.

Of course, what works best with your hypervisor is a complicated question, as it would be for bare metal FreeNAS.

(any ZFS concepts in general really)

ZFS is pretty easy to understand and incredibly easy to administer. I'd recommend Lucas and Jude's ZFS books to get a good notion of how the various features work together to simplify pretty much everything. The FreeBSD handbook also has a good ZFS primer.

We should get something like that for the forums, and people (me included, I must admit) have worked on a new one to replace Cyberjock's old guide, but I don't think anyone's really close to finishing one.

 

[ebzero89]

As long as you have enough RAM, and CPU cores, along with a HBA that you can pass through, you should be set, hardware-wise.

Of course, what works best with your hypervisor is a complicated question, as it would be for bare metal FreeNAS.


ZFS is pretty easy to understand and incredibly easy to administer. I'd recommend Lucas and Jude's ZFS books to get a good notion of how the various features work together to simplify pretty much everything. The FreeBSD handbook also has a good ZFS primer.

We should get something like that for the forums, and people (me included, I must admit) have worked on a new one to replace Cyberjock's old guide, but I don't think anyone's really close to finishing one.

Yeah compatibility matrices are always somehow full of information except the one piece you really really need lol. Thanks for the book recommendations I'll give them a read while I wait for my parts to come in! I really appreciate everyone's help and comments. Thanks so much! I'll post up how the build went once it's done.