First FreeNAS Build

[analyst79]

Hi Everyone! This is my first official build. I'm not 100% new to FreeNAS, but it has been some years and want to get back in and reap the benefits! The following upfront needs for this server are:

• NFS/CIFS file shares for my computers
• Media Storage movies/tvshows/music, the typical data
• Virtual Machine Storage ~10-15VM's with maybe 5-7 on max (Test lab)
• Plex Server with between 3-5 streams initially
• Nextcloud (Future Project)
• Zoneminder Storage (Future Project)

Here is what I am looking at in terms of hardware:

• Xeon E3-1230v6 1151
• Supermicro X11SSM-F
• 4x 16GB PC4-2400 CL17 ECC Memory (Maxed)
• 2x Dell H310 cards - Will be flashed to IT mode
  
• 2x Kingston 120GB Mirrored for boot
• 5x 4TB WD NAS Red Sata (RaidZ2) 1 pool
• 5x 2TB WD NAS Red Sata (RaidZ2) 1 pool
• 5x ??? Enterprise SSD pool (Future want/pool) ~suggestions on drives?
• Chelsio T520-SO-CR for maximum bandwidth straight to the ESXi Server
• HP NC364T PCIe 4Pt Gigabit Server Adptr
• Rosewill RSV-LV4500 Case
• Corsair RM850x 80PLUS Gold Certified

My plans are to direct connect my ESXi server via a Twinax cable from my Chelsio Card to my Mellanox 10GBe card in my ESXi server and use the 1GB connections to the network for Plex, shares, etc.. as well as storage vlan (1GB). Also have a Synology DS410 that once this server is build and complete, I will upgrade with 4X 8TB WD NAS Sata drives for backing up critical data on the FreeNAS server.

Is there anything that I am missing? As stated there will be VM's running from the FreeNAS server and won't be hammering any resources on a consistent basis except when testing. The Plex streaming will be sent to Kodi Media Centers with the Plex plugin/app, Fire TV and Stick, and mobile devices (Fire Tablet's and android phones) with the current 3-5 streams which may or may not be simultaneous (more of the latter). Will most likely do the Plex Pass once this has been all completed as the offline sync capability sold me.

Realistically for the drives, I will have the 5x 4TB drives for the first pool, then build the additional pools later as I purchase those drives.

Any thoughts, recommendations, etc... is greatly appreciated.

Thank you in advance!

 

[Inxsible]

Everything looks good. Seems you have done good research. You could save some money by going with only 1 SSD for boot, but in the grand scheme of things, it won't save you a whole lot.

You could also save some money by going with an older board with DDR3 RAM as DDR4 is currently very expensive but that is only a suggestion as you haven't mentioned any budget. If you do have enough money to go with X11 boards and DDR4, then go with that.

 

[analyst79]

Everything looks good. Seems you have done good research. You could save some money by going with only 1 SSD for boot, but in the grand scheme of things, it won't save you a whole lot.

You could also save some money by going with an older board with DDR3 RAM as DDR4 is currently very expensive.

Thanks Inxsible. In doing my further reading and research, with what I have in terms of RAM for this server to store VM's for usage across my 10GBe DAC link to my 1 ESXi server with the considered 5-7 VM's potentially being on, but not all being constantly being actively in use be sufficient? Most of these VM's will be minimal server installs with maybe 3-4 server development builds.

 

[Inxsible]

with what I have in terms of RAM for this server to store VM's for usage across my 10GBe DAC link to my 1 ESXi server with the considered 5-7 VM's potentially being on, but not all being constantly being actively in use?

That would depend on quite a number of things. Will your VMs have dynamically assigned resources or static? If static how much are you planning to give to each VM? You have a total of 64GB of RAM. You would just need to divide that by the number of VMs + FreeNAS.

Another thing is, I am not completely familiar with the HP quad NIC. It says "Two Intel 82571EB" -- Do they use intel NICs? If so, it would be good. Another option is Intel i340-t4 or Intel i350-t4 for about the same price.

Plex should be fine for the 3 streams. If you are transcoding 3 streams and running multiple VMs at the same time it might be a bit of a stretch.

 

[Inxsible]

Do you have a budget in mind?

If you have enough money, you might want to look into a Supermicro 3U/4U hot-swap chassis. Better build quality than Rosewill overall but for a fair bit of coin too.
Another option is Norco 4U or Chenbro. You can get great deals on these on ebay if you are willing to go the used routes, but even new Norcos/Chenbros will cost less than the Supermicros. These will also give you more drives available -- approx. 16 for 3U and 24 for 4U compared to 15 for the Rosewill case.

 

[analyst79]

That would depend on quite a number of things. Will your VMs have dynamically assigned resources or static? If static how much are you planning to give to each VM? You have a total of 64GB of RAM. You would just need to divide that by the number of VMs + FreeNAS.

Another thing is, I am not completely familiar with the HP quad NIC. It says "Two Intel 82571EB" -- Do they use intel NICs? If so, it would be good. Another option is Intel i340-t4 or Intel i350-t4 for about the same price.

Plex should be fine for the 3 streams. If you are transcoding 3 streams and running multiple VMs at the same time it might be a bit of a stretch.

I keep the Plex transcoding and VM usage in mind. Anything that I have "production" is on local array of ESXi server anyways.

 

[Inxsible]

I keep the Plex transcoding and VM usage in mind. Anything that I have "production" is on local array of ESXi server anyways.

Plex transcoding is more on the CPU than RAM. The amount of RAM shouldn't be a problem if you are not running all the VMs at the same time.

 

[Chris Moore]

Any thoughts, recommendations, etc... is greatly appreciated.

I have several suggestions but I have a good number of questions too.

My plans are to direct connect my ESXi server via a Twinax cable from my Chelsio Card to my Mellanox 10GBe card in my ESXi server

So, you are thinking of running iSCSI share over to the ESXi system? How is it connecting? Have you considered that you would need SLOG or L2ARC?

5x ??? Enterprise SSD pool (Future want/pool) ~suggestions on drives?

What is it for? How much capacity do you need?

Xeon E3-1230v6 1151
Supermicro X11SSM-F
4x 16GB PC4-2400 CL17 ECC Memory (Maxed)

Would you consider using a X9 series board with a Xeon E5 v2 processor? It would allow a lot more RAM max and you could use the Registered ECC memory that is much more economical on eBay right now? I would be happy to point out a good option if it would be something you would consider.

2x Dell H310 cards - Will be flashed to IT mode

You could use a single one to control all your drives if you use a server chassis that has expander backplanes instead of direct connect.

5x 4TB WD NAS Red Sata (RaidZ2) 1 pool
5x 2TB WD NAS Red Sata (RaidZ2) 1 pool

Do you choose these drives because you already have them?

Rosewill RSV-LV4500 Case
Corsair RM850x 80PLUS Gold Certified

This server chassis can take the place of the Rosewill and the power supply from Corsair. I happen to know the seller will take $350 for it because I bought one from him and it is a great deal because it gives plenty of room for expanding. I have a 12 drive pool for my main storage pool and a 4 drive pool for my backup pool and a 16 drive pool of mirrors for my iSCSI pool (for VMs) and still have 16 empty drive bays for future use.:
https://www.ebay.com/itm/Server-Chenbro-48-Bay-Top-Loader-4U-Chassis-NEW-in-Box-/253503074088

 

[analyst79]

Do you have a budget in mind?

If you have enough money, you might want to look into a Supermicro 3U/4U hot-swap chassis. Better build quality than Rosewill overall but for a fair bit of coin too.
Another option is Norco 4U or Chenbro. You can get great deals on these on ebay if you are willing to go the used routes, but even new Norcos/Chenbros will cost less than the Supermicros. These will also give you more drives available -- approx. 16 for 3U and 24 for 4U compared to 15 for the Rosewill case.

I'm content with the 15bays. I'm not running anything crazy as most of this will be learning, except for Splunk, which to a point may be production for my network for log aggregation and analysis (also part of my learning lab)

I thought about everything with what I have so far in terms of a data footprint (~4TB of data on my Synology). Planning on digitizing my entire DVD library roughly ~1000 dvd and ~60 3D Blu-Ray movies. I don't need ungodly amounts of storage in the near future, but this does get addicting lol. But with Zoneminder, and Nextcloud, I can understand the need for more drives. But this is for my home use and would really grab my Synology (backups) if the house, heaven forbid, burns down.

 

[analyst79]

Do you have a budget in mind?

If you have enough money, you might want to look into a Supermicro 3U/4U hot-swap chassis. Better build quality than Rosewill overall but for a fair bit of coin too.
Another option is Norco 4U or Chenbro. You can get great deals on these on ebay if you are willing to go the used routes, but even new Norcos/Chenbros will cost less than the Supermicros. These will also give you more drives available -- approx. 16 for 3U and 24 for 4U compared to 15 for the Rosewill case.

I'm content with the 15bays. I'm not running anything crazy as most of this will be learning, except for Splunk, which to a point may be production for my network for log aggregation and analysis (also part of my learning lab)

I thought about everything with what I have so far in terms of a data footprint (~4TB of data on my Synology). Planning on digitizing my entire DVD library roughly ~1000 dvd and ~60 3D Blu-Ray movies. I don't need ungodly amounts of storage in the near future, but this does get addicting lol. But with Zoneminder, and Nextcloud, I can understand the need for more drives. But this is for my home use

I have several suggestions but I have a good number of questions too.

So, you are thinking of running iSCSI share over to the ESXi system? How is it connecting? Have you considered that you would need SLOG or L2ARC?
A: This will be direct connect via Twinax Cable (Freenas - Chelsio 10GBe Card) <-> (ESXi - MNPA19-XTR). I have not considered SLOG or L2ARC because I am not anticipating running more than 7 VM's max for it's own pool with those resources being 8GB of memory max on 3-4 of those VM's and anticipated with their minimal load, would not need to use a SLOG device. (I'm just speculating at this time)


What is it for? How much capacity do you need?
A: This is for home use for media storage mostly. But want to do Plex Server, NextCloud, Splunk, VM storage, possibly Sonarr.

Would you consider using a X9 series board with a Xeon E5 v2 processor? It would allow a lot more RAM max and you could use the Registered ECC memory that is much more economical on eBay right now? I would be happy to point out a good option if it would be something you would consider.
A: from your answer, it sounds like I may have already exceeded the thresholds of my proposed build.

You could use a single one to control all your drives if you use a server chassis that has expander backplanes instead of direct connect.

Do you choose these drives because you already have them?
A: Just from experience and good reviews. Again, I am open to suggestions that won't break my wallet :)

This server chassis can take the place of the Rosewill and the power supply from Corsair. I happen to know the seller will take $350 for it because I bought one from him and it is a great deal because it gives plenty of room for expanding. I have a 12 drive pool for my main storage pool and a 4 drive pool for my backup pool and a 16 drive pool of mirrors for my iSCSI pool (for VMs) and still have 16 empty drive bays for future use.:
https://www.ebay.com/itm/Server-Chenbro-48-Bay-Top-Loader-4U-Chassis-NEW-in-Box-/253503074088

Thanks Chris for the input!

 

[analyst79]

Plex transcoding is more on the CPU than RAM. The amount of RAM shouldn't be a problem if you are not running all the VMs at the same time.

I read that the E3-1230v6 should be sufficient for this. It's only my wife and I that would be beating this up and even at that, my wife would be on it way less than I would be. My son however may increase this using this resource as he gets older (he's still a toddler, so this implementation would still be under utilized by him)

 

[Inxsible]

A: Just from experience and good reviews. Again, I am open to suggestions that won't break my wallet :)

The thing is, we don't know how big your wallet is in order to break it. You haven't told us your budget :)

 

[analyst79]

Do you have a budget in mind?

If you have enough money, you might want to look into a Supermicro 3U/4U hot-swap chassis. Better build quality than Rosewill overall but for a fair bit of coin too.
Another option is Norco 4U or Chenbro. You can get great deals on these on ebay if you are willing to go the used routes, but even new Norcos/Chenbros will cost less than the Supermicros. These will also give you more drives available -- approx. 16 for 3U and 24 for 4U compared to 15 for the Rosewill case.[/QUOT

Under $3000. Everything that I looked at (some I already have) fall just below this so I am content unless there is an alternative that would give me more for less money.

 

[Inxsible]

I read that the E3-1230v6 should be sufficient for this. It's only my wife and I that would be beating this up and even at that, my wife would be on it way less than I would be. My son however may increase this using this resource as he gets older (he's still a toddler, so this implementation would still be under utilized by him)

Then you shouldn't worry about it. The processor is plenty sufficient to do it. Plus not all streams would be transcoding either.

 

[Inxsible]

Under $3000. Everything that I looked at (some I already have) fall just below this so I am content unless there is an alternative that would give me more for less money.

There are alternatives as @Chris Moore and I have suggested (X9 or X10 with DDR3), different chassis with expander backplanes etc. but are you willing to go that route?

 

[analyst79]

Then you shouldn't worry about it. The processor is plenty sufficient to do it. Plus not all streams would be transcoding either.

Sweet thanks! I read some older reviews on E3-12xx v4/5 and heard some pretty good feedback on Plex Transcoding with these processors. I decided to go with the v6.

 

[analyst79]

There are alternatives as @Chris Moore and I have suggested (X9 or X10 with DDR3), different chassis with expander backplanes etc. but are you willing to go that route?

I have do have some of this gear already particularity the memory and CPU that was re-purposed for this as well as the board.

 

[Inxsible]

You will save a lot of money if you buy the Seagate Barracuda/Seagate Desktop drives instead of WD Reds.

Some on these forums will lose their sh!t over using desktop drives in NAS application. But @Chris Moore and I both use Seagate drives in our boxes and have not had problems.

I have a Samsung Spinpoint (a Seagate Barracuda re-badge) in my NAS and a couple of Ironwolfs as well

 

[analyst79]

As for the SSD pool, I have read on other forums of people buying used enterprise SSD drives, which is ok with me as I will be using it for testing mostly. But they are ?? at this time because I won't be actively be thinking about it for a while.

 

[Inxsible]

I have do have some of this gear already particularity the memory and CPU that was re-purposed for this as well as the board.

Ok. well then that changes things. If you already have some components, then you should stick with that platform.