Newbie trying to plan a build, need some guidance...

[c0rr0sive]

For a long time I have used hardware based RAID on a machine that's running ESXi, and as of late, my storage needs have outgrown what the machine can handle efficiently..

I have been toying with the idea of building a NAS that will accomplish the following...

1: Be somewhat energy efficient...
2: Not break my wallet - I can't go around dropping $10,000 all at once, but, I can build up over time if that's even possible. I rather not spend more than $200 on a CPU and not more than $200 on a motherboard if at all possible, however, it will need enough I/O to handle a NIC and SAS controller.
3: Be capable of handling an ESXi machine (or 3...) connecting to the storage array over iSCSI, I have 10 VM's on one host, and another host has 4 VM's, only one VM has any high disk I/O and that would be PLEX.
4: I would LIKE to have ZFS3 on any/all arrays as most of this data is actually work related. The more parity, the better, but I also need high performance as I work with both large and small files and will transfer a few hundred GB in a day most every day, some files are just a few KB, some are 30-50GB, sitting at 1Gbit or lower just isn't cutting it for me, I would like to see small file transfers go up a good bit more than 5-10MB/s as well, if at all possible!
5: I have a US-XG-16 on the way, so any recommendations on a 10Gbit NIC that doesn't break the bank? My main desktop WILL be connecting at 10Gbit to the switch, along with all servers.


I am thinking I will need two arrays, one for the ESXi hosts, and one for file storage, but all this talk about vdevs and other things just confuses me. I read, and read, but my study time on this is limited and tends to be after a 12 hour work shift.

So any recommendations on a build, such as amount of disks, what disks, setup of disks, and so on would be of great benefit, as well as links to resources that could help explain things. I basically need a full build and layout, and I know people don't like hearing, "there is no budget" when there obviously is, but at the same time I am not sure what my budget should be. I am looking at $3000-$4000 alone on disks (Been looking at 8TB Red Pros), and god knows what else for the rest of the hardware. I suppose the base build shouldn't go over the following...

CPU: $250
Mobo: $200
RAM: $200
PSU: ???
Case: Already got!
SAS Controller: I have an M5015 in one of my hosts, and an LSI 9220-8i, as well as an ancient 9650SE-12 laying around, would any of these suffice or should I look at something else?
NIC: $300? Would an Intel X710DA2 work well with FreeNAS?
HDD: I plan on WD Red Pros, probably 8TB ones, unless you have a better suggestion.

Also, please, suggest both ECC and Non-ECC RAM, I doubt I could afford to get ECC at the start of the build, but do plan to go ECC once it's up and my wallet stops throbbing.

Currently my VM's use about 6TB of storage, and I have approximately 16TB worth of files being stored on disks attached to Server 2012 for file-sharing purposes. I essentially want to double the storage space for both.

So, yeah, sorry if this post is in the wrong place, or if anything doesn't make sense, as I said, I don't have much free time, and tend to work my life away.

So, builds, ideas, suggestions for a tired grunt?

 

[Heracles]

Hi Corrosive,

In your list, you mentioned the need to host iSCSI storage for multiple ESXi and running multiple VMs. For that, RaidZ-x (either 1, 2 or 3) is not an option. A RaidZ-x vDev gives you the performance of a single drive no matter how many drives you put in it. You need the equivalent of Raid-10 here, grouping multiple mirrors in a single pool.

If a single mirror is not enough protection for you, you can always do 3-way mirrors : a mirror with 3 drives, each of them an exact copy of the others. It will speed up read requests but not write.

Local redundancy is surely something good for you, but you must realize that it will never compensante for an actual backup. No matter how robust a single FreeNAS server can be, it will always be destroyed by a physical incident like fire or by a single logical incident like misconfiguration or corruption.

You also said that your storage need is up to 22 TB as of now. To add space in a pool can be difficult, so better to plan for all the space you need form day one. Always try to keep your pool below 50% usage and plan for yearly increase in volume. You will also surely use snapshots, so remember that the space used by data you will delete will not be freed until all snapshots referring to it expire.

So that gives you : 22 TB as of now. I estimate annual growth to 2 TB per year (little less than 10%). For your server to last 5 years, that pushes the storage need up to 32 TB. To stay below 50%, you need 64 TB of storage. Better be safe than sorry, I would go with at least 80 TB.

Using the conventional mirrors (2 drives per mirrors), that gives you 10 mirrors of 8 TB each, so 20 disks. To increase performance, you may be looking at 15 mirrors of 6 TB drives. That is only for the main storage.

How will you do your backups ? Here, I have a second FreeNAS hosted hundreds of Km away and synchronized over VPN. That backup is only there to store the data and not to run any service. For that reason, it does not need the same speed, only the appropriate storage capacity. In your case, that would mean RaidZ-x could be an option for such a backup FreeNAS. Offline backups like tapes are another option.

As for RAM, I would say anything below 32 Gig would be clearly short. 64G would be appropriate, 96 or 128G would help you achieve better performance. About the ECC / Non-ECC, I am in the ECC clan.

So just here, to buy about 20 - 30 drives and 64 G of RAM, your budget is already under stress... I will let you think more about everything and up to you to post an update on your complete situation.

Have fun designing your system,

 

[pschatz100]

Why do you want Plex running in a VM? It would be more efficient and easier to manage if you run it in a jail.

 

[Heracles]

Hi pschatz,

As a security engineer, I too decided to prevent all direct connection from clients to my NAS server. The goal is to ensure there will be at least 2 or even 3 layers of security between clients / external systems and the most valuable and most sensitive asset in my network. This is why I run Nextcloud in a Docker host in ESXi and map the storage from FreeNAS to that host. These two are separate by my internal FW running pfSense. Another plus is that my Q-Radar SIEM can have a better view of the activity and flag me in case of incident long before an intruder can threaten the FreeNAS.

The same apply for multimedia distribution.

Sure a Jail is a valid security mechanism, but that's it. It is ONE valid security mechanism. By using a Dockerized Nextcloud outside of FreeNAS, a logical incident would need to break in that container, then break out of the Docker container and then break in the FreeNAS. Until that, an intruder in the Docker host could read the data but not destroy them because snapshots are out of reach of that Docker.

So Yes, there is surely a very good reason to completely insulate an asset like FreeNAS.

 

[c0rr0sive]

Hi Corrosive,

In your list, you mentioned the need to host iSCSI storage for multiple ESXi and running multiple VMs. For that, RaidZ-x (either 1, 2 or 3) is not an option. A RaidZ-x vDev gives you the performance of a single drive no matter how many drives you put in it. You need the equivalent of Raid-10 here, grouping multiple mirrors in a single pool.

If a single mirror is not enough protection for you, you can always do 3-way mirrors : a mirror with 3 drives, each of them an exact copy of the others. It will speed up read requests but not write.

That sort of sucks, but understandable, so it would look something like the following with 8TB disks?

Code:

| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |

Set those three groups to Raid 1 for extra parity, then combine the three with 0 for performance and that would give me the storage capacity of 3 disks, correct? This is just to get an understanding as I have never setup anything past 0/1/5 before.

Local redundancy is surely something good for you, but you must realize that it will never compensante for an actual backup. No matter how robust a single FreeNAS server can be, it will always be destroyed by a physical incident like fire or by a single logical incident like misconfiguration or corruption.

Sadly, offsite backup wont be possible, I live in a remote location and the only type of connectivity available with in about 10 miles is Satellite Internet, which has a very low monthly data cap. If I expand at 2TB/year, it would hammer the cap 24/7.

Semi-local backup however is an option, we have been looking at building a shed in the next few years, I could build a second system and house it there later on. My question is, can FreeNas do Replication to another server like Server 2012? Of course this wouldn't prevent major natural disaster from taking my data, but to be honest, if a tornado was to come through, I most likely wouldn't make it if my home was to collapse so the data would be moot.

You also said that your storage need is up to 22 TB as of now. To add space in a pool can be difficult, so better to plan for all the space you need form day one. Always try to keep your pool below 50% usage and plan for yearly increase in volume. You will also surely use snapshots, so remember that the space used by data you will delete will not be freed until all snapshots referring to it expire.

Never used snapshots before as they haven't really been of use or importance for me, unless you can give me a good reason to do so once this is built?

Using the conventional mirrors (2 drives per mirrors), that gives you 10 mirrors of 8 TB each, so 20 disks. To increase performance, you may be looking at 15 mirrors of 6 TB drives. That is only for the main storage.

Since I plan on 3 drives per mirror, that would total 30 drives, so between $5000 and $7500 depending on drive costs at time of purchase if I go with 8TB disks... Or if I go the 6TB route which would add upto 45 drives if im not mistaken, $9000+... I think I would probably go the 8TB with 10 mirrors route, and deal with the performance that would give me.

Going based upon what I stated earlier, the logical layout would look a bit like the following?

Code:

| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |
|                   |
| (HDD + HDD + HDD) |

To top it off, this would strictly be for the actual file server and not the iSCSI target? I would do a similar setup for the iSCSI target, yes?
Looks like I may have to build a shelf to accommodate all those disks.

How will you do your backups ? Here, I have a second FreeNAS hosted hundreds of Km away and synchronized over VPN. That backup is only there to store the data and not to run any service. For that reason, it does not need the same speed, only the appropriate storage capacity. In your case, that would mean RaidZ-x could be an option for such a backup FreeNAS. Offline backups like tapes are another option.

This is something I will have to look at in the future, currently, this build will appear to hurt a bit more than anticipated, offsite truly isn't an option for me.

As for RAM, I would say anything below 32 Gig would be clearly short. 64G would be appropriate, 96 or 128G would help you achieve better performance. About the ECC / Non-ECC, I am in the ECC clan.

I would love to go ECC at first, but like you said, my budget is under stress (more like over stressed).

Why do you want Plex running in a VM? It would be more efficient and easier to manage if you run it in a jail.

It's easy enough to manage as is? I have been running PLEX on a Win7 VM for 5 years now, I don't want or need PLEX running on a machine that's dedicated to other tasks. If PLEX was to cause some form of serious crash, that could potentially take system resources that it shouldn't and cause the FileServer to struggle during other operations, granted I am not sure how FreeNAS handles things like that, it's just something I dont want to migrate to unless it gave a serious boost, and the only boost it would give is my electrical bill having to run two machines with beefy processors. It's just as easy to manage inside a VM on another machine, where if the OS of said machine fails, the host wouldn't allow it to consume more than the maximums that I have already set AND where a single higher end CPU that isn't heavily used 24/7 is already present with a few spare cores.

 

[Heracles]

Hi again Corrosive,

Set those three groups to Raid 1 for extra parity, then combine the three with 0 for performance and that would give me the storage capacity of 3 disks, correct?

Just to be sure, because you made an example of a 3 by 3 matrix... I will re-use your other example of a 3 by 8 matrix. So Yes, that 3 by 8 matrix does reflect what I described as 3-way mirrors : each group of 3 disks represents a single vDev, offering the capacity of a single drive but having 2 redundant mirror copies of that drive. These mirrors are then grouped in an equivalent of Raid-0, so without any redundancy between each others. You can not group vDevs in a structure that would offer extra redundancy. vDevs are always grouped without redundancy. Redundancy is always only internal to each vDev. Should all the 3 members of such a single group fail, the entire pool would be lost. But honestly, should that happen, all the drives will fail from the same cause, like fire.

My question is, can FreeNas do Replication to another server like Server 2012?

For sure you can do something like rsync between a FreeNAS and a Windows server. As for me, I do ZFS send / receive from FreeNAS to FreeNAS, but there are other options for sure.

Never used snapshots before as they haven't really been of use or importance for me, unless you can give me a good reason to do so once this is built?

Snapshots are very powerful and very useful. Snapshot works like this :
Your data is in a first state at moment 0.
You take a snapshot at that moment 0. It is done instantly.
Once the snapshot is taken, you modify a first file, then add another and delete another one at moment 1.
The original version of the modified file is preserved in the snapshot. So is the deleted file. So should you change your mind or end up having deleted that file by accident, you can easily recover both.
At moment 2, you wish to preserve the new file but to recover the deleted one. All you need to do is to clone the snapshot, extract the data you need and delete the clone. The snapshot keeps existing, so does your actual data and you now have the recovered data you were looking for.

Snapshots are no backups because they are part of the same system as the data. Should something happen to the system, the snapshot will go down the drain with the system as well.

To top it off, this would strictly be for the actual file server and not the iSCSI target? I would do a similar setup for the iSCSI target, yes?

No. That pool would be designed to answer all your needs at once. It would be fast enough for your iSCSI and big enough for any other data you mentioned. You would just create different datasets in that pool for these 2 very different needs.

I have been running PLEX on a Win7 VM for 5 years now,

It is also very easy to run Plex in a jail as a plugin, but do as you wish. As I said, to prevent direct client access to the NAS is easy to justify.

So you now have a better understanding of what you are about to deal with. Feel free to ask more questions and for more validation as your design progress,

 

[c0rr0sive]

Ok, a few last questions before I go back to my drawing board...

Would an AMD R3 2300x be sufficient as far as a CPU goes? I am starting to lean towards an AsRock X470D4U2-2T for the motherboard as that would give me the two 10Gbit NIC's, and enough PCIe for an HBA or two, it would also enable the use of ECC later on for me when my wallet recovers. But that's only if I can find this board in stock, seems it's hard to find, and when you do it's out of stock.

Since it looks like I would be going with a secondary enclosure to get to the total disk count and to have extra room, any recommendations on an HBA with external and internal connectivity? Or should I just use my M1015 with a SAS expander and a few adapters to give me external ports? If I go with an expander, which one would you recommend if you could?

Also, would an SSD or nvme SSD assist any? Could it be setup as a write back cache?

 

[Heracles]

Hi again Corrosive,

Would an AMD R3 2300x be sufficient as far as a CPU goes?

any recommendations on an HBA with external and internal connectivity?

Unfortunately, I do not have experience with systems like this one. My systems are much smaller and I always used only Intel CPU... I am pretty sure many other ones will be able to provide you with better advice about these pieces of hardware...

My personal feeling is that the CPU should be OK, HBA are to be favored instead of SAS expander and that once you have these 64+ G of RAM and a pool of many mirrors, an SSD as a write cache will probably not make a big difference.

But again, on these points, you may be better to listen to someone else's opinion if they have more specific experience than mine about that.

Good luck,