Best practices / Best Way to Build a Private Cloud Environment with FreeNAS

[Alexandre Trevizoli]

Hi people. I already read a lot of posts in FreeNAS forums in more than 3 or 4 years that I'm here, I've already read a lot of documentation of ZFS, FreeNAS, Storages, but I never close my mainly doubt about what is a best way to build a private cloud environment using FreeNas(ZFS) and Windows Hyper-V or XenServer.
I want to sorry to my poor English, my main language is Portuguese(Brazil), but I really like to get this question resolved.
Maybe jgreco or others gods on ZFS could help us on it.
What I know:
1) Lots of ECC RAM
2) ZFS is COW, a lot of articles say it's not the best for block Storage
3) Use 10-25% of space
4) ISCSI is not the best to use with ZFS
5) RaidZ2 is not good for Block Storage

Ok...So if you will build a Cloud (with a lot of VMs) what should be used? Could someone explain best practices? Give some scenarios?

I did this way, but I don't think is correct way

I have 4 Servers Quad Processors 128GB RAM each, 2 SSD 120GB(Mirrored) for Windows (I'm using Windows 2012R2 on this scenario) Hyper-V
Storage FreeNas:1 Dual Processor 24 Bay, 64GB Ram, 12 x 4TB Hard Disk (2 VDEV RAIDZ2).
10Gbit Switch SFP+ and each server 10Gbit Card.
Following articles and posts, I shouldn't use RaidZ2, instead of this, Mirror should be a better option.
On this scenario we are small company, not rich, but want to do a good job. Mirror will lost 50% of total space correct? Maybe use mirror for OS and Programs and use RaidZ2 for VM Data? What we can do to have a good cost/benefit? Should use L2Arc and ZIL? Some articles say: instead to expend with L2Arc, spend with RAM.
How to deal with fragmentation? We know that it will occur right?
Like almost us is not rich, 10-25% of use of zpool space only is not a good option($$$)? Example: we have 4 x 4TB disks, so 16TB Total, but we did mirrored so we got 8TB to use 800MB to 2TB only will be very expensive. Have other option to not lost all this space? Should us not use ZFS? What FS big company like Amazon, Microsoft and others use?
So if we will use VMs which protocol should be used? CIFS, NFS or ISCSI or other?

We need good performance, because our VMs not can be slow, the expectation is they behave as if they were physical machines, so how build this environment to do it?
I know is a lot of question, but I just put they together, because we can find some answers in separate threads, never all together to build a sample scenario.

I want to thanks everyone on this forum, that help users like me and thank you FreeNAS developers to make it, to allow persons that can buy very expensive Storages(brands like EMC, DELL, HP, others) to build environments that was impossible without FreeNAS.

 

[Dice]

Following articles and posts, I shouldn't use RaidZ2, instead of this, Mirror should be a better option.

correct.

Mirror will lost 50% of total space correct?

correct. plus a fraction for ZFS overhead.

Maybe use mirror for OS and Programs and use RaidZ2 for VM Data?

Performance = mirrors.
VM data as in ...media storage for a transcoding server - fine.
VM data as in actual data shared through iSCSI or NFS with any aspiration of any sort of performance - get mirrors.

What we can do to have a good cost/benefit?

Lower expectations or throw more harddrives&SSDs&RAM at the server.

we did mirrored so we got 8TB to use 800MB to 2TB only will be very expensive.

Good. This is the way to go.
Very expensive? Not in relation to the performance you CAN achieve by proper design.

Have other option to not lost all this space?

The ZFS trade-off is:
You get outstanding data protection and performance thanks to clever use of RAM and cache, with non-enterprise storage equipment. This is traded for using more but cheaper storage devices.

Should use L2Arc and ZIL?

Yes. Begin by searching the forum.

Some articles say: instead to expend with L2Arc, spend with RAM.

Yes. Begin by searching the forum.

How to deal with fragmentation? We know that it will occur right?

Gobs of free space.

So if we will use VM's which protocol should be used? CIFS, NFS or ISCSI or other?

This you need to research.
Begin by searching the forum.

our VM's not can be slow, the expectation is they behave as if they were physical machines, so how build this environment to do it?

Mirrors.
Multiple vdevs.
Gobs of free space at all times.
Max out RAM.
Size L2ARC accordingly.
Get a NVMe quality SLOG.

The hardware guide resource will point you in direction of what SSDs that are of interest.

 

[Alexandre Trevizoli]

Thanks Dice....I will follow your suggestions

 

[Stux]

Mirrors provides the best random access performance.

For VMs, especially multiple VMs, random access performance, Ie IOPS is king.

The less full your pool, the less of a problem fragmentation is.

This is why for a VM storage pool the recommendation is mirrors and 50% empty.

An alternative is SSDs. SSDs provide orders of magnitude better IOPS meaning you could get away with less vdevs, perhaps Raidz2 and more fragmentation and usage, and still have better performance. All up it may be cheaper to go with SSDs.

Anyway, if you're serving VMs, you're probably serving iSCSI or NFS, and sync writes are probably an issue. A high performance PCIe NVMe SSD with data loss protection as a SLOG can improve sync writes.

iSCSI needs more ram than a vanilla FreeNAS and the morecram you have the more ARC cache you can have. I would not suggest 'maxing' ram on some systems as 2TB might be an excessive amount of RAM ;)

But L2ARC needs more ram to maintain so is not a way to reduce memory usage.

64 GB is probably good.

 

[Dice]

Overall we agree -

I would not suggest 'maxing' ram on some systems as 2TB might be an excessive amount of RAM ;)

heheh :P

64 GB is probably good.

Revisiting OP, I think the 128GB available is a good start. 64GB would be to settle too low.
Yet... I don't think there are additional benefits immediately by passing 256GB.
This is the range to aim for.

But L2ARC needs more ram to maintain so is not a way to reduce memory usage.

Which is also why it is appropriate to aim for the next-to-overboard RAM.
L2ARC will be key in this setup granted the (so far) very limited spindle count in relation to the workload.

cheers