Looking for configuration advice

[Eddie4]

Hello every one, am trying to figure out how to best setup my storage for my virtualization server. My hardware looks as follows:

2x Xeon six core
64 GB Ram ECC
Perc 6i inc 256MB + battery
4 sas 2TB disks
2 sata 2TB disks (Yah i was told it was 6x sas it wasn't)
1 NVMe 256GB Samsung disk

Please note that this is the entire VM platform which means there needs to be at least 80% RAM left for VM's. I have been running multiple setups but am still unhappy with the write performance.


Setup 1
ESXi + freenas on USB stick
Perc pci passthrough to freenas VM
ESXi NFSv3 mounted
Raidz1 6x2TB
8GB dedicated vMem
(NVMe drive wasn't here yet)

Setup 2
ESXi on HW raid5
ESXi direct control perc6
Raid 5 4x2TB sas disk
Raid 0 2x2TB sata disk
NVMe drive individually connected to vm's


What am looking for is a tiered storage system T1 memory T2 SSD T3 disks. ZFS seemed to be the perfect solution until I did more digging into SLOG/ZIL/ARC2L. ARC2 seems to grab a whole lot of memory if you want to use 240GB read cache. The data on the drive will be VM's which contains data that annoying if lost but not the end of the world. So am thinking to have Zil only in memory. As the server will soon be in a datacenter and has dual power supplies

I like ESXi because it's a finished product. But am considering dumping it for a hypervisor that has ZFS as native file system.


I would love advice on how you would setup the storage. Perhaps something I haven't thought of.

 

[snaptec]

Use mirrors instead of a raidz for more iops.
8gb RAM is the minimum. The more freenas has, the faster it will be.
That simple it is.
From one mirror you can expect 100-200 iops.
The feeled speed depends on the number of vms and their io workload if that's enough or not.


Gesendet von iPhone mit Tapatalk

 

[Eddie4]

Thing is I don't want to sacrifice 6TB of my 12TB.

Perhaps a LVM raid 5 with dm-cache would be a decent option?

 

[joeschmuck]

ZFS does not do RAID 5, we have RAIDZ, RAIDZ2, RAIDZ3, Mirrors, Striped.

You lost me in your question and I'm not sure what you are asking for. You are running an ESXi platform and want to run FreeNAS on it. So far so good. But you complain of throughput issues for the entire ESXi system. But do you mean that throughput is for the FreeNAS system and what protocol are you using, or is this a VM you are running on ESXi that is running slower than you desire and if that's true then where is the VM stored? If this is an ESXi issue then you might need to look into the ESXi forums for help on this, however the folks here may offer a helping hand. I don't know what the answer would be myself, I'd have to Google it to find one. I think a pair of SSDs added as a write cache would be the answer but like I said, I'm not all knowing.

 

[Robert Trevellyan]

I don't want to sacrifice 6TB of my 12TB

Then you must sacrifice performance. Unfortunately, you can't have it both ways.
https://forums.freenas.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/