Need some configuration advice

[zdenkers]

System is
Epyc 7402p 24 core.
8x8tb seagate enterprise hdd
1tb 970 evo plus
256gb 2933 mhz ecc ram
10gbe
250gb wd nvme ssd(os lives here)

I have currently 4 empty hdd bays. My thought is to do my hdd pool as a striped raidz1 to allow for a 4 drive expansion. I however intend to only use freenas as a host layer with vms running various tasks. This means i need a zil to handle all the double writes. How much of the 970 evo plus can or should i partition out for the zil? Does it make sense to boot the vms from the 970 evo in this case? Does the striped raidz1 seem like a good move? Any other reccomendations?

 

[Heracles]

Hey Zdenkers,

Because your plan is to run VM from that pool, Raid-Z is really not your friend here. You need to build your pool with mirrors so you end up with the maximum number of vDev possible. Also, it will be easy to use your 4 extra bays. When ready, just add another mirror, and then a last one.

As a rule of thumb, you can consider that a pool has the performance of as many HD as it has vDev. So a 2vDev pool like your plan will give you the performance of 2 drives. A set of 4 mirrors will give you the performance of 4 drives.

Also, Raid-Z1 is not that safe. On a failure, you need to perfectly read back 3 drives during the re-silvering process. To rebuild a mirror, you need to read back only 1 drive. After a first drive fails in your 2x Raid-Z1 config, the risk to loose it an on a second failure is 3 out of 7. That is pretty high... With mirror, you will survive a second failure 6 time out of 7. Or only 1 out of 7 drives will destroy your pool. That means you are 3 times safer with mirrors than with Raid-Z1.

So really, go for mirrors instead of Raid-Z1...

 

[zdenkers]

Dont really like the mirrors approach you're gonna need to show me data that supports it. I havent seen any benchmarks that indicate the performance you have pointed at.

In fact every benchmark i have looked at the 4x striped mirrors will actually perform worse

 

[Heracles]

Do some research in the forum... It is all over the place...

 

[zdenkers]

I really dont want to cut my storage in half. Redundancy is nice but i really need the large storage space that can keep up with a decent amount of read and write. I will likely saturate the 10gbe connection if the drives will keep up.

 

[Basil Hendroff]

Lot's of resources to you can tap into e.g. Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

 

[Heracles]

Redundancy is nice but i really need the large storage space that can keep up with a decent amount of read and write

That is the point : Raid-Z will not keep up with the Reads and Writes...

 

[zdenkers]

That is the point : Raid-Z will not keep up with the Reads and Writes...

like i said the benchmarks i have seen show striped raid z1 outperforming 4x mirror stripe. The data going in and out of the array will be sequential writes from 10 5mp ip cameras and 4k mkv files. It should also be noted that i said "vms" but i really mean one VM with docker running in it. This is why i wanted to mount the vm to the sdd to avoid OS/configuration of the vm dealing with the raidz, and dump the sequential writes of the ip cameras and the large video files to sit on the raidz. the only gain of going to mirrors from all the things i have read including the link posted above show it only gains you iops, which is not an issue for either pieces of data coming into the raidz.

 

[Heracles]

The data going in and out of the array will be sequential writes from 10 5mp IP cameras and 4k mkv files.

That is not at all what you said first. You first said that you were looking to run VMs...

For VMs, you need random access and that is mirrors. For long sequences, Raid-Z can do it. But if you need both, you should go with mirrors...

 

[zdenkers]

Like i said in the original post i want to host the vms on the ssd.

 

[zdenkers]

So now the question about the SSD slog still remains. can i, should i, partition part of it for the vm(hosting docker containers). if so how much should i partition for the slog?

 

[zdenkers]

I've decided that partitioning the sdd for part of my slog is probably not that great of an idea. I'm going to get an optane pci-e drive to handle the slog. i'm also going to ditch the vm with docker route, since it seems the two main items i had intended to host with docker have plugins already. However i am now torn on the zfs configuration. mirrors seem like a waste of storage to me but the striped raidz1 as was mentioned seems less safe. where is the middle ground here? Is there a middle ground that will provide the type of speeds i need? assume 10gbe saturation on approx 15 connections reading/writing a single file each.