RAID-Z6 + Cache vs RAID-Z striped vs more RAM

[MEOOOOOOOOOOOOW]

Hello everyone,

I am in the process of building myself a Fibre-Channel SAN to use with my ESXi hosts, along with some other stuff.
My plan is to use FreeNAS as an operating system.
This is my first time doing anything with FreeNAS, so please excuse some stupid questions or assumptions.

Here are some hardware specs of my machine:
Intel Xeon E5620
16GB DDR3 ECC RAM
12x 1TB HDDs

I have 8 1TB drives I want to use for the ESXi datastore, and I was planing to put them all into one big RAID-Z2. After doing some research I discovered, that the whole vdev will have the IOPS of a single drive, which is bad for virtualization.

That got me thinking, what would be a way around this issue.

Here are the three ideas I came up with:

- Use a SSD (probably 128GB) as a cache
- Add more RAM to my system
- Create two RAID-Z vdevs (with 4 drives each) and stripe them together, creating a RAID50 (more or less)

However, I am not exactly sure how to stripe those two vdevs together. I think I would put the two RAID-Z vdevs into one pool, right? Please correct me if I am wrong.

Which way do you guys think would be the better way to do this?

Thanks in advance!

 

[mav@]

Keep in mind that RAID-Z (especially wide) is efficient only for storing large data blocks, that is why it is not recommended for virtualization and generally block storage. Think again about using mirrored configuration if space permits.

 

[MEOOOOOOOOOOOOW]

If I ran it in mirrored, I would personally go ahead and create one big RAID10 across all twelve drives. (I am using "normal" RAID-names because it's easier for me to understand)
If I were to set the array up like that, how would that work in ZFS?
I guess I would create 2 striped vdevs across six drives each, and put them together in one pool, correct?
Thanks in advance!

 

[mav@]

There is no such thing as striped vdev. You should create 6 mirrored vdevs, and ZFS will distribute data between them. Or some people prefer tripple mirrors (especially if load is mostly read), in which case there would be 4 vdevs and pretty low space efficiency by the cost of high reliability.

 

[MEOOOOOOOOOOOOW]

And how would I tell ZFSto distribute data across the six vdevs? I guess by putting them in one pool, correct?
Sorry again, I am completely new to this.

 

[mav@]

Correct.

 

[Artion]

This may help.

 

[MEOOOOOOOOOOOOW]

Hello everyone,

I just set up the RAID10 across the twelve drives in FreeNAS and tested speeds using the following commands:

Write:
/usr/bin/time -h dd if=/dev/zero of=sometestfile bs=1024 count=10000
Read:
/usr/bin/time -h dd if=sometestfile of=/dev/null bs=1024 count=10000

I got the following results:

Write: 126 MB/s
Read: 36 MB/s

I honestly was expecting a lot more. I know pure read and write speeds don't matter too much when running VMs, but I can get better performance using RAID1 with two drives in one of my servers at the moment.

I have not spent any time with performance optimization (I don't know how and what either, will have to look that up tomorrow). Do you know if those speeds are normal for a config like this?

Something else I noticed, is that my RAM usage was pretty low. From what I heard FreeNAS/ZFS loves RAM, but here is my utilization during my tests:

https://abload.de/img/zfsramirs8k.png

Here are some other specs of my freenas-system:

Xeon E5620
16GB DDR3 ECC RAM
4 drives are connected to an LSI 9240-8i, passing through the drives in JBOD mode.
8 drives are connected to an LSI SAS3081E-R running in IT mode.

 

[mav@]

This was answered zillion times here. 1) "bs=1024" creates too high CPU load to measure any bandwidth. Use some values like 128K or 1M. 2) When writing to pool with dd from /dev/zero, make sure you have compression disabled, otherwise you measure compression performance instead of pool performance.

 

[MEOOOOOOOOOOOOW]

Thank you for your tips!
All my tests (also the ones before) were ran with compression off.
I have measured bandwith with 128K and 1M now.
Results of 128K:
Write: 35MB/s
Read: 3.4GB/s

1M;
Write: 4MB/s
Read: 5.5GB/s

 

[mav@]

3) There is ARC cache. Test size has to be few times bigger then RAM size. It explains huge read numbers.

But write is just too low, so there must be some 4).

 

[MEOOOOOOOOOOOOW]

The biggest amount of data I was able to transfer using my method was about 10.4GB. After that it wasn't able to make it any bigger by changing block size or count. Read speeds were pretty similar to those above, same with write speeds. I didn't test too much writing however, 1M and count 10000 already took 35min (which was 10.4GB too).
Do you have any idea what 4) could be?

 

[gpsguy]

We usually recommend a minimum of 32-64GB RAM for VM storage. And, a SLOG.

Have you run SMART tests on all the drives? Bad drives can cause performance issues.

 

[MEOOOOOOOOOOOOW]

One drive had a bad SMART-Status - i replaced it and for now just recreated the pool, since I dont have any data on it yet. I made sure that compression is off. Read speeds are still great (about 5.4 GB/s), however write speeds on bigger amounts of data are REALLY slow (3MB/s). On lower amounts of data (I tested with bs=1M, count=100 and bs=128K, count=100) I get write speeds about 4GB/s.
Next thing I will do is add SLOG, since I can get some SSDs pretty cheap - it will take a couple of days for them to arrive however. I will test and report back, as soon as the drives arrive.
Thanks for your help!

 

[mav@]

"Some cheap SSD" may be not a good option for SLOG. Not all SSDs are equal.

 

[MEOOOOOOOOOOOOW]

The SSD I will use for testing is a 80GB Intel 335 Series, which should do a fine job I think. Once I start to use my FreeNAS box for all my VMs, I will use a better SSD (most likelySLC, I don't have a specific one in mind yet) and reuse the 335 for something else.