Recommended setup for VMs

[liors]

Hi Everyone,

First of all I would like to thank everyone for sharing ineradicable amount of information , it's very helpful and the community here is amazing !
In the last few weeks I was collecting data from many threads but unfortunately I still have some gaps that I would like to share with you.

Currently we have 25 ESX servers and we're using a Nimble storage machine, in order to reduce the IT cost we were thinking to use FreeNAS for non-critical environments such as Dev/Tests

I was thinking to build a FreeNAS machine with the following spec :

1 * chassis : SUPERMICRO CSE-846BE1C-R1K03JBOD - here
1 * motherboard : Supermicro X10SRA-F-O - here
1 * CPU : Xeon E5 -2640 V4 - here
4 * RAM : Samsung DDR4 2400 LRDIMM 32GB M386A4G40DM1-CRC (128G total) - here
2 * controllers: LSI Logic Controller Card H5-25573-00 9300-8i
24 * Magnetic disks: hitachi He10 / seagate enterprise capacity 10tb
2 * SLOG: samsung 970 pro 512G connected via PCI adapter or Intel Optane SSD 900P
2 * L2ARC : samsung 970 evo 500G connected via sata or samsung 970 pro 512G
1 * PCI adapter for both m2 ssd (or something similar) - here

Questions :

1. What would be the best configuration, I understood that the best practice is less than 12 drives in a pool and I need a dedicated SLOG and L2ARC for each pool.
it means that we need two pools each of one of them with 12 spindles disks 1 SLOG and 1 L2ARC, am I right or I'm missing something ?

2. I'm not sure if raidz2 will be good enough or should I use mirror instead ?

3. Which hard disk is more recommended hitachi He10 or Seagate enterprise capacity 10tb or you might have other recommendations ?

4. If you have any comments about my spec it'll be great

5. I've seen that usually the recommendation for L2ARC is SSD which is not NVMe so why tactfully not use NVMe for L2ARC ?

6.What should be the L2ARC size that I need for such setup ?

Thanks a lot,
Lior.

 

[Chris Moore]

chassis : SUPERMICRO CSE-846BE1C-R1K03JBOD - here

The thing you linked to is actually just an externally attached SAS drive chassis. It is not intended to contain a system board, processor and memory. Did you need a suggestion for that?

CPU : Xeon E5 -2640 V4 - here

I would suggest a little faster CPU. If you go too slow, it can actually impede performance.

2 * controllers: LSI Logic Controller Card H5-25573-00 9300-8i

Why two?

What would be the best configuration, I understood that the best practice is less than 12 drives in a pool and I need a dedicated SLOG and L2ARC for each pool.

No. You have somehow got a confusion between pool and vdev. I suggest you review the following documentation.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

I'm not sure if raidz2 will be good enough or should I use mirror instead ?

For virtualization, you want to maximize IOPS. To do that, you want the maximum number of vdevs you can get. With only 24 drives to work with, that means 12 mirror vdevs. You can have more if you add more disk capacity. I have a system at work with 124 drives across 18 vdevs.

Which hard disk is more recommended hitachi He10 or Seagate enterprise capacity 10tb or you might have other recommendations ?

My preference would be the Seagate Exos 10TB. I have a new server we brought online in November at work that has 60 of those in it and they have performed flawlessly. I have already loaded 330TB of data into the unit and had no trouble with them at all. About two years ago I was doing a similar task with Western Digital drives and had three fail in the first three months.

If you have any comments about my spec it'll be great

I would say, regarding your drive selection for SLOG, that if you are spending all this money on other hardware, don't go cheap on the SLOG/L2ARC.
Have a look at the test results in these forum threads:

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

SLOG benchmarking and finding the best SLOG
https://forums.freenas.org/index.ph...-and-finding-the-best-slog.63521/#post-454773

I've seen that usually the recommendation for L2ARC is SSD which is not NVMe so why tactfully not use NVMe for L2ARC ?

Here is what I am using:
https://forums.freenas.org/index.ph...g-and-finding-the-best-slog.63521/post-481728
They work well for me.

What should be the L2ARC size that I need for such setup ?

You probably don't need the 2TB size that I have in the post above, but I don't either. Depending on exactly how you are using the system, you might not need it at all. Is this going to be iSCSI or NFS?
If you give us more information, we can probably give a bit more guidance.

 

[Stux]

2 * SLOG: samsung 970 pro 512G connected via PCI adapter or Intel Optane SSD 900P

Samsung 970 pro is not a suitable SLOG drive. No PLP.

Stick to the Optane. And look at the SLOG benchmarking thread to pick your drive.

Also, a SLOG doesn't have to be big. Its the sequential sync performance that matters... IIRC you need enough SLOG to hold 10 seconds of maximum burst data... ie, look at your input networking, 10gbit = 1GB/s = 20GB is more than enough.

Of course, a 20GB fast SSD is not something you can get... so we end up using 400+GB SSDs :)

 

[liors]

So I did some homework and I would like to share more information here and thoughts :

Regarding the spec :
1. From my understood this chassis it is an externally attached SAS drive chassis however if I'm taking out the card it has inside it can contain motherboard CPU and memory like any other server, anyway if you have better recommendation I'll be happy to know.

2. I'm using two LSI controllers because the chassis I mentioned above has 4 x Mini-SAS HD and each card supports 2 X Mini-SAS, but maybe I'm missing something ?

3. The system is going to be iSCSI

4. SLOG - seems like Intel Optane SSD 900P it the best value for money especially when it comes with 280G (which is more than enough for SLOG) but I'll check the DC P4800X series

5. PLP isn't relevant for L2ARC, so I'm wondering why not to use Samsung 970 PRO 512GB ? or you can think on better options ?

Regarding the configuration:
1. looks like that 12 mirror vdevs are the best configuration for performance , but there is something that I didn't understand , assume that I have 24 disks what is the different between 12 mirror vdevs to 6 mirror vdevs ?

2.The best practice is less than 12 drives in a vdev (and not in a pool) so in my case I need only one dedicated SLOG and one L2ARC (thanks for the clarification)

and again , thanks everyone for the help !

 

[Chris Moore]

From my understood this chassis it is an externally attached SAS drive chassis however if I'm taking out the card it has inside it can contain motherboard CPU and memory like any other server, anyway if you have better recommendation I'll be happy to know.

Are you thinking to do that for cost savings? Are you purchasing new hardware or looking to source used gear?
If you are looking to buy new hardware, you can easily order new, already assembled equipment instead of putting parts together yourself.

I'm using two LSI controllers because the chassis I mentioned above has 4 x Mini-SAS HD and each card supports 2 X Mini-SAS, but maybe I'm missing something ?

The external enclosure you are looking at has a SAS Expander backplane, it can run from a single SAS Controller card. The reason the enclosure has the port configuration you are seeing is that it is SAS3 which uses two cables in and two cables out. The second two ports allow you to cascade additional enclosures from the "out" connector so that you can attach more drives. Many SAS3 interfaces can support as many as 1024 drives on a single controller by cascading SAS Expanders.

The system is going to be iSCSI

If you plan to use iSCSI, it doesn't automatically make the pool use sync writes, but it is still possible that you would benefit from SLOG or L2ARC. It is a decision you can make. Perhaps @Stux or @jgreco others will offer some advice.

PLP isn't relevant for L2ARC, so I'm wondering why not to use Samsung 970 PRO 512GB ? or you can think on better options ?

You just need something fast and reliable with a lot of endurance.

Regarding the configuration:
1. looks like that 12 mirror vdevs are the best configuration for performance , but there is something that I didn't understand , assume that I have 24 disks what is the different between 12 mirror vdevs to 6 mirror vdevs ?

I don't know why you are asking about 6 mirror vdevs, are you thinking to only install 12 drives? If you cut the number of vdevs in half, you cut the number of IOPS in half. It is not a perfect relationship, but in general terms the number of IOPS offered by a single vdev is multiplied by the number of vdevs and it scales as you add more vdevs. You can add an external disk enclosure and add more disks to get even more vdevs for even more IOPS.

The best practice is less than 12 drives in a vdev (and not in a pool) so in my case I need only one dedicated SLOG and one L2ARC (thanks for the clarification)

I am happy we cleared that up.

and again , thanks everyone for the help !

I would suggest using an actual server chassis instead of an expansion chassis. Something like this for example:
https://www.amazon.com/Supermicro-Rackmount-Server-Chassis-CSE-846BE1C-R1K28B/dp/B00TZLZ7IK
Details on the Supermicro site:
https://www.supermicro.com/products/chassis/4U/846/SC846BE1C-R1K23B

 

[liors]

Are you thinking to do that for cost savings? Are you purchasing new hardware or looking to source used gear?
If you are looking to buy new hardware, you can easily order new, already assembled equipment instead of putting parts together yourself.

Currently I have this chassis so I prefer not to spend more money , I'll install everything by myself

If you plan to use iSCSI, it doesn't automatically make the pool use sync writes, but it is still possible that you would benefit from SLOG or L2ARC. It is a decision you can make. Perhaps @Stux or @jgreco others will offer some advice.

Let's wait for more answers

I don't know why you are asking about 6 mirror vdevs, are you thinking to only install 12 drives? If you cut the number of vdevs in half, you cut the number of IOPS in half. It is not a perfect relationship, but in general terms the number of IOPS offered by a single vdev is multiplied by the number of vdevs and it scales as you add more vdevs. You can add an external disk enclosure and add more disks to get even more vdevs for even more IOPS. .

I got the point and seems like 12 mirror vdevs will the best configuration for my need