Shared storage for 2-3 ESXI's

[Seimari]

I would like to have shared storage for our ESXI's.

Our current hardware:

ESXI1: HP Proliant DL380 G6, 144GB RAM, 2x Intel Xeon X5675 @ 3,07GHz, 3x Intel 910 800GB PCI-E SSD as local storage.
ESXI2: HP Proliant DL380 G6, 72GB RAM, 2x Intel Xeon X5675 @ 3,07GHz, 2x Intel 910 800GB + Samsung 970 Evo 1TB PCI-E SSDs as local storage.
ESXI3: HP Proliant DL380 G6, 72GB RAM, 2x Intel Xeon X5675 @ 3,07GHz, 10x Western Digital Red 1TB HDDs. Currently as backup server.
ESXI4: HP Proliant DL380p Gen8, 64GB RAM, 1x Intel Xeon E5-2620 @ 2,00GHz, 1x Samsung 970 Pro, 8x HP 300GB SAS drives. (8x SFF bay version)

Does it sound good or bad idea to convert that DL380p Gen8 to Freenas shared storage?
I was thinking of buying these parts (Ebay if possible):
- Extra 8x SFF HDD drive cage kit 662883-B21
- LSI 9305-16i HBA
- 2x Intel Xeon E5-2643 V2 (3,5GHz, 6 core)
- 256GB RAM (maybe 16GB or 32GB sticks 1866MHz, LRDIMM or RDIMM)
- Intel Optane 905P 380GB SLOG, possibly another one for L2ARC if you recommend it.
- 12-16x SATA SSD's. Some kind of PLP would be nice but lets see.
- 3x Chelsio T580-LP-CR 40Gb or Chelsio 10Gb (from storage to 2 esxi's and vmotion between esxi's).

We need about 5TB usable disk space and currently we have 15 VM's.
Our budget is quite small.

What do you guys think about this?

 

[jgreco]

I don't know about the DL380p specifically.

You don't need the 6 core processors. 4 core ones are cheaper, and it is unlikely you will tax that out.

RAM is cheap and 256GB is great if you can swing it.

You probably don't need a huge SLOG device but it won't hurt you. There is probably no reason for L2ARC, and Optane is probably not the right choice if you do.

Since you need mirror pairs for VM storage, and because you probably shouldn't go too much above the 50% utilization on a pool, you probably need around 20TB of raw space. If you can find yourself 12 x 2TB SATA SSD's, that might be a solution. Budget for SSD's would be $2800-$3400 if you buy stuff like MX500 or EVO 860. If you have intensive write requirements, those will not work out and the price jumps somewhat.

 

[Seimari]

Xeon E5-2637 V2, (3,5GHz 4 core) is my other option for cpu's.
Our VM's are typical Windows 2016 & 2019 servers like domain controller, RDS server, SQL server, file server and couple of Linux based servers.
So maybe not very intensive write requirements.

I have no idea how fast this setup would be.
I will search other threads if some user have similar setup.

 

[Seimari]

Updates:

I bought these:
- 8x SFF HDD drive cage kit 662883-B21
- LSI 9305-16i HBA
- 2x Intel Xeon E5-2643 V2 (3,5GHz, 6 core), CPU cooler and 2 fans.
- 256GB RAM (16x16GB PC3-14900 ECC RDIMM)
- Intel Optane 905P 380GB and EKWB Optane cooler
- 9x Crucial MX500 2000GB SATA SSD's. I'll buy more later if needed.
- 4x Chelsio T520-CR 10Gb.
- HP PCI-E riser
- 9x HP drive caddys
- 8x sfp+ DAC cables
- 4x SAS HD (SFF-8643) to SAS (SFF-8087) -cables.
- 1x Ubiquiti Unifi 10G switch (US-16-XG)

 

[Seimari]

I have no idea which 9305-16i bios and firmware are supported in latest Freenas.
Latest bios is SAS3_UEFI_BSD_P14 (version 16.00.00.00) and latest firmware is 9305_16i_Package_P16_IT_FW_BIOS_for_MSDOS_Windows (version 16.00.01.00).

 

[Seimari]

Small change to plans, i did order one more MX500 and go with 5x two disk mirror vdevs.

 

[Seimari]

Now i have major problem with freenas.

At some random times (0-2 times per day) Freenas reboots because of PCI-E device error:
Freenas:
"local had an unscheduled system reboot. The operating system successfully came back online at Wed Feb 12 11:03:52 2020."
HP iLO4:
"Unrecoverable System Error (NMI) has occurred. System Firmware will log additional details in a separate IML entry if possible"
"Uncorrectable PCI Express Error (Slot 6, Bus 32, Device 2, Function 0, Error status 0x00004000) "

I think that slot 6 is LSI 9305-16i which has latest P16 bios & firmare (version 16.00.01.00).

I don't know that to do.

 

[Newfoundland.Republic]

A couple of non-BSD responses:

Proliant dl360p gen8An Unrecoverable SystemError (NMI)has occurred (iLO application watchdog timeout

An Unrecoverable System Error (NMI) has occurred (iLO application watchdog timeout NMI, Service Information: 0x0000002B, 0x00000000) Hi All , We have one DL380p gen 8 server runnign Red Hat Enterprise Linux Server Release 6.2, it is second time that server crashed and reboot with the following...

community.hpe.com

VMware Knowledge Base

kb.vmware.com

They may (or may not help out). Seems to be an HP issue. I saw issues on RAM mentioned too.

 

[Seimari]

Status update:

- updated Freenas to 11.3-U2.1
- changed no-execute memory protection to disabled (bios)
- changed Intel QPI Bandwidth Optimization (RTID) setting from Optimized for I/O to balanced (bios)

Now it works and it doesn't crash anymore. I don't know which one above changed this.

But now these critical alerts: Device: /dev/da1-8, 1 currently unreadable (pending) sectors.
I did read about these alerts (Crucial MX500 + Freenas) a bit but i have to study more to know are those dangerous or not.

 

[sfcredfox]

Status update:

- updated Freenas to 11.3-U2.1
- changed no-execute memory protection to disabled (bios)
- changed Intel QPI Bandwidth Optimization (RTID) setting from Optimized for I/O to balanced (bios)

Now it works and it doesn't crash anymore. I don't know which one above changed this.

But now these critical alerts: Device: /dev/da1-8, 1 currently unreadable (pending) sectors.
I did read about these alerts (Crucial MX500 + Freenas) a bit but i have to study more to know are those dangerous or not.

How well did this system run?
Did you get it stable?

I did the 25sff bay and want to use optane for the SLOG.