FreeNAS + 2 ESXi Servers: where to put new drives and spend money

[gentoo9ball]

I've created a setup (details below) primarily for two things:

A reliable Plex setup for upwards of 10 streams at a time​

A play/test domain for network administration​

​

In the near future I'll be adding 10 more WD Red 4TB, 2x10k SAS (I believe 72GB each), and a 90GB SSD. First of all I want to know where I should put the last 3 drives. I am leaning towards the 90GB SSD in ESXi 1, with local VMs. The 2x10k SAS drives in raid 1 in ESXi 2 just running the hypervisor. I could also, split up the 10k SAS drives for both hypervisors and put the SSD anywhere as an ESXi Cache or ZFS update (log or cache)... What do you think?

I'm also looking to spend a little money on upgrades of everything. The way I see it, I have a few places of slowdown that I would like to correct. I've prioritized them as follows, but help me out with what you think would be best.

New Quad Gig Server NIC for FreeNAS (Taking suggestions)​

Battery Backup so I can disable sync writes on the VM NFS share (my power isn't bad I've got uptime for >4 months through winter now and I brought it down last time for an upgrade to 9.x)​

8GB FreeNAS ram will need an upgrade - Should I do Mobo/Proc/Ram? If so, reasonable suggestions?​

My WD Blacks all have >28k hours... some are at 37k :( Could replace one of them with the 1TB random drive that is in ESXi 2 now.​

How can I max performance and longevity here, where do you see slowdowns?

The Setup:
Switch: Cisco Gigabit Small Business SG-200-26

FreeNAS - FreeNAS-9.1.1-RELEASE-x64
Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz

MSI G45M digitial MB
8GB - I know this probably isn't enough
Dual Gig Non-Server Random NICs (LACP)
Total of 20 SATA ports: LSI SAS 9201-16i SATA3 + 4 onboard SATA2's
5x WD Black 1TB - raidz1 - All other, download/repair/extraction, currently VMs for ESXi 2 as well
5x WD Red 3TB - raidz1 - Multimedia for Plex - minimal writes, shared via NFS
(not yet, soon though) 10x WD Red 4TB - raidz2 - new storage of Multimedia for Plex

ESXi 1 - primary (must always run, DC, Plex), usually 2-3 VMs
2xOpteron 2356 = 8x2.3GHz
16GB
Dual Dig Trunked (no LACP)
Single 7200rpm WD 320GB

ESXi 2 - Non-Essential (downloader, clients, play inner domain), usually 13-20 VMs
Dell PowerEdge R805
2xOpteron 2431 = 12x2.4GHz
64GB
Quad Gig Trunked (no LACP)
Single 7200rpm ? 1TB

 

[Nindustries]

8GB - I know this probably isn't enough

There.

Raid-z1 is not recommended for 5 drives, I hope you know that?
Also, you'll want a RAID10-like setup for maximum performance out of your mechanical drives.

 

[bigphil]

A couple of thoughts here.
- I run about 20 ESXi hosts and they all use SD for system boot (home I use USB stick)....7 years with zero failures. I see no reason to use the 2 sas drives for booting your hosts...total overkill and just creates more heat and uses more power. Get yourself a couple of 4Gb+ usb sticks and use those. The system loads into memory on boot and the disks just store scratch and other system partitions.

- Unless you have ESXi 5.5 to use vFRC, the SSD would only be good for swap storage if your vm workloads ever use swap. Without knowing what type of SSD it is, we cant really say weather or not it would be good for a SLOG. It might hurt performance. Again...same thing if you were to use it for L2ARC...not to mention your FreeNAS box does not have enough RAM to really support a large L2ARC device.

- I would spend some money on upgrading the FreeNAS box to server grade hardware. Your mobo appears to only support 16Gb of memory and not ECC.
- If you plan to turn off sync, a good UPS system would be crucial. Not only that, but you'll want a way to monitor the battery level and make sure to turn off the systems in proper order before it runs dead. Disabling sync is usually frowned upon.

I'm sure others will chime in here, but a few things to think about.

 

[gentoo9ball]

There.

Raid-z1 is not recommended for 5 drives, I hope you know that?
Also, you'll want a RAID10-like setup for maximum performance out of your mechanical drives.

Thanks Nindustries, better late than never. I understand now that RaidZ1 was a bad choice initially. Sadly, I don't have enough space to swap around and re-partition until I get the 10x4TB drives. Raid10 would cut down usable space too much, is there a "second best" option? Write/Read speed is only really necessary on the partition that will be a VM datastore. Speed shouldn't be a big issue on my "Multimedia" or "Random" paritions.

A couple of thoughts here.
- I run about 20 ESXi hosts and they all use SD for system boot (home I use USB stick)....7 years with zero failures. I see no reason to use the 2 sas drives for booting your hosts...total overkill and just creates more heat and uses more power. Get yourself a couple of 4Gb+ usb sticks and use those. The system loads into memory on boot and the disks just store scratch and other system partitions.

That's great news! I'll definitely go with the USB sticks. Can you provide some more information on how to create the datastore for best performance given the hardware I already have?

- Unless you have ESXi 5.5 to use vFRC, the SSD would only be good for swap storage if your vm workloads ever use swap. Without knowing what type of SSD it is, we cant really say weather or not it would be good for a SLOG.

I'm working with an OCZ Agility 3 AGT3-25SAT3-90G 2.5" 90GB SATA III MLC. I do have ESXi 5.5, however I do not have it in a vCenter. Currently neither of the ESXi hosts swap out much if any. BTW I'll definitely start upgrading that hardware in FreeNAS. What should I shoot for, 32GB/64GB, i5, i7, Xeon (not currently doing encryption due to processor constraint)?

 

[Nindustries]

Thanks Nindustries, better late than never. I understand now that RaidZ1 was a bad choice initially. Sadly, I don't have enough space to swap around and re-partition until I get the 10x4TB drives. Raid10 would cut down usable space too much, is there a "second best" option? Write/Read speed is only really necessary on the partition that will be a VM datastore. Speed shouldn't be a big issue on my "Multimedia" or "Random" paritions

Well, you could use a simple mirror of two drives. Or use a single SSD for blazing-fast access, and back this up to your regular pool.

That's great news! I'll definitely go with the USB sticks. Can you provide some more information on how to create the datastore for best performance given the hardware I already have?

He means that ESXi can perfectly run off a USB stick, so you don't need to waste your SAS drives on this. (You could use your SAS drives in a mirror for your datastores.)
ESXi is loaded into memory on boot, and then leaves the thumbdrive alone for a long lifespan of the thumbdrive.

 

[bigphil]

That's great news! I'll definitely go with the USB sticks. Can you provide some more information on how to create the datastore for best performance given the hardware I already have?

I'm not sure you can boost the performance of the datastore much more than you already have done. If they are mounted via NFS, you could increase performance by adding the SSD as a SLOG device, but you'd have to see if you actually gain any performance or if it inhibits your current performance.

I'm working with an OCZ Agility 3 AGT3-25SAT3-90G 2.5" 90GB SATA III MLC. I do have ESXi 5.5, however I do not have it in a vCenter. Currently neither of the ESXi hosts swap out much if any. BTW I'll definitely start upgrading that hardware in FreeNAS. What should I shoot for, 32GB/64GB, i5, i7, Xeon (not currently doing encryption due to processor constraint)?

It just depends on how much you want to spend. Personally, I'd go for 64GB of ram and a Xeon. You could certainly get away with the desktop chips but make sure they and the motherboard support ECC memory. It's like getting a lap dance from the Wednesday night girls...its better than nothing, but I'd rather get the Friday night delight :)

 

[gentoo9ball]

To Do:
Install ESXi to USB sticks, boot from those
Use 10k SAS in a local Mirror on ESXi 1 (not big enough anyway for datastore of ESXi 2, only going to take up space in the NAS)

To Buy:
Friday Night lap dance... 64GB ECC + Xeon (should also take care of having a SLOG device if I understand it correctly)

 

[gentoo9ball]

Friday night lap dance arrives today,

72GB ECC + Dual Xeon 5520s (on supermicro mobo) + intel quad nic + lsi 9211-4i (combined with my 9201-16i, this should cover all 20 bays with no expanders)

Now, does anyone have a line on 10x WD 4TB drives?? I've been using their 3TB reds and I've liked them a lot. However I'm looking at around 1790 for 10x 4TB WD Reds. Anyone out there bargain shop well enough to find them cheaper?