Help me put the finishing touches on my first FreeNAS build!

[tvsjr]

humm. that's not bad. but...WHAT THE HELL ARE YOU DOING THAT YOU NEED ALL THAT?!? dual E5s with ****ing 192GB ram? did you rob a colo? I mean I guess for dedupe, but even then that's ****in' overkill. and dual procs? for what. there's no way you got enough encryption/compression going on to necessitate that...I mean...I GUESS if that's also your plex server and you're encoding multiple concurrent streams from a pretty damn raw source...

p.s. ...am I seeing what I think I'm seeing? does that case have more hot swap in the rear?!?

No dedupe, no encryption, no jails, no Plex. When you start running large iSCSI/NFS workloads, big RAM helps. Even the CPU can be helpful... if I'm hitting the VM pool hard (like overnight when virus scans run) I'll see the load averages jump to 5-6.
I've also got:
2 ESXi 6.5 nodes (2xE5-2670, 128GB RAM, 40GB SSD for boot, all main storage via NFS - about 40 VMs across the two nodes continuously, up to 60-70 at times for testing/research purposes)
1 pfSense box (E3-1270, 32GB RAM, dual 40GB SSD for boot)
2 domain controllers (E3-1270, 16GB RAM, dual 1TB drives in RAID-1)
1 Blue Iris NVR server (i7-8700K, 32GB RAM, 512GB SSD for boot/clip storage, 4x4TB drives for video storage... storing 11 1080p cameras for ~30 days)
1 3KVa Cyberpower dual-conversion online UPS
1 extended runtime battery pack (gives me about 2 hours total)
1 Cyberpower 16-position metered/switched PDU
1 48-port gigabit core switch
1 48-port gigabit distribution switch with PoE

And yes, the case I have has a 24-port backplane in the front, a 12-port backplane in the rear, and 2 internal drive bays which each handle 1 3.5" drive or 2 2.5" drives. I have my two boot devices, my SLOG, and my L2ARC in the internal drive bays cabled to the motherboard, and the LSI 9211-8i talks to the two backplanes.

Not bad for a 25U rack in a 4x8 closet, hmm?

 

[shr00mie]

good lord. ok. i'm probably going to be, at most, running a THIRD of that, maybe like a year down the line.

i'm looking at maybe 12-20 concurrent max at some point. would like to have 3 hosts total running E3s with 32GB RAM each. nowhere near your load. at most i'm probably going to have ~5-10 concurrent fulltime and then grow for lab/dev use as necessary.

closet?!? is the closet outside in the alaskan wilderness or in a basement fridge? how are you keeping all that cool?

 

[tvsjr]

good lord. ok. I'm probably going to be, at most, running a THIRD of that, maybe like a year down the line.

I'm looking at maybe 12-20 concurrent max at some point. would like to have 3 hosts total running E3s with 32GB RAM each. nowhere near your load. at most I'm probably going to have ~5-10 concurrent fulltime and then grow for lab/dev use as necessary.

closet?!? is the closet outside in the alaskan wilderness or in a basement fridge? how are you keeping all that cool?

I'd actually suggest going E5, just because of the RAM. Quite often, your issue with VM host oversubscription is RAM consumption, not with CPU load. Older E3s only support 32GB.

It's all in an upstairs closet. I've worked on the cooling issue quite a bit... it started with simply putting a fan in the door. I tried a basic exhaust vent (150CFM) in the ceiling with a vent in the door, which dramatically reduced heat buildup but didn't cure it entirely - the temperatures slowly rose and showed no signs of stopping, so I was concerned about thermal runaway. I considered a mini-split AC, but don't really want to drop that kind of coin... plus, it seems silly to run an AC all winter to eliminate the excess heat, while also running the heater.

I puzzled on this for quite a while, then found what I hope is my solution in a weird place... the world of "indoor gardening". Seems such growers have needs to move large amounts of air around. They have attic-mounted fans that are variable speed and can be had easily up to 1,000CFM (which would exchange the air in the room entirely every 6 seconds). My plan is to remove the exhaust fan that didn't work and use that hole for a ~12"x12" air return. From there, to the attic-mounted fan, then out through another vent into the largest part of the house (the front entryway, which is roughly 15x30x30). In the winter, I get the benefit of using the heat rather than spending MORE money to vent it off. In the summer, the house has 10 tons of AC (2900sqft. 2-story in north Texas) which are already handling the heat load (since it's just exhausting into the house right now). Then I'll replace the closet door with something solid and have it cut a few inches short to let fresh house air into the closet.

Now, to hope that the cops don't want to talk to me because I'm buying fans commonly used for growing the Devil's lettuce... :D

 

[wblock]

touche. I laughed. but I don't know if you can get a jab in there with an uncommitted PR to the source doc that happened like 2 weeks ago. first place I look for documentation is the PR section. obviously. ;)

point taken. what I was driving at was the fact that of all the components in a FreeNAS build, the OS, assuming backed up config, is pretty much drag and drop. even if both crap out, you reinstall on a new device and import/rebuild config. data's not going anywhere.

the point that was being made was AN ssd over mirrored USBs as being more reliable, which is...a poorly framed argument.

That PR was not committed as-is. But I got tired of explaining to people why an SSD is better, and integrated that. It is in the current documentation: http://doc.freenas.org/11/intro.html#the-operating-system-device.

 

[Stux]

I'd get a pair or two pair of 1TB SSDs. Carve out those high-speed, high-IOPS SSDs to create your VM pool. You'll thank me later. You'll also still have spare slots available for expansion or live drive replacements.

Good tip this. And it'd be even better if you uses a PLP enterprise grade SSD. They have very high performance for sync writes, which means you basically get better than SLOG enabled write performance, since the drives can securely/safely/quickly write sync writes all by themselves without using a SLOG.