-Must have a large cage since I need 2x the amount of storage as i planned for. Reason: I never want to exceed 50% usage
Correct. But also remember that what really counts is your pool fill percentage, so if you make a typical array, make some LUNs that occupy 50% of them, but have nothing on them, then you've not yet allocated those blocks and you're still very close to 0%. This is why things like TRIM and compression are very useful.
-Use 2 way mirrors not Raidz2. Go for 3 way if budget can tolerate it.
-I might want to trade off, More Ram for less CPU.
-I only need 2 pci slots - 1 for disk controller - 1 for 10G Ethernet - so i can get a lesser motherboard
-Use Large l2arc SSD's.
Yes to all that, but also be sure not to go too small. For a VM system, really, an E3 with 32GB is the smallest you should go, and you're immediately maxxed out on RAM, so that's all a bad choice. A small E5-1620V3 plus 32GB RAM is only a mistake in that it is too small to make good use of L2ARC, but if you buy that RAM in 16 or 32GB sticks, then you're left with a bunch of slots for future expansion.
- Why does 3way mirrors have a lot of overhead? Parity Calculations? Is this CPU intensive or Write intensive.
It has a lot of overhead in terms of disk used. If your design requirement includes something to the effect that a single drive failure should not compromise redundancy, then two-way mirrors is not acceptable. It should be nearly free in every other way, and in fact increases potential read IOPS.
Guys, I really learned a lot today.. FreeNas is the way to go, basically for the community.. With two years of Nexenta with full support I did not received 1/10 the amount of knowledge. Thanks for all the great info!
So, yeah, if you buy a Ford Mustang at the dealership, they'll do a decent job of supporting your car for you, but if you really want to learn about what goes on under the hood, you don't try to engage them, instead you go find the local motorheads where they'll probably be happy to gas on about it (... waits for the boo's ...)