BUILD Yet another ESXi FreeNAS build

[MrMadMan]

Hi!
My first post on this forum. There seems to be some seriously knowledgable people around, so my expectations are high for this one :)

I'm planning on building an ESXi server running FreeNAS + a handful of other servers and I'd like to hear your comments on hardware setup and configuration, as well as ask a few more specific questions.
Before you mention it, I'd like to point out that I HAVE read the guide in Please do not run FreeNAS in production as a Virtual Machine! - a couple of times - and I've still (like many others?) pretty much decided to do it anyway... So here we go!

Vision
A fairly powerful virtualising server with tons of disk space for personal (family and friends) use.

Aim
A quiet ATX case, with room for at least 6 storage drives (+ 2 for VM storage), server grade motherboard, Xeon quad core and 32GB ECC RAM.

Hardware

• Motherboard: Intel S1200V3RPS / Intel S1200V3RPL / Supermicro X10SLH-F
  • I'm thinking server grade Haswell here. These 3 boards are all based on the C22X and sport dual Intel I210-AT NICs that should be compatible with ESXi with additional drivers provided. The S1200V3RPS is by far the cheapest at around $230, followed by the S1200V3RPL at $290 and the X10SLH-F at $350. Feature wise they all seem very similar to me. Is there any reason I shouldn't go for the cheapest one here?
• CPU: Intel Xeon E3-1230v3
  • No competition price/performance wise.
• Storage Controller: IBM ServeRAID M1015
  • This seems to be THE controller to get. And I don't think any integrated controller could really match it.
• RAM: Kingston ValueRAM DDR3 PC10600/1333MHz ECC Reg CL9 Intel 2x16GB
  • Can the above mentioned motherboards handle 16GB sticks or do I have to go 4x8GB? What about chip compatibility? I'm kind of struggling with finding what RAM are compatible...
• Storage (HyperV): Kingston DataTraveler 100 G3 8GB
  • ESXi goes here.
• Storage (HyperV backup before upgrade): Kingston DataTraveler 100 G3 8GB
  • Clone ESXi to this one before attempting an upgrade.
• Storage (Fallback FreeNAS): Kingston DataTraveler 100 G3 8GB
  • A fallback FreeNAS that's installed first and thoroughly tested. Can later be used if for some reason the ESXi installation goes nuts.
• Storage (VM:s and snaps): 2x 500GB SATA HDD
  • Use on-chip RAID-1 for these and store VM:s and possibly snapshots on them. Are the on-chip controllers okay for ESXi datastores?
• Storage (Pool): 5x WD Green 2TB
  • I have these in a RAID-Z configuration on my current server. Will in time upgrade to a larger RAID-Z2 or something
• Case: Fractal Design Define R4
  • Reasonably priced and fairly quiet. Takes 8 drives which allows room for an upgrade to 6 disk RAID-Z2
• PSU: Fractal Design Integra 400W
  • I have one of these laying around. Is it sufficient or should I aim for something more reliable and efficient?

Migration and configuration
After recommendations in "Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data. I came up with the following plan.

1. Export the zpool on old server and remove the disks
2. Flash the M1015 with IT firmware
3. Configure the two 500GB drives as RAID-1 (onboard controller)
4. Install FreeNAS 9.1.1 on USB key.
5. Import the zpool
6. Leave it running for a week while doing some benchmarking/stress testing to see if it seems stable.
7. Export the zpool from the USB key FreeNAS
8. Install ESXi 5.5 on the second USB key
9. Install FreeNAS 9.1.1 as a VM on the RAID-1 setup.
10. Enable passthrough on the M1015 and import the zpool in the FreeNAS VM.
11. Share the zfs datasets with other VM:s and physical computers

Questions
What do you think about the hardware I've considered? Does it seem like a good fit? Should I go for the safer route with Socket 1155?

What about the migration and configuration? Does it seem like a safe way of migrating from my current setup (Debian box running zfs-on-linux)?

Also note that there are specific questions for each hardware item as well.

Thank you for your help :)

 

[jonnn]

FYI, OmniOS + napp-it is basically intended for VM environment... if Freenas doesn't work out.

 

[MrMadMan]

I haven't really made a final decision on what NAS software to run, so thanks for the suggestion :)

 

[MrMadMan]

I've seen a lot of people recommending the Supermicro X10SL7-F because of its integrated LSI 2308 controller.
Would choosing that board let me skip the M1015? What about ESXi compatibility and passthrough?

 

[jgreco]

FYI, OmniOS + napp-it is basically intended for VM environment... if Freenas doesn't work out.

By "intended for VM environment" you mean ... what, exactly? The problems people have with FreeNAS as a VM are not actually unique to FreeNAS, as far as I'm aware. We just actively discourage it because we're tired of seeing people do things they didn't realize were stupid until after their data has been shredded. If you cannot make FreeNAS work safely, you cannot make other NAS platforms work safely either.

 

[MrMadMan]

By "intended for VM environment" you mean ... what, exactly? The problems people have with FreeNAS as a VM are not actually unique to FreeNAS, as far as I'm aware. We just actively discourage it because we're tired of seeing people do things they didn't realize were stupid until after their data has been shredded. If you cannot make FreeNAS work safely, you cannot make other NAS platforms work safely either.

I realize they're both fine to run in a VM as long as you know what you're doing. But this isn't really an issue right now. Is there anything you can say about the actual build?

 

[jgreco]

Not really. We've not bought any Haswell as we've got sufficient capacity at the current time. I couldn't figure out what you were planning to do to acquire the RAID1 capability. I guess I could note that the M1015, and LSI2108, and 2308 in RAID1 will be slow for ESXi, as ESXi works best with a write cache with BBU for datastore. All the options there pretty much suck. We ordered a nice X9DR7-TF+ (dual E5 based) but that's not going to be reasonable for the average user... but it basically gets you a great dual board with 2208 and 10GbE too. It is one of the few onboard controllers that I'd rate "great for ESXi." So were you planning to use an M1015 in IR mode or something like that?

 

[MrMadMan]

Not really. We've not bought any Haswell as we've got sufficient capacity at the current time. I couldn't figure out what you were planning to do to acquire the RAID1 capability. I guess I could note that the M1015, and LSI2108, and 2308 in RAID1 will be slow for ESXi, as ESXi works best with a write cache with BBU for datastore. All the options there pretty much suck. We ordered a nice X9DR7-TF+ (dual E5 based) but that's not going to be reasonable for the average user... but it basically gets you a great dual board with 2208 and 10GbE too. It is one of the few onboard controllers that I'd rate "great for ESXi." So were you planning to use an M1015 in IR mode or something like that?

My hope was that I would be able to run the 2x500GB disks in a RAID-1 on the onboard (non LSI) controller and keep the VM:s on it.
The RAID-Z would be run on the M1015 or 2308 in passthrough IT mode.
Is that not likely to work reliably?

 

[jgreco]

I'm failing to understand what controller you intend to run RAID-1 on. You need some sort of controller for that...

 

[MrMadMan]

I'm failing to understand what controller you intend to run RAID-1 on. You need some sort of controller for that...

I was thinking about the C222 SW RAID "Matrix Storage Technology". But after reading up on this, I guess SW RAID is not supported in ESXi...

Thanks for making me aware of that!

So back to square 1. You're saying all the options suck... but which ones suck the least? I'm up for suggestions... I'd LIKE to avoid buying another controller, but I can be convinced to.
How do most people solve this?

Worth mentioning also is that the VM:s themselves are not THAT important. All important data on them will be backed up to the NAS, but the configuration would be a pain to lose should something happen...

 

[jgreco]

Mainboard has six ports and you have five drives? Flash the M1015 to IR mode (or the 2308) and set up RAID1, use for ESXi, then use VT-d to pass the mainboard controller to FreeNAS. You lose two FreeNAS ports using the smaller controller but even with the M1015 you only had three spare, so any realistic pool expansion probably meant another controller anyways.

 

[MrMadMan]

Mainboard has six ports and you have five drives? Flash the M1015 to IR mode (or the 2308) and set up RAID1, use for ESXi, then use VT-d to pass the mainboard controller to FreeNAS. You lose two FreeNAS ports using the smaller controller but even with the M1015 you only had three spare, so any realistic pool expansion probably meant another controller anyways.

Mainboard has 6 ports on the C222 and 8 ports on the 2308 (image).

Anyway, it shouldn't really matter which one of the 2308 and M1015 I use for VM:s and which one I flash to IT and pass to FreeNAS, right?

So if I understand you correctly, this would be a good solution:
2x500GB on the 2308 in RAID-1 for VM:s
5x2TB on the M1015 (flashed to IT mode) and pass it to FreeNAS using VT-d

What do you think?

 

[Nindustries]

Do note that ESXi can only properly use/vt-d hardware raid controllers (some), and not software ones. (Most)

- Sent from mobile.

 

[MrMadMan]

Do note that ESXi can only properly use/vt-d hardware raid controllers (some), and not software ones. (Most)

- Sent from mobile.

Yeah... I just realized and mentioned that in a post further up :)

 

[KMR]

First thoughts about the build:
PSU: grab a seasonic 80+gold PSU. They work really well and have an excellent reputation.
Drives: Grab 3tb drives. They have a better price per TB ratio. If you go with the green drives make sure to use the utility. Cyberjock has referenced this in a post somewhere.
CPU: The Xeon 1230vX is an excellent choice.
MB: Is there a significant cost savings where you live for 1155 based boards? If so build an 1155 based setup. Spend the extra money on drives.
RAM: As far as I know the haswell stuff only supports 32GB of RAM (someone please correct me if I'm wrong), same as the 1155. Don't bother with 16GB sticks, especially where they are likely much more expensive and may also be registered ECC. You will need to check your boards RAM compatibility closely to ensure you get the right parts. If the 1150 stuff only supports 32GB of ram then just fully populate with 8GB sticks.
Controllers: The M1015 is great for passthrough in IT mode for ZFS. As for the RAID-1 setup you will likely be better off just using a single drive for VM storage and backing the VM's up regularly to the pool in case your drive dies. As Jgreco said ESXi is picky about controllers. I haven't done much with ESXi and controllers so he is likely a better person to ask if you are set on using RAID-1 for VM storage.

Other than that it looks like a great build and is very similar to mine. Good job to reading the wisdom of our senior members and following through with the plans!
I may drop back with other comments as I think of them.

 

[MrMadMan]

First thoughts about the build:
PSU: grab a seasonic 80+gold PSU. They work really well and have an excellent reputation.
Drives: Grab 3tb drives. They have a better price per TB ratio. If you go with the green drives make sure to use the utility. Cyberjock has referenced this in a post somewhere.
CPU: The Xeon 1230vX is an excellent choice.
MB: Is there a significant cost savings where you live for 1155 based boards? If so build an 1155 based setup. Spend the extra money on drives.
RAM: As far as I know the haswell stuff only supports 32GB of RAM (someone please correct me if I'm wrong), same as the 1155. Don't bother with 16GB sticks, especially where they are likely much more expensive and may also be registered ECC. You will need to check your boards RAM compatibility closely to ensure you get the right parts. If the 1150 stuff only supports 32GB of ram then just fully populate with 8GB sticks.
Controllers: The M1015 is great for passthrough in IT mode for ZFS. As for the RAID-1 setup you will likely be better off just using a single drive for VM storage and backing the VM's up regularly to the pool in case your drive dies. As Jgreco said ESXi is picky about controllers. I haven't done much with ESXi and controllers so he is likely a better person to ask if you are set on using RAID-1 for VM storage.

Other than that it looks like a great build and is very similar to mine. Good job to reading the wisdom of our senior members and following through with the plans!
I may drop back with other comments as I think of them.

PSU: I decided to go with a Seasonic G-450
Drives: I already have the 2TB drives since my previous build so I'll keep them for now
MB: The price difference between 1155 and 1150 is negligible so there is no money to save going 1155
RAM: I'll go with 4x8GB sticks once I find some that are affordable and compatible
Controllers: I ordered a M1015, so I will run the VM drives in RAID-1 on the 2308 and run the NAS drives on the M1015

 

[Dr.T]

Doesn't ESXi have internal features to replicate VMs (High availibility/Fault tolerance)? So you can use another disks as yet another datastores for VM replicating and save money on M1015 .

 

[MrMadMan]

Doesn't ESXi have internal features to replicate VMs (High availibility/Fault tolerance)? So you can use another disks as yet another datastores for VM replicating and save money on M1015 .

That sounds pretty neat! Too bad the M1015 is already on it's way... But if that is true, I could always save it until the day I need to expand the zpool!

 

[KMR]

I believe HA / Fault tolerance are only available with the paid version of ESXi and may require vCenter and multiple nodes (someone please correct me if I'm wrong here). I'm not sure about the integrated 2308 controller on those boards so I can't really offer any advice although I'm curious to see if you get it working. Please keep us up to date and do all the reading you can!

 

[MrMadMan]

I believe HA / Fault tolerance are only available with the paid version of ESXi and may require vCenter and multiple nodes (someone please correct me if I'm wrong here). I'm not sure about the integrated 2308 controller on those boards so I can't really offer any advice although I'm curious to see if you get it working. Please keep us up to date and do all the reading you can!

I'll keep you posted!