I posted a bit ago that I would be acquiring some old enterprise grade parts and would be building a home lab. Well that time is finally here and I've some questions on how to get the most out of what I have for my use case. First the relevant hardware:
2 x Intel X5570 4c/8t @ 2.93 GHz
2 x Intel X5670 6c/12t @2.93 GHz
Intel X520 NIC
~30 x HGST Ultrastar DK7SAA200 2TB SATA2 7200rpm
4 x Micron M500 128GB SATA3 SSD
2 x Intel 710 100GB SATA2 SSD
18 x 8GB ECC RAM
24 x 4GB ECC RAM
8 x 16GB ECC RAM
I picked up a used Supermicro X8DT6-F, heatsinks, and fans off ebay. I've already built most of the server but its still open on my bench while I play with things a bit. I went ahead and installed the 5670 CPUs and 96GB of RAM. I've flashed the onboard LSI controller to IT mode as well.
The onboard 1Gb NICs will be used to serve CIFS shares for 5 or so Windows clients for storage and backups. I have about 2TB of existing data on those computers that will be stored on a share.
Both ports of the X520 will be directly cabled to the VM host with one port dedicated to iSCSI. I have both Twinax and multimode SFP+ with fiber for use. I haven't decided if I will use ESXi or Xenserver.
My VM guest is a Cisco C220M3 server with dual Intel E5-2650 and 64GB of RAM. NICs are 6 x Intel 1Gb onboard and a dual port HP NC522SFP 10Gb. I was very iffy on the NC522SFP at first but I got ESXi to recognize and use it after some manual driver installs and firmware flashing. It links fine over copper or fiber to the Intel 10Gb NIC in the FreeNAS box. I've mounted iSCSI LUNs over it, installed an Ubuntu guest and ran dd read and write for quite some time with no errors.
Guests I currently have planned are a TV DVR and streaming backend (probably Myth) for 6 set top boxes and a home security DVR. There may be other support guests (database?) for those and probably another couple of Windows and Linux guests for various lightweight things. I'll also do Minecraft and Teamspeak or similar servers for my kids but I know those are very light on resources as well.
My questions mainly revolve around how to best utilize the disks I have for my goals and exactly how much RAM should I really put in this. I've come up with a couple scenarios but would like input on the best way to go as I know it can be difficult to change things around once you get live data on the disks. I am limited to 12 drives in my chassis for now as my budget didn't allow for anything better than a cheapo Rosewill case at the moment. It is VERY high on my list to replace but it gets the job done for the time being.
Storage:
My current plan is a RAIDZ2 of 6 x 2TB disks attached to the onboard SATA2 ports for a pool for client shares and use the 4 x 128GB SSDs attached to the onboard LSI as a separate pool of two mirrored vdevs for the iSCSI LUNs (zvol) for the OS drives of my guests. I'll use the second port on the X520 for the guests to mount any additional storage needed from the RAIDZ2 pool. That would give me ~200GB for guests (zvol of 80% of the available space) which would be sufficient for approximately 10 guests with a 20GB boot drive each, right?
I could also do the same 6 x 2TB attached to onboard SATA for CIFS, but use 4 x 2TB attached to the LSI for my iSCSI exports (mirrored vdev again) and leave myself 4 open ports for L2ARC or whatever.
I figure I want the block storage for the guests to boot but then the file level storage so I can easily share files between desktop clients and guests. Also, my set top boxes will likely be Raspberry Pi's and I may want file level access to the recordings and such depending on the client I use.
RAM:
Should I just pop in 12 x 8GB (96GB) and go on about my build? Or is there a more scientific way than trial and error of various quantities to decide how much to install? Memtest86+ will be run for a sufficient time of course.
Sorry this is so wordy but I tried to capture all of my thoughts so as to answer any basic questions about what I have and my goals right off the bat.
2 x Intel X5570 4c/8t @ 2.93 GHz
2 x Intel X5670 6c/12t @2.93 GHz
Intel X520 NIC
~30 x HGST Ultrastar DK7SAA200 2TB SATA2 7200rpm
4 x Micron M500 128GB SATA3 SSD
2 x Intel 710 100GB SATA2 SSD
18 x 8GB ECC RAM
24 x 4GB ECC RAM
8 x 16GB ECC RAM
I picked up a used Supermicro X8DT6-F, heatsinks, and fans off ebay. I've already built most of the server but its still open on my bench while I play with things a bit. I went ahead and installed the 5670 CPUs and 96GB of RAM. I've flashed the onboard LSI controller to IT mode as well.
The onboard 1Gb NICs will be used to serve CIFS shares for 5 or so Windows clients for storage and backups. I have about 2TB of existing data on those computers that will be stored on a share.
Both ports of the X520 will be directly cabled to the VM host with one port dedicated to iSCSI. I have both Twinax and multimode SFP+ with fiber for use. I haven't decided if I will use ESXi or Xenserver.
My VM guest is a Cisco C220M3 server with dual Intel E5-2650 and 64GB of RAM. NICs are 6 x Intel 1Gb onboard and a dual port HP NC522SFP 10Gb. I was very iffy on the NC522SFP at first but I got ESXi to recognize and use it after some manual driver installs and firmware flashing. It links fine over copper or fiber to the Intel 10Gb NIC in the FreeNAS box. I've mounted iSCSI LUNs over it, installed an Ubuntu guest and ran dd read and write for quite some time with no errors.
Guests I currently have planned are a TV DVR and streaming backend (probably Myth) for 6 set top boxes and a home security DVR. There may be other support guests (database?) for those and probably another couple of Windows and Linux guests for various lightweight things. I'll also do Minecraft and Teamspeak or similar servers for my kids but I know those are very light on resources as well.
My questions mainly revolve around how to best utilize the disks I have for my goals and exactly how much RAM should I really put in this. I've come up with a couple scenarios but would like input on the best way to go as I know it can be difficult to change things around once you get live data on the disks. I am limited to 12 drives in my chassis for now as my budget didn't allow for anything better than a cheapo Rosewill case at the moment. It is VERY high on my list to replace but it gets the job done for the time being.
Storage:
My current plan is a RAIDZ2 of 6 x 2TB disks attached to the onboard SATA2 ports for a pool for client shares and use the 4 x 128GB SSDs attached to the onboard LSI as a separate pool of two mirrored vdevs for the iSCSI LUNs (zvol) for the OS drives of my guests. I'll use the second port on the X520 for the guests to mount any additional storage needed from the RAIDZ2 pool. That would give me ~200GB for guests (zvol of 80% of the available space) which would be sufficient for approximately 10 guests with a 20GB boot drive each, right?
I could also do the same 6 x 2TB attached to onboard SATA for CIFS, but use 4 x 2TB attached to the LSI for my iSCSI exports (mirrored vdev again) and leave myself 4 open ports for L2ARC or whatever.
I figure I want the block storage for the guests to boot but then the file level storage so I can easily share files between desktop clients and guests. Also, my set top boxes will likely be Raspberry Pi's and I may want file level access to the recordings and such depending on the client I use.
RAM:
Should I just pop in 12 x 8GB (96GB) and go on about my build? Or is there a more scientific way than trial and error of various quantities to decide how much to install? Memtest86+ will be run for a sufficient time of course.
Sorry this is so wordy but I tried to capture all of my thoughts so as to answer any basic questions about what I have and my goals right off the bat.
