topping
Dabbler
- Joined
- Mar 29, 2012
- Messages
- 14
EDIT: I did not realize Corral was EOL and was kindly straightened out in the thread that immediately follows. After testing, I found that these steps also work with XenServer 7.1 and FreeNAS 9.10.2-U3.
A couple of weeks ago, I finally got around to racking my old Supermicro X8ST3-F based box. Bang bang, pull the drives from my HP Gen 8 cube... screeech! The X8 BIOS can't handle 4k drives.
Several days of eBay hunting later, I ended up deciding on this server:
The main piece that I am setting up for is DC/OS, which needs a lot of RAM and CPU. I have two X7-DBTs, but 64GB of RAM is not quite enough (gulp), so I started looking at where I was going to get more sockets. Yes indeed, the new file server has RAM sockets, room for 760GB more RAM to be exact. So being short on cash and suddenly in possession of a server that had a SCU HBA and a backplane, I decided to see what might come of virtualizing the box with XenServer 7.0.
WARNING: My success here does not mean anything. It's very likely I will regret doing this. Do not read past this line.
Seriously though. Do your research on this site and make sure you understand why ECC RAM is important and what it means to your filesystems if your VM were to be suddenly destroyed.
Smarter people than me say this really should be done with VMWare ESXi, but I have been using XenServer since the days that VMWare clamped your memory and processors to try and extract money. I know XenServer, but that said, it wasn't working, either with FreeNAS 9.10 or 10.0. And that concerned me.
Last night, still unable to get things to work, I actually broke down and installed ESXi 6.5a. It was a way better experience than when I tried it ten years ago (shocking), but it was still very unclear how to use LACP for the VMs and put the management interface on that same trunk. I googled for hours on this and finally gave up this afternoon. VMWare looks great, but my skills in emergency care and feeding of the environment are with Xen, not ESX. This also concerned me.
In the process, I learned that XenServer 7.1 was out. In disappointment of having sunk more days into this project, I decided to try 7.1 on a lark. Figuring this was some kind of driver issue, I went for the FreeNAS 10.0 image that was sitting on my desktop machine. I was so used to the drill by then that I barely had to google for the steps and was able to scroll up in my terminal session to get most of the commands I needed. After that, I waited until that magical time in the boot where the machine hung, but suddenly the font color changed from white to grey... I was in!
Here are the steps that I went through:
Once you can get into the UI, set the tunables for your HBA. You'll need to look around for your card. In my case, with the LSI 9210-8I, I add two tunables as follows:
It's worth noting that when you use
From here, you should follow the instructions on how to validate your system properly, including memory and disk tests. Don't put data on a system that you are not confident of. It's your responsibility to make sure that your data is safe. I would also add that it's important that your system has good battery backup and can gracefully shut everything down in a completely automatic manner if the batteries are at risk of fully discharging.
Have fun!
A couple of weeks ago, I finally got around to racking my old Supermicro X8ST3-F based box. Bang bang, pull the drives from my HP Gen 8 cube... screeech! The X8 BIOS can't handle 4k drives.
Several days of eBay hunting later, I ended up deciding on this server:
- SUPERMICRO 6027R-E1R12N
- DUAL INTEL XEON PROCESSOR E5-2660 EIGHT CORE 20M CACHE 2.20GHZ
- MEMORY 8GB MEMORY (2X 4GB)
- RAID CONTROLLER LSI 9210-8I
- CHASSIS CSE-826
- SYSTEM BOARD X9DRI-LN4F+
- BACKPLANE BPN-SAS2-826EL1
The main piece that I am setting up for is DC/OS, which needs a lot of RAM and CPU. I have two X7-DBTs, but 64GB of RAM is not quite enough (gulp), so I started looking at where I was going to get more sockets. Yes indeed, the new file server has RAM sockets, room for 760GB more RAM to be exact. So being short on cash and suddenly in possession of a server that had a SCU HBA and a backplane, I decided to see what might come of virtualizing the box with XenServer 7.0.
WARNING: My success here does not mean anything. It's very likely I will regret doing this. Do not read past this line.
Seriously though. Do your research on this site and make sure you understand why ECC RAM is important and what it means to your filesystems if your VM were to be suddenly destroyed.
Smarter people than me say this really should be done with VMWare ESXi, but I have been using XenServer since the days that VMWare clamped your memory and processors to try and extract money. I know XenServer, but that said, it wasn't working, either with FreeNAS 9.10 or 10.0. And that concerned me.
Last night, still unable to get things to work, I actually broke down and installed ESXi 6.5a. It was a way better experience than when I tried it ten years ago (shocking), but it was still very unclear how to use LACP for the VMs and put the management interface on that same trunk. I googled for hours on this and finally gave up this afternoon. VMWare looks great, but my skills in emergency care and feeding of the environment are with Xen, not ESX. This also concerned me.
In the process, I learned that XenServer 7.1 was out. In disappointment of having sunk more days into this project, I decided to try 7.1 on a lark. Figuring this was some kind of driver issue, I went for the FreeNAS 10.0 image that was sitting on my desktop machine. I was so used to the drill by then that I barely had to google for the steps and was able to scroll up in my terminal session to get most of the commands I needed. After that, I waited until that magical time in the boot where the machine hung, but suddenly the font color changed from white to grey... I was in!
Here are the steps that I went through:
- Install 7.1. If you are eventually going to use LACP, it's best to start with it so your VM interfaces are set to what you will eventually depend on, which saves a lot of time later.
- Set up a ISO repository from which to install. The simplest form of this is to set up a directory on the server root and issue the
xe sr-createcommand. You'll need to copy the FreeNAS ISO to that directory and rescan the repository to make it available. - You will roughly follow the instructions in this Citrix forum article while building the VM. It contains two
xecommands that need to be run, one so the hypervisor does not own the PCI device and a second to pass the device through to the VM.
Once you can get into the UI, set the tunables for your HBA. You'll need to look around for your card. In my case, with the LSI 9210-8I, I add two tunables as follows:
- Variable : hw.pci.enable_msi
Value : 1
Type : sysctl - Variable : hw.pci.enable_msix
Value : 0
Type : sysctl
It's worth noting that when you use
xe vm-set-param, the VM should be fully shut down before you expect them to take effect, not simply rebooted. I may be playing it too safe, but I like to be sure that the hypervisor knows that there's no excuse not to present the options I've changed to the booting VM.From here, you should follow the instructions on how to validate your system properly, including memory and disk tests. Don't put data on a system that you are not confident of. It's your responsibility to make sure that your data is safe. I would also add that it's important that your system has good battery backup and can gracefully shut everything down in a completely automatic manner if the batteries are at risk of fully discharging.
Have fun!
Last edited:
