How I got a XenHVM kernel and xen-tools working in FreeNAS

Status
Not open for further replies.

godbolt

Cadet
Joined
Feb 9, 2013
Messages
7
This is just a summary of my notes on how I got xen integration with FreeNAS. There wasn't much I could find on google regarding how to do this so I thought someone else might be interested. I will start by assuming a working build environment capable of successfully building 9.1.1-RELEASE.
In nanobsd/FREENAS.amd64 I added:
options NO_ADAPTIVE_MUTEXES
options NO_ADAPTIVE_RWLOCKS
options NO_ADAPTIVE_SX

# Xen HVM support
options XENHVM
device xenpci

which are taken from /usr/src/sys/amd64/conf/XENHVM on my FreeBSD build system. In the file nanobsd/os-base:

add_port sysutils/xen-tools
add_port sysutils/xe-guest-utilities

When I tried the build with only these changes, I got an error during the build of xen-tools. I 'fixed' the problem by hacking nanobsd/common to remove any mention of TARGET_ARCH from the do_add_port function. Specifically the command on Line 236 is now:

CR "make \
__MAKE_CONF=${NANO_MAKE_CONF_BUILD} \
SRC_BASE=/usr/src \
WRKDIRPREFIX=/usr/workdir -C /usr/ports/$port_path \
clean package BATCH=yes -DUSE_PACKAGE_DEPENDS \
$* clean -DFORCE_PACKAGE -DFORCE_PKG_REGISTER"

With those changes I was successfully able to build FreeNAS. Of course, there is an issue with XenServer 6.2 which prevents FreeBSD from booting using the XenHVM kernel so installing from the iso is not an option. Instead, I used vhd-util to convert the raw disk image to vhd as described in this thread. I was then able to upload this vhd file to xen using the XenCenter import wizard.

My patched freenas system successfully booted however, xenguest_enable was not set in /etc/rc.conf so the guest utilities were not starting. I just edited /conf/base/etc/rc.conf to include xenguest_enable="YES" underneath the open-vm-tools lines and rebooted.

I can now see the disk and memory info in the performance tab of XenCenter. I can also shut down and restart the vm. Suspend and memory snapshots don't seem to work.
 
Last edited by a moderator:
Joined
Oct 14, 2013
Messages
5
I don't suppose you would like to upload and share the .iso you built to jump start my effort?
 

godbolt

Cadet
Joined
Feb 9, 2013
Messages
7
I don't suppose you would like to upload and share the .iso you built to jump start my effort?
I would be happy to but I don't really have a place to upload it to. Is there an obvious place to upload to?
 

godbolt

Cadet
Joined
Feb 9, 2013
Messages
7
Ok. I will try and upload the drive image in the morning. The iso doesn't help because it won't boot in Xen 6.2.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Just going to add the disclaimer that this version is not an official iXsystems release, so you are at your own risk to use the included files in this thread. There is the potential for there to be security risks, etc etc you get the idea.
 

tesiego

Cadet
Joined
Oct 25, 2013
Messages
1
I recently installed FreeNAS 9.1 but I don't have /usr/src to try this...

any idea?
 
Last edited by a moderator:

leecro

Cadet
Joined
Nov 30, 2013
Messages
3
Hi,

First off, thanks for this - I've been interested in getting this together for a while now and following your instructions was most helpful.

In one case I've used the VHD kindly provided above, but I've encountered a possibly unusual situation where the xen PV disk puts itself in as ada0 without appropriately renaming the other first drive (SATA passthrough). I've questioned this here: http://forums.freenas.org/threads/multiple-ada0-partitions-on-xen-pv-w-passthrough.16574/ - has anyone else using this technique observed any similar drive labelling issues? Ultimately this might well be a Xen-specific issue but I suspect the same problems could occur in other scenarios if insufficient 'housekeeping' is being done during the boot process.

EDIT: My passthrough-related issue was resolved in the linked thread, mav@ confirmed that the PV driver was altering CAM settings in a way that conflicted with my own 'vm-param-set' SATA controller.
hint.ada.0.at="scbus100" was the pertinent tuneable in my case, see the attached image for implementing this in the web GUI.
eOB6iJj.jpg
 

Xlot

Dabbler
Joined
Jan 3, 2014
Messages
14
Interesting - I just joined, and this post describes what I ended up doing to make my PVHVM Xen domU. I'm currently looking at both FreeNAS and NAS4free (I have both running PVHVM) to see which to go with for the next couple of years.

I built FreeNAS a couple of days ago using the github current branch (9.2.1 ALPHA) and the build worked with no errors and only the modifications to the FREENAS.amd64 file copied from the standard XENHVM kernel config in /usr/src/sys.

I do appear to have an issue with CIFS authentication - but its unclear whether its a problem with the current beta tree, the Xen modification, or my setup (I'll be trying a pure HVM domU with vanilla 9.2 tomorrow night).
 

Xlot

Dabbler
Joined
Jan 3, 2014
Messages
14
I verified that the problem with CIFS authentication is my 9.2.1-ALPHA build - I have no access issues with stock 9.2.0, so I just kicked off a 9.2.0 build with PVHVM support to run while I'm at work.
 

Xlot

Dabbler
Joined
Jan 3, 2014
Messages
14
An update - I've built Freenas 9.2.0 with Xen PVHVM support; I've built both the iso and img packages for both i386 and amd64 - although I've only tested amd64.

Its all working fine, so my CIFS authentication problem was due to the 9.2.1 alpha build I initially did.

Interestingly, I can just run the install with the Xen virtual block device drivers (phy:/dev/vg_ssd/vm_freenas,xvda,w) from the get-go, so no need to change the drivers used for "disk" access (I'm using LVM). I've also got PCI passthrough working for my LSI 9211-8i and have so far transferred 1TB onto various of the datasets I've created across a 4x2G raidz. Performance is good - I'm maxing out gigabit ethernet at around 114mB/s using the bridged xen network device, and cpu is happy too (single core w/ two threads for the VM, along with 8G of RAM).

Once I've checked my notes I'll put a write-up of the steps, and possibly upload the files (I changed the build name so no confusion with stock files e.g. FreeNAS-9.2.0-RELEASE-xen-x64.iso)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yeah, we typically don't allow links to customized versions of FreeNAS except as part of bug testing(and those are normally from developers trying to isolate a bug).
 

freeqeed

Cadet
Joined
Jan 27, 2014
Messages
6
Once I've checked my notes I'll put a write-up of the steps, and possibly upload the files (I changed the build name so no confusion with stock files e.g. FreeNAS-9.2.0-RELEASE-xen-x64.iso)

Initially I followed the first post to have the xen-tools and xe-guest-utilities ports as well but as this had failed I then reverted back to make only the change you suggested. However this was not successful either.

I'm wondering whether you are still planning to write up the steps required?

If possible, you could fork https://github.com/freenas/freenas , then clone it to your machine locally, make the changes required, commit and then push it back to GitHub. Then you would only need to share the GitHub URL of this fork. This approach might be quicker for you as you may not have to write as much and it would make it fail prove since we would only need to build by using your fork.

Your help is much appreciated!
 

freeqeed

Cadet
Joined
Jan 27, 2014
Messages
6

Somehow get the same error as previously when I follow your post. :(

Code:
Stop in /usr/ports_dir/sysutils/ataidle.
*** [checksum] Error code 1
 
Stop in /usr/ports_dir/sysutils/ataidle.
===================================================================
ERROR: build FAILED; see above or log file here: /zfsdisk/freenas/os-base/amd64/_.cust.add_port_sysutils_ataidle
to submit a build error type 'env RELEASE_LOGFILE="release.build.log" make build-bug-report '
do_build.sh: ERROR: FreeNAS /zfsdisk/freenas/nanobsd/os-base build FAILED; please check above log for more details
*** [do-release] Error code 1
 
Stop in /zfsdisk/freenas.[/CODE]
 
Below are the steps I performed, without ending up with a successful build.
 
1. Created a new FreeBSD 9.2 machine on my XenServer (4 cores, 15GB RAM, 30GB HDD)
2. Created on a separate drive the zfs disk (single disk pool)
3. Followed your steps to install the required packages.
Note: During my previous attempts I only had used the pre-compiled packages and after reading your post I was that compiling and installing might make a difference.
4. Made the changes as you outlined
5. Kicked off the make release command and left it over night
 
Attached is the entire release.build.log
 
Wonder if you know why I could get this error?
 

Attachments

  • release.build.log.txt
    106.4 KB · Views: 859

Xlot

Dabbler
Joined
Jan 3, 2014
Messages
14
Somehow get the same error as previously when I follow your post. :(

Attached is the entire release.build.log

Wonder if you know why I could get this error?


Looks to me like there's a problem with the file from the mirror, or with the checksum file (given the checksum failed comparing the file from two mirrors).

Did you try reinstalling ports? > portsnap fetch extract
 
Status
Not open for further replies.
Top