Guide: PXE: How to install FreeNAS 9.2.0 over the network

[Craig Rodrigues]

Some FreeNAS users have been asking how to do an installation of FreeNAS over PXE. While it is not currently officially supported by the FreeNAS development team, it is possible as of FreeNAS 9.2.0 and higher.

At iXsystems, the FreeNAS developers use PXE for booting new machines in our development lab and installing them over the network in new versions of FreeNAS. This saves us some time during development, since we do not have to install via a USB drive.

If you want try this out, you will need:

(1) One machine which you want to install FreeNAS on.

(2) A second machine running FreeBSD, which will be the PXE server.

(3) A FreeNAS installation ISO, obtained from http://download.freenas.org .
NOTE: PXE booting the FreeNAS installer only works for versions 9.2.0 and higher.

(4) The pxe-config.sh script.


A. RETRIEVE THE PXE CONFIG SCRIPT

1. Follow the instructions at http://doc.freenas.org/index.php/Test_Upcoming_Versions for obtaining the FreeNAS source code from github. After obtaining the source code, look in the test/pxe directory for the pxe-config.sh script.
2. Alternatively, if you do not want to download all the FreeNAS source code, you can get the script from github at https://github.com/freenas/freenas/tree/master/test/pxe/

B. SET UP FREEBSD PXE SERVER

1. Copy the pxe-config.sh script to the FreeBSD PXE server.
2. Create a directory on the FreeBSD PXE server, /usr/jails/pxeserver/images
3. Copy the FreeNAS installation ISO to /usr/jails/pxeserver/images
4. Run the pxe-config.sh script, select the ISO in the menu. This will extract the contents of the ISO file.
5. Read this document: http://www.freebsd.org/doc/handbook/network-pxe-nfs.html and use that as a guideline for setting up a PXE server.
   • make sure that /usr/jails/pxeserver/images/boot_dir is NFS exported in /etc/exports
   • make sure that you can use an NFS client to mount "pxehostname:/usr/jails/pxeserver/images/boot_dir", where "pxehostname" is the hostname or IP address of your PXE server.
   • make sure that in /etc/inetd.conf the tftp server is set up so that you the tftpd server is started with: "tftpd -l -s /usr/jails/pxeserver/images/boot_dir".
   • make sure that you can connect use a TFTP client to connect to the PXE server and retrieve the file "boot/pxeboot".
   • Configure a dhcpd.conf file so that the "filename" value is set to the value of the file you can retrieve via TFTP.
   • Make sure that option-root-path is set to the value that you can NFS mount /usr/jails/pxeserver/images/boot_dir.

C. PXE BOOTING THE FREENAS INSTALLER

If you have sucessfully configured your PXE server in section B., you should be able to configure your machine which you want to install FreeNAS on so that it can boot over the network. This can usually be done by going into the BIOS, and changing the boot order so that it boots from network first.

We recommend that for doing this, you get an Ethernet switch which is completely disconnected from any network. You can then plug in your PXE server and target machine into this switch. It is best to do this on an isolated network, since DHCP is required for PXE. You don't want the DHCP used for PXE to interfere with your home Internet connection, which also uses DHCP.

Refer to:

http://www.freebsd.org/doc/handbook/network-pxe-nfs.html

for diagnosing and debugging problems with PXE.

 

[Drew Morone]

Not all of us have FreeBSD PXE servers. It would be cool for people who administer PXE servers to just have the important files, and where they should|can be located. I have a CentOS environment, but want to PXE-boot my FreeNAS server from the network, not from a USB stick (which are prone to die).

 

[Ericloewe]

I can understand booting secondary servers off a primary server, but adding a zeroth layer server seems like an odd idea. Typically FreeNAS is the one doing the boot image serving...

The guide is meant to help install FreeNAS, as an alternative to IPMI (from what I understand of the guide), not as a way to actually boot FreeNAS from PXE.

 

[Drew Morone]

I'm not sure what a 'zeroth layer server' is. The install docs suggest you install to a flash or USB device, so why not pxeboot. I'm also not sure where IPMI fits into this.
Now that I look at it, I think they are just using it to install to their local hard drives. I don't understand why they discourage using a hard drive. Who cares if you lose some storage. Drives are cheap.

 

[Ericloewe]

I'm not sure what a 'zeroth layer server' is. The install docs suggest you install to a flash or USB device, so why not pxeboot. I'm also not sure where IPMI fits into this.
Now that I look at it, I think they are just using it to install to their local hard drives. I don't understand why they discourage using a hard drive. Who cares if you lose some storage. Drives are cheap.

Do whatever pleases you, but, at the end, something has to store the boot image - in most cases this ends up being FreeNAS, so your use case is unusual. Especially now that you can trivially mirror boot devices.

 

[Drew Morone]

Do whatever pleases you, but, at the end, something has to store the boot image - in most cases this ends up being FreeNAS, so your use case is unusual. Especially now that you can trivially mirror boot devices.

Yes. That would be my PXE server which stores a whole bunch of images. I don't have a FreeNAS box yet so I can't use that (duh).

 

[Marcos Della]

So I have a similar problem. I have a rather large data center in London... However me and my team are in Palo Alto, CA... I have a fresh box out in London, an SD card in it (blank), 16 drives, a network connect, and iLO acces... and I want to get FreeNAS on the SD card and the box running freenas. I actually have several sites around the world like this, but using this as an example.

EVERYTHING we do allows us to remotely build boxes via PXEboot. Our PXE environment is all CentOS based and currently is used to boot thousands of machines and VMs. Unfortunately most of the people that build the PXE environment are no longer here (that's what happens when your company gets bought, options expire, etc). So I have some talented people, but not the people who built it in the first place.

So... we've "moved" the 9.3.0-STABLE version out to the NFS server that serves the environment. We have both the ISO as well as the iso exploded into a mountable tree. What we *don't* know is what are the pxeclient.cfg options needed to boot the ISO off of our environment. Our goal is to boot the "install" ISO, not an operating FreeBSD. We only want the machine to perform its installation out there on its local SD card or worst case, into the ZFS itself (now that FreeBSD supports this).

So, unlike many others, we can't just "insert" a USB stick, SD card, etc into the machine without either a FedEx shipment with hours on the phone with some even lower paid data center "remote hands" to try and even find the machine to stick it into (that's dangerous for those that have worked with remote hands before), or a very expensive flight for an engineer to go out and do maintenance and roll this operation into that (we only do maintenance twice a year so that's unlikely).

So... how to boot the installation ISO over the network, not how to boot FreeNAS itself...

Any clues or help? I'll admit, we've only put a few days into figuring this out (well, hours if you count interruptions) but figured we weren't the only ones out there with this issue/problem.

Marcos

 

[Drew Morone]

Unfortunately, I gave up and ended up mounting remote media through the OOB device, and installing to local storage. It only took a few hours to boot! Woo! Ultimately I spent too much time, and since I was a one time setup (I'm not setting up a FreeNas farm or anything) I figured it wasn't worth it.

Anyway, I don't know why this sounds like such an exotic idea to some people. Maybe they've never worked in the real world with remote data centers and inept hands on people. All of our other servers will self install using pxe. I've never tried a FreeBSD pxe install, or booting a rescue disk. Redhat's kickstart works very nicely. I wish they would all use the same method.

 

[cyberjock]

Yeah, PXE is a mess unless you understand it. It's not trivial to just "pick up in a few hours".