Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

[HOWTO] AoE (ATA over Ethernet) setup using Plugins jail

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
Install the plugins jail if you haven't already.

Open a root shell prompt, and enter the jail.
Code:
jexec `jls jid` tcsh

Install vblade from packages.
Code:
pkg_add -vr vblade

Create a vblade device file. This one is ~20GB
Code:
dd if=/dev/zero of=/vblade0 count=20000 bs=1M

Find out the name of the plugin jail's virtual network adapter
Code:
ifconfig | grep BROADCAST | awk '{print $1}' 
The above command yielded "epair0b:" on my jail setup. (Don't forget to drop the colon in the next command.)

Start vblade on the FreeNAS jail;
Code:
vblade 1 1 epair0b /vblade0

You should see some output like this:
Code:
ioctl returned -1
20971520000 bytes
pid 30146: e1.1, 40960000 sectors O_RDWR


Keep the vblade command running in the shell for now. Later on, you'll probably want to kill and restart it as a daemon.
Code:
vbladed 1 1 epair0b /vblade0


Note: The following client side steps are for an Arch linux workstation with aoetools and the aoe kernel module already loaded

This client command should show the recently created vblade AoE device:
Code:
aoe-discover; aoe-stat
e1.1        20.971GB  eth0 up 

Now the AoE device should be available at /dev/etherd/e1.1.

You can partition and create filesystems on this device just like you would any other block device, for example:
Code:
sgdisk -n 1:2048:0 -c 1:"FreeNAS AoE Data Disk" /dev/etherd/e1.1
mkfs.ext4 /dev/etherd/e1.1p1
mount /dev/etherd/e1.1p1 /mnt

Now you're ready to start using the AoE disk.
 

ProtoSD

FreeNAS Guru
Joined
Jul 1, 2011
Messages
3,353
Nice hack, but it should be noted that the proper place to install ANY packages is INSIDE the plugin jail. The minimal free space available on your flash drive is needed by the OS, and it WILL crash if runs long enough and the OS needs the space. This hack should still be able to be done from the jail with some minor adjustments.
 

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
Nice hack, but it should be noted that the proper place to install ANY packages is INSIDE the plugin jail. The minimal free space available on your flash drive is needed by the OS, and it WILL crash if runs long enough and the OS needs the space. This hack should still be able to be done from the jail with some minor adjustments.
Thanks for the heads up and technical advice. I originally tried to do this in a plugin jail, however I'm not very knowledgeable about how to properly set one up. I quickly became confused and opted for the more straightforward manual approach seen.

If you have a few quick steps that I should take to convert the howto into a plugin jail method, please post.
 

cyberjock

Moderator
Joined
Mar 25, 2012
Messages
19,148
You know, it would be nice if there was a release version around somewhere that used 2GB partitions instead of 1GB partitions so you have free space. Maybe not provide the link from the webpage so that only the most advanced users will use this modified version.
 

ProtoSD

FreeNAS Guru
Joined
Jul 1, 2011
Messages
3,353
Thanks for the heads up and technical advice. I originally tried to do this in a plugin jail, however I'm not very knowledgeable about how to properly set one up. I quickly became confused and opted for the more straightforward manual approach seen.

If you have a few quick steps that I should take to convert the howto into a plugin jail method, please post.
I'll give you a hint. The jail needs permission to access devices on the host system, this is done with devfs.rules. After creating a rule, it has to be added to rc.conf. I guess a ticket should be opened (support.freenas.org) to add a field for auxiliary parameters for jails, since now all that stuff gets stuffed into the freenas database and its not as easy to tweak. Here's an old link about devfs.rules

Here's an example of the lines you'd need to add to rc.conf:

Code:
jail_Jail_devfs_enable="YES"
jail_Jail_devfs_ruleset="devfsrules_jail"
 

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
Re: Quick and dirty AoE (ATA over Ethernet) setup

Good news: Preliminary testing appears to indicate that vblade is able to use the jail virtual network adapter w/out creating any special devfs.rules. I'll update the guide soon with the steps to run this under a jail.
 

cyberjock

Moderator
Joined
Mar 25, 2012
Messages
19,148
Re: Quick and dirty AoE (ATA over Ethernet) setup

Outstanding! I'm glad you are keeping us updated on this. AoE(not Area of Effect for you MMORPG players.. I'm one of those.. :/ )

I'd love to see this implemented into FreeNAS and not be a jailed feature.

Edit: I'd also like to point out that I'd expect that using AoE would have the same potential issues as iscsi with ZFS.
 

ProtoSD

FreeNAS Guru
Joined
Jul 1, 2011
Messages
3,353
Re: Quick and dirty AoE (ATA over Ethernet) setup

It is a nice feature, almost like and alternative to iSCSI. It seems like a reasonable feature to be added. Maybe open a ticket with a feature request at support.freenas.org :)
 

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
Re: Quick and dirty AoE (ATA over Ethernet) setup

Outstanding! I'm glad you are keeping us updated on this.
Thanks. I'm glad to see that there are other people out there interested in using AoE w/FreeNAS. So much easier to use than iSCSI IMO.
I'd love to see this implemented into FreeNAS and not be a jailed feature.
I actually suggested this in person to Alfred (lead FreeNAS developer) at Scale this year. He seemed interested and asked me to email him a link to the guide, which I did.

I sent a follow-up email the other week, but haven't heard anything back yet. Alfred, if you're listening, please post with an update.
 

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
Update, something has changed on my FreeNAS system and vblade is no longer running within the jail. It runs fine outside the jail, but trying to launch an AoE instance within the jail fails like this:

Code:
jail# vblade 1 1 epair0b vblade0
ioctl returned -1
20971520000 bytes
open: No such file or directory
pid 95655: e1.1, 40960000 sectors O_RDWR
putpkt aoe id: Bad file descriptor
read: Bad file descriptor


Not sure why this is happening when it was working for several weeks beforehand. I have tried deleting the entire plugins jail and re-installing a fresh jail, to no avail.

An email to the FreeBSD vblade port maintainer has gone unanswered. Any troubleshooting suggestions welcomed.
 

ProtoSD

FreeNAS Guru
Joined
Jul 1, 2011
Messages
3,353
What version of FreeNAS are you using? Did you do any updates of anything lately?
 

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
FreeNAS-8.3.1-RELEASE-x64 (r13425). That's the thing, I can't remember for sure if I first got it working on a release candidate and then upgrading to the release and that broke it. I think this is what happened, but I'm not positive. Downgrading to a release candidate just to test would suck.
 

ProtoSD

FreeNAS Guru
Joined
Jul 1, 2011
Messages
3,353
Don't you have another USB stick you can install a release candidate on?

I haven't done it, but you're supposed to be able to import jails. That wouldn't be as bad to try.
 

guglez

Newbie
Joined
Feb 21, 2014
Messages
55
Strange. This how-to is not working for me.

Code:
Mar  2 23:22:21 aoe_jail syslogd: kernel boot file is /boot/kernel/kernel
Mar  2 23:23:21 aoe_jail vbladed: ioctl returned -1
Mar  2 23:23:21 aoe_jail vbladed: 20971520000 bytes
Mar  2 23:23:21 aoe_jail vbladed: pid 31263: e1.1, 40960000 sectors O_RDWR


can't see any AoE devices on my Linux box
Am I doing something wrong? What about the feature request?
 

sysfu

Newbie
Joined
Jun 16, 2011
Messages
71
Haven't tried this in a year or so and something broke the process in later releases. The feature request seems to have been forgotten for the mean time, maybe it would be worthwhile to bump it with a comment. If I get some time to revisit this and get it working again, I'll post back with any revisions to the process.
 

aufalien

FreeNAS Experienced
Joined
Jul 25, 2013
Messages
337
Well, I don't think its forgotten, seems to be scheduled in 9.2.3 beta which is just around the corner.
 
Top