Jail Migration Script (Tested on 11.1 -> 11.2)

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
Hello all,

EDIT: Please try the official migration script migrate_warden.py before this. This was written time ago when the official script sucked at getting jails to work out of the box. If your jails are starting and working fine after migrate_warden.py then you're good to go.

I wrote this script the other day to facilitate moving all my jails from 11.1 to 11.2 (from warden to iocage).

It's very easy to use and it will do all your jails or just a single one. It's in an ok shape. It converted all my jails successfully and without any issues. Does some minimal error checking and all that but I was too lazy to check return codes for everything. I tested this on my actual server. My networking setup is very minimal aka DHCP so that would reflect on the config.patch file. You can change the patch file however you want and you'll get anything out of the old config to the new. I believe that the patch file changes are crucial to a successful migration. Maybe stuff like release name isn't that crucial, but to be congruent as to how the 11.2 GUI generates new jails, I put them in there.

One thing people will definitely like for sure is how it converts the MAC address to the new format AND it increments the second MAC address just like many have posted to make the networking work out of the box.

If you use it please back up your jail config foremost and then anything else necessary. It uses migrate_warden.py so just like that script it will NOT mess up your old jails.

I believe everything that I did, migrate_warden.py should do by default. But of course we cannot have that.

In case you run into issues just do iocage destroy jail_name --recursive and try again.

That's how I got it to the point where it is without 0 issues.

Link to Github: warden-migration-script

Please improve it if you are so kind to do so. One thing that comes to mind is that it needs to do mount points as well. I did those manually since I felt it would be too much of a hassle to code it up.

Enjoy
 
Last edited:

Darkflame

Dabbler
Joined
Sep 25, 2016
Messages
39
would it work for 9.10 to 11.2 aswell?
 

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
I got FreeNAS when it was on 11.1 and frankly I am not familiar with 9.10 setup at all.

However, these are the conditions for it to work:

It would work if:
1. migrate_warden.py supports 9.10 jails.
2*. 9.10 and 11.1 don't have completely different config.json files.

*: When it comes to the config file you can make it look however you want. The config.patch I supplied will only work (afaik) for 11.1 to 11.2

Steps for those who don't know how to diff and patch a file (to migrate from 9.10 to 11.2):
1. Create an 11.2 default jail from GUI.
2. Get config.json from that jail (I'll call it 11.2-config.json)
3. Get config.json from your 9.10 jail (I'll call it 9.10-config.json)
4. diff 9.10-config.json 11.2-config.json > config.patch
5. Modify config.patch to your needs (stuff you want in the config you remove from patch and stuff you need you can add)
6. Save the patch file and supply it to the script I wrote.

That procedure should 100% work for any config.json.

In the end a little experiment does not hurt. Create a test jail in 9.10 and use the single jail mode on the script to migrate only that jail. See what it does. But first make sure those 2 conditions I mentioned are true.
 

enemy85

Guru
Joined
Jun 10, 2011
Messages
757
Hello all,

I wrote this script the other day to facilitate moving all my jails from 11.1 to 11.2 (from warden to iocage).

It's very easy to use and it will do all your jails or just a single one. It's in an ok shape. It converted all my jails successfully and without any issues. Does some minimal error checking and all that but I was too lazy to check return codes for everything. I tested this on my actual server. My networking setup is very minimal aka DHCP so that would reflect on the config.patch file. You can change the patch file however you want and you'll get anything out of the old config to the new. I believe that the patch file changes are crucial to a successful migration. Maybe stuff like release name isn't that crucial, but to be congruent as to how the 11.2 GUI generates new jails, I put them in there.

One thing people will definitely like for sure is how it converts the MAC address to the new format AND it increments the second MAC address just like many have posted to make the networking work out of the box.

If you use it please back up your jail config foremost and then anything else necessary. It uses migrate_warden.py so just like that script it will NOT mess up your old jails.

I believe everything that I did, migrate_warden.py should do by default. But of course we cannot have that.

In case you run into issues just do iocage destroy jail_name --recursive and try again.

That's how I got it to the point where it is without 0 issues.

Link to Github: warden-migration-script

Please improve it if you are so kind to do so. One thing that comes to mind is that it needs to do mount points as well. I did those manually since I felt it would be too much of a hassle to code it up.

Enjoy

i can't understand what to do with "config.patch" file...can u explain please? thanks
 

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
What exactly don't you understand?

This is how you migrate all your jails:
./migrate-jails.sh -p <pool_name> -f config.patch

This is how you migrate only one specified jail:
./migrate-jails.sh -p <pool_name> -f config.patch -j <jail_name>

Unless you need more config on your jails and your config.patch needs to be different please refer to my other answer above.
As for the one I have provided you simply provide it as input to the script like above. No need to change anything.
 

mattyams

Cadet
Joined
Apr 20, 2016
Messages
2
Hi, thanks for the script! I tried to run it but I get no output after attempting to execute it. Any ideas? I've tried running the script outside the pool as well.

Code:
root@homenas:/mnt/backup # ./migrate-jails.sh -p backup -f config.patch
root@homenas:/mnt/backup #
 

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
Hi, thanks for the script! I tried to run it but I get no output after attempting to execute it. Any ideas? I've tried running the script outside the pool as well.

Code:
root@homenas:/mnt/backup # ./migrate-jails.sh -p backup -f config.patch
root@homenas:/mnt/backup #

The first issue that comes to mind is the fact that your jails might not be on the standard path. The script does this:

1. You give it a pool name
2. Goes to look into /mnt/<pool_name>/jails

I notice from your reply that you are in /mnt/backup. That's not gonna work. Furthermore, migrate_warden.py will also not recognize it either. So that's why you can't just give my script a path since migrate_warden.py won't work with a random path:

Code:
migrate_warden.py <-v> -j <jail> -p <iocage-pool>


Your jails have to be in /mnt/<pool_name>/jails.

Since you have your jails in /mnt/backup and you don't wanna mess with your current pool I would then suggest making a new pool and migrating your jails there under the correct path such as /mnt/pool2/jails
 

jaccovdzaag

Dabbler
Joined
Feb 9, 2018
Messages
22
Hi,

Just updated to 11.2-RC1. Everything works great. Then I found your script, which looks amazing. However, for me, I got a strange error when I wanted to migrate all the jails in one time:

root@FREENAS:/ # ./migrate-jails.sh -p NAS -f config.patch
After giving the script a go, I got this. And it doesn't continue. Well, the migration succeeded however, it is working as well. Did it one by one in the end. Really great work, I appreciate it a lot.

Done migrating couchpotato_1
Applying patch file to couchpotato_1 config.json
Backing up old couchpotato_1 config.json...
mv: rename /mnt/iocage/jails/couchpotato_1/config.json to /mnt/iocage/jails/couchpotato_1/config.json.old: No such file or directory
Hmm... Looks like a normal diff to me...
patch: **** can't find /mnt/iocage/jails/couchpotato_1/config.json.old
Failed to patch config.json for couchpotato_1
Restoring old config...
mv: rename /mnt/iocage/jails/couchpotato_1/config.json.old to /mnt/iocage/jails/couchpotato_1/config.json: No such file or directory
 

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
Hi,

Just updated to 11.2-RC1. Everything works great. Then I found your script, which looks amazing. However, for me, I got a strange error when I wanted to migrate all the jails in one time:

root@FREENAS:/ # ./migrate-jails.sh -p NAS -f config.patch
After giving the script a go, I got this. And it doesn't continue. Well, the migration succeeded however, it is working as well. Did it one by one in the end. Really great work, I appreciate it a lot.

This seems to me like a path problem. From what I can see migration runs fine and then when the config.json needs to be modded it fails. My script renames the existing config.json to config.json.old so that it doesn't erase your config and then uses config.json.old and the config.patch to create a new config.json.

Can you check that config.json exists in the jail folder: /mnt/iocage/jails/couchpotato_1/
If it does then I have no idea why the script would fail there. It could also be that the jail folder name is different.
 

jaccovdzaag

Dabbler
Joined
Feb 9, 2018
Messages
22
This seems to me like a path problem. From what I can see migration runs fine and then when the config.json needs to be modded it fails. My script renames the existing config.json to config.json.old so that it doesn't erase your config and then uses config.json.old and the config.patch to create a new config.json.

Can you check that config.json exists in the jail folder: /mnt/iocage/jails/couchpotato_1/
If it does then I have no idea why the script would fail there. It could also be that the jail folder name is different.

Yes, the migration runs fine. I ended up (before I saw your reply) by doing one by one, as stated above already I see.
I've checked if i have a config.json in /mnt/NAS/iocage/jails/couchpotato_1, and it is present.

So, it's a rather strange error. Could be the jail folder name thing which is off. Well, migration succeeded though :)
 

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
... /mnt/NAS/iocage/jails/couchpotato_1, and it is present.

So it is a path issue. Your jails are not in the conventional path.

Per the doc: https://doc.freenas.org/11.2/jails.html#additional-storage

"New directories created must be within the jail directory structure. Example:/mnt/iocage/jails/samplejail/root/new-destination-directory."

Your path is: /mnt/NAS/iocage/jails/couchpotato_1
Conventional path would be: /mnt/iocage/jails/couchpotato_1

Thats why my script failed. I am glad the migration went through nicely however. I would thank migrate_warden.py authors for that.

P.S

I advise you to reconsider changing your jail paths. I don't remember exactly why they moved jails from /mnt/pool/warden to /mnt/iocage (security issue?) but in the future you'll have less issues if u stick with their structure and what their docs say :)

Cheers!
 

jaccovdzaag

Dabbler
Joined
Feb 9, 2018
Messages
22
Yes, I see that now. That is strange.

I did an upgrade from 11.1-U6, so it took over it's paths (ofcourse). So that's why 11.2 and the migration script doesn't get it, since it's not adjusted to it's new path.
As you say, maybe a security issue. But during such an upgrade, you would suspect them to change the paths, even if they are old warden type jails, to the new path.

However, it worked. Great work by the authors, and you for sharing and giving advice/help. As for the future, I'm thinking of doing my jails all over again, begin from scratch, to have the correct paths. Some of the jails I now have are made by the plugin option, but I might create them by myself and do proper package installs to keep it up to date.

I'll be back!
 

Randy Smyth

Dabbler
Joined
Apr 9, 2017
Messages
21
I advise you to reconsider changing your jail paths. I don't remember exactly why they moved jails from /mnt/pool/warden to /mnt/iocage (security issue?) but in the future you'll have less issues if u stick with their structure and what their docs say :)

This is my situation.
My "tank" pool is the entire drive set. My warden jails previously were part of the /mnt/tank/jails so in the upgrade to 11.2 from 11.2 the system added /mnt/tank/iocage/jails.
I would like to change the jails paths from /mnt/tank/iocage to /mnt/iocage, but I do not know of a method. Can I shrink my previous pool to allow a small 10gb pool to be created called iocage? is that how it works?
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,374
I'm confused, isn't there an official script from the developers?
 

Rico Heil

Dabbler
Joined
Jan 6, 2017
Messages
17
Thank you very much for this helpful script!
It made migrating my two jails very easy - only the config.patch threw errors, but setting up the network config was no problem.
They are up an running after only a few minutes! :cool:

So I am left with what might be a dumb question:
What do I do with the old jails in the legacy UI?
Is it safe to delete them?
 

arn0z

Dabbler
Joined
Oct 14, 2018
Messages
21
@diskdiddler My script is a wrapper to the official one. The official one did not update the jail config files correctly leading to jails not starting (MAC address updates and such). I do not know the state of the official script as of now. I'll add a statement to my post so people try that first.

@Rico Heil I'd keep them around for a while in case you need something from them. I kept mine around until recently so maybe 6 months? Then yeah I don't see the point of them being around.
 

Dalba

Cadet
Joined
Apr 12, 2014
Messages
5
Hello,
I just upgraded to version FreeNAS-11.2-U4.1 from 11.1 U7, and i'm trying to use your script to migrate my jails with no success...
I always getting this :
Code:
root@freenas:/ # ./migrate-jails.sh -p Pool1 -f config.patch
Migrating Jackett
-- Migrating: Jackett --
Traceback (most recent call last):
  File "/usr/local/sbin/migrate_warden.py", line 442, in <module>
    loop.run_until_complete(main(sys.argv[1:], loop))
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/local/sbin/migrate_warden.py", line 432, in main
    await Migrate(jail, _dir, iocage_pool, verbose, loop).migrate_jail()
  File "/usr/local/sbin/migrate_warden.py", line 192, in migrate_jail
    iocroot = self.zfs.get_dataset(f"{self.pool}/iocage").mountpoint
  File "libzfs.pyx", line 597, in libzfs.ZFS.get_dataset
libzfs.ZFSException: Dataset Pool1/iocage not found
Failed to migrate Jackett
Exiting...


Any ideas about what is happening ?
 

Dalba

Cadet
Joined
Apr 12, 2014
Messages
5
OK, nevermind, i had just to initialize and activate my pool for iocage... Now it's working good.
 

ChiknNutz

Patron
Joined
Nov 6, 2015
Messages
217
Have been trying to get this to work. Not sure why, but it only seems to recognize couchpotato, though I have several others too. This is what I get after creating the "migrate-jails.sh" and "config.patch" files in both the main pool and also within the jail directory, same results in both cases. I am very much a novice when it comes to doing this type of thing, have only really touched the code of my FreeNAS when first setting it up.

Code:
root@AdcNAS:/mnt/Rivendell/scripts # ./migrate-jails.sh -p Rivendell -f config.patch
Migrating couchpotato_1
-- Migrating: couchpotato_1 --
  couchpotato_1 is running, please stop it first.
Done migrating couchpotato_1
Applying patch file to couchpotato_1 config.json
Backing up old couchpotato_1 config.json...
mv: rename /mnt/iocage/jails/couchpotato_1/config.json to /mnt/iocage/jails/couchpotato_1/config.json.old: No such file or directory
patch: option requires an argument -- o
usage: patch [-bCcEeflNnRstuv] [-B backup-prefix] [-D symbol] [-d directory]
             [-F max-fuzz] [-i patchfile] [-o out-file] [-p strip-count]
             [-r rej-name] [-V t | nil | never | none] [-x number]
             [-z backup-ext] [--posix] [origfile [patchfile]]
       patch <patchfile
./migrate-jails.sh: /mnt/iocage/jails/couchpotato_1/config.json: not found
Failed to patch config.json for couchpotato_1
Restoring old config...
mv: rename /mnt/iocage/jails/couchpotato_1/config.json.old to /mnt/iocage/jails/couchpotato_1/config.json: No such file or directory
root@AdcNAS:/mnt/Rivendell/scripts #
 
Top