FreeNAS 11.2 all jail access failed after replicating a pool

[dmshimself]

I had all my jails running in a pool which is snapshot recursively once a day. I decide to replicate those to a different pool on the same server and having done that I couldn't access any jails. From the GUI all that showed was an activation of a pool request. From the NAS shell I got the following:

Code:

# iocage list
Pools:
  General
  OurData
  backup
  freenas-boot
You have 2 pools marked active for iocage usage.
 Run "iocage activate ZPOOL" on the preferred pool.

In my case the origonal pool activated to hold jails was General. It looks as though if you replicate a pool which holds the active jails, the resulting replicated area also has this same activated property set and bang. In my case, I activated the origonal pool, restarted all the jails and all was well. However it was worrying for a few minutes and I have stopped the replication for the moment.

Is there a safe way to stop this happening again?

 

[dlavigne]

That's not so bad that it prompts to activate the desired pool. If you think it is a bug however, create a report at bugs.freenas.org and post the issue number here.

 

[dmshimself]

Will do. I've also found that if you replicate the pool down a level to something like backup/pool, it does the same thing. The jails are no longer accessible, the pool used for the jail has to be redone and the pools restarted.

The issue is number 66853

 

[Apollo]

@dmshimself, one of the issue with replication to another location will not allow jails to work for the follwoing reasons:

- Jails must use the Pool you want the jail to be active ( not a bug and not a problem):
You can have several pools containing different Jails and you can switch back and forth. Quite useful indeed.
- If you replicate to a different Pool name or replicate to the same or different Pool within another Dataset, all your jails mount point must be updated to point to the new locations. For that, you can edit the jails mount point which are stored in the "fstab" file.

If you do the above adjustment/correction, then you will find your jails working the way they meant to be without having to reinstall them.

 

[dmshimself]

I agree having several jail mount points is handy and flicking between them is a neat facility. In my case, the selection of which one is active is apparently uncertain to FN after the replication. I'd suggest that in the case of replication, the administrator has not chosen to switch the jail to a new area, so the current area they have selected should remain the active one, IMHO. So post replication, FN could reset the jail selected to the one the administrator has chosen or exclude replicating whatever widget it is that makes the replicated area active. Leave it inactive and all should be well.

In the short term I'm very interested in your suggestion to manipulate the fstab file in some way as I know which area the jails should be using. My google skills have let me down though and I'm guessing you don't mean editing /etc/fstab on the FN box. Could you do me a favour and point me in the right direction to track down the mod and I'll for sure give it a go and report back.....

 

[Apollo]

@dmshimself,
On my test system, I have a SSD I use to host the iocage I want to play with.
I have been experimenting with Nextcloud install and update quite a bit before having them connect to my live data. Because of it, I have been creating the various environment (datasets) to be used by the iocage jail.
I have been using Danb35 script

https://forums.freenas.org/index.ph...llation-of-nextcloud-14-in-iocage-jail.62485/

and applied the following steps to reach my goals which are to install a new Nextcloud jail pointing to a brand new database and folder datasets and then once everything is working, then I can remap the db (database) and files datasets to my existing live Nextcloud db and files datasets.


the "ssd" pool is located as follow:

/mnt/ssd

I have the few iocage jail environments as such:

iocage jail jail_example1:
/mnt/ssd/iocage_setup1
/mnt/ssd/iocage_setup1/jail_example1/db
/mnt/ssd/iocage_setup1/jail_example1/files
/mnt/ssd/iocage_setup1/jail_example1/portsnap
/mnt/ssd/iocage_setup1/jail_example1/install

iocage jail jail_example2:
/mnt/ssd/iocage_setup1
/mnt/ssd/iocage_setup1/jail_example2/db
/mnt/ssd/iocage_setup1/jail_example2/files
/mnt/ssd/iocage_setup1/jail_example2/portsnap
/mnt/ssd/iocage_setup1/jail_example2/install

iocage jail jail_example3:
/mnt/ssd/iocage_setup2
/mnt/ssd/iocage_setup2/jail_example3/db
/mnt/ssd/iocage_setup2/jail_example3/files
/mnt/ssd/iocage_setup2/jail_example3/portsnap
/mnt/ssd/iocage_setup2/jail_example3/install


My live Nextcloud db and files dataset can be located as such:

/mnt/mirror-4TB/cloud
/mnt/mirror-4TB/cloud/db
/mnt/mirror-4TB/cloud/files

When the various jails have been created, even though the exist under the "ssd" pool, Freenas upon activating the necessary jail system must be told to be using "ssd" using the following command"

iocage activate ssd

upon such command, Freenas will create the "iocage" mount point under the "/mnt" folder structure.

Listing the content of "/mnt" should return something like that:

/mnt/ssd
/mnt/iocage
/mnt/mirror-4TB
/mnt/md_size


In reality, "mnt/iocage" is not a physical object but equivalent to a symlink if you get my drift, at least this is what I think.

The "fstab" assigned to the jail will be located as follow:
One single "fstab" file per jail:

/mnt/iocage/jails/jail_example1
/mnt/iocage/jails/jail_example2
/mnt/iocage/jails/jail_example3

You can then edit the relevant "fstab" using your prefered file editor, I like "nano":

nano /mnt/iocage/jails/jail_example1/fstab
You should get something similar to what follows:


/mnt/ssd/iocage_setup1/jail_example1/portsnap/ports /mnt/iocage/jails/jail_example1/root/usr/ports nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:05
/mnt/ssd/iocage_setup1/jail_example1/portsnap/db /mnt/iocage/jails/jail_example1/root/var/db/portsnap nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:06
/mnt/ssd/iocage_setup1/jail_example1/files /mnt/iocage/jails/jail_example1/root/mnt/files nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:06
/mnt/ssd/iocage_setup1/jail_example1/db /mnt/iocage/jails/jail_example1/root/var/db/mysql nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:06


Now I just need to edit "fstab" content related to the source of "files" and "db" datsets and replace them with the one where my Nextcloud datasets resides.
This should give something like such:


/mnt/ssd/iocage_setup1/jail_example1/portsnap/ports /mnt/iocage/jails/jail_example1/root/usr/ports nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:05
/mnt/ssd/iocage_setup1/jail_example1/portsnap/db /mnt/iocage/jails/jail_example1/root/var/db/portsnap nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:06
/mnt/mirror-4TB/cloud/files /mnt/iocage/jails/jail_example1/root/mnt/files nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:06
/mnt/mirror-4TB/cloud/db /mnt/iocage/jails/jail_example1/root/var/db/mysql nullfs rw 0 0 # Added by iocage on 2018-11-17 06:50:06


If doing a true script installation of Nextcloud, there will be extra elements that I need to update but for the mount point location, this is all that is needed.
Once updated, you will need to restart the corresponding jail such as:

iocage restart jail_example1

Doing the following will list the various jails recognized by the system and their status with the mapped IP address:
iocage list

If you need to change the IP address, the name of whatever is used to configure the jail, you can first have a look at the "config.json" file located in the same folder as "fstab". I beleive it is only read only and therefore cannot edit it. If not read only, it is only indicating the parameters used by the jail, and even you manage to edit it, it will not post once the jail is restarted, I think.
Instead, you will need to run some commands.
To get the parameters as show in "config.json", you can run:

iocage get all jail_example1

Let's say, I want to change the IP address of the jail which is listed on my system as follow:

ip4_addr:igb0|192.168.2.120/24

I can also do the following to get only one single parameter:

iocage get ip4_addr jail_example1
It should return:

igb0|192.168.2.120/24

All I need to do to update the IP address to "192.168.2.222" is to do the following:

iocage set ip4_addr=192.168.2.222 jail_example1

You can see the new IP address using:
iocage list

 

[dmshimself]

Thanks @Apollo - appreciated. I think this method might still run into issues though. Let be run something by you. In that "fstab" section for several of the jails I have there are a while bunch of other references to /mnt/iocage for releases, /usr/bin and so on. Vital items for correct running. If a replication occurs and the confusion on which is the master iocage happens, there is no user level opportunity to get in and re-edit those "fstab" entries automatically. Also in my case when this happens, the jails stop being available. I don't exactly why, but I'm guessing it would be like the rug being pulled from them and some of the key system areas /usr/lib being changed on the fly to something that the snapshot had as contents at the time the snapshot was taken. Whatever it is, in my use case, the jails stop working as far as the outside world is concerned.

If I'm right (!) for a production scenario, you'd still need to do some manual edit and jail restarts. I accept it could be more scripted, but you still have run the script or do the edits. In my current setup, I would let people know there would be an outage, reset the iocage location, and restart the jails, so different manual work.

At the moment the dev team I think are looking at the issue, so it would be good to get their comments too.

I'm happy there are at least a couple of ways to recover correct operation, but my feeling is that this is a bit of a ticking time bomb waiting to sneak up on the unwaring administrator who doesn't know it will happen.

 

[Apollo]

Let be run something by you. In that "fstab" section for several of the jails I have there are a while bunch of other references to /mnt/iocage for releases, /usr/bin and so on. Vital items for correct running. If a replication occurs and the confusion on which is the master iocage happens, there is no user level opportunity to get in and re-edit those "fstab" entries automatically. Also in my case when this happens, the jails stop being available. I don't exactly why, but I'm guessing it would be like the rug being pulled from them and some of the key system areas /usr/lib being changed on the fly to something that the snapshot had as contents at the time the snapshot was taken. Whatever it is, in my use case, the jails stop working as far as the outside world is concerned. [/QUOTE
If the iocage jail is not up and running, it can be caused by redundant use of an IP address or possibly having several jails pointing to the same mount points. I think I have seen it, and if correct, Freenas or iocage will prevent the jail to start as it might lock the resource to prevent uncontrolled access.
If iocage is not able to mount a jail, while being located at "/mnt/iocage/jail", I believe you can access the details of the jail and "fstab" directly under the proper iocage location.
I am not so sure of the default iocage location as I started creating jails as early as Freenas 10 I think and I might have used the iocage as the name of my Dataset back then.
There was alos one instance where the "/mnt/iocage" didn't exist, but I seem to recall jails were installed and running. This was on another test system.

I think you just need to find the location of the iocage dataset under your pool, such as "/mnt/ssd/iocage" such as in my case, and you should be able to find your various jails being listed there under "/mnt/ssd/iocage/jails".

In regard to your question about the other instances of /mnt/iocage for releases, I am unclear.
Are you saying you have other elements being mapped in your fstab that point to the "/mnt/iocage/release" location?
In my fstab files, the iocage being ran are pointing to the "db" and "ports" section within "portsnap" folder on my ssd. If you did not create the "portsnap" dataset when the jail was created, it would have defaulted to a different location on the main pool, I think.

 

[Apollo]

@dmshimself, One thought in mind: If you are worried about replication as a mena to create an image copy of the Pool on the server so that in the event somethign terribly went to happen, you are thinking about setting a new server with the pool from the replicated image. If that is the case, then you can avoid such scenario and be up and running in no time if you replicate the main pool to another server over LAN, where the name of the pool on the backup server will have the same name as the source server. Mind you that you will be loosing everything that was not including from the time of the latest snapshots.

 

[dmshimself]

Here is the list of a the "fstab" from a standard iocage jail I have running. As you can see there are *lots* of entries created automatically.

My use of replication here is not really to allow me to more easily set up another server in case the main one fails. It's really to give me a quick way to get to old versions of the data on the same server, but I take your point and it is a good one.

Code:

/mnt/iocage/releases/11.2-RELEASE/root/bin      /mnt/iocage/jails/syncthing/root/bin    nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:34
/mnt/iocage/releases/11.2-RELEASE/root/boot     /mnt/iocage/jails/syncthing/root/boot   nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:34
/mnt/iocage/releases/11.2-RELEASE/root/lib      /mnt/iocage/jails/syncthing/root/lib    nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:34
/mnt/iocage/releases/11.2-RELEASE/root/libexec  /mnt/iocage/jails/syncthing/root/libexec        nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:34
/mnt/iocage/releases/11.2-RELEASE/root/rescue   /mnt/iocage/jails/syncthing/root/rescue nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:34
/mnt/iocage/releases/11.2-RELEASE/root/sbin     /mnt/iocage/jails/syncthing/root/sbin   nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:34
/mnt/iocage/releases/11.2-RELEASE/root/usr/bin  /mnt/iocage/jails/syncthing/root/usr/bin        nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:35
/mnt/iocage/releases/11.2-RELEASE/root/usr/include      /mnt/iocage/jails/syncthing/root/usr/include    nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:35
/mnt/iocage/releases/11.2-RELEASE/root/usr/lib  /mnt/iocage/jails/syncthing/root/usr/lib        nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:36
/mnt/iocage/releases/11.2-RELEASE/root/usr/libexec      /mnt/iocage/jails/syncthing/root/usr/libexec    nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:36
/mnt/iocage/releases/11.2-RELEASE/root/usr/sbin /mnt/iocage/jails/syncthing/root/usr/sbin       nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:36
/mnt/iocage/releases/11.2-RELEASE/root/usr/share        /mnt/iocage/jails/syncthing/root/usr/share      nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:40
/mnt/iocage/releases/11.2-RELEASE/root/usr/libdata      /mnt/iocage/jails/syncthing/root/usr/libdata    nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:40
/mnt/iocage/releases/11.2-RELEASE/root/usr/lib32        /mnt/iocage/jails/syncthing/root/usr/lib32      nullfs  ro      0       0 # Added by iocage on 2018-12-10 21:05:40
/mnt/OurData/OurImages/syncthing        /mnt/iocage/jails/syncthing/root/Sync   nullfs  rw      0       0 # Added by iocage on 2018-12-10 21:16:40

 

[Apollo]

@dmshimself,
From what I can tell, all but the below line reference the iocage jail itself.
/mnt/OurData/OurImages/syncthing /mnt/iocage/jails/syncthing/root/Sync nullfs rw 0 0 # Added by iocage on 2018-12-10 21:16:40
What seems to be the issue is that you are missing the pool "OurData".
Do an import from existing volume under Storage Web GUI and see where this is leading you to.
I have never used this "syncthing" jail so I can't really comment on it.

 

[Apollo]

It's really to give me a quick way to get to old versions of the data on the same server, but I take your point and it is a good one.

Then why creating replication in the first place?
If you have snapshots in place, and I believe you do, otherwise there would be no replication, you can create a clone of the snapshot.
I was unsuccessful doing just that and having a Nextcloud instance access the data on the clone, but I think I fell through the crack due to Nextcloud 14 bugs with apps causing the upgrade to take place.

The point is that with a clone, the data will be available at the time of the snapshot being cloned. That is, if you do not intend on updating the clone with new live data.

 

[Apollo]

@dmshimself,I was able to install the plugin for "syncthing" and it was installed under the follwoing path on my syste:

Code:

 /mnt/ssd/iocage/jails/syncthing

The content of "fstab" is:

Code:

/mnt/ssd/iocage/releases/11.2-RELEASE/root/bin  /mnt/ssd/iocage/jails/syncthing/root/bin        nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:50
/mnt/ssd/iocage/releases/11.2-RELEASE/root/boot /mnt/ssd/iocage/jails/syncthing/root/boot       nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:50
/mnt/ssd/iocage/releases/11.2-RELEASE/root/lib  /mnt/ssd/iocage/jails/syncthing/root/lib        nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/libexec      /mnt/ssd/iocage/jails/syncthing/root/libexec    nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/rescue       /mnt/ssd/iocage/jails/syncthing/root/rescue     nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/sbin /mnt/ssd/iocage/jails/syncthing/root/sbin       nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/bin      /mnt/ssd/iocage/jails/syncthing/root/usr/bin    nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/include  /mnt/ssd/iocage/jails/syncthing/root/usr/include        nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/lib      /mnt/ssd/iocage/jails/syncthing/root/usr/lib    nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/libexec  /mnt/ssd/iocage/jails/syncthing/root/usr/libexec        nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/sbin     /mnt/ssd/iocage/jails/syncthing/root/usr/sbin   nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/share    /mnt/ssd/iocage/jails/syncthing/root/usr/share  nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/libdata  /mnt/ssd/iocage/jails/syncthing/root/usr/libdata        nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51
/mnt/ssd/iocage/releases/11.2-RELEASE/root/usr/lib32    /mnt/ssd/iocage/jails/syncthing/root/usr/lib32  nullfs  ro      0       0 # Added by iocage on 2019-01-03 03:17:51

Very much like yours except for the "OurData" Pool.
You find your Pool (imported as described in previous post) and everything should be fine, I hope.

 

[xames]

I have POOL1 and POOL2, POOL1 is replicatet to POOL2, but i cannot access jail section, thats normal? I do a replication as a backup of POOL1 if some time the phisical disk of POOL1 fails, and POOL2 is on another disk. Thanks.

 

[Apollo]

I have POOL1 and POOL2, POOL1 is replicatet to POOL2, but i cannot access jail section, thats normal? I do a replication as a backup of POOL1 if some time the phisical disk of POOL1 fails, and POOL2 is on another disk. Thanks.

Is POOL2 on the same system as POOL1 or is it installed on another system?

In short, Freenas only allow one pool where the jails is being pointed to.
This is set under GUI => Jails and can be edited by selecting the "gear" icon.
For instance, if POOL1 already contains jails, Freenas should already point to "POOL1".

If you want to run the jails from POOL2, then you will need to change it using the "gear" icon and use "POOL2" instead.
Then you will have to remap all the relevent "fstab" to point to "POOL2".

A better solution would be to unplug "POOL1" and import "POOL2" and rename it as "POOL1"

There maybe numerous issues with this process if not managed properly.

 

[xames]

Thanks.

 

[xames]

Now in another system.

i have a problem with my jail, i install new hdd on my freenas and i make my POOL1 (where jails and nextcloud is installed), and replicated to the new one to make backups, but after that i can't access nextcloud ip anymore, then i disconnect the new hdd and restart the freenas, but now it says me all jails corrupted. It says:

nextcloud is missing it's configuration, please destroy this jail and recreate it.

How i can resolve that because, all jails, plex, radarr etc... corrupted because of this replication.

I think i fail doing replication on the same system, but now i don't know how to revert all.

 

[xames]

i can't find the fstab file, where is exactly?

Thanks.

 

[Apollo]

i can't find the fstab file, where is exactly?

Thanks.

Hi Xames,

If Jail location is assigned the correct pool, and your pool name is POOL1, then you should find it advertised under the following location:
/mnt/POOL1/iocage

It is possible under certain circumptances you jails are located under:
/mnt/iocage

From the above, you can descend under the iocage folder into what should be the listed names from all the installed jails:
../iocage/jails

From there, enter the subfolder corresponding to your jail and the fstab should be there.