Trying to recover config from 9.10 (ZFS) boot device

[sremick]

Before you do that, what's the output of zfs list?

Code:

# zfs list
NAME  USED  AVAIL  REFER  MOUNTPOINT
MainVolume  7.75T  2.73T  72.3G  /mnt/MainVolume
MainVolume/.system  92.3M  2.73T  320K  legacy
MainVolume/.system/configs-dc50aef459594e43b97c3504ed86a0c5  34.2M  2.73T  33.3M  legacy
MainVolume/.system/cores  1.75M  2.73T  1.56M  legacy
MainVolume/.system/rrd-dc50aef459594e43b97c3504ed86a0c5  448K  2.73T  288K  legacy
MainVolume/.system/samba4  3.89M  2.73T  2.34M  legacy
MainVolume/.system/syslog-dc50aef459594e43b97c3504ed86a0c5  50.5M  2.73T  44.0M  legacy
MainVolume/Media  7.56T  2.73T  7.16T  /mnt/MainVolume/Media
MainVolume/jails  124G  2.73T  440K  /mnt/MainVolume/jails
MainVolume/jails/.warden-template-standard  2.70G  2.73T  2.64G  /mnt/MainVolume/jails/.warden-template-standard
MainVolume/jails/.warden-template-standard-9.2-x64  325M  2.73T  317M  /mnt/MainVolume/jails/.warden-template-standard-9.2-x64
MainVolume/jails/Plex  64.2G  2.73T  55.9G  /mnt/MainVolume/jails/Plex
MainVolume/jails/Plex2  56.6G  2.73T  56.1G  /mnt/MainVolume/jails/Plex2
freenas-boot  647M  27.2G  64K  none
freenas-boot/ROOT  640M  27.2G  29K  none
freenas-boot/ROOT/Initial-Install  1K  27.2G  636M  legacy
freenas-boot/ROOT/default  640M  27.2G  637M  legacy
freenas-boot/grub  6.35M  27.2G  6.35M  legacy

Well, gee... I see a ".system" there. :) So now I'm a bit hopeful.

 

[danb35]

What about ls /var/db?

 

[sremick]

What about ls /var/db?

Well:

Code:

# ls -l /var/db
total 72
drwxr-xr-x  19 root  wheel  1536 May 24 17:58 ./
drwxr-xr-x  29 root  wheel  1792 May 24 15:58 ../
drwxr-xr-x  3 root  wheel  64 May 24 15:58 collectd/
drwxr-xr-x  2 root  wheel  64 May 24 15:58 dbus/
----------  1 root  wheel  355 May 24 18:13 dhclient.leases.igb0
drwx------  2 operator  operator  0 May 22 19:04 entropy/
drwx------  2 root  wheel  0 May 22 19:04 freebsd-update/
drwx------  2 root  wheel  0 May 22 19:04 hyperv/
drwx------  2 root  wheel  0 May 22 19:04 ipf/
-rw-r--r--  1 nobody  wheel  0 May 22 19:03 locate.database
drwx------  2 root  wheel  0 May 22 19:04 ntp/
-rw-r--r--  1 root  wheel  7 May 24 17:58 ntpd.drift
-rw-r--r--  1 root  wheel  8795 May 22 19:03 ntpd.leap-seconds.list
drwxrwx---  2 root  uucp  0 May 22 19:04 nut/
drwxr-xr-x  9 root  wheel  576 May 24 15:58 pbi/
drwxr-xr-x  2 root  wheel  192 May 22 19:03 pkg/
drwxr-xr-x  2 root  wheel  0 May 22 19:04 ports/
drwxr-xr-x  2 root  wheel  0 May 22 19:04 portsnap/
drwxr-xr-x  2 root  wheel  0 May 24 15:58 samba/
drwxr-xr-x  3 root  wheel  256 May 24 15:58 samba4/
drwxr-xr-x  2 root  wheel  0 May 24 15:58 sss/
drwxr-xr-x  2 root  wheel  0 May 24 15:58 sss_mc/
drwx--x--x  3 root  wheel  64 May 22 19:04 sudo/
srwxr-xr-x  1 root  wheel  0 May 24 15:58 syslog-ng.ctl=
-rw-------  1 root  wheel  16384 May 24 18:13 syslog-ng.persist
-rw-r--r--  1 root  wheel  20 May 24 15:58 zoneinfo

But of course that's the new USB boot device, not any of the old pool's data. Not sure how it helps, but providing because you asked. :)

 

[danb35]

But of course that's the new USB boot device

What makes you think this?

Edit: Try ls /var/db/system

 

[sremick]

What makes you think this?

*sheepishly*.... because the 6-disk data array pool is mounted under /mnt/MainVolume ...? So I figured everything not beneath that is part of the USB drive. But I get the feeling I'm about to be schooled here...

 

[danb35]

*sheepishly*.... because the 6-disk data array pool is mounted under /mnt/MainVolume ...?

That'd be a reasonable assumption, except that @Ericloewe and I have been going back and forth on this thread about the mount point for the .system dataset. Datasets behave in many ways like independent filesystems, and though it's common (most common, I'm sure) for them to be mounted as their dataset hierarchy would suggest (such that, if your pool is at /mnt/MainVolume, the dataset MainVolume/.system would appear as /mnt/MainVolume/.system), it doesn't have to be that way. And, specifically, in the case of the .system dataset, it isn't. It should be at /var/db/system/.

So, what's the output of ls /var/db/system? And if that doesn't work, what about df | grep configs?

 

[sremick]

Code:

# ls /var/db/system
ls: /var/db/system: No such file or directory

Well, boo. So:

Code:

# df | grep configs
#

Nada. So, for reference:

Code:

# df
Filesystem  1K-blocks  Used  Avail Capacity  Mounted on
freenas-boot/ROOT/default  29194501  652051  28542450  2%  /
devfs  1  1  0  100%  /dev
tmpfs  32768  8764  24004  27%  /etc
tmpfs  4096  8  4088  0%  /mnt
tmpfs  5569264  90144  5479120  2%  /var
freenas-boot/grub  28548948  6498  28542450  0%  /boot/grub
fdescfs  1  1  0  100%  /dev/fd
MainVolume  3008461787  75794749 2932667038  3%  /mnt/MainVolume
MainVolume/Media  10622480194 7689813156 2932667038  72%  /mnt/MainVolume/Media
MainVolume/jails  2932667478  440 2932667038  0%  /mnt/MainVolume/jails
MainVolume/jails/.warden-template-standard  2935431568  2764530 2932667038  0%  /mnt/MainVolume/jails/.warden-template-standard
MainVolume/jails/.warden-template-standard-9.2-x64  2932992040  325002 2932667038  0%  /mnt/MainVolume/jails/.warden-template-standard-9.2-x64
MainVolume/jails/Plex  2991309002  58641964 2932667038  2%  /mnt/MainVolume/jails/Plex
MainVolume/jails/Plex2  2991517974  58850936 2932667038  2%  /mnt/MainVolume/jails/Plex2

Doesn't look promising. But curiously:

Code:

# zfs list | grep configs
MainVolume/.system/configs-dc50aef459594e43b97c3504ed86a0c5  34.2M  2.73T  33.3M  legacy

(for what it's worth: the "Plex" jail I don't care about, while the "Plex2" jail is the one created after my most-recent properly-made config backup. So if we end up SOL trying to find a copy of the current config, it's that jail I'll be looking to salvage after applying the old config)

 

[danb35]

Curiouser and curiouser. Ah, legacy mountpoints.

OK, try this:

Code:

# mkdir -p /mnt/MainVolume/configs
# mount -t zfs MainVolume/.system/configs-dc50aef459594e43b97c3504ed86a0c5 /mnt/MainVolume/configs
# ls /mnt/MainVolume/configs

 

[sremick]

Ok, so the last ls command:

Code:

# ls -l /mnt/MainVolume/configs
total 99
drwxr-xr-x  6 root  wheel  6 May  9 00:45 ./
drwxr-xr-x  8 root  wheel  8 May 24 18:55 ../
drwxr-xr-x  2 root  wheel  40 Aug  7  2016 FreeNAS-9.10-STABLE-201606270534 (dd17351)/
drwxr-xr-x  2 root  wheel  234 Mar 29 00:45 FreeNAS-9.10.1 (d989edd)/
drwxr-xr-x  2 root  wheel  42 May  8 00:45 FreeNAS-9.10.2-U2 (e1497f2)/
drwxr-xr-x  2 root  wheel  12 May 18 00:45 FreeNAS-9.10.2-U3 (e1497f269)/

I think this is where I start to get excited.

Code:

# ls -l /mnt/MainVolume/configs/FreeNAS-9.10.2-U3\ \(e1497f269\)/
total 1158
drwxr-xr-x  2 root  wheel  12 May 18 00:45 ./
drwxr-xr-x  6 root  wheel  6 May  9 00:45 ../
-rw-r-----  1 root  wheel  342016 May  9 00:45 20170509.db
-rw-r-----  1 root  wheel  342016 May 10 00:45 20170510.db
-rw-r-----  1 root  wheel  342016 May 11 00:45 20170511.db
-rw-r-----  1 root  wheel  342016 May 12 00:45 20170512.db
-rw-r-----  1 root  wheel  342016 May 13 00:45 20170513.db
-rw-r-----  1 root  wheel  342016 May 14 00:45 20170514.db
-rw-r-----  1 root  wheel  342016 May 15 00:45 20170515.db
-rw-r-----  1 root  wheel  342016 May 16 00:45 20170516.db
-rw-r-----  1 root  wheel  342016 May 17 00:45 20170517.db
-rw-r-----  1 root  wheel  342016 May 18 00:45 20170518.db

Are those what I think/hope they are?

 

[danb35]

Are those what I think/hope they are?

They are indeed. Copy the latest one to an accessible location, cd /, umount MainVolume/.system/configs-dc50aef459594e43b97c3504ed86a0c5, and upload the config to the web GUI. You should be good to go.

 

[Ericloewe]

The GUI really should have a "restore config from system dataset" option. I have to remember to file a bug report in the morning.

 

[danb35]

...and it would appear that you were correct about the .system dataset's mountpoint not being stored in its properties, such that it would be mounted automatically in the right place. Wonder what the reason was for that design choice.

 

[sremick]

YES. IT WORKS.

Thank you so much for your patience and assistance. I am hoping that this thread will be of some use to someone else in the future. I also love that my pain has inspired some improvements to the FreeNAS GUI that perhaps all will be able to enjoy sometime in the future.

Needless to say, I just made an updated backup of my config to a safe offsite location.

 

[Ericloewe]

...and it would appear that you were correct about the .system dataset's mountpoint not being stored in its properties, such that it would be mounted automatically in the right place. Wonder what the reason was for that design choice.

Maybe it is, but the pool was imported with the override mount point option, which I guess could cause it to be ignored.

 

[danb35]

but the pool was imported with the override mount point option

True (and I don't understand why that was done, but it doesn't really matter), but the problem is that the .system dataset (and all its child datasets) has a mountpoint of "legacy". That means it doesn't get automatically mounted with the rest of the pool, and that the zfs mount/ zfs unmount commands won't work on it.

 

[sremick]

If any new feature requests or bug reports get filed as a result of what was learned from this, can you post the bug #s here so I can (silently) follow their progress/discussion? Thanks!

 

[Ericloewe]

https://bugs.freenas.org/issues/24173

For the restore from system dataset functionality.