Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SOLVED ***RESOLVED*** Recover Configuration.db From Failing Boot Device. Almost There. Help!

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Hi all,

I did a regular update to my FreeNAS server on Friday, and afterward I could not access it. A trip to the basement server rack and a reboot showed a Grub error on the boot USB stick, a Sandisk Cruzr. I found this thread in my search for a solution:

https://forums.freenas.org/index.php?threads/recover-configure-from-failing-boot-device.28032/

and I installed FreeNAS onto a new 16Gb Sandisk USB stick using the latest .iso from the repositories. I want to copy freenas-v1.db from the old disk to the new. I followed this step:
  • Use zpool import to see the ID of the pool (you don't want to use the name to mount the pool as it's the same name than the current system pool)
  • Use zpool import -R /mnt the_ID_of_the_pool new_pool_name to import the pool which should be moutned as /mnt/new_pool_name
so far so good...and that's where I'm stuck. Pardon the noob question, but I've searched and I can't find the info for the next step: how to copy the old db onto the new drive? I can't find /mnt, and cp commands from the new poolname/subdir result in a 'file not found' error. I'm missing something gawdawful basic. I've also learned my lesson and bought a second USB stick to mirror the fresh stick, once I figure out how to copy the config file over.
I really appreciate any help that you can offer.
 
Last edited:

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,890
What do you mean that you can't find /mnt? Does /mnt not exist? If not, try creating it--'mkdir /mnt'.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Have you tried ls -l /mnt/poolname/ to see if it exists ?
 

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Hi Bidule0hm,

Thanks for the command! I can see the pool and its files and subdirs. I Installed the server 2 and bit years ago and its been chugging along ever since. I haven't had much experience at the BSD command line. I'm not seeing the /Data subdir where freenas-v1.db is supposed to reside. Once I find it, I can cp the file over to the new drive. I'll try a recursive "find" and see what happens. Please let me know if I', missing anything.

danb35, I created the subdir, but for some reason I wasn't seeing anything in it when I typed the "ls /mnt" command.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
It should be here: /mnt/poolname/data/freenas-v1.db

If it's not can you post the output of the ls and zpool status please?
 

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Thanks BiduleOhm,

ls -l:

[root@freenas /mnt/old_pool]# ls -l
total 98
-rw-rw-rw- 1 nobody wheel 14340 Mar 15 05:10 .DS_Store
drwxrwxrwx 3 nobody wheel 4 Jan 8 2015 .TemporaryItems
-rw-rw-rw- 1 nobody wheel 4096 Dec 27 2014 ._.DS_Store
-rw-rw-rw- 1 nobody wheel 4096 Jan 8 2015 ._.TemporaryItems
-rw-rw-rw- 1 nobody wheel 4096 Jan 8 2015 ._.apdisk
-rw-rw-rw- 1 nobody wheel 291 Jan 8 2015 .apdisk
drwxr-xr-x 2 www www 2 Apr 25 2015 .freenas
drwxrwxrwx 53 root www 68 Aug 29 2015 MP3 Songs
drwxrwxrwx 13 root www 15 Nov 17 16:48 Video
drwxr-xr-x 12 root wheel 12 Dec 20 08:35 jails

zpool status:

da0p2 ONLINE 0 0 0

errors: No known data errors

pool: old_pool
state: ONLINE
scan: scrub repaired 0 in 4h26m with 0 errors on Sun Mar 13 01:26:53 2016
config:

NAME STATE READ WRITE CKS
UM
old_pool ONLINE 0 0
0
gptid/355230bd-87fe-11e4-b6f7-0013215be4ac ONLINE 0 0
0
mirror-1 ONLINE 0 0
0
gptid/35c1b7c1-87fe-11e4-b6f7-0013215be4ac ONLINE 0 0
0
gptid/363d8b02-87fe-11e4-b6f7-0013215be4ac ONLINE 0 0
0

errors: No known data errors
=================

My reading through the forums says that the .db file is on a different partition, not the root file structure? or has that changed in 9.3?
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
This pool seems like it's the data pool, not the system pool.

However IIRC there's a backup of the config in the .freenas dataset so can you post the output of ls -l /mnt/old_pool/.freenas/ please (between code tags, it's easier to read).
 

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Hmmmm...not good:

[root@freenas ~]# ls -l /mnt/old_pool/.freenas
total 0

Finding the system pool looks like it's my only option at this point. If I installed BSD in a VM, would it show all partitions on the drive, assuming that they still existed?
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,474
The file used to be in the root directory of the 4th partition of the boot device.

IIRC, at some point, a feature was added in 9.3?, such that if your system dataset resided on your pool a backup copy of the configuration was saved there at ~3:15am daily.


Sent from my iPhone using Tapatalk
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,890
The file used to be in the root directory of the 4th partition of the boot device.
I think you're thinking of pre-9.3 versions. In 9.3+, the boot device is a ZFS pool.
IIRC, at some point, a feature was added in 9.3?, such that if your system dataset resided on your pool a backup copy of the configuration was saved there at ~3:15am daily.
At one point, a feature was silently added that made backups of your configuration file whenever a change was made. At a subsequent time, that feature was silently dropped.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,474
I downloaded my 9.3 PDF version of the docs. In section 5.2 - General, in the "Save Config" section it states that FreeNAS backs up the configuration database to the system dataset every morning at 3:45.

It's recommended to save the confirmation after making changes. This has been a suggestion since the 8.x days.


Sent from my iPhone using Tapatalk
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,890
Interesting. I looked at my system (in /var/db/system/configs-$LONGSTRING), and there are indeed backups there, going back to release 201509282017, so I would suppose that's the release when they started doing this. Another silent change, I guess.

And yes, saving a copy of the config file when the config changes has always been a good idea. But of course people don't do that, which is why there are a bunch of how-to documents on setting up automatic backups of that file. Doesn't look like that's been necessary for the last six months or so, though.
 

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,474
It was documented in the manual ;-) but we know users don't RTFM.

I probably saw it since I look to see if changes have been made to the docs and generate a new PDF if it's been updated.


Sent from my iPhone using Tapatalk
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,890
I think I'm better than most at R'ingTFM, but I don't generally re-read it for point releases of a product. I do usually look through the release notes on the updates, though, and I don't recall seeing anything about any of at least three changes (make backup on any change, stop making backup on any change, start doing backup daily) made to the "config backup" feature since 9.3 was released.

In any event, it's a good feature, and it should give OP somewhere else to look for a config db backup.
 

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Thanks for the discussion and leads. I did just try installing PCBSD into a VM, and I set it to recognize the USB controller, but I get no love when I plug the USB stick into my PC. I'll take a look at the avenues of research above and see where it takes me.

I set this up about 1.5 years ago on a HP ML350 server w/ 24Gb RAM. The past year has gotten crazy busy w/ life and work, so not a lot of time was available to keep up to date on the OS. It "just worked" and was taken for granted in the basement server rack. My bad on that.
 

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Hmmmm...I installed PC-BSD in another VM in VirtualBox after issues with VMWare Workstation, and when I try and mount the drive in VB, I get an "unknown special file or file system" error. Could this be because in 9.3 the boot device is a ZFS pool?
 

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
Ok, I'm making some progress. I found this informative article on recovering a FreeNAS 9.3 server:

https://headcrash.industries/reference/recovering-freenas-configuration-from-zfs-boot-drive/

I did a zfs list command on my mounted old drive:

freenas-boot 939M 13.0G 31K none
freenas-boot/ROOT 931M 13.0G 31K none
freenas-boot/ROOT/default 931M 13.0G 931M legacy
freenas-boot/grub 7.77M 13.0G 7.77M legacy
old_pool 1.35T 866G 1.34T /mnt/old_pool
old_pool/.system 15.5M 866G 116K legacy
old_pool/.system/configs-f36704f2fe794cb6a75657843255655d 9.08M 866G 9.08M legacy
old_pool/.system/cores 3.17M 866G 3.17M legacy
old_pool/.system/rrd-f36704f2fe794cb6a75657843255655d 96K 866G 96K legacy
old_pool/.system/samba4 2.14M 866G 2.14M legacy
old_pool/.system/syslog-f36704f2fe794cb6a75657843255655d 876K 866G 876K legacy
old_pool/jails 11.1G 866G 86.9M /mnt/old_pool/jails
old_pool/jails/.warden-template-pluginjail 452M 866G 449M /mnt/Media/jails/.warden-template-pluginjail
old_pool/jails/.warden-template-pluginjail--x64 723M 866G 719M /mnt/old_pool/jails/.warden-template-pluginjail--x64
old_pool/jails/crashplan_1 373M 866G 1.06G /mnt/old_pool/jails/crashplan_1
old_pool/jails/maraschino_1 172M 866G 886M /mnt/old_pool/jails/maraschino_1
old_pool/jails/plexmediaserver_1 9.37G 866G 10.1G /mnt/old_pool/jails/plexmediaserver_1

Now to see if I can find my freenas-v1.db. I'm not seeing a ".../ROOT" in my old_pool drive. Thoughts on where I should look for my config db?
 
Last edited:

Astronomer

Newbie
Joined
Oct 5, 2013
Messages
12
**** RESOLVED ****

From the link directly above (It describes 3 options. I used the third):
================

Besides reimporting the configuration database there is an even simpler solution, and this is the one I would ultimately recommend, because no files need to be backed up, and no extra USB drive is required: the FreeNAS installer has an option to upgrade an existing installation. It can perform this in place with the backup USB drive containing the damaged copy of the original boot drive. It will also retain the old configuration database:

  1. Download the FreeNAS ISO, burn it to a CD or USB drive, and boot it
  2. In the boot menu, chose Install/Upgrade
  3. In the destination media menu, select the drive containing the copy of the damaged boot image
  4. Click OK. The installer will offer you to upgrade instead of reinstall, and to keep the original configuration
The installer will make a copy of the configuration database, then wipe the file system and reinstall FreeNAS, and finally restore the database on the drive. The drive is now ready to be used in the server. When booting it for the first time, FreeNAS will migrate the database to the installed version. After that, your system should be up and running, and all settings, including user accounts, network shares, plug-ins and jails should be working again.
=================

It worked and my server is functioning normally again. I will now build a pair of boot sticks and import the database to them.
 
Last edited:

RichieB

Newbie
Joined
Jan 12, 2017
Messages
16
I did a zfs list command on my mounted old drive:

freenas-boot 939M 13.0G 31K none
freenas-boot/ROOT 931M 13.0G 31K none
freenas-boot/ROOT/default 931M 13.0G 931M legacy
freenas-boot/grub 7.77M 13.0G 7.77M legacy
old_pool 1.35T 866G 1.34T /mnt/old_pool
old_pool/.system 15.5M 866G 116K legacy
old_pool/.system/configs-f36704f2fe794cb6a75657843255655d 9.08M 866G 9.08M legacy
old_pool/.system/cores 3.17M 866G 3.17M legacy
old_pool/.system/rrd-f36704f2fe794cb6a75657843255655d 96K 866G 96K legacy
old_pool/.system/samba4 2.14M 866G 2.14M legacy
old_pool/.system/syslog-f36704f2fe794cb6a75657843255655d 876K 866G 876K legacy
old_pool/jails 11.1G 866G 86.9M /mnt/old_pool/jails
old_pool/jails/.warden-template-pluginjail 452M 866G 449M /mnt/Media/jails/.warden-template-pluginjail
old_pool/jails/.warden-template-pluginjail--x64 723M 866G 719M /mnt/old_pool/jails/.warden-template-pluginjail--x64
old_pool/jails/crashplan_1 373M 866G 1.06G /mnt/old_pool/jails/crashplan_1
old_pool/jails/maraschino_1 172M 866G 886M /mnt/old_pool/jails/maraschino_1
old_pool/jails/plexmediaserver_1 9.37G 866G 10.1G /mnt/old_pool/jails/plexmediaserver_1

Now to see if I can find my freenas-v1.db. I'm not seeing a ".../ROOT" in my old_pool drive. Thoughts on where I should look for my config db?
I am amazed that the answer to this question is not documented anywhere. I'm just settling up my FreeNAS and am preparing for disaster. Making regular backups of the configuration database is part of that. As noted above FreeNAS already takes care of that. Why they make it a secret where those backups are is beyond me. This thread was the closest I got to finding the answer.

FreeNAS makes nightly backups in
Code:
/var/db/system/configs-ea02119b0df4495ba64ec1dbdd61ed06/FreeNAS-9.10.2-U1
(86c7ef5)]

of course the long hex string and FreeNAS version might be different for you. If you mount your pool at a different location as part of recovery you should be able to find these backups at
Code:
/mnt/old_pool/.system/configs-ea02119b0df4495ba64ec1dbdd61ed06/FreeNAS-9.10.2-U1

From there you can get the latest backup of your configuration database needed for recovering a failed boot disk.
 
Top