Upgrading from 9.2 to 9.3 - can not import Pool and Datasets anymore

[jakal]

Hi,

my config:
- Supermicro X10SL7-F with 16GB ECC
- 6x 2TB Samsung F4 in a raidz2 environment
- no encryption
- freenas user since v8

my steps of the evil beginning:
- from 9.2.8 initiating a GUI upgrade with FreeNAS-9.3-RELEASE.GUI_Upgrade.txz and using the pool as temporary space for the installation (free space was min 300GB or more)
- after several reboots 9.3 is booting up, login in GUI successful - but no jails, syslog and sharing services are comming up
- storage tab says, there is no available space left in the datasets (AVAIL 0 at all datasets), very bad, cause the system dataset is also in this pool
strange behavior: there was a first level dataset with the same name of the pool. the pool was at 260GB free space or so, the dataset was at zero space and so all other child datasets
- there are warnings about the zfs version and the /dev/mps0 firmware in the GUI, so i do an 'zpool upgrade' and a firmware upgrade on my LSI 2308 to v16IT - maybe
- after reboot same symptoms, no space left in the datasets, so i do a fresh install from ISO with no config restore
- in the fresh system i used the wizard and tried to import the pool, same thing: no space left
- so i had to move the system dataset on another pool i created temporarily on a second usb stick, so after that i have the syslog and services again
- in the shell zpool import says:

Code:

[root@sleipnir] ~# zpool import
   pool: timmy
     id: 15208926062868032888
  state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        timmy                                           ONLINE
          raidz2-0                                      ONLINE
            gptid/33c405da-aebd-11e2-9c37-00012e38c26a  ONLINE
            gptid/12022056-6ca6-11e4-9cb4-002590f3c778  ONLINE
            gptid/13bbe724-f1ab-11e0-9aee-00012e38c26a  ONLINE
            gptid/7c29f863-6bd2-11e4-9cb4-002590f3c778  ONLINE
            gptid/146b56bd-f1ab-11e0-9aee-00012e38c26a  ONLINE
            gptid/14cab269-f1ab-11e0-9aee-00012e38c26a  ONLINE

- but it wont import again through the GUI with the following error:

Code:

Dec 13 03:06:51 sleipnir manage.py: [middleware.notifier:3520] Importing timmy [15208926062868032888] failed with: cannot mount '/mnt/timmy/DATA': failed to create mountpoint
cannot mount '/mnt/timmy/DATA/Applications': failed to create mountpoint
cannot mount '/mnt/timmy/DATA/Ebooks': failed to create mountpoint
cannot mount '/mnt/timmy/MEDIA': failed to create mountpoint
cannot mount '/mnt/timmy/MEDIA/Filme': failed to create mountpoint
cannot mount '/mnt/timmy/MEDIA/Musik': failed to create mountpoint
cannot mount '/mnt/timmy/MEDIA/Serien': failed to create mountpoint
cannot mount '/mnt/timmy/home': failed to create mountpoint
cannot mount '/mnt/timmy/home/Julia': failed to create mountpoint
cannot mount '/m
Dec 13 03:06:51 sleipnir manage.py: nt/timmy/incoming': failed to create mountpoint
cannot mount '/mnt/timmy/jails': failed to create mountpoint
cannot mount '/mnt/timmy/jails/.warden-template-pluginjail-9.2-x64': failed to create mountpoint
cannot mount '/mnt/timmy/jails/plexmediaserver_1': failed to create mountpoint
cannot mount '/mnt/timmy/jails/pyload': failed to create mountpoint
cannot mount '/mnt/timmy/jails/transmission_1': failed to create mountpoint
Dec 13 03:06:51 sleipnir manage.py: [middleware.exceptions:38] [MiddlewareError: The volume "timmy" failed to import, for futher details check pool status]

- so i did a zpool status:

Code:

[root@sleipnir] ~# zpool status -v
  pool: dada
state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        dada                                          ONLINE       0     0     0
          gptid/2ee7b936-825d-11e4-9dc5-002590f3c778  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        freenas-boot                                  ONLINE       0     0     0
          gptid/f650e296-8257-11e4-966f-002590f3c778  ONLINE       0     0     0

errors: No known data errors

  pool: timmy
state: ONLINE
  scan: scrub repaired 928K in 18h31m with 0 errors on Sun Nov 30 18:31:17 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        timmy                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/33c405da-aebd-11e2-9c37-00012e38c26a  ONLINE       0     0     0
            gptid/12022056-6ca6-11e4-9cb4-002590f3c778  ONLINE       0     0     0
            gptid/13bbe724-f1ab-11e0-9aee-00012e38c26a  ONLINE       0     0     0
            gptid/7c29f863-6bd2-11e4-9cb4-002590f3c778  ONLINE       0     0     0
            gptid/146b56bd-f1ab-11e0-9aee-00012e38c26a  ONLINE       0     0     0
            gptid/14cab269-f1ab-11e0-9aee-00012e38c26a  ONLINE       0     0     0

errors: No known data errors

('dada' is the temporary pool for the system dataset)

- the storage section in the GUI doesn't show the pool or any dataset from the raidz2 either nor there is an option to import the pool again, so i used an zfs list to make sure that my pool and datasets aren't already gone:

Code:

[root@sleipnir] ~# zfs list
NAME                                                    USED  AVAIL  REFER  MOUNTPOINT
dada                                                   6.32M  1.66G    96K  /mnt/dada
dada/.system                                           4.98M  1.66G  2.35M  legacy
dada/.system/cores                                       96K  1.66G    96K  legacy
dada/.system/rrd-f36704f2fe794cb6a75657843255655d      2.15M  1.66G  2.15M  legacy
dada/.system/samba4                                     104K  1.66G   104K  legacy
dada/.system/syslog-f36704f2fe794cb6a75657843255655d    288K  1.66G   288K  legacy
freenas-boot                                            963M  6.27G    31K  none
freenas-boot/ROOT                                       954M  6.27G    31K  none
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201412091831       954M  6.27G   931M  /
freenas-boot/ROOT/default                                47K  6.27G   931M  legacy
freenas-boot/grub                                      7.81M  6.27G  7.77M  legacy
timmy                                                  7.08T      0  13.5M  /mnt/timmy
timmy/.system                                           248M      0   320K  legacy
timmy/.system/cores                                    31.1M      0  31.1M  legacy
timmy/.system/rrd-c282aadcd9aa4260adb7b1137606b762     55.6M      0  55.6M  legacy
timmy/.system/samba4                                   6.19M      0  6.19M  legacy
timmy/.system/syslog-c282aadcd9aa4260adb7b1137606b762   155M      0   155M  legacy
timmy/DATA                                              470G      0   304K  /mnt/timmy/DATA
timmy/DATA/Applications                                 249G      0   249G  /mnt/timmy/DATA/Applications
timmy/DATA/Ebooks                                      19.6G      0  19.6G  /mnt/timmy/DATA/Ebooks
timmy/MEDIA                                            5.90T      0   304K  /mnt/timmy/MEDIA
timmy/MEDIA/Filme                                      2.92T      0  2.92T  /mnt/timmy/MEDIA/Filme
timmy/MEDIA/Musik                                       179G      0   179G  /mnt/timmy/MEDIA/Musik
timmy/MEDIA/Serien                                     2.42T      0  2.42T  /mnt/timmy/MEDIA/Serien
timmy/home                                              280G      0   256K  /mnt/timmy/home
timmy/home/Julia                                       85.4G      0  85.4G  /mnt/timmy/home/Julia
timmy/incoming                                         28.7G      0  28.7G  /mnt/timmy/incoming
timmy/jails                                            21.4G      0   791K  /mnt/timmy/jails
timmy/jails/.warden-template-pluginjail-9.2-x64         880M      0   874M  /mnt/timmy/jails/.warden-template-pluginjail-9.2-x64
timmy/jails/plexmediaserver_1                          17.2G      0  18.1G  /mnt/timmy/jails/plexmediaserver_1
timmy/jails/pyload                                     3.11G      0  3.86G  /mnt/timmy/jails/pyload
timmy/jails/transmission_1                              241M      0  1.08G  /mnt/timmy/jails/transmission_1

Whats going wrong here and is there a chance to get my data available back again? I'm rather at a loss.

Thanks for trying to help me and sorry for my confuse spelling after 8 hours with this problem at this time of the day ( 3am ;) )

 

[dlavigne]

Hmm, haven't seen that error before. Please create a bug report at bugs.freenas.org and post the issue number here.

 

[Ericloewe]

Can you confirm the LSI firmware is P16? Odd stuff has happened with other versions (though this is a new kind of odd).

 

[mjws00]

Someone else recently fixed a pool that would not create a mount point by chmodding mnt or something similar. That thread may help ya. I'm on my phone so can't find it easily.

 

[jakal]

#7123

the LSI controller is indeed at v16 IT mode. also the warning in the GUI about the wrong firmware version disappeared.

i tried the 'chmod -R 777 /mnt' from the other thread, but unfortunately mounting the datasets is still fails..

 

[mjws00]

Blech. The 'zpool upgrade' may screw rolling back easily. If it were mine I would have tried a clean 9.2 install and an import just to land access to the data. I can't tell if it's just the GUI messing with you or system. How can the status be ONLINE and not be accessible?

In addition 'zfs list' clearly knows what's up. Did you try a clean 9.3 install? maybe I missed it. zpool export the pools and import them to a clean config might be worth a shot.

At least the pools still seem viable. So that is good.

 

[jakal]

9.2 downgrade didn't work cause of the 'zpool upgrade' earlier. there is nothing to import for the Auto-Importer there, 9.2 didn't see my pool anymore.