Fatal trap 12 page fault while in kernel mode while importing pool

Status
Not open for further replies.
Z

zang3tsu

Dabbler
Joined
Feb 17, 2013
Messages
14
I got the ff. error while importing the pool at boot after a restart:
Code:
Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address   = 0x29
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff804053f4
stack pointer           = 0x28:0xfffffe3fd106ccb0
frame pointer           = 0x28:0xfffffe3fd106cd00
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 10982 (zpool)


The server specs are as follows:
  • 24 logical CPUs @ 2.1GHz
  • 256GB RAM
  • 180x 4TB HDDs
I tried reinstalling FreeNAS and I still got the same error. The pool was degraded before this and I'm wondering if it's still possible to recover the pool and if so, what are the steps? I have also attached the full crash dump. I have also tried running Memtest and I got no errors after 1 pass.

Here's the output of zpool import:

Code:
   pool: geostorage
     id: 3404713725681892846
  state: DEGRADED
status: One or more devices were being resilvered.
action: The pool can be imported despite missing or damaged devices.  The
    fault tolerance of the pool may be compromised if imported.
config:

    geostorage                                          DEGRADED
     raidz2-0                                          ONLINE
       gptid/fcdcb585-ccd2-11e5-9a4e-cad7727acf4a      ONLINE
       gptid/68b6b491-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/38fc2c3e-b6c5-11e5-b864-cad7727acf4a      ONLINE
       gptid/69a970d5-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6a1d1dbf-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6a901720-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6b03dc50-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6b772c4b-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6beabcd9-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6c600a7f-9e13-11e3-ad0f-00505683e462      ONLINE
     raidz2-1                                          ONLINE
       gptid/6cd9ba76-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/b08f973e-1412-11e5-be9a-cad7727acf4a      ONLINE
       gptid/6dca0d22-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6e4039a7-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6eba50a9-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6f329f6b-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/6fab58f6-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/5532cdae-9645-11e4-aa3c-00505683ace4      ONLINE
       gptid/709d2d30-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/71142109-9e13-11e3-ad0f-00505683e462      ONLINE
     raidz2-2                                          DEGRADED
       gptid/71a155dd-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/72194443-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/7291ed7c-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/730ef1ce-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/738700e6-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/ea97d975-ea35-11e4-bc5f-cad7727acf4a      ONLINE
       gptid/e8735f4c-cb10-11e5-867c-cad7727acf4a      ONLINE
       gptid/74fb6f87-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/7575a1c7-9e13-11e3-ad0f-00505683e462      ONLINE
       spare-9                                         DEGRADED
         9340407383384096899                           UNAVAIL  cannot open
         gptid/e7f3f005-0e6b-11e6-88d8-0cc47a524dd8    ONLINE
     raidz2-3                                          DEGRADED
       gptid/76a0e943-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/77207cad-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/77a1966f-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/7821c0e3-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/78a2d7bc-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/792686bf-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/79a8619d-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/7a2bf7bc-9e13-11e3-ad0f-00505683e462      ONLINE
       spare-8                                         DEGRADED
         7740330037582776592                           UNAVAIL  cannot open
         gptid/f8037171-0e6b-11e6-88d8-0cc47a524dd8    ONLINE
       gptid/c278abb4-7933-11e4-a84b-00505683ace4      ONLINE
     raidz2-4                                          DEGRADED
       gptid/04b3e884-5fa3-11e5-bfc0-cad7727acf4a      ONLINE
       gptid/7c9c5231-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/7d251c78-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/7dae3460-9e13-11e3-ad0f-00505683e462      ONLINE
       spare-4                                         DEGRADED
         7269800102247609207                           UNAVAIL  cannot open
         gptid/eb889d0c-0e6b-11e6-88d8-0cc47a524dd8    ONLINE
       gptid/7ebf1ef5-9e13-11e3-ad0f-00505683e462      ONLINE
       spare-6                                         ONLINE
         gptid/e0a48ca2-b6ba-11e5-b864-cad7727acf4a    ONLINE
         gptid/4b9b405a-f47a-11e5-9be3-fe825bd4f517    ONLINE
       gptid/7fd430de-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/d998fa3b-7931-11e4-a84b-00505683ace4      ONLINE
       gptid/2ed81d53-ea61-11e4-bc5f-cad7727acf4a      ONLINE
     raidz2-5                                          DEGRADED
       spare-0                                         ONLINE
         gptid/81f43343-9e13-11e3-ad0f-00505683e462    ONLINE
         gptid/50beb9d2-f47a-11e5-9be3-fe825bd4f517    ONLINE
       spare-1                                         ONLINE
         gptid/827f0ba2-9e13-11e3-ad0f-00505683e462    ONLINE
         gptid/533f132e-f47a-11e5-9be3-fe825bd4f517    ONLINE
       replacing-2                                     DEGRADED
         spare-0                                       DEGRADED
           7749537349230275809                         UNAVAIL  cannot open
           gptid/4e3060d7-f47a-11e5-9be3-fe825bd4f517  ONLINE
         gptid/ba3456f9-f46d-11e5-9be3-fe825bd4f517    ONLINE
       spare-3                                         ONLINE
         gptid/83970f6e-9e13-11e3-ad0f-00505683e462    ONLINE
         gptid/0c3a9f8f-0825-11e6-8dd6-15ab60ecee7e    ONLINE
       spare-4                                         ONLINE
         gptid/8423fbbe-9e13-11e3-ad0f-00505683e462    ONLINE
         gptid/1b15ae6a-0825-11e6-8dd6-15ab60ecee7e    ONLINE
       spare-5                                         ONLINE
         gptid/ea4b78e7-d977-11e5-9fab-fe825bd4f517    ONLINE
         gptid/e823a9df-0887-11e6-90cd-0cc47a524dd8    ONLINE
       spare-6                                         ONLINE
         gptid/85439210-9e13-11e3-ad0f-00505683e462    ONLINE
         gptid/2a6c85a7-0825-11e6-8dd6-15ab60ecee7e    ONLINE
       spare-7                                         ONLINE
         gptid/ff2b48e4-3ca2-11e4-b541-005056834a59    ONLINE
         gptid/e1569fba-0887-11e6-90cd-0cc47a524dd8    ONLINE
       spare-8                                         ONLINE
         gptid/01cd3b0d-ccc0-11e5-9a4e-cad7727acf4a    ONLINE
         gptid/f40ef431-0e6b-11e6-88d8-0cc47a524dd8    ONLINE
       spare-9                                         ONLINE
         gptid/86f38569-9e13-11e3-ad0f-00505683e462    ONLINE
         gptid/e42e49ac-0887-11e6-90cd-0cc47a524dd8    ONLINE
     raidz2-6                                          ONLINE
       gptid/8842da5b-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/88d4836d-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/8967b1a0-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/b3d2d840-cb24-11e5-867c-cad7727acf4a      ONLINE
       gptid/8a8faa1d-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/8b22b5af-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/8bb74818-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/8c4da3cd-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/9f91b77b-6a60-11e4-b445-00505683ace4      ONLINE
       gptid/8d7cfc05-9e13-11e3-ad0f-00505683e462      ONLINE
     raidz2-7                                          ONLINE
       gptid/8f13bdf7-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/8faeb6e2-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/904a5700-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/90e6044d-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/9182ed4d-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/921f2e5a-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/92c25e98-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/cbfe7b75-cb19-11e5-867c-cad7727acf4a      ONLINE
       gptid/9401be46-9e13-11e3-ad0f-00505683e462      ONLINE
       gptid/94a2d372-9e13-11e3-ad0f-00505683e462      ONLINE
     raidz2-8                                          ONLINE
       gptid/109d2c09-3a72-11e4-939d-005056834a59      ONLINE
       gptid/117ebf79-3a72-11e4-939d-005056834a59      ONLINE
       gptid/125be125-3a72-11e4-939d-005056834a59      ONLINE
       gptid/134772ae-3a72-11e4-939d-005056834a59      ONLINE
       gptid/14534a28-3a72-11e4-939d-005056834a59      ONLINE
       gptid/155620f3-3a72-11e4-939d-005056834a59      ONLINE
       gptid/1632d86c-3a72-11e4-939d-005056834a59      ONLINE
       gptid/173ac95e-3a72-11e4-939d-005056834a59      ONLINE
       gptid/1835744a-3a72-11e4-939d-005056834a59      ONLINE
       gptid/191b54af-3a72-11e4-939d-005056834a59      ONLINE
     raidz2-9                                          ONLINE
       gptid/7f5a9a89-3a72-11e4-939d-005056834a59      ONLINE
       gptid/804d6b1b-3a72-11e4-939d-005056834a59      ONLINE
       gptid/81346bae-3a72-11e4-939d-005056834a59      ONLINE
       gptid/8233dbb6-3a72-11e4-939d-005056834a59      ONLINE
       gptid/831dd8b6-3a72-11e4-939d-005056834a59      ONLINE
       gptid/84297c7d-3a72-11e4-939d-005056834a59      ONLINE
       gptid/85201d14-3a72-11e4-939d-005056834a59      ONLINE
       gptid/863e6093-3a72-11e4-939d-005056834a59      ONLINE
       gptid/872bec6e-3a72-11e4-939d-005056834a59      ONLINE
       gptid/885acb36-3a72-11e4-939d-005056834a59      ONLINE
     raidz2-10                                         DEGRADED
       gptid/f10fa953-3a72-11e4-939d-005056834a59      ONLINE
       gptid/f2a2971a-3a72-11e4-939d-005056834a59      ONLINE
       gptid/f472779a-3a72-11e4-939d-005056834a59      ONLINE
       gptid/f74ba73c-3a72-11e4-939d-005056834a59      ONLINE
       gptid/fa16ee5e-3a72-11e4-939d-005056834a59      ONLINE
       spare-5                                         DEGRADED
         14980889269527436465                          UNAVAIL  cannot open
         gptid/b44a833d-2576-11e6-9ef3-0cc47a524dd8    ONLINE
       gptid/fea5e8b4-3a72-11e4-939d-005056834a59      ONLINE
       gptid/00a5338a-3a73-11e4-939d-005056834a59      ONLINE
       gptid/02a91b96-3a73-11e4-939d-005056834a59      ONLINE
       gptid/0482b1cc-3a73-11e4-939d-005056834a59      ONLINE
     raidz2-11                                         ONLINE
       gptid/7448c883-3a73-11e4-939d-005056834a59      ONLINE
       gptid/7565c245-3a73-11e4-939d-005056834a59      ONLINE
       gptid/768f2d01-3a73-11e4-939d-005056834a59      ONLINE
       gptid/77d2dc42-3a73-11e4-939d-005056834a59      ONLINE
       gptid/790e2efa-3a73-11e4-939d-005056834a59      ONLINE
       gptid/7a617f86-3a73-11e4-939d-005056834a59      ONLINE
       gptid/140df83e-cce1-11e5-9a4e-cad7727acf4a      ONLINE
       gptid/7cfc2538-3a73-11e4-939d-005056834a59      ONLINE
       gptid/7e469b7f-3a73-11e4-939d-005056834a59      ONLINE
       gptid/7f8b0f07-3a73-11e4-939d-005056834a59      ONLINE
     raidz2-12                                         ONLINE
       gptid/6782199c-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/24ce6cc3-7884-11e5-94b8-cad7727acf4a      ONLINE
       gptid/6a858478-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/6c27a0d0-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/6d9c375d-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/6f0ae027-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/70c14ee0-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/72152d1d-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/738cf21e-d518-11e4-9559-76fe6da4b092      ONLINE
       gptid/753af2db-d518-11e4-9559-76fe6da4b092      ONLINE
     raidz2-13                                         ONLINE
       gptid/a6f8d0f8-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/a8984ec4-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/aa3c1826-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/abea32cb-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/ad5d6312-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/af244b97-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/b0cbe968-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/b2616962-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/b4334557-d519-11e4-9559-76fe6da4b092      ONLINE
       gptid/b5b67baf-d519-11e4-9559-76fe6da4b092      ONLINE
     raidz2-14                                         ONLINE
       gptid/e3c71985-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/e58a1abd-d51a-11e4-9559-76fe6da4b092      ONLINE
       spare-2                                         ONLINE
         gptid/e7496fd0-d51a-11e4-9559-76fe6da4b092    ONLINE
         gptid/efd77553-0e6b-11e6-88d8-0cc47a524dd8    ONLINE
       gptid/e8c934a8-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/ea50698a-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/eba7c2fb-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/ed933b15-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/ef01d125-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/f056ad88-d51a-11e4-9559-76fe6da4b092      ONLINE
       gptid/f192f84f-d51a-11e4-9559-76fe6da4b092      ONLINE
     raidz2-15                                         ONLINE
       gptid/06ea6215-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/0889fff6-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/0a33d190-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/0bfb725b-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/0d858ba9-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/0ef3705c-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/1054cf67-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/12087976-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/13ca9bb3-d51c-11e4-9559-76fe6da4b092      ONLINE
       gptid/1527c8ef-d51c-11e4-9559-76fe6da4b092      ONLINE
    spares
     17886976228965645538
     13625686331579142137
     958615736657255826
     4672799073486101347
     17154147026914963643
     13979255892235385664
     4095870112142717487
     12133705865953680600
     17981433701222750059
     13088820351863643008
     13001546594261122203
     6228860620740359793
     7154730018238934862
     12748887184868880695
     4517004712639587403
     13336623583950581355
     gptid/b93d806d-2576-11e6-9ef3-0cc47a524dd8
     gptid/bebf99d6-2576-11e6-9ef3-0cc47a524dd8
     gptid/c38c6b45-2576-11e6-9ef3-0cc47a524dd8
     gptid/c7e85013-2576-11e6-9ef3-0cc47a524dd8
 

Attachments

  • textdump.tgz
    59 KB · Views: 178
Last edited:
m0nkey_

m0nkey_

MVP
Joined
Oct 27, 2015
Messages
2,739
You could try booting into single user mode to see if the pool is mountable. Once in single user mode, try to import the pool
Code:
zpool import -R /mnt poolname

If it panics, just restart again in single user mode, then try mounting your pool read-only:
Code:
zpool import -o readonly=on -R /mnt poolname

If it mounts read-only, backup your data right away. Your pool is in danger of being lost.

If it fails to mount read-only, you can try the shotgun approach, but this can cause your pool to blow up if it doesn't work and data on it lost forever (Use this one at your own risk. I accept zero responsibility.)
Code:
zpool import -R /mnt -FX poolname
 
T

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Holy pile of drives, Batman. Out of curiosity, did all of these drive failures occur simultaneously? That could point to a power, controller, backplane, etc. problem which could manifest itself in weird ways.

If the failures didn't all occur at once, someone has been neglecting this system. An array with 180 drives isn't something you should neglect...
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yeah, either you had a bunch of sudden failures (which would indicate much bigger problems to deal with) or you've been mis-managing your zpool and should make it a very high priority to do a better job in the future.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Why are there spares randomly assigned to individual drives? I didn't even realize that was possible.
 
T

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Ericloewe said:
Why are there spares randomly assigned to individual drives? I didn't even realize that was possible.
Click to expand...
I saw something similar when I had a drive die and a hotspare take over... it showed as a mirrored pair within a mirrored pair. Of course, I promptly cleaned that up once resilvering was complete.

Having so many of them, and having raidz2-5 showing spares within spares within the vdev, seems to indicate much Bad Juju. I would suggest making appropriate sacrifices to the data deities in hopes a seasoned vet like cyberjock, jgreco, DrKK, etc. would be willing to step through this with you. I would *not* run off half-cocked trying things. IMO, you need a methodical approach to walk through removing all of the failed drives physically, bringing up the zpool, replacing any other failed drives, then cleaning up the mess.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
tvsjr said:
I saw something similar when I had a drive die and a hotspare take over... it showed as a mirrored pair within a mirrored pair. Of course, I promptly cleaned that up once resilvering was complete.
Click to expand...
I figured it would be something of the sort, but the pairs with both of them showing "ONLINE" aren't making any sense to me.

tvsjr said:
Having so many of them, and having raidz2-5 showing spares within spares within the vdev
Click to expand...
No, it seems to be a real resilver because the drive must've failed outright or.... You know what, I'm confused, and that's a bad sign.
 
Status
Not open for further replies.

Similar threads

P
  • Locked
S.M.A.R.T. Test / Scrubs for large pool's
Replies
6
Views
1K
Johnny Fartpants
Johnny Fartpants
Z
  • Locked
Drive "UNAVAILABLE" -- but which one?
Replies
13
Views
2K
Bidule0hm
Bidule0hm
M
  • Locked
Problem with disk in zvol
Replies
4
Views
1K
mykolaq
M
H
  • Locked
I accidentally extended a pool...
Replies
7
Views
2K
joeschmuck
joeschmuck
L
  • Locked
Files missing but space still consumed
Replies
5
Views
1K
Liviu D
L
Top