I lost a pool and don't know how to bring it back.

rebl · Apr 22, 2020

I lost a pool, here's the traceback:

Code:

Environment:

Software Version: FreeNAS-11.2-U8 (06e1172340)
Request Method: GET
Request URL: http://xxx.xxx.xxx.xxx/legacy/storage/dataset/create/Data/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  163.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in dataset_create
  414.         form = forms.ZFSDatasetCreateForm(initial=defaults, fs=fs)
File "./freenasUI/storage/forms.py" in __init__
  1406.         super(ZFSDatasetCreateForm, self).__init__(*args, fs=fs, **kwargs)
File "./freenasUI/storage/forms.py" in __init__
  1301.                 self.parentdata['atime'][0]

Exception Type: KeyError at /legacy/storage/dataset/create/Data/
Exception Value: 'atime'

I`m a newbie and don't know, what I should do now. Can you help me?
I built my system in 2018. After the setup my system ran without problems until yesterday. I didn't install any update at this time.

System:
Freenas 11.2-U8
16TB / 4x4TB

Please find infos to HDDs and Pools attached.

How can I repair "KeyError at /legacy/storage/dataset/create/Data/" ?
What is the Exception Value "atime"?
Did I loose my pool forever?

TIA and Regards,
Frank

PhiloEpisteme · Apr 22, 2020

Can you provide the output of zpool status in code tags like this
[code]
code here
[/code]

rebl · Apr 23, 2020

Code:

 
pool: AudioVideo                                                                                                                 
 state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0 days 08:25:53 with 0 errors on Sun Apr  5 08:25:54 2020                                               
config:                                                                                                                             
                                                                                                                                    
        NAME                                            STATE     READ WRITE CKSUM                                                 
        AudioVideo                                      ONLINE       0     0     0                                                 
          raidz1-0                                      ONLINE       0     0     0                                                 
            gptid/84e0ef3d-961a-11e8-a90c-bc5ff4c79295  ONLINE       0     0     0                                                 
            gptid/86905065-961a-11e8-a90c-bc5ff4c79295  ONLINE       0     0     0                                                 
            gptid/8892f00a-961a-11e8-a90c-bc5ff4c79295  ONLINE       0     0     0                                                 
            gptid/8a6d5ff8-961a-11e8-a90c-bc5ff4c79295  ONLINE       0     0     0                                                 
                                                                                                                                    
errors: No known data errors                                                                                                       
                                                                                                                                    
  pool: freenas-boot                                                                                                               
 state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0 days 00:00:24 with 0 errors on Sun Apr 19 03:45:24 2020                                               
config:                                                                                                                             
                                                                                                                                    
        NAME        STATE     READ WRITE CKSUM                                                                                     
        freenas-boot  ONLINE       0     0     0                                                                                   
          da0p2     ONLINE       0     0     0                                                                                     
                                                                                                                                    
errors: No known data errors

Theres no pool "Data"....

Yorick · Apr 23, 2020

KeyError: Was this an encrypted pool? If so, was it being exported? There's a known issue there, fixed in 11.3-U2.1, see https://jira.ixsystems.com/browse/NAS-105782

PhiloEpisteme · Apr 23, 2020

So I'm clear, you expect there to be three pools? AudioVideo, Data, and freenas-boot? What hardware are you using? How are the drives connected? How many total drives do you expect? What is the output of gpart show? I bunch of questions, I know. I'm trying to figure out what happened. If you could also let us know any other recent things you did to/with your system that might help.

Yorick said:
KeyError: Was this an encrypted pool? If so, was it being exported? There's a known issue there, fixed in 11.3-U2.1, see https://jira.ixsystems.com/browse/NAS-105782

I thought 11.2-U8 was not affected?

Yorick · Apr 23, 2020

PhiloEpisteme said:
I thought 11.2-U8 was not affected

You are right, I should have paid better attention. Never mind me :).

rebl · Apr 23, 2020

Yorick said:
KeyError: Was this an encrypted pool?

No encryption, but raid-z

PhiloEpisteme said:
So I'm clear, you expect there to be three pools? AudioVideo, Data, and freenas-boot? What hardware are you using? How are the drives connected? How many total drives do you expect? What is the output of gpart show? I'm trying to figure out what happened. If you could also let us know any other recent things you did to/with your system that might help.

Thanks for your help. No, there should be two pools, but there are only one, because by Data pool is lost. freenas-boot is the start-partition on an external SSD-drive.

It's a self-built NAS with Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz, 4x4 TB WD RED, so 16 TB and a start Pool/partition on an external SSD on USB3.
The pools look like this:

The system ran perfectly for over 2 years (the most time 24/7). I opened the NAS last year to clean the dust. and updated to 11.2-U3 without problems. the DATA-pool was there until 4 weeks ago, I don't exactly know what happened, at least four weeks ago was the momen I realized the loss. There is importand data on it (for most data I have backups), but I used the pool rarely compüared to the other pool, which I use daily.
Here's the boot history:

Because The raid-Z1 includes all four HDDs for all pools and I didn't notice a problem on my main pool I locked out a hardware problem with my HDDs.
Here are the results of "gpart show":

Code:

=>        40  7814037088  ada0  GPT  (3.6T)                                     
          40          88        - free -  (44K)                                 
         128     4194304     1  freebsd-swap  (2.0G)                           
     4194432  7809842696     2  freebsd-zfs  (3.6T)                             
                                                                                
=>        40  7814037088  ada1  GPT  (3.6T)                                     
          40          88        - free -  (44K)                                 
         128     4194304     1  freebsd-swap  (2.0G)                           
     4194432  7809842696     2  freebsd-zfs  (3.6T)                             
                                                                                
=>        40  7814037088  ada2  GPT  (3.6T)                                     
          40          88        - free -  (44K)                                 
         128     4194304     1  freebsd-swap  (2.0G)                           
     4194432  7809842696     2  freebsd-zfs  (3.6T)                             
                                                                                
=>        40  7814037088  ada3  GPT  (3.6T)                                     
          40          88        - free -  (44K)                                 
         128     4194304     1  freebsd-swap  (2.0G)                           
     4194432  7809842696     2  freebsd-zfs  (3.6T)                             
                                                                                
=>       40  117231328  da0  GPT  (56G)                                         
         40       1024    1  freebsd-boot  (512K)                               
       1064  117230304    2  freebsd-zfs  (56G)

Please let me know if you need another info. I`m a complete layman in UNIX, BSD so please be forgiving if I don't understand it straight away.

rebl · Apr 23, 2020

Yorick said:
First build.
SuperMicro X11SSH-F, Intel E3-1225 v6 (Plex hw transcode works)
32 GiB (2x16) Crucial ECC DDR4 UDIMM 2400Mhz
Patriot SCORCH 128GB M.2 NVMe boot device
5 x 8TB HGST Ultrastar white-label @5400rpm
Seasonic SSR-650PX
Fractal Design Node 804

Nice, mine is also a Fractal Design Node 804. Good case!

PhiloEpisteme · Apr 23, 2020

@rebl I think I am misunderstanding something.

You said your system is composed of 5 disks total, 4 HDDs for the data and 1 USB SSD for the boot pool.

rebl said:
It's a self-built NAS with Intel(R) Core(TM) i3-4130T CPU @ 2.90GHz, 4x4 TB WD RED, so 16 TB and a start Pool/partition on an external SSD on USB3.

From your output from zpool status and gpart show it looks like the pool AudioVideo is a 4-disk pool and that all 4 of those disks are showing up in the system.

rebl said:

So, my question is, if you have 4 disks in your system; and all 4 of them are being used by AudioVideo what disks make up the Data pool?

rebl · Apr 23, 2020

PhiloEpisteme said:
@rebl
So, my question is, if you have 4 disks in your system; and all 4 of them are being used by AudioVideo what disks make up the Data pool?

Thank you for your question. Back in the days when it worked (>4weeks ago) I had 1 Raid-Z1 System with 2 Pools in this virtual drive: AudioVideo with 9.6 TB andf a small one, Data with 1 TB. You know, just for small "data", text, letters, PDFs, and so on. TRo sum it up: 1 RaiD-Z, 2 pools with 10.54 GB (RAID Z1 of 16 TB).

I`m curious, is it possible that I expanded my main pool and destroyed the small pool while I was doing this? Wouldn`t the system warn me? Because I don`t remember anything like that.

K_switch · Apr 23, 2020

@rebl Is there currently a mount point at

Code:

/mnt/Data

From what i can see your AudioVideo pool is currently assigned all 12TB of usable storage.

PhiloEpisteme · Apr 23, 2020

rebl said:
I had 1 Raid-Z1 System with 2 Pools in this virtual drive: AudioVideo with 9.6 TB andf a small one, Data with 1 TB

Can you please expand on EXACTLY how you did that? The reason I am asking is that FreeNAS uses ZFS and it would require special configuration to use 4 disks to back 4 pools.

Pools are made of vdevs
vdevs cannot be shared between pools

Currently you have 4 drives, and all of them are in AudioVideo. Did you do some sort of special configuration to get them to support two pools? Did you do some sort of manual partitioning? How are they connected to your system?

Heracles · Apr 23, 2020

Hey Rebl,

Are you sure you are not confused between pools and datasets ? From what you posted, you probably always had a single pool that itsels was divided in different datasets.

Pools and datasets are very different things in FreeNAS...

rebl · Apr 23, 2020

I can`t explain it, because I`m an idiot. :) You are right, but, as you know, I built the system nearly two years ago and I just forgot it, I`m sorry. Opening the case revealed the real problem: I have a pool of 16TB (RAID5) and second pool of 4TB (RAID1). This was my Data pool Thanks god I wrote it on the case. I completely forgot the additional 2 HDDs. I suppose I`ve lost one HDD.

Im I right, when one HDD of RAID1 is defect the partition doesn`t work anymore?
Thank you to you Expert-guys and...sorry

It`s great to have a place to go for any Freenas questions.

rebl · Apr 23, 2020

Heracles said:
Hey Rebl,
Are you sure you are not confused between pools and datasets ?

Sure, as a complete layman I was confused, you are right. I assumed pools are datasets. Right. I know know I lost a pool with some datasets. :)

PhiloEpisteme · Apr 23, 2020

No worries @rebl. I think you may get a lot of good info out of the terminology primer. :)

rebl said:
Im I right, when one HDD of RAID1 is defect the partition doesn`t work anymore?

Just for clarification zfs has no raid1 or raid5. In zfs you store data in datasets. Datasets live on pools. Pools are made up of 1 or more vdevs. vdevs are made of disks.

vdevs can be either a single disk, a mirror of 2 or more disks, or a group of disks as raidz1|2|3. Traditional raid1 is most similar to a vdev with a 2-disk mirror (this sounds like what data was). Traditional raid5 is most similar to a vdev using raidz1.

rebl said:
second pool of 4TB (RAID1). This was my Data pool Thanks god I wrote it on the case. I completely forgot the additional 2 HDDs

If you used a 2-disk mirror vdev losing 1 drive would not have caused you to lose the pool. If you used a single drive or experienced both drives going offline for some reason that would certainly explain the issue. Have you checked the cable connections? By chance did you have email alerting set up and did you see any alerts there about bad disks or degraded pools?

What is the output of zpool import?

Yorick · Apr 23, 2020

rebl said:
Im I right, when one HDD of RAID1 is defect the partition doesn`t work anymore?

When a single drive in a mirror fails, then the mirror will continue in degraded state.

Next what you'll want to look at is the status of those two drives. Am I right in thinking they don't even show up on gpart show? If you are lucky, this is merely an issue with cable / controller / power / BIOS, something or anything that keeps the drives from showing up: And that once you get them to be visible again, your data will still be there.

K_switch · Apr 23, 2020

@rebl

Im I right, when one HDD of RAID1 is defect the partition doesn`t work anymore?

Like @Yorick pointed out if you have them setup in a Mirror (Raid1 not correct in this regard but same level of redundancy) then you should be able to lose one of the disks with no issues... what it looks like happened is both of your Data pool disks got disconnected or removed from the system... however you should still see the Data pool in a very bad state... try running

Code:

 geom disk list

This should list all disks that the FreeNAS system can see... if it does not show these devices then i would boot into BIOS and see if they are recognized there... if they are still not seen then you could add those drives into other SATA connections and attempt to import the Pool as you would have to lose both disks to truly lose all data.

Thanks

rebl · Apr 23, 2020

Thank you "geom disk list" doesn`t show the two drives. No cable issue. I have to boot into BIOS. So I`m looking for display and keyboard and try. Thanks!

Yorick · Apr 23, 2020

To-Do list for afterwards:
- Set up email alerts in FreeNAS and make sure those work
- Set up scrub tasks for both pools
- Set up short and long SMART test tasks
- Set up daily config backup; optionally and recommended sync config backup to cloud

Important Announcement for the TrueNAS Community.

I lost a pool and don't know how to bring it back.

Dabbler

Guru

Dabbler

Wizard

Guru

Wizard

Dabbler

Dabbler

Guru

Dabbler

Dabbler

Guru

Wizard

Dabbler

Dabbler

Guru

Wizard

Dabbler

Dabbler

Wizard

Similar threads