SOLVED Zpool imported but files missing

[Ericloewe]

What hardware are we talking about, anyway?

This one, at least, is addressed above. And since the dataset was mounted previously, I think the second possibility was also addressed.

So it seems...

 

[saikee]

I have a Sabertooth Z77 mobo with a i7-3770K CPU populated with 32GB ram.

The mirror udev has 2x3x8TB.

I know I am not using the recommended hardware because I am still in the stage of trying out the FreeNAS. However FreeNAS has worked for every of the 3 old PC I have tried. Absolutely no complaint. I was about to mirror the boot drive but got this problem first. The new V9.10 installation has 2x32GB USB boot drive but it is too late.

My other other FreNAS box has a Gigabyte EX58 UD5 mobo, a i7-920 CPU, 12GB ram and 3x2TB stripe udev. It is my pilot scheme working flawlessly.

I have found in the past the udev can be transferred from one PC to another with the files intact after import. I wouldn't mind to rebuild my data again but it would be nice if I know the mistake and learn from it.

 

[saikee]

You can try browsing the snapshots directly.
ex. ls -alh /mnt/i7-3770K/.zfs/snapshot/manual-20170625/Summerhill

Code:

[root@Summerhill ~]# ls -alh /mnt/i7-3770K/.zfs/snapshot/manual-20170625/Summerhill												 
total 18																															
drwxr-xr-x  2 root  wheel	 2B Jun 24 17:12 .																					 
drwxrwxr-x  8 root  wheel	 8B Jun 24 17:13 ..   

The snapshot was created after my problem developed so it does not have much information.

 

[saikee]

I think I'm stumped here. I don't think your data is gone--something's using that space on your pool. But it seems to be hidden for some reason, and I think I'm out of ideas why. @Ericloewe? @rs225?

Edit: Actually, I have one other idea--try reporting a bug using the Support button in your web GUI. That will attach a debug file, which might help the devs figure it out.

I have filed a bug report as recommended.

Thanks for all the help. It is a great forum here. The quality of the community is reflective of the quality of FreeNAS.

 

[DrKK]

This is very odd.

This is either something ridiculously stupid, and we're all going feel like idiots in about a minute, or this is really complicated.

We got down this road by a boot pool upgrade being interrupted, right?

 

[DrKK]

So we're talking in the Mumble here, @Ericloewe, me, and @m0nkey_ , and we have a crazy idea.

Go ahead and "ADD STORAGE" to one of your jails, FROM a directory that you think is supposed to have a bunch of files in it, mount it somewhere inside the jail, and just for the hell of it, let's see if they show up in the jail.

 

[saikee]

This is very odd.

This is either something ridiculously stupid, and we're all going feel like idiots in about a minute, or this is really complicated.

We got down this road by a boot pool upgrade being interrupted, right?

The exact sequence was I switched off the PC, pulled out the FreeNAS 9.10 boot drive, inserted a ssd and installed the new FreNAS 11.0 in it. Then I couldn't find my files after importing. This is a fresh install of FreNAS 11.0 and not a upgrade. The FreNAS 11.0 asked me to update the storage but I declined as it said the process is a one-way street that I cannot go back to V9.10.

When reading the guide I realised I could upgrade from 9.10 to 11.0 using the old drive. Therefore I was trying to upgrade the 9.10. However I did not reach the part indicated in the Guide and realized the installer was start writing on the old boot drive. I stopped the installation. The v9.10 boot drive could no longer boot. I therefore have to re-install 9.10 again but could not find my files after the import.

If my zpool was damaged accidentally it could possibly be inflicted by the new V11.0 when I imported the zpool. I couldn't use it because the files were not there. In V9.10 I was able to transfer the zpool between PCs before and a new installation could always import back a zpool without problem.

 

[saikee]

So we're talking in the Mumble here, @Ericloewe, me, and @m0nkey_ , and we have a crazy idea.

Go ahead and "ADD STORAGE" to one of your jails, FROM a directory that you think is supposed to have a bunch of files in it, mount it somewhere inside the jail, and just for the hell of it, let's see if they show up in the jail.

The existing jails do not show up in the jails section.
I created a new jail /mnt/i7-3770K/jails_5/OwnFault by pointing the source from /mnt/i7-3770K/Summerhill
to a target
/mnt/i7-3770K/jails_5/OwnFault/usr/ownFault/

CLI indicates both are empty

 

[Ericloewe]

The existing jails do not show up in the jails section.
I created a new jail /mnt/i7-3770K/jails_5/OwnFault by pointing the source from /mnt/i7-3770K/Summerhill
to a target
/mnt/i7-3770K/jails_5/OwnFault/usr/ownFault/

CLI indicates both are empty
View attachment 19166

You're not actually inside the jail there, are you? In other words, jexec into the jail.

 

[saikee]

You're not actually inside the jail there, are you? In other words, jexec into the jail.

This is a new jail visible in the GUI only after I created following DrKK suggestion. In CLI I could see the filing systems of all the jails. May be there are other ways to do it the files in my jails seem to be symbolic links to the original files created automatically with plugins but the corresponding mounting points are now empty. I can confirm this by comparison with my other smaller FreeNAS installation.

In my other FreeNAS box the files are at the mounting point and the folder has 5.7TB data. The jails also have identical files at the corresponding mounting point with another 5.7TB size. Thus filing system tree has 11TB even the zpool itself is only 5.73TB large as shown in my smaller healthy FreeNAS installation in another Box.

Code:

[root@freenas ~]# ls /mnt/New_films/																								
jails   saikee																													
[root@freenas ~]# ls /mnt/New_films/saikee/																						
.cshrc				  .mail_aliases		   .rhosts				 Foreign films		   Sam-Plex							
.login				  .mailrc				 .shrc				   Movie-viewed			Sam-photo_Video_source			
.login_conf			 .profile				Blockbusters			Mp3-1													 
[root@freenas ~]# du -hs /mnt/New_films/																							
11T	/mnt/New_films/																											
[root@freenas ~]# ls /mnt/New_films/jails/plexmediaserver_1/usr/share/saikee/													 
.cshrc				  .mail_aliases		   .rhosts				 Foreign films		   Sam-Plex							
.login				  .mailrc				 .shrc				   Movie-viewed			Sam-photo_Video_source			
.login_conf			 .profile				Blockbusters			Mp3-1													 
[root@freenas ~]# du -hs  /mnt/New_films/jails/plexmediaserver_1/usr/share/saikee/												
5.7T	/mnt/New_films/jails/plexmediaserver_1/usr/share/saikee/																	
[root@freenas ~]# du -hs /mnt/New_films/saikee/																					
5.7T	/mnt/New_films/saikee/																									
[root@freenas ~]# zpool list																										
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT												 
New_films	 7.25T  5.73T  1.52T		 -	43%	79%  1.00x  ONLINE  /mnt													
freenas-boot   222G   879M   221G		 -	  -	 0%  1.00x  ONLINE  -														
[root@freenas ~]#

Please note the above is from another FreeNAS installation running normally with a zpool New_films. My current problem without the files is the zpool i7-3770K.

 

[saikee]

I think I'm stumped here. I don't think your data is gone--something's using that space on your pool. But it seems to be hidden for some reason, and I think I'm out of ideas why. @Ericloewe? @rs225?

Edit: Actually, I have one other idea--try reporting a bug using the Support button in your web GUI. That will attach a debug file, which might help the devs figure it out.

Thanks a million for your advice.

I sent a bug report to the Support team as advised. A day later I received the advice

It seems you still have a snapshot, so try this:

zfs clone i7-3770K@manual-20170625 i7-3770K/backup

Then check /mnt/i7-3770K/backup

I followed the commands and got 14TB data back.

Code:

[root@Summerhill ~]# ls /mnt/i7-3770K/backup/																					
Summerhill	  jails		   jails_2		 jails_3		 jails_4		 saikee											
[root@Summerhill ~]# ls /mnt/i7-3770K/backup/saikee/																				
.cshrc		  .login_conf	 .mailrc		 .rhosts		 Disk1-series	Disk3-viewed	Disk5-latest	z-Disk0-oddies	
.login		  .mail_aliases   .profile		.shrc		   Disk2-new	   Disk4-Torrent   temp
[root@Summerhill ~]# du -hs /mnt/i7-3770K/backup/saikee/																			
14T	/mnt/i7-3770K/backup/saikee/																				

The data are Disk1-series, Disk2-new, Disk3-viewed, Disk4-Torrent, Disk5-latest....etc.

 

[danb35]

That's... very strange. But glad it worked, and your data is back.

Edit: Well, one discrepancy that I'm seeing is that we'd been focusing primarily on Summerhill, while it seems the data was in saikee.

 

[styno]

You can try browsing the snapshots directly.

This should've worked! There was 14TB as refer in that snapshot!

Edit: Well, one discrepancy that I'm seeing is that we'd been focusing primarily on Summerhill, while it seems the data was in saikee.

Oh ffs

Glad it worked out in the end !

 

[saikee]

Thanks again guys. I can watch the videos via smb from another PC in Linux so I think everything is back to normal. I shall mark the thread solved.

 

[Apollo]

Some suggestion maybe.
Sometime, when a dataset has been created and files have been copied, there are some scenario for which the maximum length of folder and file length can cause the dataset to not be mountable upon reboot or after volume has been detached then attached again.

 

[saikee]

Apollo,

I shall remember your pointer as I do have long file folders and huge amount of scanned and digital photo files.

 

[flingwing]

Hi, just wanted to say that zfs clone command help make visible a missing dataset as well. I did not get into the situation the same way, however, I was transferring the contents of one pool to a new one using the snapshot --> replication through the GUI. After it was completed I wasn't able to see any files and after fiddling around for the day this was the thing that did it.

The target folder that the replication was pointed to I named data_cache_copy, not sure if the length of that might have been the issue...

 

[scp]

I have a very similar problem to this. I recently updated to 11.3-U1, everything seemed to go well, but I can no longer see my files. I have followed the suggestions in this thread but nothing seems to work.

The only snapshot I have is post update. The dashboard and output from command line du suggest that the files are there, its just I can see them.

==========================================================
Results from zpool status is:
==========================================================
freenas# zpool status
pool: freenas-boot state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:28 with 0 errors on Mon Mar 30 03:46:28 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

pool: norfield
state: ONLINE
scan: scrub repaired 0 in 0 days 00:33:01 with 0 errors on Mon Mar 30 12:48:46 2020
config:

NAME STATE READ WRITE CKSUM
norfield ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/2af67338-7b13-11e8-bf7e-2c27d7ef69f1 ONLINE 0 0 0
gptid/2be8afb2-7b13-11e8-bf7e-2c27d7ef69f1 ONLINE 0 0 0


==========================================================
Results from zfs list is:
==========================================================
freenas# zfs list -r norfield
NAME USED AVAIL REFER
MOUNTPOINTnorfield 340G 3.07T 128K
/norfieldnorfield/.system 161M 3.07T 140K
legacynorfield/.system/configs-c0708eef149b49a396f66530a6f1b80a 39.0M 3.07T 38.9M
legacynorfield/.system/cores 86.5M 3.07T 86.4M
legacynorfield/.system/rrd-c0708eef149b49a396f66530a6f1b80a 34.8M 3.07T 19.7M
legacynorfield/.system/samba4 564K 3.07T 407K
legacynorfield/.system/syslog-c0708eef149b49a396f66530a6f1b80a 128K 3.07T 128K
legacynorfield/.system/webui 128K 3.07T 128K
legacynorfield/iocage 1.10G 3.07T 6.27M
/norfield/iocagenorfield/iocage/download 137M 3.07T 128K
/norfield/iocage/download
norfield/iocage/download/11.3-RELEASE 137M 3.07T 137M/norfield/iocage/download/11.3-RELEASE
norfield/iocage/images 128K 3.07T 128K/norfield/iocage/images
norfield/iocage/jails 559M 3.07T 128K/norfield/iocage/jails
norfield/iocage/jails/plexmediaserver 559M 3.07T 581K/norfield/iocage/jails/plexmediaserver
norfield/iocage/jails/plexmediaserver/root 558M 3.07T 552M/norfield/iocage/jails/plexmediaserver/root
norfield/iocage/log 134K 3.07T 134K/norfield/iocage/log
norfield/iocage/releases 423M 3.07T 128K/norfield/iocage/releases
norfield/iocage/releases/11.3-RELEASE 423M 3.07T 128K/norfield/iocage/releases/11.3-RELEASE
norfield/iocage/releases/11.3-RELEASE/root 423M 3.07T 423M/norfield/iocage/releases/11.3-RELEASE/root
norfield/iocage/templates 128K 3.07T 128K/norfield/iocage/templates
norfield/media 336G 3.07T 3.60G/norfield/media
norfield/media/video 333G 3.07T 143G/norfield/media/video
norfield/media/video/movies 189G 3.07T 189G/norfield/media/video/movies
==========================================================

Have I lost my data?! :-(

Any help would be most welcome.

Thanks.

 

[Apollo]

Your zfs list result doesn't show the entire content, You are missing disks within a RAIDZ2 setup. Currently you are only showing 2. However the pool looks healthy.

What worries me is the fact you are using Dedup on your Media dataset. I don't know the consequence of that for your setup.

PS. I think you should start a new thread specific to your use case.

 

[scp]

Hi Apollo,

Thanks for your response.

Just checked the missing disks from the zpool status output were a cut and paste error. I think that the dedup got switched on accidentally when I was trying to figure out what happened. I have switched it off.

I'll start a new thread as you suggest.