SOLVED Imported Volume is Missing Data

[dudeinco]

My organization had a catastrophic failure of the USB boot device on a power outage. I installed v9.3 on a new USB Device, and on setup, it asked me which volume to import - chose the volume, and it mounted. The problem is that the volume was empty! It shows 9.9MB used out of 460GB, which is definitely not correct.

I created a new USB drive (v11), and it asked the same questions, did the same thing, and same results.

There are four hard drives on this system, I'm not sure what two of them were for, but I know they had some sort of redundant raid running, and this volume. I don't know if the raid was copied to/part of the ZFS Volume or not. I know this was all on an NTFS share (SMB, I believe - with one named user).

I know this isn't offering up much, but how can I piece this thing back together? Can I? I have so little to go off of here, but the volumes being empty makes me think this thing is hosed.

Boot details:
(2x) secondary GPT header is not in the last LBA
Device mirror Raid1 Loaded

 

[Stux]

Did you cancel the wizard or did you follow through and erase your drives?

 

[SweetAndLow]

Along with the above question, your full hardware specs would be great information. Motherboard, cpu, memory, psu, hba card, power supply and chassie.

 

[dudeinco]

Did you cancel the wizard or did you follow through and erase your drives?

I went through the wizard, but was stumped (we don't have a domain controller, etc), so cancelled. I later used import volume to import the volume.

 

[dudeinco]

Along with the above question, your full hardware specs would be great information. Motherboard, cpu, memory, psu, hba card, power supply and chassie.

Xeon 5650 - 12GB Ram - don't remember the m/b (old school top end gaming mb) or psu (same). No hba card.

 

[danb35]

Output of zpool status, zpool import, and zpool list? All in code tags, please.

 

[dudeinco]

Output of zpool status, zpool import, and zpool list? All in code tags, please.

Code:

[root@freenas ~]# zpool status																									 
  pool: LifeZFSVolume																											   
state: ONLINE																													 
status: Some supported features are not enabled on the pool. The pool can														   
		still be used, but some features are unavailable.																		   
action: Enable all features using 'zpool upgrade'. Once this is done,															   
		the pool may no longer be accessible by software that does not support													 
		the features. See zpool-features(7) for details.																			
  scan: none requested																											 
config:																															 
																																	
		NAME										  STATE	 READ WRITE CKSUM													
		LifeZFSVolume								 ONLINE	   0	 0	 0													
		  gptid/7317532e-04ee-11e7-ac5b-002354aa0a24  ONLINE	   0	 0	 0													
																																	
errors: No known data errors																										
																																	
  pool: freenas-boot																												
state: ONLINE																													 
  scan: none requested																											 
config:																															 
																																	
		NAME		STATE	 READ WRITE CKSUM																					 
		freenas-boot  ONLINE	   0	 0	 0																					
		  da0p2	 ONLINE	   0	 0	 0																					 
																																	
errors: No known data errors					 

Code:

[root@freenas ~]# zpool import																									 
[root@freenas ~]#

Code:

NAME			SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT												 
LifeZFSVolume   460G  11.1M   460G		 -	 0%	 0%  1.00x  ONLINE  /mnt													 
freenas-boot   7.56G   744M  6.84G		 -	  -	 9%  1.00x  ONLINE  -		 

 

[dudeinco]

Output of zpool status, zpool import, and zpool list? All in code tags, please.

I'll note that there are two drives that are not part of this pool.

What I'm seeing is:
ada0, ada1, and ada2 - all 500.1 GB.

 

[danb35]

You say there are four hard drives running on the system, but only one of them is part of this volume (LifeZFSVolume). What's the output of camcontrol devlist and gpart show?

 

[dudeinco]

You say there are four hard drives running on the system, but only one of them is part of this volume (LifeZFSVolume). What's the output of camcontrol devlist and gpart show?

Looks like it's 3 (I was counting the USB drive I think)

Code:

[root@freenas ~]# camcontrol devlist											
<ST3500830AS AAE485>			   at scbus7 target 0 lun 0 (pass0,ada0)		
<ST3500830AS AAE485>			   at scbus8 target 0 lun 0 (pass1,ada1)		
<ST500DM002-1BD142 KC48>		   at scbus12 target 0 lun 0 (pass2,ada2)	  
<SMI USB DISK 1100>				at scbus14 target 0 lun 0 (pass3,da0)	

Code:

[root@freenas ~]# gpart show													
=>	   34  976773101  ada2  GPT  (466G)									  
		 34		 94		- free -  (47K)								  
		128	4194304	 1  freebsd-swap  (2.0G)							
	4194432  972578696	 2  freebsd-zfs  (464G)							  
  976773128		  7		- free -  (3.5K)								
																				
=>	  40  15974320  da0  GPT  (7.6G)										
		40	  1024	1  bios-boot  (512K)									
	  1064  15973288	2  freebsd-zfs  (7.6G)								
  15974352		 8	   - free -  (4.0K)									
																				
=>	   34  976773100  mirror/LifeRaid1  GPT  (466G)						  
		 34  976773100				 1  freebsd-ufs  (466G)	

Maybe it's on that mirror, but no clue how to import it.

 

[danb35]

So da0 is your boot device, and ada2 is your ZFS pool (LifeZFSVolume). ada0 and ada1 comprise a UFS mirror that FreeNAS 9.3+ doesn't fully support any more. You can probably do "import disk" to copy their contents onto your ZFS pool, but you might be safer reverting to 9.2.1.9 and importing your UFS volume that way.

 

[dudeinco]

So da0 is your boot device, and ada2 is your ZFS pool (LifeZFSVolume). ada0 and ada1 comprise a UFS mirror that FreeNAS 9.3+ doesn't fully support any more. You can probably do "import disk" to copy their contents onto your ZFS pool, but you might be safer reverting to 9.2.1.9 and importing your UFS volume that way.

I'll give that a shot - thanks!

 

[Stux]

And I should probably point out, that your ZFS pool is a single disk pool and has no redundancy.

Once you've rescued your data from the UFS mirror, then you should look into resolving that.

 

[dudeinco]

So da0 is your boot device, and ada2 is your ZFS pool (LifeZFSVolume). ada0 and ada1 comprise a UFS mirror that FreeNAS 9.3+ doesn't fully support any more. You can probably do "import disk" to copy their contents onto your ZFS pool, but you might be safer reverting to 9.2.1.9 and importing your UFS volume that way.

I tried 9.2.1.9, and got an error 19 that dropped me to a mount prompt.

I went back to 11, and tried import disk. UFS (which was my guess as to what this was) - said filesystem not validated. So I chose NTFS (as this was an SMB share between Windows computers) - it tells me the device doesn't seem to have a valid NTFS.

I don't know at this point. I've been reading around, and 9.2.1.9 had issues with some people with Haswell processors? Perhaps unrelated, but the menu that pops up in 9.2.1.9 looks like the one I had in the past. I think we're getting closer to a version that reflects what was on the original USB.

Edit: Tried a second (different manufacturer) USB drive, and same result (error 19 - mountroot prompt)
Edit 2: Tried with 9.10 - back to same scenario as 9.3+ (of course - my bad)
Edit 3: Trying 9.1 - if no change, will fall back to the 8 series to see if I can get something different. I did notice when booting 9.10 it mentioned an mbr errror on one of the drives - didn't have time to catch which one.

 

[rs225]

Maybe the UFS needs to have fsck run?

 

[dudeinco]

At this point, I'd be good with mounting the UFS mirror and copying the files to the ZFS manually

Maybe the UFS needs to have fsck run?

If 9.1 doesn't run, and drops me to the mount prompt, I'll look into that.

9.1 Same thing - drop to mountroot prompt - looking into fsck

 

[dudeinco]

Maybe the UFS needs to have fsck run?

So here's where we're at - 9.1 (like 9.2.1.9) drops to mountroot prompt. Hitting enter, results in panic, dropping me to a db> prompt. I am going to try to load up 9.10, and drop to shell there for fsck.

Edit 1: 9.10 drop to shell. fsck ada1/ada2/etc.: Can't determine filesystem type. ....Dropping to 8.x
Edit 2: Specifying ufs gives: Inappropriate ioctl for device - can't read disk label
Edit 3: 8.2 - drop to mountroot prompt... Error 19 while waiting for /dev/ufs/FreeNASs1a

That's about it - I don't know where to go from here.

 

[dudeinco]

OK, this just gets weirder - I booted 9.10 again, and started poking around in the file system. Went to /mnt/LifeShare (our share), and found all the files (I think). At this point, no volume has been imported, and nothing has been done in 9.10, so... (Remember that when I imported the volume in 11, it was empty)

What the heck is going on here? Is there something I should do to get these things off of here, and start fresh? Dare I try to import the volume again, but this time in 9.10?

 

[rs225]

Do a df -h and zfs list while you've got it running.

 

[dudeinco]

Do a df -h and zfs list while you've got it running.

Code:

[root@freenas ~]# df -h																											 
Filesystem				   Size	Used   Avail Capacity  Mounted on															 
freenas-boot/ROOT/default	 14G	471M	 13G	 3%	/																	   
devfs						1.0K	1.0K	  0B   100%	/dev																	
tmpfs						 32M	8.1M	 24M	25%	/etc																	
tmpfs						4.0M	8.0K	4.0M	 0%	/mnt																	
tmpfs						4.0G	 62M	3.9G	 2%	/var																	
freenas-boot/grub			 13G	6.3M	 13G	 0%	/boot/grub															 
linprocfs					4.0K	4.0K	  0B   100%	/compat/linux/proc													 
/dev/ufs/LifeRaid1p1		 451G	 18G	397G	 4%	/mnt																	
[root@freenas ~]# zfs list																										 
NAME								USED  AVAIL  REFER  MOUNTPOINT																 
freenas-boot						481M  13.5G	31K  none																		
freenas-boot/ROOT				   474M  13.5G	25K  none																		
freenas-boot/ROOT/Initial-Install	 1K  13.5G   471M  legacy																	 
freenas-boot/ROOT/default		   474M  13.5G   471M  legacy																	 
freenas-boot/grub				  6.33M  13.5G  6.33M  legacy																	 
[root@freenas ~]#