SOLVED Volume Degraded After 2 Offline Drives

[DanPrs]

ZFS2 so data is still good. However I had a loose power SATA cable that was feeding two drives, come loose. The two drives are now plugged in but I'm not sure how I need to add them back. And if they need to resilver.

Code:

root@freenas:~ # zpool status
  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  da1p2	 ONLINE	   0	 0	 0

errors: No known data errors

  pool: vol01
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: none requested
config:

	NAME												STATE	 READ WRITE CKSUM
	vol01											   DEGRADED	 0	 0	 0
	  raidz2-0										  DEGRADED	 0	 0	 0
		gptid/ca735441-68ea-11e7-b079-bc5ff4e79bf9.eli  ONLINE	   0	 0	 0
		gptid/cb1a547d-68ea-11e7-b079-bc5ff4e79bf9.eli  ONLINE	   0	 0	 0
		gptid/cbceb3e9-68ea-11e7-b079-bc5ff4e79bf9.eli  ONLINE	   0	 0	 0
		15721754969528194695							UNAVAIL	  0	 0	 0  was /dev/gptid/cc84d917-68ea-11e7-b079-bc5ff4e79bf9.eli
		7762318063434369017							 UNAVAIL	  0	 0	 0  was /dev/gptid/cd3cc582-68ea-11e7-b079-bc5ff4e79bf9.eli

errors: No known data errors
root@freenas:~ # 
root@freenas:~ # 
root@freenas:~ # 
root@freenas:~ # zpool online vol01 cc84d917-68ea-11e7-b079-bc5ff4e79bf9.eli
cannot online cc84d917-68ea-11e7-b079-bc5ff4e79bf9.eli: no such device in pool
root@freenas:~ # zpool online vol01 
missing device name
usage:
	online [-e] <pool> <device> ...
root@freenas:~ # zpool online -e vol01 7762318063434369017
warning: device '7762318063434369017' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
root@freenas:~ # 

 

[m0nkey_]

Rebooted? Have you loaded the key before attempting to put them online? What does camcontrol devlist show?

 

[Ericloewe]

ZFS2

RAIDZ2. There is no such thing as "ZFS2". Please use proper terminology.

 

[Stux]

Yeay for encryption.

You need to mount/decrypt the partitions first.

 

[DanPrs]

Code:

root@freenas:~ #
root@freenas:~ # camcontrol devlist
<ST3000VN000-1HJ166 SC60>		  at scbus0 target 0 lun 0 (pass0,ada0)
<ST3000VN000-1HJ166 SC60>		  at scbus0 target 1 lun 0 (pass1,ada1)
<ST3000VN000-1H4167 SC43>		  at scbus1 target 0 lun 0 (pass2,ada2)
<ST3000VN000-1H4167 SC43>		  at scbus2 target 0 lun 0 (pass3,ada3)
<ST3000VN000-1H4167 SC43>		  at scbus3 target 0 lun 0 (pass4,ada4)
<SanDisk Cruzer Fit 1.00>		  at scbus5 target 0 lun 0 (pass5,da0)
<Lexar USB Flash Drive 1100>	   at scbus6 target 0 lun 0 (pass6,da1)
root@freenas:~ #

I have rebooted. Also pretty sure the volume is mounted and encrypted as I am able to navigate to the data via smb or ssh.

 

[Stux]

Probably the quickest thing to do is follow the manual on replacing the unavail drives with them selves.

If it won't let you, the use the GUI to wipe the drives first.

 

[joeschmuck]

You are in a predicament here... If you drop one more drive then all your data is toast. If you haven't already done so, backup all your important data. Even if the resilvering starts you should have a copy of that data should one of those other drives fail.

Also, if you do not need encrypted drives, I'd get rid of them.

 

[DanPrs]

Also, if you do not need encrypted drives, I'd get rid of them.

The entire volume is backed up to crashplan. I also have some of the more critical stuff on an external locally. Because it's on crashplan, I like to encrypt it before uploading.

 

[joeschmuck]

Because it's on crashplan, I like to encrypt it before uploading.

I can't argue that one bit, I'd do the same thing. Hopefully you are able to fix this up fairly easily.

 

[Ericloewe]

I can't argue that one bit, I'd do the same thing.

I can. It's disk encryption, not file encryption. Everything is still sent in the clear.

Normal operation (data access excepted, naturally) on encrypted pools will only be possible with native ZFS encryption, which shouldn't take much longer now. But even then, sending encrypted data implies zfs send, which implies either a ZFS pool on the other side or clumsily handling the raw output of zfs send.

 

[joeschmuck]

I can. It's disk encryption, not file encryption. Everything is still sent in the clear.

I'm assuming he is encrypting the data before it leaves his computer. I tested out Crashplan myself and the program allowed me to encrypt it on my end before it flys over the wire. Maybe I shouldn't assume.

 

[Ericloewe]

I'm assuming he is encrypting the data before it leaves his computer. I tested out Crashplan myself and the program allowed me to encrypt it on my end before it flys over the wire. Maybe I shouldn't assume.

Well, the disks look mighty encrypted, as evidenced by the .eli tacked at the end of the GPTIDs.

 

[danb35]

Well, the disks look mighty encrypted, as evidenced by the .eli tacked at the end of the GPTIDs.

Disks being encrypted, and data being encrypted in transit to CrashPlan, are not mutually exclusive.

 

[Evi Vanoost]

You have to issue a zpool replace on the affected drives, not online because the devices are still missing. You can replace them with their 'new' .eli counterparts, if the /dev/gptid/xxx still exists. You may need to issue the command with force and/or labelclear the drives, if it keeps complaining, export the ZFS pool first, then remove the first and last ~1M to be safe of the disk using dd and/or make the kernel re-read the partition table using sfdisk.

 

[Stux]

Or just use the UI to wipe the disks. Then replace with them.

 

[DanPrs]

Or just use the UI to wipe the disks. Then replace with them.

This seems to be working. Initiated the first disk, will do the second tomorrow. Thank you for your help!!

Also yay for backups. And backups of backups :)

 

[Ericloewe]

Disks being encrypted, and data being encrypted in transit to CrashPlan, are not mutually exclusive.

Yes, of course not, but what matters here is that the disk encryption has, at best, dubious value.

 

[joeschmuck]

Yes, of course not, but what matters here is that the disk encryption has, at best, dubious value.

I guess that depends on the end user and how they are using it. I hope he understands that his data is not encrypted when it comes off the machine. As I said, I assume the user understands this and is encrypting the data as it flows to Crashplan. Maybe that isn't the case, there is not enough information here to know that.

@DanPrs are you encrypting your data via Crashplan or some other third party encryption program before sending the backups to Crashplan for storage? @Ericloewe brings up a good point if you are assuming that the FreeNAS encryption means any data you pull off and send to Crashplan is encrypted, because it would not be encrypted. You would need to setup the Crashplan application to encrypt it as it leaves your computer. I believe Crashplan allows you to select your own passphrase or let them select a passphrase. I've only used the Windows version of Crashplan and I selected my own passphrase when I was testing it out.

And I'm glad to hear that you are making progress with getting those drive resilvered. I hope that you have really securred that electrical connection from coming loose again in the future. You know what works great! Small zip-ties. One small zip-tie going around both of the connectors will keep them from pulling apart.

 

[DanPrs]

I recently upgraded the NAS with two additional drives. Those two drive bays were not being previously, hence I did not have them properly wired. I had a chance to tie down the cables this weekend though.

I have encryption both locally and with Crashplan, but thanks for pointing out that I should not rely on drive encryption for over the wire data protection.