SOLVED Cannot Import Zpool - Insufficient Replicas

[NadoNate]

Hi all,
I have 12 Seagate 4TB's in a Z1 I am unable to import the pool.

Here is the output of zpool import:

Code:

[root@freenas] ~# zpool import
   pool: BEAST
	 id: 10419953668961628712
  state: UNAVAIL
 status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

		BEAST										   UNAVAIL  missing device
		  mirror-0									  DEGRADED
			gptid/c8bfdd13-b5f2-11e4-bafb-00235402667d  ONLINE
			5405074622095566651						 UNAVAIL  cannot open

   pool: RENAULT
	 id: 7829655522934002613
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
		devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

		RENAULT										 UNAVAIL  insufficient replicas
		  raidz1-0									  UNAVAIL  insufficient replicas
			296370703986508029						  UNAVAIL  cannot open
			gptid/e268ccdf-d2e6-11e6-96e2-74d02b29b743  ONLINE
			3929311607626872838						 UNAVAIL  cannot open
			gptid/e3101f77-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e396c418-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e2f12b56-d2e6-11e6-96e2-74d02b29b743  ONLINE
		  raidz1-1									  ONLINE
			gptid/e3dbfd10-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e34b2c1d-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e44badda-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e416d51f-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e52b93a0-d2e6-11e6-96e2-74d02b29b743  ONLINE
			gptid/e599f47d-d2e6-11e6-96e2-74d02b29b743  ONLINE

To troubleshoot, I have swapped SATA & power cables, & made sure the connectors were mated properly. This didn't help, I removed the disks that were identified as UNAVAILABLE, than ran DST, Short, and Long generic tests on a separate system to each of these drives (using Seatools). Both drives passed all tests, and feel like they are spinning up properly without any unusual noises.

Its unfortunate if two drives died at the same time, because of the type of raid i chose...

Is there any tests or tools anyone can recommend? I'm very hesitant to do anything that may destroy the data on these disks because they are needed to complete the pool...

-Thanks,

FreeNAS 9.10.2
32675MB Memory
Asus Maximus Formula Motherboard
SAS 9211-8i card
12 4TB Seagate HDD- RAIDZ1

 

[bigphil]

Try zpool import BEAST -f to force import of that pool and then you can replace the faulted disk. As for the RENAULT pool...not sure what to do there, but it doesn't look good. How are the disks connected to the system? Motherboard controller, HBA, expanders, etc?

 

[DrKK]

I have 12 Seagate 4TB's in a Z1...

...and DrKK stops reading.

 

[NadoNate]

...and DrKK stops reading.

Are 4tb drives unreliable or does my Z1 RAID choice amuse you?

 

[NadoNate]

Try "zpool import BEAST -f" to force import of that pool and then you can replace the faulted disk. As for the RENAULT pool...not sure what to do there, but it doesn't look good. How are the disks connected to the system??? Motherboard controller, HBA, expanders, etc?

8 drives are connected via the SAS card, the other 4 MB.

I have tried swapping connections, it seems like the drives at this point :(

 

[rs225]

If the drives seem to work on another system, then there is a hardware problem in your FreeNAS system. You will need to find it, or move all the drives to a working system. Consider your power supply as a possible cause.

 

[cobrakiller58]

Are 4tb drives unreliable or does my Z1 RAID choice amuse you?

My heart sank when I read that second line if @DrKK reacted the same way I did he probably really did stop reading lol I'd probaby edit that line.
A 12 wide RAIDZ1 is discouraged but you have a saner(IMO) setup than that according to the reset of your post. I'm hoping you have a back-up of your valuable data and that one of the knowledgeable members here will be able to help you.

 

[DrKK]

I literally stopped reading.

I still don't know what you guys are talking about. I read "12-wide RAID-Z1" and instantly disengaged.

 

[rs225]

It's actually 6-wide raidz1, two vdevs.(Which is no better, as one vdev is missing two disks) The problem may be in the system rather than the drives, since three disks going UNAVAIL at the same time is rather suspicious.

 

[tvsjr]

Unless you've just got a weird hardware issue, your pool is dead. What's the output of smartctl -a /dev/xxxx, where xxxx is one of the affected drives?

 

[NadoNate]

Ok, I was able to verify the two drives were in fact defective. This was a system I set up for a friend several years ago, one of the drives was throwing ignored errors for sometime, so this could have been avoided.

I have replaced the defective drive, upgraded to FreeNAS 11 and created a new RAIDZ3 pool per @DrKK's recommendation... :))). I have also added myself to email alerts on routine maintenance so we don't end up in this situation again.

I really appreciate everyone's help and quick responses, you guys (gals?) are awesome!

 

[tvsjr]

You should also configure short and long SMART tests and periodic scrubs. Drive temperature monitoring wouldn't hurt either.

 

[NadoNate]

All set, bi-weekly short tests, a weekly long test, and monthly scrubs. Also, periodic snapshots set for critical datasets. :)