Degraded RAID-10 Array

loknar28 · Dec 7, 2017

I have been running FreeNAS 9.10 containing a RAID10 volume comprised of six 3TiB WD datacenter drives for about a year now. A few days ago I saw that one of the mirrors had degraded causing the array to degrade. So I added a new 7th disk and instructed freenas through the GUI to replace the drive. Just after selecting the new drive in the presented drop down menu the Webportal went offline. However, I saw that the array had become very active, so assuming that the mirror was replicating to the new drive I left the system alone for awhile.
When I came back to the array about a day later the portal was still unavailable, so I ran 'zpool status' from the cli which provided the following output. Looking at this output it a appears I now have two which have been removed from the mirror sets.

Code:

[root@freenas] ~# zpool status
  pool: UberStore
 state: DEGRADED
status: One or more devices has been removed by the administrator.
   Sufficient replicas exist for the pool to continue functioning in a
   degraded state.
action: Online the device using 'zpool online' or replace the device with
   'zpool replace'.
  scan: resilvered 237G in 2h9m with 0 errors on Sun Dec  3 20:28:37 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   UberStore									   DEGRADED	 0	 0	 0
	 mirror-0									  DEGRADED	 0	 0	 0
	   gptid/d90afcc5-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   1235246432796506539						 REMOVED	  0	 0	 0  was /dev/gptid/d9a3adfe-19ae-11e7-b492-d05099c1cb7b
	 mirror-1									  DEGRADED	 0	 0	 0
	   4075242487269000551						 REMOVED	  0	 0	 0  was /dev/gptid/dcba356c-19ae-11e7-b492-d05099c1cb7b
	   gptid/dd6c9ecd-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/de0d95b2-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/54304a63-d880-11e7-bc9b-d05099c1cb7b  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  4 03:45:19 2017
config:

   NAME		STATE	 READ WRITE CKSUM
   freenas-boot  ONLINE	   0	 0	 0
	 ada5p2	ONLINE	   0	 0	 0

errors: No known data errors

I have been researching this hoping I would not need to bother the community but I really could use some help.
Additionally, I am fine proceeding with the cli for now but at some point I would like to get the GUI running again. I am completely unfamiliar with nginx, so I will probably need help with that as well.

tvsjr · Dec 7, 2017

First, you should back this pool up immediately. A single additional drive loss has a 66.67% chance in rendering your pool unrecoverable.
The GUI going unavailable is weird... someone more knowledgeable will have to comment on that one. Honestly, I'd just reboot the thing... AFTER you back it up.

As far as what's going on, it appears that the 7th drive was never added to anything. You haven't provided any system information, so it's hard to know what you're working with... but I would be somewhat suspicious of a hardware failure. List your hardware specs (per forum rules) and that'll give people a better clue.

Stux · Dec 7, 2017

GUI non response would be explained by it being swapped out to the failed disk

loknar28 · Dec 7, 2017

Thank you for the quick reply. As requested here are my system specs.....See signature block..

tvsjr · Dec 7, 2017

That's far from the full specs. Are you using an HBA or connecting directly to the MB? Case? Power supply? Do you have enough headroom on your power supply to accommodate adding that 7th drive?

ethereal · Dec 8, 2017

loknar28 said:
Thank you for the quick reply. As requested here are my system specs.....See signature block..

many people using phones cannot see the signature - always better to provide full system specs in the post

Ericloewe · Dec 8, 2017

ethereal said:
many people using phones cannot see the signature - always better to provide full system specs in the post

Yes. When asking for help that benefits from the system specs, always include them explicitly in your posts.

It helps those on mobile see signatures (even if phablets can often use landscape to get the desktop layout, it's a pain in the ass) and it doesn't ruin threads down the line after you change out your system.

rs225 · Dec 8, 2017

Does each drive have it's own power connection, or are any of them using splitters?

loknar28 · Dec 8, 2017

I am using a case and powersupply designed for this purpose. The Silverstone DS380 chassis which has a backplane 8 hotswap bays. The power supply used is a 300W Silverstone ST30SF.
Based on the drive specs this should be enough since each drive is rated for 10-11Watts
Average power requirements (W)
Sequential read = 10.2
Sequential write =9.7
Random read/write = 10.9 Idle = 8.6

The Avoton CPU is soldered to the motherboard

Johnnie Black · Dec 9, 2017

The drives will require the most power during spin-up, still and since it's got a 25A single 12v rail it's enough, though I didn't check the specs each disk shouldn't need more than 2/2.5A for spin-up, but not a lot to spare.

tvsjr · Dec 9, 2017

300-watt supply? I think you've found your problem. Read jgreco's treatise on power supply sizing here:
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

In short, I suspect 6 drives was pushing your power supply to the max. You added #7, and have now introduced a heisen-bug.

I would:
A. back up your pool - you're playing a dangerous game here with already-limited redundancy
B. replace your power supply with something more properly sized... Seasonic being the hands-down favorite
C. restart your troubleshooting

loknar28 · Dec 9, 2017

For those of you who are unable to see my signature I am posting a copy of it here.
FreeNAS 9.10
ASRock C2750D4I AVOTON
Crucial 16GB Kit (8GBx2) DDR3/DDR3L-1600MT/s (PC3-12800) DR x8 ECC UDIMM
6 x WD Re Datacenter drives(WD3000FYYZ)
Silverstone DS380 Case w/300w Silverstone ST30SF power supply

I have a Silverstone SFX 650SX-G on the way.

loknar28 · Dec 12, 2017

So I replaced the power supply and restarted the NAS. This seems to have fixed everything but the 7th disk that I added which is not part of the pool is not visible. If I want to make it a hotspare for the array do I have to do a zpool import first?

Code:

pool: UberStore
 state: ONLINE
  scan: resilvered 8.09G in 0h8m with 0 errors on Mon Dec 11 22:39:00 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   UberStore									   ONLINE	   0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/d90afcc5-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/d9a3adfe-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-1									  ONLINE	   0	 0	 0
	   gptid/dcba356c-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/dd6c9ecd-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/de0d95b2-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/54304a63-d880-11e7-bc9b-d05099c1cb7b  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  4 03:45:19 2017
config:

   NAME		STATE	 READ WRITE CKSUM
   freenas-boot  ONLINE	   0	 0	 0
	 ada5p2	ONLINE	   0	 0	 0

errors: No known data errors

loknar28 · Dec 12, 2017

Here are all of the disks. I am not clear on how to identify which of these is not part of the pool in the previous post.

Code:

[root@freenas] ~# camcontrol devlist
<WDC WD3000FYYZ-01UL1B1 01.01K02>  at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD3000FYYZ-01UL1B1 01.01K02>  at scbus1 target 0 lun 0 (pass1,ada1)
<WD3000FYYX 00.0D1K4>			  at scbus2 target 0 lun 0 (pass2,ada2)
<WDC WD3000FYYZ-01UL1B0 01.01K01>  at scbus3 target 0 lun 0 (pass3,ada3)
<WD3000FYYX 00.0D1K4>			  at scbus5 target 0 lun 0 (pass4,ada4)
<Marvell Console 1.01>			 at scbus9 target 0 lun 0 (pass5)
<SAMSUNG SSD UM410 Series 2.5" 16GB VAM12D1Q>  at scbus10 target 0 lun 0 (pass6,ada5)
<WDC WD3000FYYZ-01UL1B0 01.01K01>  at scbus14 target 0 lun 0 (pass7,ada6)
<WDC WD3000FYYZ-01UL1B0 01.01K01>  at scbus15 target 0 lun 0 (pass8,ada7)

Ericloewe · Dec 12, 2017

loknar28 said:
If I want to make it a hotspare for the array do I have to do a zpool import first?

That's a weird question. The pool you're adding a spare to has to be online during the process, yes.

loknar28 said:
I am not clear on how to identify which of these is not part of the pool in the previous post.

There are some documents in the Resources section that should help.

loknar28 · Dec 12, 2017

Ericloewe said:
That's a weird question. The pool you're adding a spare to has to be online during the process, yes.

The pool is already online. I'm asking about adding the extra disk to the pool. Then hopefully having it designated as a hot spare. Maybe zpool import is not the command to be referencing. I have been searching in the documentation(RTFM) but not finding anything that clearly explains what I am looking to accomplish. I realize freeNAS works differently than a typical Hardware RAID NAS so I may just not be understanding how hotspares are defined.

Ericloewe · Dec 12, 2017

I was going to link the manual section, but it turns out that it doesn't actually tell you how to add a hot spare. Someone who's done it before will probably show up and tell you the exact process, so that you don't have to play around looking for it.

loknar28 said:
Maybe zpool import is not the command to be referencing.

No. The command would be zpool add -n poolname spare driveID, but don't do that or you'll make FreeNAS sad.

danb35 · Dec 12, 2017

Ericloewe said:
Someone who's done it before will probably show up and tell you the exact process,

I haven't done it, but I'm 99% sure it's just another form of extending a pool. Volume Manager, select pool to extend, add the disk, select spare from the drop-down, and you're good.

loknar28 · Dec 12, 2017

danb35 said:
select spare from the drop-down, and you're good.

Nice...That is exactly what I needed!!

Code:

[root@freenas] ~# zpool status -v
  pool: UberStore
 state: ONLINE
  scan: resilvered 8.09G in 0h8m with 0 errors on Mon Dec 11 22:39:00 2017
config:

   NAME											STATE	 READ WRITE CKSUM
   UberStore									   ONLINE	   0	 0	 0
	 mirror-0									  ONLINE	   0	 0	 0
	   gptid/d90afcc5-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/d9a3adfe-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-1									  ONLINE	   0	 0	 0
	   gptid/dcba356c-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/dd6c9ecd-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	 mirror-2									  ONLINE	   0	 0	 0
	   gptid/de0d95b2-19ae-11e7-b492-d05099c1cb7b  ONLINE	   0	 0	 0
	   gptid/54304a63-d880-11e7-bc9b-d05099c1cb7b  ONLINE	   0	 0	 0
   spares
	 gptid/43326693-dfb4-11e7-b8eb-d05099c1cb7b	AVAIL   

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Dec  4 03:45:19 2017
config:

   NAME		STATE	 READ WRITE CKSUM
   freenas-boot  ONLINE	   0	 0	 0
	 ada5p2	ONLINE	   0	 0	 0

errors: No known data errors

Ericloewe · Dec 12, 2017

danb35 said:
I haven't done it, but I'm 99% sure it's just another form of extending a pool. Volume Manager, select pool to extend, add the disk, select spare from the drop-down, and you're good.

So was I, but I like to steer people away from the Volume Manager (such a horrible name, too) unless I'm certain that's what they need.

Anyway, there's a docs bug for this now. I'm on mobile, so no link until the morning.

Important Announcement for the TrueNAS Community.

Degraded RAID-10 Array

Dabbler

Guru

MVP

Dabbler

Guru

Guru

Server Wrangler

Guru

Dabbler

Guru

Guru

Dabbler

Dabbler

Dabbler

Server Wrangler

Dabbler

Server Wrangler

Hall of Famer

Dabbler

Server Wrangler

Similar threads