Substitution Unavailable disks RAIDz2

[OltreXR4]

Hello,

Just to be clear from the beginning I am not a deep FreeNAS expert and I am managing our internal lab storage.

Since a couple of years I have installed an IBM x3560 M3 FreeNAS server ( FreeNAS-9.10.2-U6 ) with a fiber channel connection to 12x 2TB disk 7.2K SAS HDD and 12x 450GB 15K SAS HDD.
It worked fine until I got an error in a couple of device.
I am not sure about the procedure I have followed and still I have some issue, that is why I ask you guru.

I have a couple of pool defined in raidz2 with a couple of dataset for eachone.
There was a DEGRADED pool from the console, so from the GUI console Storage > View Multipats I saw the dev in FAIL.
I had four dev in FAIL and from the shell with the command smartctl -a /dev/daxx I had two FreeBSD iSCSI disks and two real disk details.
I bought the two real new disks and substitute them.

Fromt the console I have "Replace" the multipath of the degraded Pool-2 with new one and now it seems to work but another multipath in the Pool-1 is in FAIL.

At this point I am not sure how to undestand if the new disks are already available and working or not, how to make them available and how I can understand which real disk match the different pool, the intention was to have Pool-2 using only the 15K disks and Pool-1 the 7.2K

I have found several instruction how to substitute a degraded disk but I have not found details when we use raidz2 and multipath disks, that is confusing me.

Thanks a lot for your patience and your help

Mirco

 

[sretalla]

You should start by showing us (in code tags) the output of zpool status -v so we can really see what's happening here.

Multipath isn't really the concern when replacing a disk in a zpool as long as the replacement disk is connected in some way (should be multipath if you make the same connections as the old disk).... unless the failure is actually of the multiple paths themselves and the disk may still be OK. let's start with the command above.

You can also show gmultipath list since you'll be at a command line.

 

[OltreXR4]

Thanks for the answer.
here the results of the command:

# zpool status -v Pool-1
pool: Pool-1
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub repaired 176K in 97h6m with 0 errors on Thu Aug 2 01:06:56 2018
config:

NAME STATE READ WRITE CKSUM
Pool-1 DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/62aa844a-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/65fe2f6c-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/695718da-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/6ca87a5d-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/6ff60b12-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/7337c439-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
gptid/768a8813-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
1206842781626932889 UNAVAIL 0 0 0 was /dev/gptid/79e51bf3-ed08-11e5-9b9d-e41f13e5b
4c8
gptid/7d4b42f5-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/80a54bab-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/83e45198-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/875f4693-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
logs
mirror-2 ONLINE 0 0 0
gptid/87c1dbd4-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
gptid/8814e3a8-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0
cache
gptid/e0e49045-ed08-11e5-9b9d-e41f13e5b4c8 ONLINE 0 0 0

errors: No known data errors

# zpool status -v Pool-2
pool: Pool-2
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: resilvered 468G in 16h24m with 0 errors on Wed Aug 15 09:19:46 2018
config:

NAME STATE READ WRITE CKSUM
Pool-2 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/b91e246e-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
3308045596388739653 UNAVAIL 0 0 0 was /dev/gptid/29c516c5-9fbe-11e8-bf0e-e41f13e5b
4c8
gptid/bc7c598d-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
gptid/be2b66a2-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
gptid/bfd73ce8-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
gptid/c18b1015-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
gptid/c357238b-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
gptid/c5080709-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
8903519689664840524 UNAVAIL 0 0 0 was /dev/gptid/8170caa5-9fbe-11e8-bf0e-e41f13e5b
4c8
gptid/c86354be-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
gptid/ca154ea4-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
16049171543103426295 UNAVAIL 0 0 0 was /dev/gptid/0c3bcc9f-9fc9-11e8-bf0e-e41f13e5b
4c8
logs
mirror-2 ONLINE 0 0 0
gptid/cc2b104e-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
gptid/cc77c45f-f0cc-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0
cache
gptid/167c8afe-f0cd-11e5-885e-e41f13e5b4c8 ONLINE 0 0 0

errors: No known data errors


gmultipath list

Geom name: disk13
Type: AUTOMATIC
Mode: Active/Passive
UUID: 5793db22-9fae-11e8-bf0e-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk13
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da47
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
2. Name: da46
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e1
State: PASSIVE

Geom name: disk3
Type: AUTOMATIC
Mode: Active/Passive
UUID: 577b2220-9fae-11e8-bf0e-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk3
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da45
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
2. Name: da44
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r1w1e1
State: PASSIVE

Geom name: disk1
Type: AUTOMATIC
Mode: Active/Passive
UUID: c409dba6-f015-11e5-a668-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk1
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da43
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da42
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk8
Type: AUTOMATIC
Mode: Active/Passive
UUID: 47ee9f4b-f009-11e5-acb4-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk8
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da41
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da34
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk9
Type: AUTOMATIC
Mode: Active/Passive
UUID: 480999e2-f009-11e5-acb4-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk9
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da40
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da35
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk15
Type: AUTOMATIC
Mode: Active/Passive
UUID: 485f66f0-f009-11e5-acb4-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk15
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da39
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da37
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk14
Type: AUTOMATIC
Mode: Active/Passive
UUID: 484323c6-f009-11e5-acb4-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk14
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da38
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da36
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk5
Type: AUTOMATIC
Mode: Active/Passive
UUID: bc3f328f-ed0a-11e5-abd9-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk5
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da33
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da32
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk4
Type: AUTOMATIC
Mode: Active/Passive
UUID: bc237c51-ed0a-11e5-abd9-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk4
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da31
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da30
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk12
Type: AUTOMATIC
Mode: Active/Passive
UUID: 23af4440-df9d-11e5-b0d8-b499bab74d24
State: OPTIMAL
Providers:
1. Name: multipath/disk12
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da29
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da11
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

 

[OltreXR4]

Geom name: disk11
Type: AUTOMATIC
Mode: Active/Passive
UUID: 237f6d75-df9d-11e5-b0d8-b499bab74d24
State: OPTIMAL
Providers:
1. Name: multipath/disk11
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da28
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da10
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk10
Type: AUTOMATIC
Mode: Active/Passive
UUID: 235b5ebd-df9d-11e5-b0d8-b499bab74d24
State: OPTIMAL
Providers:
1. Name: multipath/disk10
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da27
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da8
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk18
Type: AUTOMATIC
Mode: Active/Passive
UUID: 37142689-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk18
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da26
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da13
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk2
Type: AUTOMATIC
Mode: Active/Passive
UUID: 2250d629-df9d-11e5-b0d8-b499bab74d24
State: OPTIMAL
Providers:
1. Name: multipath/disk2
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da25
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da9
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk17
Type: AUTOMATIC
Mode: Active/Passive
UUID: 36fd1843-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk17
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da24
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da12
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk19
Type: AUTOMATIC
Mode: Active/Passive
UUID: 3724e82e-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk19
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da23
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da14
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk24
Type: AUTOMATIC
Mode: Active/Passive
UUID: 379899d6-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk24
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da22
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da18
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk23
Type: AUTOMATIC
Mode: Active/Passive
UUID: 37863570-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk23
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da21
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da17
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk21
Type: AUTOMATIC
Mode: Active/Passive
UUID: 3756490d-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk21
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da20
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da15
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk22
Type: AUTOMATIC
Mode: Active/Passive
UUID: 376dac1c-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk22
Mediasize: 2000398933504 (1.8T)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da19
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da16
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

 

[OltreXR4]

Geom name: disk7
Type: AUTOMATIC
Mode: Active/Passive
UUID: 3671abd1-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk7
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da6
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da3
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk6
Type: AUTOMATIC
Mode: Active/Passive
UUID: 365f1a6b-ed04-11e5-a0fa-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk6
Mediasize: 450098159104 (419G)
Sectorsize: 512
Mode: r2w2e5
State: OPTIMAL
Consumers:
1. Name: da4
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: ACTIVE
2. Name: da2
Mediasize: 450098159616 (419G)
Sectorsize: 512
Mode: r3w3e6
State: PASSIVE

Geom name: disk13-15530672
Type: AUTOMATIC
Mode: Active/Passive
UUID: 8e382980-6606-11e7-b1bc-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk13-15530672
Mediasize: 644245093888 (600G)
Sectorsize: 512
Stripesize: 131072
Stripeoffset: 0
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da56
Mediasize: 644245094400 (600G)
Sectorsize: 512
Stripesize: 131072
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
2. Name: da57
Mediasize: 644245094400 (600G)
Sectorsize: 512
Stripesize: 131072
Stripeoffset: 0
Mode: r1w1e1
State: PASSIVE

Geom name: disk25
Type: AUTOMATIC
Mode: Active/Passive
UUID: 8e4ea3d6-6606-11e7-b1bc-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk25
Mediasize: 1099511627264 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da58
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
2. Name: da59
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: PASSIVE

Geom name: disk26
Type: AUTOMATIC
Mode: Active/Passive
UUID: 8e8bece7-6606-11e7-b1bc-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk26
Mediasize: 1099511627264 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da60
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
2. Name: da61
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: PASSIVE

Geom name: disk27
Type: AUTOMATIC
Mode: Active/Passive
UUID: 8fdb05ae-6606-11e7-b1bc-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk27
Mediasize: 1099511627264 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da62
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
2. Name: da63
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: PASSIVE

Geom name: disk28
Type: AUTOMATIC
Mode: Active/Passive
UUID: 904b1b45-6606-11e7-b1bc-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk28
Mediasize: 1099511627264 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da64
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
2. Name: da65
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: PASSIVE

Geom name: disk29
Type: AUTOMATIC
Mode: Active/Passive
UUID: 9228de3f-6606-11e7-b1bc-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk29
Mediasize: 1099511627264 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r0w0e0
State: OPTIMAL
Consumers:
1. Name: da66
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: ACTIVE
2. Name: da67
Mediasize: 1099511627776 (1.0T)
Sectorsize: 512
Stripesize: 16384
Stripeoffset: 0
Mode: r1w1e1
State: PASSIVE

 

[OltreXR4]

I add also a glabel status:

glabel status
Name Status Components
gptid/87c1dbd4-ed08-11e5-9b9d-e41f13e5b4c8 N/A da48p2
gptid/8814e3a8-ed08-11e5-9b9d-e41f13e5b4c8 N/A da49p2
gptid/de2a8091-ed01-11e5-9b19-e41f13e5b4c8 N/A da50p1
gptid/de33f2ed-ed01-11e5-9b19-e41f13e5b4c8 N/A da50p2
gptid/de5cc480-ed01-11e5-9b19-e41f13e5b4c8 N/A da51p1
gptid/de6635fc-ed01-11e5-9b19-e41f13e5b4c8 N/A da51p2
gptid/167c8afe-f0cd-11e5-885e-e41f13e5b4c8 N/A da52p1
gptid/cc2b104e-f0cc-11e5-885e-e41f13e5b4c8 N/A da53p2
gptid/cc77c45f-f0cc-11e5-885e-e41f13e5b4c8 N/A da54p2
gptid/e0e49045-ed08-11e5-9b9d-e41f13e5b4c8 N/A da55p1
gptid/b91e246e-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk1p2
gptid/c357238b-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk8p2
gptid/c86354be-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk9p2
gptid/ca154ea4-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk15p2
gptid/c5080709-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk14p2
gptid/be2b66a2-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk5p2
gptid/bc7c598d-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk4p2
gptid/6ca87a5d-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk12p2
gptid/695718da-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk11p2
gptid/65fe2f6c-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk10p2
gptid/7337c439-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk18p2
gptid/62aa844a-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk2p2
gptid/6ff60b12-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk17p2
gptid/768a8813-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk19p2
gptid/875f4693-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk24p2
gptid/83e45198-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk23p2
gptid/7d4b42f5-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk21p2
gptid/80a54bab-ed08-11e5-9b9d-e41f13e5b4c8 N/A multipath/disk22p2
gptid/c18b1015-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk7p2
gptid/bfd73ce8-f0cc-11e5-885e-e41f13e5b4c8 N/A multipath/disk6p2
gptid/cdcde654-9fb0-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk13-15530672p1
gptid/cde12fc7-9fb0-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk13-15530672p2
gptid/27d697dd-9bad-11e8-9d14-e41f13e5b4c8 N/A multipath/disk25p1
gptid/27e56672-9bad-11e8-9d14-e41f13e5b4c8 N/A multipath/disk25p2
gptid/98043678-9fd0-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk26p1
gptid/9821b84b-9fd0-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk26p2
gptid/0c21928f-9fc9-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk27p1
gptid/0c3bcc9f-9fc9-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk27p2
gptid/81557aae-9fbe-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk28p1
gptid/8170caa5-9fbe-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk28p2
gptid/29bdd466-9fbe-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk29p1
gptid/29c516c5-9fbe-11e8-bf0e-e41f13e5b4c8 N/A multipath/disk29p2

 

[OltreXR4]

and a gmultipath status:

# gmultipath status
Name Status Components
multipath/disk13 OPTIMAL da47 (ACTIVE)
da46 (PASSIVE)
multipath/disk3 OPTIMAL da45 (ACTIVE)
da44 (PASSIVE)
multipath/disk1 OPTIMAL da43 (ACTIVE)
da42 (PASSIVE)
multipath/disk8 OPTIMAL da41 (ACTIVE)
da34 (PASSIVE)
multipath/disk9 OPTIMAL da40 (ACTIVE)
da35 (PASSIVE)
multipath/disk15 OPTIMAL da39 (ACTIVE)
da37 (PASSIVE)
multipath/disk14 OPTIMAL da38 (ACTIVE)
da36 (PASSIVE)
multipath/disk5 OPTIMAL da33 (ACTIVE)
da32 (PASSIVE)
multipath/disk4 OPTIMAL da31 (ACTIVE)
da30 (PASSIVE)
multipath/disk12 OPTIMAL da29 (ACTIVE)
da11 (PASSIVE)
multipath/disk11 OPTIMAL da28 (ACTIVE)
da10 (PASSIVE)
multipath/disk10 OPTIMAL da27 (ACTIVE)
da8 (PASSIVE)
multipath/disk18 OPTIMAL da26 (ACTIVE)
da13 (PASSIVE)
multipath/disk2 OPTIMAL da25 (ACTIVE)
da9 (PASSIVE)
multipath/disk17 OPTIMAL da24 (ACTIVE)
da12 (PASSIVE)
multipath/disk19 OPTIMAL da23 (ACTIVE)
da14 (PASSIVE)
multipath/disk24 OPTIMAL da22 (ACTIVE)
da18 (PASSIVE)
multipath/disk23 OPTIMAL da21 (ACTIVE)
da17 (PASSIVE)
multipath/disk21 OPTIMAL da20 (ACTIVE)
da15 (PASSIVE)
multipath/disk22 OPTIMAL da19 (ACTIVE)
da16 (PASSIVE)
multipath/disk7 OPTIMAL da6 (ACTIVE)
da3 (PASSIVE)
multipath/disk6 OPTIMAL da4 (ACTIVE)
da2 (PASSIVE)
multipath/disk13-15530672 OPTIMAL da56 (ACTIVE)
da57 (PASSIVE)
multipath/disk25 OPTIMAL da58 (ACTIVE)
da59 (PASSIVE)
multipath/disk26 OPTIMAL da60 (ACTIVE)
da61 (PASSIVE)
multipath/disk27 OPTIMAL da62 (ACTIVE)
da63 (PASSIVE)
multipath/disk28 OPTIMAL da64 (ACTIVE)
da65 (PASSIVE)
multipath/disk29 OPTIMAL da66 (ACTIVE)
da67 (PASSIVE)

 

[sretalla]

So, first of all, you need to act fast to avoid losing your pool as you're out of fault tolerance on one of the VDEVs in Pool-2.

It looks to me that overall, you have 3 failed/dead disks in Pool-2 (1 in the first VDEV and 2 in the second) and 1 failed/dead disk in Pool-1 (2nd VDEV).

Have you been able to identify those disks physically in your server?

Do you have spare slots (and spare disks of equal or greater size)?

I see you show resilvering in the Pool-2 status, so I presume that's the disk you replaced (around 500GB resilvered).

Is this an old server (maybe you're going to be losing some more disks soon)?

It really should be as simple as replacing the disks (all 4 of them) in the GUI as you probably did once already to get back to "normal" (and then start to consider what your options are in terms of how you got to 5 simultaneous drive failures... maybe these have accumulated over some time and not been dealt with as they happened?), the muiltipath stuff should handle itself as you go.

 

[OltreXR4]

I agree with you that I need to act fast, I am trying it as much as possible.
The first error was related to four multipath dev, I have checked them one by one.

[root@intl-nas-001 ~]# smartctl -a /dev/da59 | head -15
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: FreeBSD
Product: iSCSI Disk
Revision: 0123
Compliance: SPC-4
User Capacity: 1,099,511,627,776 bytes [1.09 TB]
Logical block size: 512 bytes
LU is thin provisioned, LBPRZ=1
Rotation Rate: 15000 rpm
Logical Unit id: 0x6589cfc000000d5472afe3937b0a8d9b
Serial number: e41f13e5b4c804
Device type: disk


[root@intl-nas-001 ~]# smartctl -a /dev/da57 | head -15
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: FreeBSD
Product: iSCSI Disk
Revision: 0123
Compliance: SPC-4
User Capacity: 644,245,094,400 bytes [644 GB]
Logical block size: 512 bytes
LU is thin provisioned, LBPRZ=1
Rotation Rate: 15000 rpm
Logical Unit id: 0x6589cfc000000af4b17dcb63ed9b1c85
Serial number: e41f13e5b4c800
Device type: disk
[root@intl-nas-001 ~]#

[root@intl-nas-001 ~]# smartctl -a /dev/da24 | head -15
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST32000444SS
Revision: XRMA
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50025e03b2f
Serial number: 9WM2Y8WL0000C12346KM
Device type: disk
Transport protocol: SAS (SPL-3)
[root@intl-nas-001 ~]#

[root@intl-nas-001 ~]# smartctl -a /dev/da40 | head -15
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST3450857SS
Revision: XREC
User Capacity: 450,098,159,616 bytes [450 GB]
Logical block size: 512 bytes
Rotation Rate: 15000 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50042b67b73
Serial number: 6SK149AM0000N203BYVS
Device type: disk
Transport protocol: SAS (SPL-3)
[root@intl-nas-001 ~]#

so I changed physically two disks:

- 9WM2Y8WL0000C12346KM 2TB 7.2K
- 6SK149AM0000N203BYVS 450GB 15K

after the restart I am not sure if these disks are available.

From the GUI Storage -> Disks I have two size 0 disks but if I check with smartctl I got:

Vendor: SEAGATE
Product: ST3450857SS
Revision: XREC
Rotation Rate: 15000 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50042b20f1b
Serial number: 6SK0GYZQ0000N204AWWN
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Thu Aug 16 12:59:27 2018 CEST
device is NOT READY (e.g. spun down, busy)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.


[root@intl-nas-001] ~# smartctl -a /dev/da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST3450857SS
Revision: XREC
Rotation Rate: 15000 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5003ba6ee6b
Serial number: 6SK0Y8MB0000N150MJED
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Thu Aug 16 12:59:34 2018 CEST
device is NOT READY (e.g. spun down, busy)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

and I am not able to wipe them because I got the error:

"Failed to wipe da7: dd: /dev/da7 device not configured 1+0 record in 0+0 out of 0 bytes transfered in 0.000775 secs (0 bytes/sec)"

So, initially I was able to identify the failing disks from the Storage -> Multipath view, because there were 4 /dev/daxx in FAIL status, but now there are not any FAIL status as you can see from my previous post.

So it is not clear to me

- if I have to substitute other disks or not
- how to identify them
- how to be sure that the new two disks are available and to make them available, is it enough to wipe? do I have to do more things?

Thanks in advance for your precious help.

 

[OltreXR4]

About the resilvering, I have tried initially to Replace some multipath and it seemed to work, but after a restart I lost them and the pool are now Degraded.

If I try to "Replace" the multipath with the available from the drop down menu list I got this error:

Disk is not clear, partitions or ZFS labels were found.

so How can I identify the multipath avilable to replace?

Thanks

 

[OltreXR4]

for example if I select multipath/disk3 to replace a disk in the pool I receive the previous error
If I look the gmultipath list for disk3 I have:

Code:

# gmultipath list disk3
Geom name: disk3
Type: AUTOMATIC
Mode: Active/Passive
UUID: 577b2220-9fae-11e8-bf0e-e41f13e5b4c8
State: OPTIMAL
Providers:
1. Name: multipath/disk3
   Mediasize: 450098159104 (419G)
   Sectorsize: 512
   Mode: r0w0e0
   State: OPTIMAL
Consumers:
1. Name: da45
   Mediasize: 450098159616 (419G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
2. Name: da44
   Mediasize: 450098159616 (419G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE


If I look the glabel status I cannot find any da44 or da45 or disk3

# glabel status
									  Name  Status  Components
gptid/87c1dbd4-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  da48p2
gptid/8814e3a8-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  da49p2
gptid/de2a8091-ed01-11e5-9b19-e41f13e5b4c8	 N/A  da50p1
gptid/de33f2ed-ed01-11e5-9b19-e41f13e5b4c8	 N/A  da50p2
gptid/de5cc480-ed01-11e5-9b19-e41f13e5b4c8	 N/A  da51p1
gptid/de6635fc-ed01-11e5-9b19-e41f13e5b4c8	 N/A  da51p2
gptid/167c8afe-f0cd-11e5-885e-e41f13e5b4c8	 N/A  da52p1
gptid/cc2b104e-f0cc-11e5-885e-e41f13e5b4c8	 N/A  da53p2
gptid/cc77c45f-f0cc-11e5-885e-e41f13e5b4c8	 N/A  da54p2
gptid/e0e49045-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  da55p1
gptid/b91e246e-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk1p2
gptid/c357238b-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk8p2
gptid/c86354be-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk9p2
gptid/ca154ea4-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk15p2
gptid/c5080709-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk14p2
gptid/be2b66a2-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk5p2
gptid/bc7c598d-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk4p2
gptid/6ca87a5d-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk12p2
gptid/695718da-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk11p2
gptid/65fe2f6c-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk10p2
gptid/7337c439-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk18p2
gptid/62aa844a-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk2p2
gptid/6ff60b12-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk17p2
gptid/768a8813-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk19p2
gptid/875f4693-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk24p2
gptid/83e45198-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk23p2
gptid/7d4b42f5-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk21p2
gptid/80a54bab-ed08-11e5-9b9d-e41f13e5b4c8	 N/A  multipath/disk22p2
gptid/c18b1015-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk7p2
gptid/bfd73ce8-f0cc-11e5-885e-e41f13e5b4c8	 N/A  multipath/disk6p2
gptid/cdcde654-9fb0-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk13-15530672p1
gptid/cde12fc7-9fb0-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk13-15530672p2
gptid/27d697dd-9bad-11e8-9d14-e41f13e5b4c8	 N/A  multipath/disk25p1
gptid/27e56672-9bad-11e8-9d14-e41f13e5b4c8	 N/A  multipath/disk25p2
gptid/98043678-9fd0-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk26p1
gptid/9821b84b-9fd0-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk26p2
gptid/0c21928f-9fc9-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk27p1
gptid/0c3bcc9f-9fc9-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk27p2
gptid/81557aae-9fbe-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk28p1
gptid/8170caa5-9fbe-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk28p2
gptid/29bdd466-9fbe-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk29p1
gptid/29c516c5-9fbe-11e8-bf0e-e41f13e5b4c8	 N/A  multipath/disk29p2

does it mean that I can "force" the replace of the multipath without any risk?

Thanks

 

[danb35]

tl;dr. Code tags are your friend. Learn them.

 

[OltreXR4]

At the end I have understood the reference and created a shell script to extract all the related info (disk, serial, gptid, dev) for each disk.
I have substituted the failing disks and now I have another issue.
I have a disk I was able to see that is not working properly, it is a new just replaced disk.

It looks like it is not ready:

# smartctl -a /dev/da41
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST3450857SS
Revision: XREC
Rotation Rate: 15000 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5003ba681ef
Serial number: 6SK0XZT50000N151CL4A
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Thu Aug 23 10:37:21 2018 CEST
device is NOT READY (e.g. spun down, busy)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

but I can see it from the camcontrol devlist:

<SEAGATE ST3450857SS XREC> at scbus0 target 94 lun 0 (pass44,da40)
<SEAGATE ST3450857SS XREC> at scbus0 target 95 lun 0 (pass45,da41)

from the inquiry:

# camcontrol inquiry da40
pass44: <SEAGATE ST3450857SS XREC> Fixed Direct Access SPC-3 SCSI device
pass44: Serial Number 6SK0XZT50000N151CL4A
pass44: 600.000MB/s transfers, Command Queueing Enabled
# camcontrol inquiry da41
pass45: <SEAGATE ST3450857SS XREC> Fixed Direct Access SPC-3 SCSI device
pass45: Serial Number 6SK0XZT50000N151CL4A
pass45: 600.000MB/s transfers, Command Queueing Enabled

it is available also from the reporting (admin gui) but without any i/o of course.

We have already turn the unit off, extract and reinsert the disk but nothing change.
No valid suggestions found in the forum

Any idea how can I solve this issue?

Thanks