remove and re-add l2arc + hotspare problems

[rolandk]

i have setup a new system with freenas 9.10 and currently i`m testing a little bit

i have 2 pools raidz2 9+2 (9 disks + 2 hotspares)

zil/slog is mirrored across enterprise ssd`s, cache/l2arc is striped across the same disks (as i do not want to add extra ssd i partitioned the disks appropriately)

i came across two issues:

1.
- when pulling one of the ssd`s, zil mirror goes to degrade state as expected and cache device goes unavail. as expected

- when re-inserting the disk zill mirror automatically resilvers - but cache device remain unavail.

- i cannot bring back the cache device online. i`m getting:

[root@fnask-bnkw] ~# zpool online zfspool_sas_hitachi /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150
cannot online /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150: cannot expand cache devices

[root@fnask-bnkw] ~# glabel status|grep ed8c
gptid/ed8c9680-ee27-11e6-a3b4-00259004b150 N/A da21p3

- what do do here ?

2.
- when pulling an unused hotspare disk, it`s not being correctly detected as missing nor is it being correctly detected as
available by zfs when it was offline and disk being re-inserted.
- i cannot bring it back online manually to zfs again. any clues, why ?

[root@fnask-bnkw] ~# zpool online zfspool_sas_seagate /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150
cannot online /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150: device is reserved as a hot spare
[root@fnask-bnkw] ~# glabel status | grep 9b9
gptid/9b95873a-ee2b-11e6-a3b4-00259004b150 N/A multipath/disk24p2
[root@fnask-bnkw] ~# zpool status|grep 9b95
1682681741557551889 UNAVAIL was /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150

regards
roland

Code:

  pool: zfspool_sas_hitachi
state: ONLINE
status: One or more devices has been removed by the administrator.
   Sufficient replicas exist for the pool to continue functioning in a
   degraded state.
action: Online the device using 'zpool online' or replace the device with
   'zpool replace'.
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Feb  9 04:31:13 2017
config:

   NAME  STATE  READ WRITE CKSUM
   zfspool_sas_hitachi  ONLINE  0  0  0
    raidz2-0  ONLINE  0  0  0
    gptid/c907b699-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/c98a0ca1-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/ca0ba737-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/ca8ee926-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/cb0fc998-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/cb9c7ef9-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/cc310971-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/ccafe07d-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/cd33f6db-ee29-11e6-a3b4-00259004b150  ONLINE  0  0  0
   logs
    mirror-1  ONLINE  0  0  0
    gptid/713f0662-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/78d3e971-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
   cache
    gptid/e9d3a97b-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
    242793518834828308  REMOVED  0  0  0  was /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150
   spares
    gptid/841cb167-ee2a-11e6-a3b4-00259004b150  AVAIL
    gptid/849e25a9-ee2a-11e6-a3b4-00259004b150  AVAIL

errors: No known data errors

  pool: zfspool_sas_seagate
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Feb  9 04:55:42 2017
config:

   NAME  STATE  READ WRITE CKSUM
   zfspool_sas_seagate  ONLINE  0  0  0
    raidz2-0  ONLINE  0  0  0
    gptid/6f6df9f4-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/701396d5-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/7102205b-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/71ac1e3e-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/7254274d-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/72fd6fee-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/73a6dc80-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/744ee081-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/74f7524d-ee2b-11e6-a3b4-00259004b150  ONLINE  0  0  0
   logs
    mirror-1  ONLINE  0  0  0
    gptid/98bdd972-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/9d75100f-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
   cache
    gptid/f4becdc9-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
    gptid/f94b4bc7-ee27-11e6-a3b4-00259004b150  ONLINE  0  0  0
   spares
    1682681741557551889  UNAVAIL  was /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150
    gptid/9aef2b1b-ee2b-11e6-a3b4-00259004b150  AVAIL

errors: No known data errors

 

[rolandk]

regarding removal and re-add of hotspare, apparently this must be a porting issue, as i found in zfsonlinux bugtracker, linux has a similar problem https://github.com/zfsonlinux/zfs/issues/1105

 

[dlavigne]

It wouldn't hurt to put in a feature request at bugs.freenas.org that mentions https://github.com/zfsonlinux/zfs/pull/4673 so that the FreeNAS devs can track the FreeBSD port of this. Post your issue number here after you make the request.

 

[rolandk]

https://bugs.freenas.org/issues/21606 Hotspare removal/reinsertion not detected
https://bugs.freenas.org/issues/21608 cannot bring back cache device online