[FreeNAS] multipaths, geli: Cannot open gptid after adding another external Storage

[JSunny]

Dear Community,

i need your help :)

I just FreeNAS for more than 6 month and everything went fine.

No I did 2 changes and ran into a bigger problem:

1. Change: I removed my SSD cache (because it didnt really increase the speed) and added it as ZFS Volume
2. Change: I added another external storage (identical to the one i already used) in order to increase the storage.

System Setup:

FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
i3-4010U CPU @ 1.70GHz
8 GB RAM
30GB SSD (ada0)
8TB external RAID Station (now two of them) via USB3.0 (da1, da2)

I had to set hw.usb.no_shutdown_wait to 1, due to the trouble of external storage but that shouldnt effect us here.

So, now after a reboot I retrieve the following error, if I want to attach the first encrypted volume, i justed the last 6 month:

CRITICAL: The following multipaths are not optimal: disk1

Mar 9 18:36:44 storage manage.py: [middleware.notifier:1314] Failed to geli attach gptid/661f6b6e-50b0-[...]: geli: Cannot open gptid/661f6b6e-50b0-[...]: No such file or directory.
Mar 9 18:36:45 storage manage.py: [middleware.notifier:3428] Importing main [17451066640[...]] failed with: cannot import '174510666409[...]': no such pool available
Mar 9 18:36:45 storage manage.py: [middleware.exceptions:38] [MiddlewareError: Volume could not be imported: 1 devices failed to decrypt]

In parallel I get a new option within the storage tab "view multipaths" with (da0)

My Question: what happened after I plugged in the additional storage ?
How can I handle this problem?

Maybe freenas detected those two external storage as multipath falsely . Is there a way to "remove" this multipath and let freenas interprets both connections individual?

FreeNAS® uses gmultipath(8) to provide multipath I/O support on systems containing hardware that is capable of multipath. An example would be a dual SAS expander backplane in the chassis or an external JBOD.

Multipath hardware adds fault tolerance to a NAS as the data is still available even if one disk I/O path has a failure.

FreeNAS® automatically detects active/active and active/passive multipath-capable hardware. Any multipath-capable devices that are detected will be placed in multipath units with the parent devices hidden. The configuration will be displayed in Storage ‣ Volumes ‣ View Multipaths. Note that this option will not be displayed in the Storage ‣ Volumes tree on systems that do not contain multipath-capable hardware.

Thank you and best regards
JSunny

 

[Robert Trevellyan]

Unfortunately your hardware is not suitable for FreeNAS. You should probably find a way to backup all your data and start over, or look for another solution. If you want to understand why, start reading the documentation and the numerous pinned posts in the forums.

The key points are that you're not giving ZFS direct access to the hard drives, and you're using USB3.

 

[JSunny]

thank you Robert Trevellyan for your response. I appreciate it!

May I say, that it looks in my eyes more a trouble with multipath than with wrong hardware. It worked already, so there seem to be no incompatibilities. Freenas "just" detects an 8TB drive so far and uses it in stripe.

But maybe you can tell me how to disable the automatic multipath detection?
And how to reset / delete the wrong detected multipath?

Thank you so much!

 

[Robert Trevellyan]

Freenas "just" detects an 8TB drive so far and uses it in stripe.

And herein lies the first serious issue with your hardware. ZFS can't see the disks, so it can't make best use of them and it can't tell you if they're failing.

But maybe you can tell me how to disable the automatic multipath detection?

I don't know how to do that. I could try to figure it out, but it would be a waste of time for both of us. It's possible someone who knows has already looked at your first post and decided not to bother responding because your hardware is so unsuitable, or because you didn't follow forum rules, which upsets the long-time members.

If you're committed to using your RAID Stations for storage (nothing inherently wrong with that if they meet your needs), I suggest you install a different OS and connect them to that. It just doesn't make sense to use them with FreeNAS.

As for how you get from here to a solution, the first thing I would try is to shut down the box and disconnect the 2nd RAID Station, then start up and find out if you can still see your data so you can transfer it somewhere else.

 

[SweetAndLow]

Ha! Encryption, raid, strips and usb3 all in the same thread. You have pretty much created the perfect storm for yourself. Now if you have 4gb of memory and realtek nic you probably just made a legendary thread. You need to get things mounted and remove your data asap or you are going to lose all your data.

 

[mav@]

Probably explanation to your situation is that two of your USB storages report equal SCSI LUN IDs. It is quite typical for cheap USB devices, but seems not only cheap. There is no way to selectively disable multipath for some disks in FreeNAS. You could delete ix-multipath scriptx from your system and manually destroy multipath devices, but be ready they reappear after the next FreeNAS update.

Out of curiosity it would be interesting to see output of `sg_vpd -p di /dev/da1` and `sg_vpd -p di /dev/da2` commands.

 

[JSunny]

thanks again for all your responses!

Out of curiosity it would be interesting to see output of `sg_vpd -p di /dev/da1` and `sg_vpd -p di /dev/da2` commands.

I tried but sg_vpd seems to be not installed (Command not found.). Do I have to download and compile it (http://sg.danny.cz/sg/sg3_utils.html)? Or is there something like a hidden package manager ;) ?

Probably explanation to your situation is that two of your USB storages report equal SCSI LUN IDs.

yes, I think so too!

There is no way to selectively disable multipath for some disks in FreeNAS.

Even not for all disks?

You could delete ix-multipath scriptx from your system and manually destroy multipath devices, but be ready they reappear after the next FreeNAS update.

That is exactly what i am looking for! Could you tell me how to manually destroy multipath devices? :)

So I went ahead and tried to export the affected volume, but it seems that the multipath settings are still in place. at least I cant import it again.

Code:

Mar 10 18:24:38 storage kernel: usb_alloc_device: Failure selecting configuration index 0:USB_ERR_STALLED, port 3, addr 3 (ignored)
Mar 10 18:24:38 storage kernel: ugen0.3: <JMicron> at usbus0
Mar 10 18:24:38 storage kernel: ugen0.3: <JMicron> at usbus0 (disconnected)
Mar 10 18:24:43 storage kernel: ugen0.3: <JMicron> at usbus0
Mar 10 18:24:43 storage kernel: umass0: <MSC Bulk-Only Transport> on usbus0
Mar 10 18:24:43 storage kernel: umass0:  SCSI over Bulk-Only; quirks = 0x4100
Mar 10 18:24:43 storage kernel: umass0:2:0:-1: Attached to scbus2
Mar 10 18:24:43 storage kernel: (probe0:umass-sim0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Mar 10 18:24:43 storage kernel: (probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
Mar 10 18:24:43 storage kernel: (probe0:umass-sim0:0:0:0): SCSI status: Check Condition
Mar 10 18:24:43 storage kernel: (probe0:umass-sim0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
Mar 10 18:24:43 storage kernel: (probe0:umass-sim0:0:0:0): Error 22, Unretryable error
Mar 10 18:24:43 storage kernel: da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
Mar 10 18:24:43 storage kernel: da0: <H/W RAID 5 0X08> Fixed Direct Access SCSI-6 device
Mar 10 18:24:43 storage kernel: da0: Serial Number 000000000000
Mar 10 18:24:43 storage kernel: da0: 40.000MB/s transfers
Mar 10 18:24:43 storage kernel: da0: 7630720MB (15627714560 512 byte sectors: 255H 63S/T 972780C)
Mar 10 18:24:43 storage kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 10 18:24:43 storage kernel: GEOM_MULTIPATH: disk1 created
Mar 10 18:24:43 storage kernel: GEOM_MULTIPATH: da0 added to disk1
Mar 10 18:24:43 storage kernel: GEOM_MULTIPATH: da0 is now active path in disk1
Mar 10 18:24:43 storage snmpd[3798]: disk_OS_get_disks: device 'da0' not in device list
Mar 10 18:24:43 storage kernel: GEOM: multipath/disk1: corrupt or invalid GPT detected.   <!---
Mar 10 18:24:43 storage kernel: GEOM: multipath/disk1: GPT rejected -- may not be recoverable.

As soon as I can access the volume again I ll follow your advice and move the data. I ll try out eSATA with clean RAID so FreeNAS manage the HDDs directly. Hope that is better :)

 

[mav@]

I tried but sg_vpd seems to be not installed (Command not found.). Do I have to download and compile it (http://sg.danny.cz/sg/sg3_utils.html)? Or is there something like a hidden package manager ;) ?

It is present in 9.3, but probably not in 9.2.1.8. At least you may show output of `geom disk list` and `camcontrol cmd da1 -c '12 01 83 00 ff 00' -i 255 - | hd`.

That is exactly what i am looking for! Could you tell me how to manually destroy multipath devices? :)

After deleting ix-multipath script you may destroy multipath devices with `gmultipath destroy disk1`. After that you may need to recover corrupted backup gpt partition table with `gpart recover da1`.

 

[Robert Trevellyan]

I ll try out eSATA with clean RAID so FreeNAS manage the HDDs directly. Hope that is better

If doing so gives FreeNAS 100% transparent access to all the individual hard drives in each RAID Station, including things like SMART data, then it will be better, but I think that's pretty unlikely to be the outcome. Good luck!

 

[cyberjock]

Yeah, I'm with Robert on this. I don't think deleting some scripts is going to do you any good. It may even create bigger problems because who knows what else might be touching that script for various reasons and fail when that script doesn't exist.

 

[JSunny]

Thanks again for your advices & responses.

After deleting ix-multipath script you may destroy multipath devices with `gmultipath destroy disk1`. After that you may need to recover corrupted backup gpt partition table with `gpart recover da1`.

So, I followed your advice and executed `gmultipath destroy disk1` and `gpart recover da1` without deleting ix-multipath script which worked great. My Freenas detects the disks again. I went ahead and tried to import the volume (selected the device, pw, key, volume) and after some minutes the process exit with the following error:

Code:

Mar 11 18:42:59 storage kernel: GEOM_ELI: Device gptid/661f6b6e-[...]-c03fd562ded5.eli created.
Mar 11 18:42:59 storage kernel: GEOM_ELI: Encryption: AES-XTS 128
Mar 11 18:42:59 storage kernel: GEOM_ELI:  Crypto: hardware
[...]
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 02 b7 56 de d0 00 00 00 80 00 00
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): CAM status: SCSI Status Error
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): SCSI status: Check Condition
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): SCSI sense: ABORTED COMMAND asc:0,0 (No additional sense information)
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): Retrying command (per sense data)
[...]
Mar 11 18:50:40 storage notifier: Stopping collectd.
Mar 11 18:50:44 storage notifier: Waiting for PIDS: 2956.
Mar 11 18:50:44 storage notifier: Starting collectd.
Mar 11 18:50:46 storage manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set passphrase: geli: Cannot read metadata from /dev/gptid/661f6b6e-[...]-c03fd562ded5: No such file or directory. ]

Is that related to destroying the desk1 ? another `gpart recover da1` didnt help.


----Additional informations like requested ---

At least you may show output of `geom disk list` and `camcontrol cmd da1 -c '12 01 83 00 ff 00' -i 255 - | hd`.

`geom disk list`

Code:

[root@storage] ~# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
  Mediasize: 30016659456 (28G)
  Sectorsize: 512
  Mode: r1w1e2
  descr: KINGSTON SMS200S330G
  lunid: 50026b72480c5b80
  ident: 50026B72480C5B80
  fwsectors: 63
  fwheads: 16

Geom name: da0
Providers:
1. Name: da0
  Mediasize: 8004304896 (7.5G)
  Sectorsize: 512
  Mode: r2w1e4
  descr: SanDisk Cruzer Fit
  lunname: SanDisk Cruzer Fit  4C530002070706120503
  lunid: SanDisk Cruzer Fit  4C530002070706120503
  ident: 4C530002070706120503
  fwsectors: 63
  fwheads: 255

Geom name: da1
Providers:
1. Name: da1
  Mediasize: 8001389854720 (7.3T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e3
  descr: H/W RAID 5
  lunname: H/W RAID5
  lunid: H/W RAID5
  ident: 000000000000
  fwsectors: 63
  fwheads: 255

`camcontrol cmd da1 -c '12 01 83 00 ff 00' -i 255 - | hd`

 

[mav@]

I don't see how shown error messages can be related to destroying of multipath. From the other side I have no clue what is your problem now. I guess those errors can be the problem, but I have no idea what causes them.

What is about provided device data -- they've confirmed that your USB devices are standard-violating crap, that is not surprising for USB. :(

 

[JSunny]

thank you mav@ for your input. Maybe let me clarify my question:

Code:

Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): READ(16). CDB: 88 00 00 00 00 02 b7 56 de d0 00 00 00 80 00 00
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): CAM status: SCSI Status Error
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): SCSI status: Check Condition
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): SCSI sense: ABORTED COMMAND asc:0,0 (No additional sense information)
Mar 11 18:45:57 storage kernel: (da1:umass-sim1:1:0:0): Retrying command (per sense data)
[...]
Mar 11 18:50:40 storage notifier: Stopping collectd.
Mar 11 18:50:44 storage notifier: Waiting for PIDS: 2956.
Mar 11 18:50:44 storage notifier: Starting collectd.
Mar 11 18:50:46 storage manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set passphrase: geli: Cannot read metadata from /dev/gptid/661f6b6e-[...]-c03fd562ded5: No such file or directory. ]

looks like FreeNAS has trouble reading the volume content. It - however - can detect the volume. What causes could be, that FreeNAS is not able to read the metadata?

• Is it possible that FreeNAS removed metadata during the export of the volume? Is there a way to recover / repair these metadata?
• Is it possible that `gmultipath remove` corrupted the metadata?

The multipath device can be configured using two different methods: ``manual'' or ``automatic''. [..] The ``automatic'' method uses on-disk meta-data to detect device and all it's paths. Metadata use the last sector of the underlying disk device and include device name and UUID. [..]

Update: So I did some digging and it seems to be possible to access the data of the volume via shell after auto-import on /mnt/main like it is supposed to be. But the volume is not listed in the UI. How does that work out?

Code:

[root@storage] ~# gpart show da0
=>  34  15627714493  da0  GPT  (7.3T)
  34  94  - free -  (47k)
  128  4194304  1  freebsd-swap  (2.0G)
  4194432  15623520088  2  freebsd-zfs  (7.3T)
  15627714520  7  - free -  (3.5k)

[root@storage] ~# gpart status da0
Name  Status  Components
da0p1  OK  da0
da0p2  OK  da0
[root@storage] ~# zpool status
  pool: main
state: ONLINE
[..]
  NAME  STATE  READ WRITE CKSUM
  main  ONLINE  0  0  0
  gptid/661f6b6e-[..]-c03fd562ded5.eli  ONLINE  0  0  0

[root@storage] ~# zpool list
NAME  SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
main  7.25T  5.98T  1.27T  82%  1.00x  ONLINE  /mnt

Thanks again, i really appreciate your help!