wtfuar
Dabbler
- Joined
- Jun 25, 2013
- Messages
- 36
Following all done by GUI - FreeBSD 8.3-RELEASE-p7 #1 r249203M: Sat Apr 6 09:28:27 PDT 2013
A drive had errors and after scrubbing - with no dataloss and successfull repair - I decided to replace the drive in question.
I followed the manual: 6.3.11
- zfs RaidZ1 was mounted using the recovery-key and in unknown state
- [Offline] the drive - state changed to degraded
- hot swaped it with the new one (same type and place)
- [Replace] button hit adn resilvering toke 12h and was without errors finished - state degraded
- [Detache] button hit on old drive - status healthy
Now as the drive has changed I needed to recreate the keys but:
I wont switch off the box until this problem is solved as I think I would not able to mount my zfspool again.
I read the thread cyberjock started and I'd like to here a more official work around or better a solution as the manual take this problem not in account.
It seems the [Detache] button does NOT remove the old drive from the config.
It seems there have to be an Information when starting the procedure of replacing drives on an encrypted zfs pool how to proceed - better in manual.
When replacing drives on an encrypted zfs pool it is necessary to add further more steps that require
- change of passphrase
- download new key
- encryption re-key
Following actual pool status:
Following the complete syslog:
Edit:
If I take a look into the freenas-v1.db:
And deleted ID 3 of the storage_encrypteddisk and could create new key and recovery key.
I did not changed the passphrase as I think it is not needed.
How can I test if this solves the problem without dismounting the zfs pool and try to mount it again? Could some official please confirm or describe how to validate this solution?
A drive had errors and after scrubbing - with no dataloss and successfull repair - I decided to replace the drive in question.
I followed the manual: 6.3.11
- zfs RaidZ1 was mounted using the recovery-key and in unknown state
- [Offline] the drive - state changed to degraded
- hot swaped it with the new one (same type and place)
- [Replace] button hit adn resilvering toke 12h and was without errors finished - state degraded
- [Detache] button hit on old drive - status healthy
Now as the drive has changed I needed to recreate the keys but:
Code:
Aug 24 08:44:37 freenas notifier: 1+0 records in Aug 24 08:44:37 freenas notifier: 1+0 records out Aug 24 08:44:37 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec) Aug 24 08:44:44 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ] Aug 24 08:45:07 freenas notifier: 1+0 records in Aug 24 08:45:07 freenas notifier: 1+0 records out Aug 24 08:45:07 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec) Aug 24 08:45:07 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ] Aug 24 08:47:18 freenas notifier: 1+0 records in Aug 24 08:47:18 freenas notifier: 1+0 records out Aug 24 08:47:18 freenas notifier: 64 bytes transferred in 0.000025 secs (2556528 bytes/sec) Aug 24 08:47:18 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]
I wont switch off the box until this problem is solved as I think I would not able to mount my zfspool again.
I read the thread cyberjock started and I'd like to here a more official work around or better a solution as the manual take this problem not in account.
It seems the [Detache] button does NOT remove the old drive from the config.
It seems there have to be an Information when starting the procedure of replacing drives on an encrypted zfs pool how to proceed - better in manual.
When replacing drives on an encrypted zfs pool it is necessary to add further more steps that require
- change of passphrase
- download new key
- encryption re-key
Following actual pool status:
Code:
[root@freenas] ~# zpool status pool: 000 state: ONLINE scan: resilvered 1.25T in 12h32m with 0 errors on Sat Aug 24 07:37:39 2013 config: NAME STATE READ WRITE CKSUM zfs1 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/27c029db-0c16-11e3-b6f5-002215ab012f.eli ONLINE 0 0 0 gptid/622428ec-9d54-11e2-9312-000c29cec408.eli ONLINE 0 0 0 gptid/634a7ede-9d54-11e2-9312-000c29cec408.eli ONLINE 0 0 0 gptid/64712091-9d54-11e2-9312-000c29cec408.eli ONLINE 0 0 0 gptid/65977639-9d54-11e2-9312-000c29cec408.eli ONLINE 0 0 0 gptid/66bc5d54-9d54-11e2-9312-000c29cec408.eli ONLINE 0 0 0 errors: No known data errors
Following the complete syslog:
Code:
Aug 23 11:06:22 freenas newsyslog[1402]: logfile first created Aug 23 11:06:22 freenas syslogd: kernel boot file is /boot/kernel/kernel Aug 23 11:06:22 freenas kernel: Copyright (c) 1992-2012 The FreeBSD Project. Aug 23 11:06:22 freenas kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Aug 23 11:06:22 freenas kernel: The Regents of the University of California. All rights reserved. Aug 23 11:06:22 freenas kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Aug 23 11:06:22 freenas kernel: FreeBSD 8.3-RELEASE-p7 #1 r249203M: Sat Apr 6 09:28:27 PDT 2013 Aug 23 11:06:22 freenas kernel: root@build.ixsystems.com:/tank/home/jpaetzel/fn8.3/freenas/os-base/amd64/tank/home/jpaetzel/fn8.3/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64 Aug 23 11:06:22 freenas kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Aug 23 11:06:22 freenas kernel: CPU: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz (3292.54-MHz K8-class CPU) Aug 23 11:06:22 freenas kernel: Origin = "GenuineIntel" Id = 0x00000 Family = 6 Model = 2a Stepping = 7 Aug 23 11:06:22 freenas kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Aug 23 11:06:22 freenas kernel: Features2=0x179ae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,TSCDLT,AESNI,XSAVE,AVX> Aug 23 11:06:22 freenas kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> Aug 23 11:06:22 freenas kernel: AMD Features2=0x1<LAHF> Aug 23 11:06:22 freenas kernel: TSC: P-state invariant Aug 23 11:06:22 freenas kernel: real memory = 17179869184 (16384 MB) Aug 23 11:06:22 freenas kernel: avail memory = 16452546560 (15690 MB) Aug 23 11:06:22 freenas kernel: ACPI APIC Table: <INTEL DH67BL > Aug 23 11:06:22 freenas kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Aug 23 11:06:22 freenas kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) Aug 23 11:06:22 freenas kernel: cpu0 (BSP): APIC ID: 0 Aug 23 11:06:22 freenas kernel: cpu1 (AP): APIC ID: 2 Aug 23 11:06:22 freenas kernel: cpu2 (AP): APIC ID: 4 Aug 23 11:06:22 freenas kernel: cpu3 (AP): APIC ID: 6 Aug 23 11:06:22 freenas kernel: WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. Aug 23 11:06:22 freenas kernel: ioapic0 <Version 2.0> irqs 0-23 on motherboard Aug 23 11:06:22 freenas kernel: kbd1 at kbdmux0 Aug 23 11:06:22 freenas kernel: hpt27xx: RocketRAID 27xx controller driver v1.0 (Apr 6 2013 09:28:15) Aug 23 11:06:22 freenas kernel: cryptosoft0: <software crypto> on motherboard Aug 23 11:06:22 freenas kernel: aesni0: <AES-CBC,AES-XTS> on motherboard Aug 23 11:06:22 freenas kernel: acpi0: <INTEL DH67BL> on motherboard Aug 23 11:06:22 freenas kernel: acpi0: [ITHREAD] Aug 23 11:06:22 freenas kernel: acpi0: Power Button (fixed) Aug 23 11:06:22 freenas kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 Aug 23 11:06:22 freenas kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 Aug 23 11:06:22 freenas kernel: cpu0: <ACPI CPU> on acpi0 Aug 23 11:06:22 freenas kernel: cpu1: <ACPI CPU> on acpi0 Aug 23 11:06:22 freenas kernel: cpu2: <ACPI CPU> on acpi0 Aug 23 11:06:22 freenas kernel: cpu3: <ACPI CPU> on acpi0 Aug 23 11:06:22 freenas kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 Aug 23 11:06:22 freenas kernel: pci0: <ACPI PCI bus> on pcib0 Aug 23 11:06:22 freenas kernel: pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 Aug 23 11:06:22 freenas kernel: pci1: <ACPI PCI bus> on pcib1 Aug 23 11:06:22 freenas kernel: mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfe5c0000-0xfe5c3fff,0xfe580000-0xfe5bffff irq 16 at device 0.0 on pci1 Aug 23 11:06:22 freenas kernel: mps0: Firmware: 16.00.00.00, Driver: 13.00.00.00-fbsd Aug 23 11:06:22 freenas kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> Aug 23 11:06:22 freenas kernel: mps0: [ITHREAD] Aug 23 11:06:22 freenas kernel: vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 Aug 23 11:06:22 freenas kernel: pci0: <simple comms> at device 22.0 (no driver attached) Aug 23 11:06:22 freenas kernel: em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x0000-0x0000 mem 0xfe600000-0x0000000,0x00000000-0x00000000 irq 20 at device 25.0 on pci0 Aug 23 11:06:22 freenas kernel: em0: Using an MSI interrupt Aug 23 11:06:22 freenas kernel: em0: [FILTER] Aug 23 11:06:22 freenas kernel: em0: Ethernet address: 00:00:00:00:00:00 Aug 23 11:06:22 freenas kernel: ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe623000-0xfe6233ff irq 16 at device 26.0 on pci0 Aug 23 11:06:22 freenas kernel: ehci0: [ITHREAD] Aug 23 11:06:22 freenas kernel: usbus0: EHCI version 1.0 Aug 23 11:06:22 freenas kernel: usbus0: <EHCI (generic) USB 2.0 controller> on ehci0 Aug 23 11:06:22 freenas kernel: pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0 Aug 23 11:06:22 freenas kernel: pci2: <ACPI PCI bus> on pcib2 Aug 23 11:06:22 freenas kernel: pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2 Aug 23 11:06:22 freenas kernel: pci3: <ACPI PCI bus> on pcib3 Aug 23 11:06:22 freenas kernel: pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 Aug 23 11:06:22 freenas kernel: pci4: <ACPI PCI bus> on pcib4 Aug 23 11:06:22 freenas kernel: pci4: <serial bus, USB> at device 0.0 (no driver attached) Aug 23 11:06:22 freenas kernel: ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe622000-0xfe6223ff irq 23 at device 29.0 on pci0 Aug 23 11:06:22 freenas kernel: ehci1: [ITHREAD] Aug 23 11:06:22 freenas kernel: usbus1: EHCI version 1.0 Aug 23 11:06:22 freenas kernel: usbus1: <EHCI (generic) USB 2.0 controller> on ehci1 Aug 23 11:06:22 freenas kernel: isab0: <PCI-ISA bridge> at device 31.0 on pci0 Aug 23 11:06:22 freenas kernel: isa0: <ISA bus> on isab0 Aug 23 11:06:22 freenas kernel: ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf060-0xf07f mem 0xfe621000-0xfe6217ff irq 19 at device 31.2 on pci0 Aug 23 11:06:22 freenas kernel: ahci0: [ITHREAD] Aug 23 11:06:22 freenas kernel: ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported Aug 23 11:06:22 freenas kernel: ahcich0: <AHCI channel> at channel 0 on ahci0 Aug 23 11:06:22 freenas kernel: ahcich0: [ITHREAD] Aug 23 11:06:22 freenas kernel: ahcich1: <AHCI channel> at channel 1 on ahci0 Aug 23 11:06:22 freenas kernel: ahcich1: [ITHREAD] Aug 23 11:06:22 freenas kernel: ahcich2: <AHCI channel> at channel 2 on ahci0 Aug 23 11:06:22 freenas kernel: ahcich2: [ITHREAD] Aug 23 11:06:22 freenas kernel: ahcich3: <AHCI channel> at channel 3 on ahci0 Aug 23 11:06:22 freenas kernel: ahcich3: [ITHREAD] Aug 23 11:06:22 freenas kernel: pci0: <serial bus, SMBus> at device 31.3 (no driver attached) Aug 23 11:06:22 freenas kernel: acpi_button0: <Power Button> on acpi0 Aug 23 11:06:22 freenas kernel: acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Aug 23 11:06:22 freenas kernel: Timecounter "HPET" frequency 14318180 Hz quality 900 Aug 23 11:06:22 freenas kernel: atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 Aug 23 11:06:22 freenas kernel: orm0: <ISA Option ROM> at iomem 0xce800-0xcefff on isa0 Aug 23 11:06:22 freenas kernel: sc0: <System console> at flags 0x100 on isa0 Aug 23 11:06:22 freenas kernel: sc0: VGA <16 virtual consoles, flags=0x300> Aug 23 11:06:22 freenas kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Aug 23 11:06:22 freenas kernel: atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 Aug 23 11:06:22 freenas kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0 Aug 23 11:06:22 freenas kernel: kbd0 at atkbd0 Aug 23 11:06:22 freenas kernel: atkbd0: [GIANT-LOCKED] Aug 23 11:06:22 freenas kernel: atkbd0: [ITHREAD] Aug 23 11:06:22 freenas kernel: ppc0: cannot reserve I/O port range Aug 23 11:06:22 freenas kernel: coretemp0: <CPU On-Die Thermal Sensors> on cpu0 Aug 23 11:06:22 freenas kernel: est0: <Enhanced SpeedStep Frequency Control> on cpu0 Aug 23 11:06:22 freenas kernel: p4tcc0: <CPU Frequency Thermal Control> on cpu0 Aug 23 11:06:22 freenas kernel: coretemp1: <CPU On-Die Thermal Sensors> on cpu1 Aug 23 11:06:22 freenas kernel: est1: <Enhanced SpeedStep Frequency Control> on cpu1 Aug 23 11:06:22 freenas kernel: p4tcc1: <CPU Frequency Thermal Control> on cpu1 Aug 23 11:06:22 freenas kernel: coretemp2: <CPU On-Die Thermal Sensors> on cpu2 Aug 23 11:06:22 freenas kernel: est2: <Enhanced SpeedStep Frequency Control> on cpu2 Aug 23 11:06:22 freenas kernel: p4tcc2: <CPU Frequency Thermal Control> on cpu2 Aug 23 11:06:22 freenas kernel: coretemp3: <CPU On-Die Thermal Sensors> on cpu3 Aug 23 11:06:22 freenas kernel: est3: <Enhanced SpeedStep Frequency Control> on cpu3 Aug 23 11:06:22 freenas kernel: p4tcc3: <CPU Frequency Thermal Control> on cpu3 Aug 23 11:06:22 freenas kernel: fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8 Aug 23 11:06:22 freenas kernel: Timecounters tick every 1.000 msec Aug 23 11:06:22 freenas kernel: hpt27xx: no controller detected. Aug 23 11:06:22 freenas kernel: GEOM_RAID5: Module loaded, version 1.1.20110927.40 (rev 00ce00e5abb4) Aug 23 11:06:22 freenas kernel: usbus0: 480Mbps High Speed USB v2.0 Aug 23 11:06:22 freenas kernel: usbus1: 480Mbps High Speed USB v2.0 Aug 23 11:06:22 freenas kernel: ugen0.1: <Intel> at usbus0 Aug 23 11:06:22 freenas kernel: uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 Aug 23 11:06:22 freenas kernel: ugen1.1: <Intel> at usbus1 Aug 23 11:06:22 freenas kernel: uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 Aug 23 11:06:22 freenas kernel: uhub0: 2 ports with 2 removable, self powered Aug 23 11:06:22 freenas kernel: uhub1: 2 ports with 2 removable, self powered Aug 23 11:06:22 freenas kernel: ugen1.2: <vendor 0x8087> at usbus1 Aug 23 11:06:22 freenas kernel: uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1 Aug 23 11:06:22 freenas kernel: ugen0.2: <vendor 0x8087> at usbus0 Aug 23 11:06:22 freenas kernel: uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0 Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): CAM status: SCSI Status Error Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): SCSI status: Check Condition Aug 23 11:06:22 freenas kernel: (probe14:mps0:0:14:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) Aug 23 11:06:22 freenas kernel: uhub3: 6 ports with 6 removable, self powered Aug 23 11:06:22 freenas kernel: uhub2: 8 ports with 8 removable, self powered Aug 23 11:06:22 freenas kernel: ugen0.3: <vendor 0x13fe> at usbus0 Aug 23 11:06:22 freenas kernel: umass0: <vendor 0x13fe product 0x3600, class 0/0, rev 2.00/1.00, addr 3> on usbus0 Aug 23 11:06:22 freenas kernel: da0 at mps0 bus 0 scbus0 target 21 lun 0 Aug 23 11:06:22 freenas kernel: da0: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 11:06:22 freenas kernel: da0: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: da0: Command Queueing enabled Aug 23 11:06:22 freenas kernel: da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 11:06:22 freenas kernel: da1 at mps0 bus 0 scbus0 target 22 lun 0 Aug 23 11:06:22 freenas kernel: da1: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 11:06:22 freenas kernel: da1: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: da1: Command Queueing enabled Aug 23 11:06:22 freenas kernel: da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 11:06:22 freenas kernel: da2 at mps0 bus 0 scbus0 target 23 lun 0 Aug 23 11:06:22 freenas kernel: da2: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 11:06:22 freenas kernel: da2: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: da2: Command Queueing enabled Aug 23 11:06:22 freenas kernel: da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 11:06:22 freenas kernel: da3 at mps0 bus 0 scbus0 target 24 lun 0 Aug 23 11:06:22 freenas kernel: da3: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 11:06:22 freenas kernel: da3: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: da3: Command Queueing enabled Aug 23 11:06:22 freenas kernel: da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 11:06:22 freenas kernel: da4 at mps0 bus 0 scbus0 target 25 lun 0 Aug 23 11:06:22 freenas kernel: da4: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 11:06:22 freenas kernel: da4: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: da4: Command Queueing enabled Aug 23 11:06:22 freenas kernel: da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 11:06:22 freenas kernel: da5 at mps0 bus 0 scbus0 target 26 lun 0 Aug 23 11:06:22 freenas kernel: da5: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 11:06:22 freenas kernel: da5: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: da5: Command Queueing enabled Aug 23 11:06:22 freenas kernel: da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 11:06:22 freenas kernel: ses0 at mps0 bus 0 scbus0 target 14 lun 0 Aug 23 11:06:22 freenas kernel: ses0: <ESG-SHV. SCA HSBP M1..... 2.14> Fixed Enclosure Services SCSI-3 device da6 at umass-sim0 bus 0 scbus5 target 0 lun 0 Aug 23 11:06:22 freenas kernel: da6: < DL03> Removable Direct Access SCSI-0 device Aug 23 11:06:22 freenas kernel: da6: 40.000MB/s transfers Aug 23 11:06:22 freenas kernel: da6: 3825MB (7833600 512 byte sectors: 255H 63S/T 487C) Aug 23 11:06:22 freenas kernel: Aug 23 11:06:22 freenas kernel: ses0: 300.000MB/s transfers Aug 23 11:06:22 freenas kernel: ses0: SCSI-3 SES Device Aug 23 11:06:22 freenas kernel: SMP: AP CPU #1 Launched! Aug 23 11:06:22 freenas kernel: SMP: AP CPU #2 Launched! Aug 23 11:06:22 freenas kernel: SMP: AP CPU #3 Launched! Aug 23 11:06:22 freenas kernel: GEOM: da6s1: geometry does not match label (16h,63s != 255h,63s). Aug 23 11:06:22 freenas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a Aug 23 11:06:22 freenas kernel: ZFS filesystem version 5 Aug 23 11:06:22 freenas kernel: ZFS storage pool version 28 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da0p1.eli created. Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da1p1.eli created. Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da2p1.eli created. Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da3p1.eli created. Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da4p1.eli created. Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:06:22 freenas kernel: GEOM_ELI: Device da5p1.eli created. Aug 23 11:06:22 freenas kernel: GEOM_ELI: Encryption: AES-XTS 256 Aug 23 11:06:22 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:06:22 freenas root: /etc/rc: WARNING: Dump device does not exist. Savecore not run. Aug 23 11:06:22 freenas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd Aug 23 11:06:23 freenas ntpd[1713]: ntpd 4.2.4p5-a (1) Aug 23 11:06:32 freenas avahi-daemon[2117]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Aug 23 11:06:31 freenas ntpd[1714]: time reset -0.918430 s Aug 23 11:07:01 freenas autosnap.py: [tools.autosnap:56] Popen()ing: /sbin/zfs snapshot 000/000@auto-20130823.1107-1w Aug 23 11:07:01 freenas autosnap.py: [tools.autosnap:247] Failed to create snapshot '000/000@auto-20130823.1107-1w': cannot open '000/000': dataset does not exist Aug 23 11:07:17 freenas manage.py: [storage.forms:1794] <InMemoryUploadedFile: geli_recovery.key (application/octet-stream)> Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/65977639-9d54-11e2-9312-000c29cec408.eli created. Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 11:07:17 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/66bc5d54-9d54-11e2-9312-000c29cec408.eli created. Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 11:07:17 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/6125e604-9d54-11e2-9312-000c29cec408.eli created. Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 11:07:17 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/622428ec-9d54-11e2-9312-000c29cec408.eli created. Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 11:07:17 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:07:17 freenas kernel: GEOM_ELI: Device gptid/634a7ede-9d54-11e2-9312-000c29cec408.eli created. Aug 23 11:07:17 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 11:07:17 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:07:18 freenas kernel: GEOM_ELI: Device gptid/64712091-9d54-11e2-9312-000c29cec408.eli created. Aug 23 11:07:18 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 11:07:18 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 11:07:40 freenas notifier: Stopping dbus. Aug 23 11:07:40 freenas avahi-daemon[2117]: Disconnected from D-Bus, exiting. Aug 23 11:07:40 freenas notifier: dbus not running? (check /var/run/dbus/dbus.pid). Aug 23 11:07:40 freenas notifier: Starting dbus. Aug 23 11:07:40 freenas notifier: Stopping avahi-daemon. Aug 23 11:07:40 freenas notifier: Failed to kill daemon: No such file or directory Aug 23 11:07:40 freenas notifier: Stopping avahi-daemon. Aug 23 11:07:40 freenas notifier: Failed to kill daemon: No such file or directory Aug 23 11:07:40 freenas notifier: Starting avahi-daemon. Aug 23 11:07:40 freenas avahi-daemon[3029]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Aug 23 11:07:40 freenas notifier: winbindd not running? (check /var/run/samba/winbindd.pid). Aug 23 11:07:40 freenas notifier: Stopping smbd. Aug 23 11:07:40 freenas notifier: Waiting for PIDS: 1976. Aug 23 11:07:40 freenas notifier: Stopping nmbd. Aug 23 11:07:40 freenas notifier: Waiting for PIDS: 1973. Aug 23 11:07:40 freenas notifier: Performing sanity check on Samba configuration: OK Aug 23 11:07:40 freenas notifier: smbd not running? (check /var/run/samba/smbd.pid). Aug 23 11:07:40 freenas notifier: nmbd not running? (check /var/run/samba/nmbd.pid). Aug 23 11:07:40 freenas notifier: Removing stale Samba tdb files: ....... done Aug 23 11:07:40 freenas notifier: Starting nmbd. Aug 23 11:07:40 freenas notifier: Starting smbd. Aug 23 12:54:53 freenas kernel: em0: link state changed to DOWN Aug 23 12:56:07 freenas kernel: em0: link state changed to UP Aug 23 18:53:08 freenas kernel: GEOM_ELI: Detached da0p1.eli on last close. Aug 23 18:53:08 freenas notifier: geli: No such device: /dev/da0p1. Aug 23 18:53:57 freenas kernel: mps0: mpssas_alloc_tm freezing simq Aug 23 18:53:58 freenas kernel: mps0: mpssas_remove_complete on handle 0x000a, IOCStatus= 0x0 Aug 23 18:53:58 freenas kernel: mps0: mpssas_free_tm releasing simq Aug 23 18:53:58 freenas kernel: (da0:mps0:0:21:0): lost device - 0 outstanding, 1 refs Aug 23 18:53:58 freenas kernel: GEOM_ELI: Device gptid/6125e604-9d54-11e2-9312-000c29cec408.eli destroyed. Aug 23 18:53:58 freenas kernel: (da0:mps0:0:21:0): removing device entry Aug 23 19:03:19 freenas kernel: da0 at mps0 bus 0 scbus0 target 27 lun 0 Aug 23 19:03:19 freenas kernel: da0: <ATA WDC WD20EARX-00P AB51> Fixed Direct Access SCSI-6 device Aug 23 19:03:19 freenas kernel: da0: 150.000MB/s transfers Aug 23 19:03:19 freenas kernel: da0: Command Queueing enabled Aug 23 19:03:19 freenas kernel: da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) Aug 23 19:05:04 freenas notifier: 1+0 records in Aug 23 19:05:04 freenas notifier: 1+0 records out Aug 23 19:05:04 freenas notifier: 1048576 bytes transferred in 0.009222 secs (113703374 bytes/sec) Aug 23 19:05:04 freenas notifier: dd: /dev/da0: short write on character device Aug 23 19:05:04 freenas notifier: dd: /dev/da0: end of device Aug 23 19:05:04 freenas notifier: 5+0 records in Aug 23 19:05:04 freenas notifier: 4+1 records out Aug 23 19:05:04 freenas notifier: 4284416 bytes transferred in 0.080944 secs (52930579 bytes/sec) Aug 23 19:05:16 freenas kernel: GEOM_ELI: Device gptid/27c029db-0c16-11e3-b6f5-002215ab012f.eli created. Aug 23 19:05:16 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 19:05:16 freenas kernel: GEOM_ELI: Crypto: hardware Aug 23 19:05:21 freenas kernel: GEOM_ELI: Device da0p1.eli created. Aug 23 19:05:21 freenas kernel: GEOM_ELI: Encryption: AES-XTS 128 Aug 23 19:05:21 freenas kernel: GEOM_ELI: Crypto: hardware Aug 24 08:44:37 freenas notifier: 1+0 records in Aug 24 08:44:37 freenas notifier: 1+0 records out Aug 24 08:44:37 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec) Aug 24 08:44:44 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ] Aug 24 08:45:07 freenas notifier: 1+0 records in Aug 24 08:45:07 freenas notifier: 1+0 records out Aug 24 08:45:07 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec) Aug 24 08:45:07 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ] Aug 24 08:47:18 freenas notifier: 1+0 records in Aug 24 08:47:18 freenas notifier: 1+0 records out Aug 24 08:47:18 freenas notifier: 64 bytes transferred in 0.000025 secs (2556528 bytes/sec) Aug 24 08:47:18 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to set recovery key: geli: Cannot open gptid/6125e604-9d54-11e2-9312-000c29cec408: No such file or directory. ]
Edit:
If I take a look into the freenas-v1.db:
Code:
[root@freenas] /data# sqlite3 /data/freenas-v1.db "select * from storage_disk;" 1|Maximum|30|WD-WCAZAF9XXXXX||{serial}WD-WCAZAF9XXXXX|1|128|Auto||||1|da0 1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||2|da1 1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||3|da2 1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||4|da3 1|Maximum|30|WD-WCAZAK9XXXXX||{serial}WD-WCAZAK9XXXXX|1|128|Auto||||5|da4 1|Maximum|30|WD-WCAZAK8XXXXX||{serial}WD-WCAZAK8XXXXX|1|128|Auto||||6|da5 sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;" 1|1|5|gptid/65977639-9d54-11e2-9312-000c29cec408 2|1|6|gptid/66bc5d54-9d54-11e2-9312-000c29cec408 3|1|1|gptid/6125e604-9d54-11e2-9312-000c29cec408 4|1|2|gptid/622428ec-9d54-11e2-9312-000c29cec408 5|1|3|gptid/634a7ede-9d54-11e2-9312-000c29cec408 6|1|4|gptid/64712091-9d54-11e2-9312-000c29cec408 7|1|1|gptid/27c029db-0c16-11e3-b6f5-002215ab012f sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where id=3;" sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;" 1|1|5|gptid/65977639-9d54-11e2-9312-000c29cec408 2|1|6|gptid/66bc5d54-9d54-11e2-9312-000c29cec408 4|1|2|gptid/622428ec-9d54-11e2-9312-000c29cec408 5|1|3|gptid/634a7ede-9d54-11e2-9312-000c29cec408 6|1|4|gptid/64712091-9d54-11e2-9312-000c29cec408 7|1|1|gptid/27c029db-0c16-11e3-b6f5-002215ab012f
And deleted ID 3 of the storage_encrypteddisk and could create new key and recovery key.
Code:
Aug 24 11:38:53 freenas notifier: 1+0 records in Aug 24 11:38:53 freenas notifier: 1+0 records out Aug 24 11:38:53 freenas notifier: 64 bytes transferred in 0.000024 secs (2657777 bytes/sec) Aug 24 11:39:33 freenas notifier: 1+0 records in Aug 24 11:39:33 freenas notifier: 1+0 records out Aug 24 11:39:33 freenas notifier: 64 bytes transferred in 0.000023 secs (2767376 bytes/sec)
I did not changed the passphrase as I think it is not needed.
How can I test if this solves the problem without dismounting the zfs pool and try to mount it again? Could some official please confirm or describe how to validate this solution?
Last edited: