Multiple drives showing up REMOVED during resilvering

[Alan W. Smtih]

I've got some badness going on. I'm hoping for some guidance for how to approach.

My box is running:

- FreeNAS-9.10.2-U3 (e1497f269)
- ASRock C2550D4I motherboard
- Updated 0.30.0 BIOS firmware to prevent the watchdog bug
- 16GB Memory (2x Crucial 8GB 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1600 - PC3 12800)
- Eleven HDs in a single RAIDZ3

Last week I started getting some SMART errors and prepped disks for replacement. When I went to do the replacement of the first one today, I saw that three were showing up as "REMOVED" (including the one I was going to replace).

After physically replacing the drive and powering on the machine everything showed up as "ONLINE".

I started the resilver process of the drive I replaced. I checked it a few times during the day and everything was fine until just a little while ago. The same three slots (two with the original hard drives and one with the new drive) showing up as "REMOVED" again. The one with the new drive even still said `resilvering`.

Code:

freenas% zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Sat May 27 03:46:40 2017
config:

  NAME										  STATE	 READ WRITE CKSUM
  freenas-boot								  ONLINE	   0	 0	 0
	gptid/07bec8cd-0ab3-11e7-9ecb-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors

  pool: z
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jun 21 12:02:55 2017
		5.93T scanned out of 34.3T at 203M/s, 40h43m to go
		539G resilvered, 17.26% done
config:

  NAME											STATE	 READ WRITE CKSUM
  z											   DEGRADED	 0	 0	 0
	raidz3-0									  DEGRADED	 0	 0	 0
	  gptid/c6171db6-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c68a4186-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  8431330591788471819						 REMOVED	  0	 0	 0  was /dev/gptid/c701c120-4668-11e4-b49f-d05099264f68
	  17501151778533256183						REMOVED	  0	 0	 0  was /dev/gptid/c779c670-4668-11e4-b49f-d05099264f68
	  11668331602481738125						REMOVED	  0	 0	 0  was /dev/gptid/0d977d05-569b-11e7-8777-d05099264f68  (resilvering)
	  gptid/c8688176-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c8e2f1e9-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c9cc35d5-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/ca449ed0-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cabb7d11-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cb39f216-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors

I went to reboot the machine but got a warning that it was in the middle of resilver process that suggest I wait before rebooting. I'm guessing it wasn't actually, since the disk is showing as "REMOVED".

I was going to ask here if it was safe to reboot, but tried ended up doing it on accident while trying to get a screenshot of the warning message. Thankfully that seems not to have been a critical error.

After the reboot, all three drives are showing up as "ONLINE" again and it looks like the resilvering has picked back up:

Code:

freenas% zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Sat May 27 03:46:40 2017
config:

  NAME										  STATE	 READ WRITE CKSUM
  freenas-boot								  ONLINE	   0	 0	 0
	gptid/07bec8cd-0ab3-11e7-9ecb-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors

  pool: z
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jun 21 12:02:55 2017
		6.17T scanned out of 34.3T at 232M/s, 35h19m to go
		548G resilvered, 17.96% done
config:

  NAME											STATE	 READ WRITE CKSUM
  z											   ONLINE	   0	 0	 0
	raidz3-0									  ONLINE	   0	 0	 0
	  gptid/c6171db6-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c68a4186-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c701c120-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 2
	  gptid/c779c670-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 2
	  gptid/0d977d05-569b-11e7-8777-d05099264f68  ONLINE	   0	 0	 2  (resilvering)
	  gptid/c8688176-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c8e2f1e9-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c9cc35d5-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/ca449ed0-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cabb7d11-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cb39f216-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors

The system seems to things things are fine right now, so I'm hoping that's the case. But, I need to solve the overall issue with the drives dropping out.

I read through these other posts about "REMOVED" drives but they don't seem to be the same problem.

- https://forums.freenas.org/index.php?threads/zfs-keeps-removing-my-drive-after-resilvering.24829/
- https://forums.freenas.org/index.php?threads/mixed-messages-in-resilver.24576/

A few pieces of info:

The drives are plugged into the ASRock C2550D4I motherboard on ports:

- SATAIII_M0
- SATAIII_M1
- SATAIII_M2

According to the manual (http://www.asrockrack.com/general/productdetail.asp?Model=C2550D4I#Manual), those are all on the "Marvell 9230" controller.

I built this machine in 2014 and it's largely been running fine. I was on FreeNAS 9.3 until a few months ago when the USB crapped out and I moved to `FreeNAS-9.10.2-U3 (e1497f269)`. It's been fine for a few months and I know it's done some scrubs. So, this seems really weird.

Another piece of info is that these message showed up in `dmesg` after I put in the new drive and it was resilvering (i.e. when all three drives were showing "ONLINE" again)

Code:

ahcich2: Timeout on slot 25 port 0
ahcich2: is 00000000 cs 02000000 ss 00000000 rs 02000000 tfd 40 serr 00000000 cmd 10009817
ahcich2: Timeout on slot 12 port 0
ahcich2: is 00000000 cs 00001000 ss 00000000 rs 00001000 tfd 50 serr 00000000 cmd 10008b17
ahcich2: Timeout on slot 25 port 0
ahcich2: is 00000000 cs 02000000 ss 00000000 rs 02000000 tfd 40 serr 00000000 cmd 10009817

The three issue drives are also all using the same cord from the power supply. I'm going to replace in case it's the culprit but I've got a few questions:

Question 1 - Is the system really okay as while the drives aren't showing as "REMOVED"?

Question 2 - Is it safe to shut down the machine during a resilver?

(It seems like it's safe since I just did it, but I don't know if I just got lucky. If it is safe, I'll power down the machine and replace the power cable to see if that's the issue)

Question 3 - Any other ideas about what's going on here or things I need to watch out for?

 

[Robert Trevellyan]

1. The reboot cleared the warnings, but there's no reason to believe the underlying problem is resolved.
2. Define "safe". ZFS will do its best to deal with it, but I would avoid it if at all possible.
3. "Don't use the Marvell ports" is a pretty common refrain here. Replacing the power cable makes sense too.

 

[rs225]

3. Also consider whether the power supply isn't able to power that many drives on one rail, or perhaps total.

 

[Stux]

Could be a power issue. Seriously look into your PSU.

How many rails? Are all the drives evenly distributed to each rail?

What's this cable?

Marvell controllers are yucky :) but I believe there are two on that board. Maybe the other one is better?

 

[Alan W. Smtih]

Update: The same three disk slots (with two old disks and one new one) went "REMOVED" again sometime early this morning.

Also, the WebGUI died. I had to shutdown via the console. (I can't work out in my brain how that would be related, but I've been in tech long enough to know paths aren't always visible until someone points them out.)

---

The reboot cleared the warnings, but there's no reason to believe the underlying problem is resolved.

Understood. Though, something seems better for a time. The new disk was being resilvered. So, it was there and legit "ONLINE" for at least some period of time. There is definitely a problem but it doesn't seem to be there after a reboot and only comes back intermittently.

---

Define "safe". ZFS will do its best to deal with it, but I would avoid it if at all possible.

I should have clarified on safe. I'm transferring some data onto a new box that isn't otherwise backed up. So, the question really is: am I more likely to have a catastrophic failure and lose the data if I reboot the machine or if I leave it running until I've got the rest of the data pulled off?

(I wouldn't be surprised if there's no way to answer this, but figured it's worth asking just in case. Also, since it survived one reboot now, I'm going with that strategy unless someone here throws the big red warning flag. Not to mention, I certainly like the look of "ONLINE" better than "DEGRADED" for the pool even if it's only for short bursts and possibly overstating the stability.)

---

"Don't use the Marvell ports" is a pretty common refrain here. Replacing the power cable makes sense too.

Regarding the general advice on Marvell ports, can/do those go bad over time? (I ask because the machine has been running for about three years without issue.)

Related: Could it be that some versions of FreeNAS don't play as well with the controller?

---

3. Also consider whether the power supply isn't able to power that many drives on one rail, or perhaps total.

The power supply is a Seasonic SS-560KM Active PFC F3, 560W ATX12V V2.3/EPS 12V V2.91, 80Plus Gold Certified.

The manual is no longer online, but the Newegg page lists "+12V Rails" as "Single".

I'm pretty sure I did the math back when I setup the box to make sure that would be okay for eleven drives and it's been running fine for a few years.

Is there any way to test it independently to see if it's starting to fail?

---

Could be a power issue. Seriously look into your PSU.

Right on. Starting with the power supply is also the easiest. That's my next step.

---

How many rails? Are all the drives evenly distributed to each rail?

Just one rail.

The cables are modular and I'm using all but one of the plugs on the back of the PSU. When I switch the cable, I'll switch to the currently unused plug.

Does distribution of the connections into the PSU matter if there is only one rail?

What's this cable?

The cable is the stock cable that came with the PSU. Plugs straight into the PSU and has three SATA plugs for the three drives.

Marvell controllers are yucky :) but I believe there are two on that board. Maybe the other one is better?

- Welp. I've got 11 drives and 12 SATA ports. So, the other adapter is already in play and spoken for. :(

---

I'm off to switch out the cable.

 

[Stux]

The most likely cause of your issue is that the PSU or cable is causing a power glitch which is knocking out your three drives in one go.

 

[Johnnie Black]

There's are known issues with the Marvell ports on those boards and dropped disks, if you want to continue to use them make sure they have the latest firmware available on the Asrock support site, it can fix or at least considerably mitigate the problem.

 

[Robert Trevellyan]

Regarding the general advice on Marvell ports, can/do those go bad over time? (I ask because the machine has been running for about three years without issue.)

My expectation is not that those ports "go bad" over time, but rather that the system eventually ends up in a state where their weaknesses become an issue. In my case, I had no issues with the Marvell controller in my Rocket 640L until I tried replacing two disks attached to it at the same time (heavy read load, see link in my signature).

Related: Could it be that some versions of FreeNAS don't play as well with the controller?

Could be.

 

[Alan W. Smtih]

dmesg reference 1 (see below for details)

Code:

 
freenas% uptime
12:53PM  up  2:21, 2 users, load averages: 8.71, 7.17, 7.04
freenas% dmesg
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
  The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 r295946+21897e6695f(9.10.2-STABLE): Thu Apr 20 17:58:31 UTC 2017
	root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Atom(TM) CPU  C2550  @ 2.40GHz (2400.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 19327352832 (18432 MB)
avail memory = 16542715904 (15776 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  2
cpu2 (AP): APIC ID:  4
cpu3 (AP): APIC ID:  6
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20160527/tbfadt-733)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80e489b0, 0) error 19
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdf740000-0xdf75ffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> mem 0xdf720000-0xdf73ffff irq 20 at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xdf500000-0xdf51ffff irq 22 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 23 at device 1.0 on pci3
pci4: <PCI bus> on pcib4
ahci0: <Marvell 88SE9172 AHCI SATA controller> port 0xc040-0xc047,0xc030-0xc033,0xc020-0xc027,0xc010-0xc013,0xc000-0xc00f mem 0xdf410000-0xdf4101ff irq 23 at device 0.0 on pci4
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib5: <PCI-PCI bridge> irq 23 at device 5.0 on pci3
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> irq 23 at device 0.0 on pci5
pci6: <PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 23 at device 0.0 on pci6
vgapci0: Boot video device
pcib7: <PCI-PCI bridge> irq 21 at device 7.0 on pci3
pci7: <PCI bus> on pcib7
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xa000-0xa01f mem 0xdf300000-0xdf37ffff,0xdf380000-0xdf383fff irq 21 at device 0.0 on pci7
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: d0:50:99:26:4f:68
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib8: <PCI-PCI bridge> irq 23 at device 9.0 on pci3
pci8: <PCI bus> on pcib8
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x9000-0x901f mem 0xdf200000-0xdf27ffff,0xdf280000-0xdf283fff irq 23 at device 0.0 on pci8
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: d0:50:99:26:4f:69
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
pcib9: <ACPI PCI-PCI bridge> mem 0xdf700000-0xdf71ffff at device 4.0 on pci0
pci9: <ACPI PCI bus> on pcib9
ahci1: <Marvell 88SE9230 AHCI SATA controller> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xdf610000-0xdf6107ff irq 23 at device 0.0 on pci9
ahci1: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported
ahci1: quirks=0x900<NOBSYRES,ALTSIG>
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich7: <AHCI channel> at channel 5 on ahci1
ahcich8: <AHCI channel> at channel 6 on ahci1
ahcich9: <AHCI channel> at channel 7 on ahci1
pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached)
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf763000-0xdf7633ff irq 23 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
ahci2: <Intel Avoton AHCI SATA controller> port 0xe0d0-0xe0d7,0xe0c0-0xe0c3,0xe0b0-0xe0b7,0xe0a0-0xe0a3,0xe040-0xe05f mem 0xdf762000-0xdf7627ff irq 19 at device 23.0 on pci0
ahci2: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich10: <AHCI channel> at channel 0 on ahci2
ahcich11: <AHCI channel> at channel 1 on ahci2
ahcich12: <AHCI channel> at channel 2 on ahci2
ahcich13: <AHCI channel> at channel 3 on ahci2
ahci3: <Intel Avoton AHCI SATA controller> port 0xe090-0xe097,0xe080-0xe083,0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem 0xdf761000-0xdf7617ff irq 19 at device 24.0 on pci0
ahci3: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich14: <AHCI channel> at channel 0 on ahci3
ahcich15: <AHCI channel> at channel 1 on ahci3
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
random: unblocking device.
usbus0: 480Mbps High Speed USB v2.0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub0: 8 ports with 8 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0
uhub1: 4 ports with 4 removable, self powered
ugen0.3: <SanDisk> at usbus0
umass0: <SanDisk Cruzer Fit, class 0/0, rev 2.10/1.00, addr 3> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:17:0:-1: Attached to scbus17
ugen0.4: <SanDisk> at usbus0
umass1: <SanDisk Cruzer Fit, class 0/0, rev 2.10/1.00, addr 4> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0x8100
umass1:18:1:-1: Attached to scbus18
ugen0.5: <PI Engineering> at usbus0
uhub2: <PI Engineering Kinesis Keyboard Hub, class 9/0, rev 1.10/3.20, addr 5> on usbus0
uhub2: 4 ports with 2 removable, bus powered
ugen0.6: <vendor 0x05f3> at usbus0
ukbd0: <vendor 0x05f3 product 0x0007, class 0/0, rev 1.10/3.20, addr 6> on usbus0
kbd2 at ukbd0
ugen0.7: <American Megatrends Inc.> at usbus0
uhub3: <7-port Hub> on usbus0
uhub3: 5 ports with 5 removable, self powered
ugen0.8: <American Megatrends Inc.> at usbus0
ukbd1: <Keyboard Interface> on usbus0
kbd3 at ukbd1
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada0: Serial Number WD-WCC4E1941920
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada1: Serial Number WD-WCC4EK8ZS9NF
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada2: Serial Number WD-WCC4EJXKYRYV
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada3: Serial Number WD-WCC4ELZ7294R
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 3815447MB (7814037168 512 byte sectors)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada4: Serial Number VK1B1ZXY
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 7630885MB (15628053168 512 byte sectors)
ada4: Previously was known as ad12
ada5 at ahcich10 bus 0 scbus10 target 0 lun 0
ada5: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada5: Serial Number WD-WCC4EE3LXRDF
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 3815447MB (7814037168 512 byte sectors)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad24
ada6 at ahcich11 bus 0 scbus11 target 0 lun 0
ada6: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada6: Serial Number WD-WCC4E1956067
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 3815447MB (7814037168 512 byte sectors)
ada6: quirks=0x1<4K>
ada6: Previously was known as ad26
ada7 at ahcich12 bus 0 scbus12 target 0 lun 0
ada7: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada7: Serial Number WD-WCC4E0242645
ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada7: Command Queueing enabled
ada7: 3815447MB (7814037168 512 byte sectors)
ada7: quirks=0x1<4K>
ada7: Previously was known as ad28
ada8 at ahcich13 bus 0 scbus13 target 0 lun 0
ada8: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada8: Serial Number WD-WCC4EH72PYJY
ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada8: Command Queueing enabled
ada8: 3815447MB (7814037168 512 byte sectors)
ada8: quirks=0x1<4K>
ada8: Previously was known as ad30
ada9 at ahcich14 bus 0 scbus14 target 0 lun 0
ada9: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada9: Serial Number WD-WCC4E1946451
ada9: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada9: Command Queueing enabled
ada9: 3815447MB (7814037168 512 byte sectors)
ada9: quirks=0x1<4K>
ada9: Previously was known as ad32
ada10 at ahcich15 bus 0 scbus15 target 0 lun 0
ada10: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada10: Serial Number WD-WCC4E1936067
ada10: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada10: Command Queueing enabled
ada10: 3815447MB (7814037168 512 byte sectors)
ada10: quirks=0x1<4K>
ada10: Previously was known as ad34
da0 at umass-sim0 bus 0 scbus17 target 0 lun 0
da0: <SanDisk Cruzer Fit 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C530001180202112543
da0: 40.000MB/s transfers
da0: 59976MB (122830848 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
da1 at umass-sim1 bus 1 scbus18 target 0 lun 0
da1: <SanDisk Cruzer Fit 1.00> Removable Direct Access SPC-4 SCSI device
da1: Serial Number 4C530001200202106405
da1: 40.000MB/s transfers
da1: 59976MB (122830848 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
pass5 at ahcich9 bus 0 scbus9 target 0 lun 0
pass5: <Marvell Console 1.01> Removable Processor SCSI device
pass5: Serial Number HKDP221516WL
pass5: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1200071484 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-0449fefcfe064b30b82ac037398dc224 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
igb0: link state changed to UP
uhid0: <vendor 0x05f3 product 0x0007, class 0/0, rev 1.10/3.20, addr 6> on usbus0
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada10p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 30 port 0
ahcich2: is 00000000 cs c0000000 ss 00000000 rs c0000000 tfd 40 serr 00000000 cmd 10009d17
ahcich2:  ... waiting for slots 80000000
ahcich2: Timeout on slot 31 port 0
ahcich2: is 00000000 cs c0000000 ss 00000000 rs c0000000 tfd 40 serr 00000000 cmd 10009d17
(ada2:ahcich2:0:0:0): READ_DMA48. ACB: 25 00 a8 de 2e 40 98 01 00 00 00 01
(ada2:ahcich2:0:0:0): CAM status: Command timeout
(ada2:ahcich2:0:0:0): Retrying command
ahcich2: stopping AHCI engine failed
ahcich4: Timeout on slot 5 port 0
ahcich4: is 00000000 cs 00000020 ss 00000020 rs 00000020 tfd 40 serr 00000000 cmd 10008417
(ada4:ahcich4:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 20 68 de 2e 40 98 01 00 00 00 00
(ada4:ahcich4:0:0:0): CAM status: Command timeout
(ada4:ahcich4:0:0:0): Retrying command
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 0 port 0
ahcich2: is 00000000 cs c0000001 ss 00000000 rs 00000001 tfd 40 serr 00000000 cmd 10009d17
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Retrying command
ahcich2: stopping AHCI engine failed
ahcich4: Timeout on slot 6 port 0
ahcich4: is 00000000 cs 00000060 ss 00000020 rs 00000040 tfd 40 serr 00000000 cmd 10008417
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Retrying command
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 1 port 0
ahcich2: is 00000000 cs c0000003 ss 00000000 rs 00000002 tfd 40 serr 00000000 cmd 10009d17
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich2: stopping AHCI engine failed
ahcich4: Timeout on slot 7 port 0
ahcich4: is 00000000 cs 000000e0 ss 00000020 rs 00000080 tfd 40 serr 00000000 cmd 10008417
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 2 port 0
ahcich2: is 00000000 cs c0000007 ss 00000000 rs 00000004 tfd 40 serr 00000000 cmd 10009d17
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retry was blocked
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD40EFRX-68WT0N0 80.00A80> s/n WD-WCC4EJXKYRYV detached
ahcich2: stopping AHCI engine failed
ahcich4: Timeout on slot 8 port 0
ahcich4: is 00000000 cs 000001e0 ss 00000020 rs 00000100 tfd 40 serr 00000000 cmd 10008417
ahcich2: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: Poll timeout on slot 3 port 0
ahcich2: is 00000000 cs c000000f ss 00000000 rs 00000008 tfd 40 serr 00000000 cmd 10009d17
ahcich2: stopping AHCI engine failed
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retry was blocked
(aprobe0:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD80EFZX-68UW8N0 83.H0A83> s/n VK1B1ZXY detached
ahcich2: Timeout on slot 4 port 0
ahcich2: is 00000000 cs c000003f ss 00000000 rs 00000030 tfd 40 serr 00000000 cmd 10009d17
ahcich2:  ... waiting for slots 00000020
ahcich2: Timeout on slot 5 port 0
ahcich2: is 00000000 cs c000003f ss 00000000 rs 00000030 tfd 40 serr 00000000 cmd 10009d17
(ada2:ahcich2:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Command timeout
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
ahcich2: stopping AHCI engine failed
ahcich4: Timeout on slot 9 port 0
ahcich4: is 00000000 cs 000003e0 ss 00000020 rs 00000200 tfd 40 serr 00000000 cmd 10008417
ahcich2: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: Poll timeout on slot 6 port 0
ahcich2: is 00000000 cs c000007f ss 00000000 rs 00000040 tfd 40 serr 00000000 cmd 10009d17
ahcich2: stopping AHCI engine failed
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Retrying command
(aprobe0:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich3: Timeout on slot 12 port 0
ahcich3: is 00000000 cs 00001000 ss 00000000 rs 00001000 tfd 40 serr 00000000 cmd 10008b17
ahcich3: stopping AHCI engine failed
ahcich4: Timeout on slot 10 port 0
ahcich4: is 00000000 cs 000007e0 ss 00000020 rs 00000400 tfd 40 serr 00000000 cmd 10008417
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 7 port 0
ahcich2: is 00000000 cs c00000ff ss 00000000 rs 00000080 tfd 40 serr 00000000 cmd 10009d17
ahcich4: stopping AHCI engine failed
ahcich2: stopping AHCI engine failed
ahcich3: Timeout on slot 13 port 0
ahcich3: is 00000000 cs 00003000 ss 00000000 rs 00002000 tfd 40 serr 00000000 cmd 10008b17
ahcich3: stopping AHCI engine failed
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich4: Poll timeout on slot 11 port 0
ahcich4: is 00000000 cs 00000fe0 ss 00000020 rs 00000800 tfd 40 serr 00000000 cmd 10008417
ahcich4: stopping AHCI engine failed
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 a8 df 2e 40 98 01 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Unconditionally Re-queue Request
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
(ada2:ahcich2:0:0:0): READ_DMA48. ACB: 25 00 a8 de 2e 40 98 01 00 00 00 01
(ada2:ahcich2:0:0:0): CAM status: Command timeout
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
ahcich2: stopping AHCI engine failed
GEOM_ELI: Device ada2p1.eli destroyed.
GEOM_ELI: Detached ada2p1.eli on last close.
ahcich2: Poll timeout on slot 8 port 0
ahcich2: is 00000000 cs c00001ff ss 00000000 rs 00000100 tfd 40 serr 00000000 cmd 10009d17
ahcich2: stopping AHCI engine failed
(aprobe0:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich3:0:0:0): CAM status: Command timeout
(aprobe0:ahcich3:0:0:0): Retrying command
(aprobe1:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retries exhausted
(aprobe2:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe2:ahcich2:0:0:0): CAM status: Command timeout
(aprobe2:ahcich2:0:0:0): Error 5, Retries exhausted
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich3: Timeout on slot 14 port 0
ahcich3: is 00000000 cs 00007000 ss 00000000 rs 00004000 tfd 40 serr 00000000 cmd 10008b17
(aprobe0:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich3:0:0:0): CAM status: Command timeout
(aprobe0:ahcich3:0:0:0): Error 5, Retries exhausted
ahcich3: stopping AHCI engine failed
ahcich4: Timeout on slot 12 port 0
ahcich4: is 00000000 cs 00001fe0 ss 00000020 rs 00001000 tfd 40 serr 00000000 cmd 10008417
(ada4:ahcich4:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada4:ahcich4:0:0:0): CAM status: Command timeout
(ada4:ahcich4:0:0:0): Error 5, Periph was invalidated
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich4: Poll timeout on slot 13 port 0
ahcich4: is 00000000 cs 00003fe0 ss 00000020 rs 00002000 tfd 40 serr 00000000 cmd 10008417
ahcich4: stopping AHCI engine failed
(aprobe1:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich3: Timeout on slot 15 port 0
ahcich3: is 00000000 cs 0000f000 ss 00000000 rs 00008000 tfd 40 serr 00000000 cmd 10008b17
(aprobe0:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich3:0:0:0): CAM status: Command timeout
(aprobe0:ahcich3:0:0:0): Error 5, Retry was blocked
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD40EFRX-68WT0N0 80.00A80> s/n WD-WCC4ELZ7294R detached
ahcich3: stopping AHCI engine failed
GEOM_ELI: Device ada3p1.eli destroyed.
GEOM_ELI: Detached ada3p1.eli on last close.
ahcich3: stopping AHCI engine failed
ahcich3: Poll timeout on slot 16 port 0
ahcich3: is 00000000 cs 0001f000 ss 00000000 rs 00010000 tfd 40 serr 00000000 cmd 10008b17
ahcich3: stopping AHCI engine failed
(aprobe0:ahcich3:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
ahcich4: (aprobe0:ahcich3:0:0:0): CAM status: Command timeout
Timeout on slot 14 port 0
(aprobe0:ahcich4: ahcich3:0:is 00000000 cs 00007fe0 ss 00004020 rs 00004000 tfd 40 serr 00000000 cmd 10008417
0:0): Error 5, Retries exhausted
(ada4:ahcich4:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 20 68 de 2e 40 98 01 00 00 00 00
(ada4:ahcich4:0:0:0): CAM status: Command timeout
(ada4:ahcich4:0:0:0): Error 5, Periph was invalidated
ahcich4: stopping AHCI engine failed
GEOM_ELI: Device ada4p1.eli destroyed.
GEOM_ELI: Detached ada4p1.eli on last close.
(ada2:ahcich2:0:0:0): Periph destroyed
ahcich4: stopping AHCI engine failed
ahcich4: Poll timeout on slot 15 port 0
ahcich4: is 00000000 cs 0000ffe0 ss 00004020 rs 00008000 tfd 40 serr 00000000 cmd 10008417
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich3: Timeout on slot 17 port 0
ahcich3: is 00000000 cs 0003f000 ss 00000000 rs 00020000 tfd 40 serr 00000000 cmd 10008b17
(ada3:ahcich3:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: Unconditionally Re-queue Request
(ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
(ada4:ahcich4:0:0:0): Periph destroyed
(ada3:ahcich3:0:0:0): Periph destroyed
ahcich3: stopping AHCI engine failed
freenas%

 

[Alan W. Smtih]

dmesg reference 2 - Part 1 (see below for details)

Code:

freenas% uptime
9:33PM  up 12:17, 1 user, load averages: 10.43, 6.84, 5.08
freenas% dmesg
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
  The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 r295946+21897e6695f(9.10.2-STABLE): Thu Apr 20 17:58:31 UTC 2017
	root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Atom(TM) CPU  C2550  @ 2.40GHz (2400.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 19327352832 (18432 MB)
avail memory = 16542715904 (15776 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  2
cpu2 (AP): APIC ID:  4
cpu3 (AP): APIC ID:  6
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20160527/tbfadt-733)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80e489b0, 0) error 19
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdf740000-0xdf75ffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> mem 0xdf720000-0xdf73ffff irq 20 at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xdf500000-0xdf51ffff irq 22 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 23 at device 1.0 on pci3
pci4: <PCI bus> on pcib4
ahci0: <Marvell 88SE9172 AHCI SATA controller> port 0xc040-0xc047,0xc030-0xc033,0xc020-0xc027,0xc010-0xc013,0xc000-0xc00f mem 0xdf410000-0xdf4101ff irq 23 at device 0.0 on pci4
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib5: <PCI-PCI bridge> irq 23 at device 5.0 on pci3
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> irq 23 at device 0.0 on pci5
pci6: <PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 23 at device 0.0 on pci6
vgapci0: Boot video device
pcib7: <PCI-PCI bridge> irq 21 at device 7.0 on pci3
pci7: <PCI bus> on pcib7
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xa000-0xa01f mem 0xdf300000-0xdf37ffff,0xdf380000-0xdf383fff irq 21 at device 0.0 on pci7
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: d0:50:99:26:4f:68
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib8: <PCI-PCI bridge> irq 23 at device 9.0 on pci3
pci8: <PCI bus> on pcib8
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x9000-0x901f mem 0xdf200000-0xdf27ffff,0xdf280000-0xdf283fff irq 23 at device 0.0 on pci8
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: d0:50:99:26:4f:69
igb1: Bound queue 0 to cpu 0
igb1: Bound queue 1 to cpu 1
igb1: Bound queue 2 to cpu 2
igb1: Bound queue 3 to cpu 3
pcib9: <ACPI PCI-PCI bridge> mem 0xdf700000-0xdf71ffff at device 4.0 on pci0
pci9: <ACPI PCI bus> on pcib9
ahci1: <Marvell 88SE9230 AHCI SATA controller> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xdf610000-0xdf6107ff irq 23 at device 0.0 on pci9
ahci1: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported
ahci1: quirks=0x900<NOBSYRES,ALTSIG>
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich7: <AHCI channel> at channel 5 on ahci1
ahcich8: <AHCI channel> at channel 6 on ahci1
ahcich9: <AHCI channel> at channel 7 on ahci1
pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached)
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf763000-0xdf7633ff irq 23 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
ahci2: <Intel Avoton AHCI SATA controller> port 0xe0d0-0xe0d7,0xe0c0-0xe0c3,0xe0b0-0xe0b7,0xe0a0-0xe0a3,0xe040-0xe05f mem 0xdf762000-0xdf7627ff irq 19 at device 23.0 on pci0
ahci2: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich10: <AHCI channel> at channel 0 on ahci2
ahcich11: <AHCI channel> at channel 1 on ahci2
ahcich12: <AHCI channel> at channel 2 on ahci2
ahcich13: <AHCI channel> at channel 3 on ahci2
ahci3: <Intel Avoton AHCI SATA controller> port 0xe090-0xe097,0xe080-0xe083,0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem 0xdf761000-0xdf7617ff irq 19 at device 24.0 on pci0
ahci3: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich14: <AHCI channel> at channel 0 on ahci3
ahcich15: <AHCI channel> at channel 1 on ahci3
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
random: unblocking device.
usbus0: 480Mbps High Speed USB v2.0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub0: 8 ports with 8 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0
uhub1: 4 ports with 4 removable, self powered
ugen0.3: <SanDisk> at usbus0
umass0: <SanDisk Cruzer Fit, class 0/0, rev 2.10/1.00, addr 3> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:17:0:-1: Attached to scbus17
ugen0.4: <SanDisk> at usbus0
umass1: <SanDisk Cruzer Fit, class 0/0, rev 2.10/1.00, addr 4> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0x8100
umass1:18:1:-1: Attached to scbus18
ugen0.5: <PI Engineering> at usbus0
uhub2: <PI Engineering Kinesis Keyboard Hub, class 9/0, rev 1.10/3.20, addr 5> on usbus0
uhub2: 4 ports with 2 removable, bus powered
ugen0.6: <vendor 0x05f3> at usbus0
ukbd0: <vendor 0x05f3 product 0x0007, class 0/0, rev 1.10/3.20, addr 6> on usbus0
kbd2 at ukbd0
ugen0.7: <American Megatrends Inc.> at usbus0
uhub3: <7-port Hub> on usbus0
uhub3: 5 ports with 5 removable, self powered
ugen0.8: <American Megatrends Inc.> at usbus0
ukbd1: <Keyboard Interface> on usbus0
kbd3 at ukbd1
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada0: Serial Number WD-WCC4E1941920
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 3815447MB (7814037168 512 byte sectors)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada1: Serial Number WD-WCC4EK8ZS9NF
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada2: Serial Number WD-WCC4EJXKYRYV
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada3: Serial Number WD-WCC4ELZ7294R
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 3815447MB (7814037168 512 byte sectors)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
ada4: Serial Number VK1B1ZXY
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 7630885MB (15628053168 512 byte sectors)
ada4: Previously was known as ad12
ada5 at ahcich10 bus 0 scbus10 target 0 lun 0
ada5: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada5: Serial Number WD-WCC4EE3LXRDF
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 3815447MB (7814037168 512 byte sectors)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad24
ada6 at ahcich11 bus 0 scbus11 target 0 lun 0
ada6: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada6: Serial Number WD-WCC4E1956067
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 3815447MB (7814037168 512 byte sectors)
ada6: quirks=0x1<4K>
ada6: Previously was known as ad26
ada7 at ahcich12 bus 0 scbus12 target 0 lun 0
ada7: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada7: Serial Number WD-WCC4E0242645
ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada7: Command Queueing enabled
ada7: 3815447MB (7814037168 512 byte sectors)
ada7: quirks=0x1<4K>
ada7: Previously was known as ad28
ada8 at ahcich13 bus 0 scbus13 target 0 lun 0
ada8: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada8: Serial Number WD-WCC4EH72PYJY
ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada8: Command Queueing enabled
ada8: 3815447MB (7814037168 512 byte sectors)
ada8: quirks=0x1<4K>
ada8: Previously was known as ad30
ada9 at ahcich14 bus 0 scbus14 target 0 lun 0
ada9: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada9: Serial Number WD-WCC4E1946451
ada9: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada9: Command Queueing enabled
ada9: 3815447MB (7814037168 512 byte sectors)
ada9: quirks=0x1<4K>
ada9: Previously was known as ad32
ada10 at ahcich15 bus 0 scbus15 target 0 lun 0
ada10: <WDC WD40EFRX-68WT0N0 80.00A80> ACS-2 ATA SATA 3.x device
ada10: Serial Number WD-WCC4E1936067
ada10: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada10: Command Queueing enabled
ada10: 3815447MB (7814037168 512 byte sectors)
ada10: quirks=0x1<4K>
ada10: Previously was known as ad34
da0 at umass-sim0 bus 0 scbus17 target 0 lun 0
da0: <SanDisk Cruzer Fit 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 4C530001180202112543
da0: 40.000MB/s transfers
da0: 59976MB (122830848 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
da1 at umass-sim1 bus 1 scbus18 target 0 lun 0
da1: <SanDisk Cruzer Fit 1.00> Removable Direct Access SPC-4 SCSI device
da1: Serial Number 4C530001200202106405
da1: 40.000MB/s transfers
da1: 59976MB (122830848 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>
pass5 at ahcich9 bus 0 scbus9 target 0 lun 0
pass5: <Marvell Console 1.01> Removable Processor SCSI device
pass5: Serial Number HKDP221516WL
pass5: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1200072444 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-0449fefcfe064b30b82ac037398dc224 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
igb0: link state changed to UP
uhid0: <vendor 0x05f3 product 0x0007, class 0/0, rev 1.10/3.20, addr 6> on usbus0
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
GEOM_ELI: Device ada10p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:	 Crypto: hardware
ahcich2: Timeout on slot 13 port 0
ahcich2: is 00000000 cs 00002000 ss 00000000 rs 00002000 tfd 40 serr 00000000 cmd 10008c17
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 50 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 18 port 0
ahcich2: is 00000000 cs 00040000 ss 00000000 rs 00040000 tfd 50 serr 00000000 cmd 10009117
ahcich2: Timeout on slot 28 port 0
ahcich2: is 00000000 cs 10000000 ss 00000000 rs 10000000 tfd 40 serr 00000000 cmd 10009b17
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 16 port 0
ahcich2: is 00000000 cs 00010000 ss 00000000 rs 00010000 tfd 40 serr 00000000 cmd 10008f17
ahcich2: Timeout on slot 13 port 0
ahcich2: is 00000000 cs 00002000 ss 00000000 rs 00002000 tfd 50 serr 00000000 cmd 10008c17
ahcich2: Timeout on slot 27 port 0
ahcich2: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd 40 serr 00000000 cmd 10009a17
ahcich2: Timeout on slot 21 port 0
ahcich2: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd 40 serr 00000000 cmd 10009417


	... Clipped - Removed 650 lines of the same thing ...

 

[Alan W. Smtih]

dmesg reference 2 - Part 2 (see below for details)

Code:

ahcich2: Timeout on slot 23 port 0
ahcich2: is 00000000 cs 00800000 ss 00000000 rs 00800000 tfd 40 serr 00000000 cmd 10009617
ahcich2: Timeout on slot 23 port 0
ahcich2: is 00000000 cs 00800000 ss 00000000 rs 00800000 tfd 40 serr 00000000 cmd 10009617
ahcich2: Timeout on slot 7 port 0
ahcich2: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd 40 serr 00000000 cmd 10008617
ahcich2: Timeout on slot 27 port 0
ahcich2: is 00000000 cs 08000000 ss 00000000 rs 08000000 tfd 40 serr 00000000 cmd 10009a17
ahcich2: Timeout on slot 4 port 0
ahcich2: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 40 serr 00000000 cmd 10008317
ahcich2: Timeout on slot 21 port 0
ahcich2: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd 40 serr 00000000 cmd 10009417
ahcich2: Timeout on slot 1 port 0
ahcich2: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 40 serr 00000000 cmd 10009f17
ahcich2: Timeout on slot 1 port 0
ahcich2: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 40 serr 00000000 cmd 10009f17
ahcich2: Timeout on slot 3 port 0
ahcich2: is 00000000 cs 00000008 ss 00000000 rs 00000008 tfd 40 serr 00000000 cmd 10008217
ahcich2: Timeout on slot 19 port 0
ahcich2: is 00000000 cs 00080000 ss 00000000 rs 00080000 tfd 40 serr 00000000 cmd 10009217
ahcich2: Timeout on slot 21 port 0
ahcich2: is 00000000 cs 00200000 ss 00000000 rs 00200000 tfd 50 serr 00000000 cmd 10009417
ahcich2: Timeout on slot 11 port 0
ahcich2: is 00000000 cs 00000800 ss 00000000 rs 00000800 tfd 40 serr 00000000 cmd 10008a17
ahcich2: Timeout on slot 3 port 0
ahcich2: is 00000000 cs 00000008 ss 00000000 rs 00000008 tfd 40 serr 00000000 cmd 10008217
ahcich2: Timeout on slot 22 port 0
ahcich2: is 00000000 cs 00400000 ss 00000000 rs 00400000 tfd 40 serr 00000000 cmd 10009517
ahcich2: Timeout on slot 15 port 0
ahcich2: is 00000000 cs 00008000 ss 00000000 rs 00008000 tfd 40 serr 00000000 cmd 10008e17
ahcich2: Timeout on slot 30 port 0
ahcich2: is 00000000 cs 40000000 ss 00000000 rs 40000000 tfd 40 serr 00000000 cmd 10009d17
ahcich2: Timeout on slot 1 port 0
ahcich2: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 50 serr 00000000 cmd 10008017
ahcich2: Timeout on slot 12 port 0
ahcich2: is 00000000 cs 00001000 ss 00000000 rs 00001000 tfd 40 serr 00000000 cmd 10008b17
ahcich2: Timeout on slot 19 port 0
ahcich2: is 00000000 cs 00080000 ss 00000000 rs 00080000 tfd 40 serr 00000000 cmd 10009217
ahcich2: Timeout on slot 4 port 0
ahcich2: is 00000000 cs 00000010 ss 00000000 rs 00000010 tfd 40 serr 00000000 cmd 10008317
ahcich2: Timeout on slot 0 port 0
ahcich2: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 50 serr 00000000 cmd 10009f17
ahcich2: Timeout on slot 29 port 0
ahcich2: is 00000000 cs 20000000 ss 00000000 rs 20000000 tfd 50 serr 00000000 cmd 10009c17
ahcich2: Timeout on slot 31 port 0
ahcich2: is 00000000 cs 80000000 ss 00000000 rs 80000000 tfd 50 serr 00000000 cmd 10009e17
ahcich2: Timeout on slot 13 port 0
ahcich2: is 00000000 cs 00002000 ss 00000000 rs 00002000 tfd 40 serr 00000000 cmd 10008c17
ahcich4: Timeout on slot 23 port 0
ahcich4: is 00000000 cs 00800000 ss 00000000 rs 00800000 tfd 40 serr 00000000 cmd 10009617
ahcich2: Timeout on slot 3 port 0
ahcich2: is 00000000 cs 00000008 ss 00000000 rs 00000008 tfd 50 serr 00000000 cmd 10008217
ahcich2: Timeout on slot 8 port 0
ahcich2: is 00000000 cs 00000100 ss 00000000 rs 00000100 tfd 40 serr 00000000 cmd 10008717
ahcich2: Timeout on slot 6 port 0
ahcich2: is 00000000 cs 00000040 ss 00000000 rs 00000040 tfd 40 serr 00000000 cmd 10008517
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Timeout on slot 7 port 0
ahcich2: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Retrying command
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Timeout on slot 8 port 0
ahcich2: is 00000000 cs 00000180 ss 00000000 rs 00000100 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Timeout on slot 9 port 0
ahcich2: is 00000000 cs 00000380 ss 00000000 rs 00000200 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retry was blocked
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD40EFRX-68WT0N0 80.00A80> s/n WD-WCC4EJXKYRYV detached
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich3: Timeout on slot 26 port 0
ahcich3: is 00000000 cs 04000000 ss 00000000 rs 04000000 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
ahcich2: Timeout on slot 10 port 0
ahcich2: is 00000000 cs 00000780 ss 00000000 rs 00000400 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Retrying command
ahcich3: Timeout on slot 27 port 0
ahcich3: is 00000000 cs 0c000000 ss 00000000 rs 08000000 tfd 40 serr 00000000 cmd 10009917
(aprobe1:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich3:0:0:0): CAM status: Command timeout
(aprobe1:ahcich3:0:0:0): Retrying command
ahcich3: stopping AHCI engine failed
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich3: Timeout on slot 28 port 0
ahcich3: is 00000000 cs 1c000000 ss 00000000 rs 10000000 tfd 40 serr 00000000 cmd 10009917
(aprobe1:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich3:0:0:0): CAM status: Command timeout
(aprobe1:ahcich3:0:0:0): Error 5, Retries exhausted
ahcich3: stopping AHCI engine failed
ahcich2: Timeout on slot 11 port 0
ahcich2: is 00000000 cs 00000f80 ss 00000000 rs 00000800 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich3: Timeout on slot 29 port 0
ahcich3: is 00000000 cs 3c000000 ss 00000000 rs 20000000 tfd 40 serr 00000000 cmd 10009917
(aprobe1:ahcich3:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich3:0:0:0): CAM status: Command timeout
(aprobe1:ahcich3:0:0:0): Error 5, Retry was blocked
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD40EFRX-68WT0N0 80.00A80> s/n WD-WCC4ELZ7294R detached
ahcich3: stopping AHCI engine failed
GEOM_ELI: Device ada3p1.eli destroyed.
GEOM_ELI: Detached ada3p1.eli on last close.
ahcich3: stopping AHCI engine failed
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich3: Poll timeout on slot 30 port 0
ahcich3: is 00000000 cs 7c000000 ss 00000000 rs 40000000 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
(aprobe1:ahcich3:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich3:0:0:0): CAM status: Command timeout
(aprobe1:ahcich3:0:0:0): Error 5, Retries exhausted
ahcich2: Poll timeout on slot 12 port 0
ahcich2: is 00000000 cs 00001f80 ss 00000000 rs 00001000 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich3: Timeout on slot 31 port 0
ahcich3: is 00000000 cs fc000000 ss 00000000 rs 80000000 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
ahcich4: Timeout on slot 20 port 0
ahcich4: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009317
ahcich3: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 13 port 0
ahcich2: is 00000000 cs 00003f80 ss 00000000 rs 00002000 tfd 80 serr 00000000 cmd 10000517
ahcich3: Poll timeout on slot 0 port 0
ahcich3: is 00000000 cs fc000001 ss 00000000 rs 00000001 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
(aprobe0:ahcich3:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich3:0:0:0): CAM status: Command timeout
(aprobe0:ahcich3:0:0:0): Error 5, Retries exhausted
ahcich3: Timeout on slot 1 port 0
ahcich3: is 00000000 cs fc000003 ss 00000000 rs 00000002 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
ahcich3: stopping AHCI engine failed
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich3: Poll timeout on slot 2 port 0
ahcich3: is 00000000 cs fc000007 ss 00000000 rs 00000004 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
(aprobe0:ahcich3:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich3:0:0:0): CAM status: Command timeout
(aprobe0:ahcich3:0:0:0): Error 5, Retries exhausted
ahcich2: Poll timeout on slot 14 port 0
ahcich2: is 00000000 cs 00007f80 ss 00000000 rs 00004000 tfd 80 serr 00000000 cmd 10000517
(aprobe2:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe2:ahcich2:0:0:0): CAM status: Command timeout
ahcich4: (aprobe2:Timeout on slot 21 port 0
ahcich2:0:ahcich4: 0:is 00000000 cs 00300000 ss 00000000 rs 00200000 tfd 40 serr 00000000 cmd 10009317
0): Error 5, Retries exhausted
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Retrying command
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 15 port 0
ahcich2: is 00000000 cs 0000ff80 ss 00000000 rs 00008000 tfd 80 serr 00000000 cmd 10000517
ahcich3: Timeout on slot 3 port 0
ahcich3: is 00000000 cs fc00000f ss 00000000 rs 00000008 tfd 40 serr 00000000 cmd 10009917
(ada3:ahcich3:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: Command timeout
(ada3:ahcich3:0:0:0): Error 5, Periph was invalidated
ahcich3: stopping AHCI engine failed
ahcich3: stopping AHCI engine failed
ahcich4: Timeout on slot 22 port 0
ahcich4: is 00000000 cs 00700000 ss 00000000 rs 00400000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich3: Poll timeout on slot 4 port 0
ahcich3: is 00000000 cs fc00001f ss 00000000 rs 00000010 tfd 40 serr 00000000 cmd 10009917
ahcich3: stopping AHCI engine failed
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retries exhausted
(aprobe2:ahcich3:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe2:ahcich3:0:0:0): CAM status: Command timeout
(aprobe2:ahcich3:0:0:0): Error 5, Retries exhausted
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Poll timeout on slot 16 port 0
ahcich2: is 00000000 cs 0001ff80 ss 00000000 rs 00010000 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich4: Timeout on slot 23 port 0
ahcich4: is 00000000 cs 00f00000 ss 00000000 rs 00800000 tfd 40 serr 00000000 cmd 10009317
(aprobe1:ahcich4:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retry was blocked
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4: <WDC WD80EFZX-68UW8N0 83.H0A83> s/n VK1B1ZXY detached
ahcich4: stopping AHCI engine failed
GEOM_ELI: Device ada4p1.eli destroyed.
GEOM_ELI: Detached ada4p1.eli on last close.
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 17 port 0
ahcich2: is 00000000 cs 0003ff80 ss 00000000 rs 00020000 tfd 80 serr 00000000 cmd 10000517
ahcich4: Poll timeout on slot 24 port 0
ahcich4: is 00000000 cs 01f00000 ss 00000000 rs 01000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Poll timeout on slot 18 port 0
ahcich2: is 00000000 cs 0007ff80 ss 00000000 rs 00040000 tfd 80 serr 00000000 cmd 10000517
(aprobe1:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich2:0:0:0): CAM status: Command timeout
(aprobe1:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich4: Timeout on slot 25 port 0
ahcich4: is 00000000 cs 03f00000 ss 00000000 rs 02000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich4: Poll timeout on slot 26 port 0
ahcich4: is 00000000 cs 07f00000 ss 00000000 rs 04000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich2: Timeout on slot 19 port 0
ahcich2: is 00000000 cs 000fff80 ss 00000000 rs 00080000 tfd 80 serr 00000000 cmd 10000517
ahcich4: Timeout on slot 27 port 0
ahcich4: is 00000000 cs 0ff00000 ss 00000000 rs 08000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich4: Poll timeout on slot 28 port 0
ahcich4: is 00000000 cs 1ff00000 ss 00000000 rs 10000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe1:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich4:0:0:0): CAM status: Command timeout
(aprobe1:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich2: Poll timeout on slot 20 port 0
ahcich2: is 00000000 cs 001fff80 ss 00000000 rs 00100000 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich4: Timeout on slot 29 port 0
ahcich4: is 00000000 cs 3ff00000 ss 00000000 rs 20000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 21 port 0
ahcich2: is 00000000 cs 003fff80 ss 00000000 rs 00200000 tfd 80 serr 00000000 cmd 10000517
ahcich4: Poll timeout on slot 30 port 0
ahcich4: is 00000000 cs 7ff00000 ss 00000000 rs 40000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich4: Timeout on slot 31 port 0
ahcich4: is 00000000 cs fff00000 ss 00000000 rs 80000000 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich4: Poll timeout on slot 0 port 0
ahcich4: is 00000000 cs fff00001 ss 00000000 rs 00000001 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich2: Poll timeout on slot 22 port 0
ahcich2: is 00000000 cs 007fff80 ss 00000000 rs 00400000 tfd 80 serr 00000000 cmd 10000517
(aprobe1:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich2:0:0:0): CAM status: Command timeout
(aprobe1:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich4: Timeout on slot 1 port 0
ahcich4: is 00000000 cs fff00003 ss 00000000 rs 00000002 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: Timeout on slot 23 port 0
ahcich2: is 00000000 cs 00ffff80 ss 00000000 rs 00800000 tfd 80 serr 00000000 cmd 10000517
ahcich4: Poll timeout on slot 2 port 0
ahcich4: is 00000000 cs fff00007 ss 00000000 rs 00000004 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(ada2:ahcich2:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Command timeout
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich4: Timeout on slot 3 port 0
ahcich4: is 00000000 cs fff0000f ss 00000000 rs 00000008 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich4: Poll timeout on slot 4 port 0
ahcich4: is 00000000 cs fff0001f ss 00000000 rs 00000010 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich2: Poll timeout on slot 24 port 0
ahcich2: is 00000000 cs 01ffff80 ss 00000000 rs 01000000 tfd 80 serr 00000000 cmd 10000517
(aprobe1:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich2:0:0:0): CAM status: Command timeout
(aprobe1:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich4: Timeout on slot 5 port 0
ahcich4: is 00000000 cs fff0003f ss 00000000 rs 00000020 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich2: Timeout on slot 25 port 0
ahcich2: is 00000000 cs 0fffff80 ss 0e000000 rs 0e000000 tfd 80 serr 00000000 cmd 10000517
ahcich4: Poll timeout on slot 6 port 0
ahcich4: is 00000000 cs fff0007f ss 00000000 rs 00000040 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 20 30 ef 5f 40 49 01 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Command timeout
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 20 50 ef 5f 40 49 01 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Unconditionally Re-queue Request
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 f8 6a b9 40 6d 01 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Unconditionally Re-queue Request
(ada2:ahcich2:0:0:0): Error 5, Periph was invalidated
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
GEOM_ELI: Device ada2p1.eli destroyed.
GEOM_ELI: Detached ada2p1.eli on last close.
(ada3:ahcich3:0:0:0): Periph destroyed
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich4: Timeout on slot 7 port 0
ahcich4: is 00000000 cs fff000ff ss 00000000 rs 00000080 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich4: Poll timeout on slot 8 port 0
ahcich4: is 00000000 cs fff001ff ss 00000000 rs 00000100 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
ahcich2: Poll timeout on slot 28 port 0
ahcich2: is 00000000 cs 1fffff80 ss 0e000000 rs 10000000 tfd 80 serr 00000000 cmd 10000517
(aprobe1:ahcich2:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich2:0:0:0): CAM status: Command timeout
(aprobe1:ahcich2:0:0:0): Error 5, Retries exhausted
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich4: Timeout on slot 9 port 0
ahcich4: is 00000000 cs fff003ff ss 00000000 rs 00000200 tfd 40 serr 00000000 cmd 10009317
(ada2:ahcich2:0:0:0): Periph destroyed
ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich4: Poll timeout on slot 10 port 0
ahcich4: is 00000000 cs fff007ff ss 00000000 rs 00000400 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich4: Timeout on slot 11 port 0
ahcich4: is 00000000 cs fff00fff ss 00000000 rs 00000800 tfd 40 serr 00000000 cmd 10009317
(ada4:ahcich4:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada4:ahcich4:0:0:0): CAM status: Command timeout
(ada4:ahcich4:0:0:0): Error 5, Periph was invalidated
(ada4:ahcich4:0:0:0): Periph destroyed
ahcich4: stopping AHCI engine failed
freenas%

 

[Alan W. Smtih]

Pesky day-job has kept me from spending as much time troubleshooting as I'd like. Here's the latest:

- All three disks that showed up as removed were on the same power cable. I replaced the cable with a new one and plugged it into a different port on the PSU. That didn't fix the issue.

- I upgraded the BIOS from the original `C2550D4I P1.40` to the latest `C2550D4I P2.30`. (The BMC firmware was updated to the latest `0.30.0` two months ago.)

- I wanted the resilver to finish before messing with much. So, I'd let the machine run for a few hours at a time and then reboot it in an attempt to catch it before the drives went "REMOVED". There's usually a window of several hours so I managed to get it move of the time. However, the drives went "REMOVED" once after just an hour or so.

- I did manage to make it through the resilver with the frequent reboots. The `zpool status` report showed everything as "ONLINE"

Code:

  pool: z
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
  attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
  using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 2.45T in 104h26m with 0 errors on Sun Jun 25 20:29:44 2017
config:

  NAME											STATE	 READ WRITE CKSUM
  z											   ONLINE	   0	 0	 0
	raidz3-0									  ONLINE	   0	 0	 0
	  gptid/c6171db6-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c68a4186-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c701c120-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 2
	  gptid/c779c670-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 1
	  gptid/0d977d05-569b-11e7-8777-d05099264f68  ONLINE	   0	 0	 1
	  gptid/c8688176-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c8e2f1e9-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c9cc35d5-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/ca449ed0-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cabb7d11-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cb39f216-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors  

- I shut the machine down for a couple days after the resilver since I wasn't going to have a chance to work on it.

- The next time I turned on the machine, all three drives that had suffered "REMOVED" showed up as being resilvered. (Note: I'm pretty sure it was the next time I booted the machine, but there may have been one other power cycle before that.)

Code:

  pool: z
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jul  2 22:57:09 2017
		61.8G scanned out of 34.3T at 175M/s, 56h59m to go
		5.42G resilvered, 0.18% done
config:

  NAME											STATE	 READ WRITE CKSUM
  z											   ONLINE	   0	 0	 0
	raidz3-0									  ONLINE	   0	 0	 0
	  gptid/c6171db6-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c68a4186-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c701c120-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0  (resilvering)
	  gptid/c779c670-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0  (resilvering)
	  gptid/0d977d05-569b-11e7-8777-d05099264f68  ONLINE	   0	 0	 0  (resilvering)
	  gptid/c8688176-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c8e2f1e9-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c9cc35d5-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/ca449ed0-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cabb7d11-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cb39f216-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors

- That was pretty late and I didn't want to leave it running overnight and having the "REMOVED" thing happen again so I shut down the machine.

- Started the next morning, and only one drive showing up as being resilvered. (The same drive that was replaced and had already gone through the resilver)

Code:

	pool: z
	state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
	scan: resilver in progress since Sun Jul  2 22:57:09 2017
		344G scanned out of 34.3T at 291M/s, 34h3m to go
		30.8G resilvered, 0.98% done
config:

  NAME											STATE	 READ WRITE CKSUM
  z											   ONLINE	   0	 0	 0
	raidz3-0									  ONLINE	   0	 0	 0
	  gptid/c6171db6-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c68a4186-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c701c120-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c779c670-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/0d977d05-569b-11e7-8777-d05099264f68  ONLINE	   0	 0	 0  (resilvering)
	  gptid/c8688176-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c8e2f1e9-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/c9cc35d5-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/ca449ed0-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cabb7d11-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0
	  gptid/cb39f216-4668-11e4-b49f-d05099264f68  ONLINE	   0	 0	 0

errors: No known data errors

- Looking at `dmesg` logs from times when the drives went "REMOVED", it certinaly looks to be related to `ahcich#: Timeout` errors. (See: dmesg reference 1 above)


- That one was a little different. Most of the time, there are a few hundred of these:

Code:

ahcich2: Timeout on slot 13 port 0
ahcich2: is 00000000 cs 00002000 ss 00000000 rs 00002000 tfd 40 serr 00000000 cmd 10008c17
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 50 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 18 port 0
ahcich2: is 00000000 cs 00040000 ss 00000000 rs 00040000 tfd 50 serr 00000000 cmd 10009117
ahcich2: Timeout on slot 28 port 0
ahcich2: is 00000000 cs 10000000 ss 00000000 rs 10000000 tfd 40 serr 00000000 cmd 10009b17
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 20 port 0
ahcich2: is 00000000 cs 00100000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009317
ahcich2: Timeout on slot 16 port 0
ahcich2: is 00000000 cs 00010000 ss 00000000 rs 00010000 tfd 40 serr 00000000 cmd 10008f17

prior to messages like these showing up:

Code:

ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Timeout on slot 7 port 0
ahcich2: is 00000000 cs 00000080 ss 00000000 rs 00000080 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Retrying command
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Timeout on slot 8 port 0
ahcich2: is 00000000 cs 00000180 ss 00000000 rs 00000100 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retries exhausted
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich2: Timeout on slot 9 port 0
ahcich2: is 00000000 cs 00000380 ss 00000000 rs 00000200 tfd 80 serr 00000000 cmd 10000517
(aprobe0:ahcich2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich2:0:0:0): CAM status: Command timeout
(aprobe0:ahcich2:0:0:0): Error 5, Retry was blocked
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD40EFRX-68WT0N0 80.00A80> s/n WD-WCC4EJXKYRYV detached
ahcich2: AHCI reset: device not ready after 31000ms (tfd = 00000080)

and ending up with this which is where the drives become "REMOVED" in `zpool status`.

Code:

ahcich4: stopping AHCI engine failed
ahcich4: stopping AHCI engine failed
ahcich4: Poll timeout on slot 10 port 0
ahcich4: is 00000000 cs fff007ff ss 00000000 rs 00000400 tfd 40 serr 00000000 cmd 10009317
ahcich4: stopping AHCI engine failed
(aprobe0:ahcich4:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich4:0:0:0): CAM status: Command timeout
(aprobe0:ahcich4:0:0:0): Error 5, Retries exhausted
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
xptioctl: pass driver is not in the kernel
xptioctl: put "device pass" in your kernel config file
ahcich4: Timeout on slot 11 port 0
ahcich4: is 00000000 cs fff00fff ss 00000000 rs 00000800 tfd 40 serr 00000000 cmd 10009317
(ada4:ahcich4:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 00 00 00 00 00 00
(ada4:ahcich4:0:0:0): CAM status: Command timeout
(ada4:ahcich4:0:0:0): Error 5, Periph was invalidated
(ada4:ahcich4:0:0:0): Periph destroyed
ahcich4: stopping AHCI engine failed

(See dmesg reference 2 above for a fully copy representative of how it went down most of the time)


- I found this post (https://forums.freebsd.org/threads/51868/page-2) about folks having similar `ahcich#: Timeout` issues. Some report a fix by adding:

Code:

hint.ahci.0.msi=0
hint.ahci.1.msi=0
...
hint.ahcich.0.sata_rev=2
hint.ahcich.1.sata_rev=2

- It didn't work for everyone though.


__ Current Plan __

- Do the repeated reboot dance until the resilver completes. (Logic being: Don't mess with stuff while it's rebuilding.)

- Upgrade form `9.10.2-U3` to the latest `9.10.2-U5` to see if that makes the error to away.

- If not, downgrade to a `9.3.x` version that never seemed to have this issue.

- If that doesn't do it, try the `hint.ahci.0.msi=0` and `hint.ahcich.0.sata_rev=2` switches.

- I'm also considering adding hardware to move the drives to new ports (assuming that's possible), but am will to try a few more software fixes first.


__ Questions __


Question 1: Any thoughts/gotchas on the above details and/or plan?

Question 2: Any idea why the three drives showed up being resilvered for one boot, but only one on the next reboot? (And, more to the point, should I be worried about that?)

Question 3: I'm assuming the `0`, `1`, etc... in `hint.ahci.#.msi=0` and `hint.ahcich.#.sata_rev=2` are indexes for drives (e.g. `hint.ahci.0.msi=0` corresponds to `/dev/ada0`, `hint.ahci.1.msi=0` corresponds to `/dev/ada1`, etc...) and that I'd need to make one entry for each drive. Is that correct?

 

[Alan W. Smtih]

Quick update for interested parties:

- I finished the resilver by rebooting the machine every two hours. After a reboot everything is showing ONILNE and seems in good shape there.

- I updated to `9.10.2-U5`.

- After the resilver and update, I'm still getting the timeout alerts, but they are way less frequent. (My guess is the drop-off is related to no longer having the resilver I/O hitting the system.)

- I ended up trying the the `hint.ahci.0.msi=0` and `hint.ahcich.0.sata_rev=2` switches, while the resilver was underway. Neither seemed to have an impact on reducing or eliminating the issue.

- I dropped the idea of falling back to `9.3.x`. I'd rather do whatever it takes to get this solved in a way that allows me to stay up to date. Even if that ends up requiring a new motherboard and supporting hardware.

- I've decided my next step is to switch out the hard drives. I'm running `badblocks` on fresh disks on another machine now. I'll report back when they are swapped in.

 

[Robert Trevellyan]

Are you still using the Marvel controller?

 

[Alan W. Smtih]

Are you still using the Marvel controller?

Yep, but I'm not married to it.

I'm going one step at a time and trying the easiest stuff first.

I'd gotten some S.M.A.R.T. warnings about the same time this started. While I don't know if the issues are related (or have a causal link in either direction), I'd be replacing the drives regardless.

I figure I'll do that first. It would be great if that resolves the Timeout issue. If it doesn't, the next step is taking that controller out of the mix.

 

[Alan W. Smtih]

Just to close this thread instead of leaving it hanging...

I replaced two of the three drives and let them finish their resilver (stopping the processes every couple hours for a reboot to avoid the REMOVED issue).

I was hoping that something wrong with a drive would be the cause of the TIMEOUT issues. And, that if I could resolve it, it would give me a little more elbow room to migrate off the Marvel controller. But that doesn't seem to be the case.

While I could try replacing the third drive, it just doesn't seem worth it.

So, I'm off to find replacement/upgrade parts, and we can add one more checkmark to the "yeah, probably want to avoid that marvel controller" tally.

:)

 

[reaktor]

Hi Alan and others;)
(how) did u fixed the issue?

I ve the same... dropped disks with this sucking mainboard