11.1-RELEASE IOC Fault

Status
Not open for further replies.
J

jmck

Cadet
Joined
Dec 28, 2017
Messages
2
Hello,

I upgraded from 11.0-U4 to 11.1-RELEASE yesterday morning. I did not encounter any problems during the upgrade. I received emails about a pool being degraded, a disk being removed and unexpected shutdown. I didn't find anything in /data/crash and zpool status showed 8MB had been resilvered. I checked the smart status of all the drives and did not find anything with any of the drives. Later, I received more emails about a drive being removed and degraded pool for a different pool than last night. I checked everything again and found no issues, everything was showing healthy and nothing was resilvered. Same thing happened again about 30 minutes later to the same pool from last night and again by the time I was able to check, everything was fine and 17.8MB had been resilvered. I checked out /var/log/messages and found the following

Code:
Dec 28 03:36:11 freenas-1 mps0: IOC Fault 0x40000d04, Resetting
Dec 28 03:36:11 freenas-1 mps0: Reinitializing controller,
Dec 28 03:36:11 freenas-1 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Dec 28 03:36:11 freenas-1 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
Dec 28 03:36:11 freenas-1 mps0: mps_reinit finished sc 0xfffffe0000ecd000 post 4 free 3
Dec 28 03:36:11 freenas-1 (da8:mps0:0:42:0): Invalidating pack
Dec 28 03:36:11 freenas-1 da8 at mps0 bus 0 scbus0 target 42 lun 0
Dec 28 03:36:11 freenas-1 da8: <ATA WDC WD80EFAX-68L 0A83> s/n 7SGJEKMC			 detached
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 384935409bae9a74
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 484b29409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 464a404c9db98e5c
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 49463e409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 494f4a409bae9a74
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 454b304c9db98e5c
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = dd5143409bae9a74
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = d14f3d409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 3d481a4c9db98d7f
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = db5149409bae9a71
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 484f2a409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 463d364c9db98e5c
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 384935409bae9a74
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 484b29409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 464a404c9db98e5c
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 49463e409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 494f4a409bae9a74
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 454b304c9db98e5c
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = dd5143409bae9a74
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = d14f3d409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 3d481a4c9db98d7f
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = db5149409bae9a71
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 484f2a409bae9a73
Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 463d364c9db98e5c
Dec 28 03:36:19 freenas-1 ZFS: vdev state changed, pool_guid=13824347541065318658 vdev_guid=17157989435552045004
Dec 28 03:36:19 freenas-1 (da8:mps0:0:42:0): Periph destroyed
Dec 28 03:36:19 freenas-1 da8 at mps0 bus 0 scbus0 target 42 lun 0
Dec 28 03:36:19 freenas-1 da8: <ATA WDC WD80EFAX-68L 0A83> Fixed Direct Access SPC-4 SCSI device
Dec 28 03:36:19 freenas-1 da8: Serial Number 7SGJEKMC			
Dec 28 03:36:19 freenas-1 da8: 600.000MB/s transfers
Dec 28 03:36:19 freenas-1 da8: Command Queueing enabled
Dec 28 03:36:19 freenas-1 da8: 7630885MB (15628053168 512 byte sectors)
Dec 28 03:36:19 freenas-1 ZFS: vdev state changed, pool_guid=13824347541065318658 vdev_guid=17157989435552045004


I thought it was da8 but continuing to go through the logs, it includes basically every drive at some point. Sometimes no drives are included, like this from just after the above event.

Code:
Dec 28 03:37:19 freenas-1 mps0: IOC Fault 0x40000d04, Resetting
Dec 28 03:37:19 freenas-1 mps0: Reinitializing controller,
Dec 28 03:37:19 freenas-1 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Dec 28 03:37:19 freenas-1 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
Dec 28 03:37:19 freenas-1 mps0: mps_reinit finished sc 0xfffffe0000ecd000 post 4 free 3
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 384935409bae9a74
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 484b29409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 464a404c9db98e5c
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 49463e409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 494f4a409bae9a74
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 454b304c9db98e5c
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = dd5143409bae9a74
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = d14f3d409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 3d481a4c9db98d7f
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = db5149409bae9a71
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 484f2a409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 463d364c9db98e5c
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 384935409bae9a74
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 484b29409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 464a404c9db98e5c
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 49463e409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 494f4a409bae9a74
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 454b304c9db98e5c
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = dd5143409bae9a74
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = d14f3d409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 3d481a4c9db98d7f
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = db5149409bae9a71
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 484f2a409bae9a73
Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 463d364c9db98e5c


I restarted and was still encountering this issue. It occurs anywhere from several times a minute to every 10 minutes or so. I rolled back to 11.0-U4 and I'm no longer seeing the events. This server has run without issue for about 18 months now. The hardware is a S2600IP4 with a single E5-2670, 32GB EEC, RES2SV240 expander and H200 flashed to P20 in IT.

Any ideas?

Thank you
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Try updating the expander's firmware.
 
J

jmck

Cadet
Joined
Dec 28, 2017
Messages
2
I checked the version before updating and it was already at 254.13.00.00. I went ahead and flashed it with 254.13.00.00 again but I was still encountering the errors. I swapped the H200, expander, and cables this afternoon and as soon as I started coping something from a pool, same problem.

Code:
Jan  2 18:18:55 freenas-1 mps0: IOC Fault 0x40000d04, Resetting
Jan  2 18:18:55 freenas-1 mps0: Reinitializing controller,
Jan  2 18:18:55 freenas-1 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Jan  2 18:18:55 freenas-1 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
Jan  2 18:18:55 freenas-1 mps0: mps_reinit finished sc 0xfffffe0000ecd000 post 4 free 3
Jan  2 18:18:55 freenas-1 (da10:mps0:0:44:0): Invalidating pack
Jan  2 18:18:55 freenas-1 da10 at mps0 bus 0 scbus0 target 44 lun 0
Jan  2 18:18:55 freenas-1 da10: <ATA WDC WD80EFAX-68L 0A83> s/n 7SGJDT9C			 detached
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 384935409bae9a74
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 484b29409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 464a404c9db98e5c
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 49463e409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 494f4a409bae9a74
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 454b304c9db98e5c
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = dd5143409bae9a74
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = d14f3d409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 3d481a4c9db98d7f
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = db5149409bae9a71
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 484f2a409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address for SATA device = 463d364c9db98e5c
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 384935409bae9a74
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 484b29409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 464a404c9db98e5c
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 49463e409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 494f4a409bae9a74
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 454b304c9db98e5c
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = dd5143409bae9a74
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = d14f3d409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 3d481a4c9db98d7f
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = db5149409bae9a71
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 484f2a409bae9a73
Jan  2 18:18:57 freenas-1 mps0: SAS Address from SATA device = 463d364c9db98e5c
Jan  2 18:19:02 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=12796817161823950944
Jan  2 18:19:02 freenas-1 (da10:mps0:0:44:0): Periph destroyed
Jan  2 18:19:02 freenas-1 da10 at mps0 bus 0 scbus0 target 44 lun 0
Jan  2 18:19:02 freenas-1 da10: <ATA WDC WD80EFAX-68L 0A83> Fixed Direct Access SPC-4 SCSI device
Jan  2 18:19:03 freenas-1 da10: Serial Number 7SGJDT9C
Jan  2 18:19:03 freenas-1 da10: 600.000MB/s transfers
Jan  2 18:19:03 freenas-1 da10: Command Queueing enabled
Jan  2 18:19:03 freenas-1 da10: 7630885MB (15628053168 512 byte sectors)
Jan  2 18:19:03 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=12796817161823950944
Jan  2 18:19:03 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=4834717325419658139
Jan  2 18:19:03 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=1056015453832170795
Jan  2 18:19:03 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=2092723076604001089
Jan  2 18:19:04 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=11899780509422130641
Jan  2 18:19:04 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=12796817161823950944
Jan  2 18:19:04 freenas-1 ZFS: vdev state changed, pool_guid=3942001365838018798 vdev_guid=14961958218660331498
 
David Henrickson

David Henrickson

Explorer
Joined
Jul 26, 2017
Messages
93
Any solution to this? I am having the same issue with a T610 and H200
 
Status
Not open for further replies.

Similar threads

ThomasDK81
What looks like a .... problem
Replies
4
Views
1K
jgreco
jgreco
ndboost
  • Locked
ISCSI Errors - logical block address out of range
Replies
1
Views
2K
Ericloewe
Ericloewe
G
  • Locked
System has started crashing repeatedly. Pls help with some troubleshooting.
Replies
6
Views
2K
cyberjock
C
Scharbag
  • Locked
Unauthorized Reboot - 11.1-RELEASE
3 4 5 6 7
Replies
125
Views
19K
Ericloewe
Ericloewe
T
  • Locked
FreeNAS 11.1 locking up roughly every 24 hours
2 3
Replies
54
Views
10K
Pointeo13
P
Top