Pool problems

[BlueScreenTT]

Hi

i lost my pool and cant get it back
i have a pool 8 X 3 TB discs
one minute it was ok and i was using the Pool 1 hour later it said degraded and then after another 30 minutes it just dissapeared.
I updated Freenas from one 11.2 to the latest 11.2 one day earlier.

today i reinstalled it from scratch and tried to import again but it still do not work.
i am using 3 lsi 9207-8i to controll my discs it that is important

i have looked in the forums for similar problems but none of the clues i found there helped me


OS Version:
FreeNAS-11.2-RELEASE-U1
(Build Date: Dec 20, 2018 22:41)
Processor:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (32 cores)
Memory:
16 GiB


-------------------------------------------------------------------------------------------------------
root@freenas[~]# zpool import
pool: Data1-8
id: 6019148157068706093
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

Data1-8 ONLINE
raidz2-0 ONLINE
gptid/bd487063-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/be90af22-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/bff6795f-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/c129b6be-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/c26ba3e5-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/c3c044ef-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/c51984c8-00b7-11e9-ad5c-0cc47a304374 ONLINE
gptid/c667f082-00b7-11e9-ad5c-0cc47a304374 ONLINE
-------------------------------------------------------------------------------------------------------
when i try to import it in Freenas 11.2 gui i get this error and then the pool dissapars from the pool table

-------------------------------------------------------------------------------------------------------
this command works

root@freenas[~]# zpool import -nfF Data1-8
root@freenas[~]#
-------------------------------------------------------------------------------------------------------
This command do not work

root@freenas[~]# zpool import -fF Data1-8
cannot import 'Data1-8': no such pool or dataset
Destroy and re-create the pool from
a backup source.
-------------------------------------------------------------------------------------------------------
root@freenas[~]# zpool status Data1-8
cannot open 'Data1-8': no such pool
-------------------------------------------------------------------------------------------------------
and this command also fails

root@freenas[~]# zpool import -fFX Data1-8
cannot import 'Data1-8': one or more devices is currently unavailable
-------------------------------------------------------------------------------------------------------

How do i proceed from here ?
\\Thomas

 

[BERKUT]

I got the same error (unknown 0 B(error)) on FreeNAS-11.2-RELEASE-U1, when 1 device (SSD slog) went down. It was a test zpool so I didn't try to recover it.

 

[BlueScreenTT]

i have 12 TB of data on this one
everything
i dont want to loose it :-(

 

[hugovsky]

Try to import using ID of pool.

 

[Chris Moore]

i have 12 TB of data on this one
everything
i don't want to loose it :-(

Did you have any backup?

 

[Chris Moore]

You need to submit a bug report and get the development team involved in this.

 

[Chris Moore]

i am using 3 lsi 9207-8i to controll my discs it that is important

Please explain more about your system hardware. Why are you using 3 cards? Because one of those cards can control 256 drives. How many srives are supposed to be here?

 

[BERKUT]

If you have a broken disk, like I had it.
This helped me:
https://unix.stackexchange.com/questions/130846/how-to-remove-broken-zil-disk-from-zfs-pool

But your "zpool import" shows that you pool is ok.

 

[joeschmuck]

If your system was working on the previous version of FreeNAS, have you tried to roll it back? This should be one of the first things you think about doing. Maybe even roll it back to 11.1-U4 or U6. The point is to get your pool mounted again so you can access your data. If you do not have a backup then you should start thinking about a backup strategy because at some point in time things will go south and all the redundancy in the world can't keep safe a single computer and group of hard drives. Think electrical storm or flood.

I wish you luck.

 

[BlueScreenTT]

ok
here goes

No backup :-(

I have reinstalled Freenas 2 times now after the first crash and it continues to crash
my setup is this

Chassis: Supermicro CSE-846TQ-R1200B
Motherboard: x9dri-ln4f+
Processor: 2 X Intel Xeon E5-2650 V2
DDR3 non ECC but i am going to upgrade to ECC soon
Controller: 3 X LSI 9207-8i
Boot drive: Samsung SSD 128 GB running on onboard sata controller
Network: Chelsio N320E
HDD's,:
8 X 3 TB WD nas drives ZFS2
1 X 8 TB WD nas drive ZFS1
2 X 8TB Ironwolf drives
and 8 more in different sizes


I have disbled 2 onboard gigabit ports, PCI-X ports not in use, seriel ports and the onboard SAS controller.
this i did in the hope that i would avoid IRQ problems.

i got a crash a couple of days ago and lost the test files on the drive
i only have one 8 TB drive attached at the moment
in the IPMI log i see a runtime error from the OS
how do i find the error in freenas los ?


I pulled the log from IPMI

Event ID
Time Stamp ​

Sensor Name
Sensor Type
Description

1
2018/12/14 20:26:47​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

2
2018/12/15 09:21:09​

OEM
AC Power On
AC Power On - Asserted

3
2018/12/15 09:27:05​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

4
2018/12/15 09:33:37​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

5
2018/12/15 10:05:11​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

6
2018/12/15 10:06:44​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

7
2018/12/15 10:08:47​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

8
2018/12/15 20:12:34​

OEM
AC Power On
AC Power On - Asserted

9
2018/12/15 20:14:52​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

10
2019/01/01 19:22:20​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Deasserted

11
2019/01/02 20:16:19​

OEM
AC Power On
AC Power On - Asserted

12
2019/01/02 20:18:00​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

13
2019/01/10 19:49:29​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

14
2019/01/10 19:57:16​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

15
2019/01/11 22:03:06​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

16
2019/01/11 22:15:44​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

17
2019/01/12 09:55:41​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

18
2019/01/12 10:00:38​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

19
2019/01/12 10:02:45​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

20
2019/01/12 10:04:27​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

21
2019/01/12 12:01:37​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

22
2019/01/12 12:03:25​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

23
2019/01/12 14:24:24​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

24
2019/01/12 14:26:10​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

25
2019/01/13 00:13:30​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

26
2019/01/13 00:15:17​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

27
2019/01/13 00:27:16​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

28
2019/01/13 00:29:08​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

29
2019/01/13 00:32:30​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

30
2019/01/13 00:36:07​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

31
2019/01/13 00:36:41​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

32
2019/01/14 19:31:14​

OEM
AC Power On
AC Power On - Asserted

33
2019/01/14 19:33:17​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

34
2019/01/14 19:33:19​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

35
2019/01/14 19:48:36​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

36
2019/01/14 19:54:36​

FAN3
Fan
Lower Critical - Going Low - Asserted

37
2019/01/14 19:54:36​

FAN3
Fan
Lower Non-Recoverable - Going Low - Asserted

38
2019/01/14 20:03:27​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

39
2019/01/14 20:05:29​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

40
2019/01/14 20:07:08​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

41
2019/01/14 20:15:30​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

42
2019/01/14 20:17:13​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

43
2019/01/14 20:31:51​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Deasserted

44
2019/01/16 18:44:10​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

45
2019/01/16 20:29:45​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

46
2019/01/18 18:31:01​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

47
2019/01/18 19:04:18​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

48
2019/01/18 19:15:49​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

49
2019/01/18 19:16:45​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

50
2019/01/18 19:18:50​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

51
2019/01/18 19:22:28​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

52
2019/01/18 20:24:21​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

53
2019/01/18 20:32:53​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

54
2019/01/18 20:34:48​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

55
2019/01/18 22:08:30​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

56
2019/01/18 22:10:43​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

57
2019/01/18 22:11:01​

OEM
CPLD
CATERR - Asserted

58
2019/01/18 22:16:50​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

59
2019/01/18 22:25:05​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

60
2019/01/18 22:31:22​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

61
2019/01/19 18:16:22​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

62
2019/01/19 18:20:12​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

63
2019/01/19 18:20:30​

OEM
CPLD
CATERR - Asserted

64
2019/01/19 19:02:45​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

65
2019/01/19 19:07:27​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

66
2019/01/19 19:09:41​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

67
2019/01/19 20:47:25​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

68
2019/01/19 20:49:48​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

69
2019/01/19 20:53:23​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

70
2019/01/19 20:54:31​

OEM
CPLD
CATERR - Asserted

71
2019/01/20 14:40:15​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

72
2019/01/20 14:45:38​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

73
2019/01/20 15:15:14​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

74
2019/01/20 15:19:59​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

75
2019/01/20 15:40:35​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

76
2019/01/20 15:42:56​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

77
2019/01/20 15:46:28​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

78
2019/01/20 15:47:29​

OEM
CPLD
CATERR - Asserted

79
2019/01/20 15:52:49​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

80
2019/01/20 15:55:56​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

81
2019/01/20 17:10:42​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

82
2019/01/20 17:12:43​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

83
2019/01/20 17:15:54​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

84
2019/01/20 17:20:41​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

85
2019/01/20 17:22:34​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

86
2019/01/20 17:22:53​

OEM
CPLD
CATERR - Asserted

87
2019/01/20 17:46:44​

OEM
AC Power On
AC Power On - Asserted

88
2019/01/20 17:47:28​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

89
2019/01/20 17:47:31​

Chassis Intru
Physical Security (Chassis Intrusion)
General Chassis Intrusion - Asserted

90
2019/01/20 17:48:32​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

91
2019/01/20 17:56:53​

OEM
CPLD
CATERR - Asserted

92
2019/01/20 18:00:28​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

93
2019/01/20 19:28:38​

OEM
CPLD
CATERR - Asserted

94
2019/01/20 19:33:16​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

95
2019/01/20 19:55:40​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

96
2019/01/20 19:57:36​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

97
2019/01/20 20:22:06​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

98
2019/01/20 20:24:06​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

99
2019/01/20 20:24:26​

OEM
CPLD
CATERR - Asserted

100
2019/01/20 22:06:04​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

101
2019/01/22 02:10:49​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

102
2019/01/22 02:15:28​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

103
2019/01/22 02:15:53​

OEM
CPLD
CATERR - Asserted

104
2019/01/22 14:18:06​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

105
2019/01/22 14:47:46​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

106
2019/01/22 14:51:04​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

107
2019/01/22 14:54:35​

OS Stop #0x00
OS Stop
OS Graceful Shutdown - Asserted

108
2019/01/22 15:46:04​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

109
2019/01/22 18:07:33​

OS Stop #0x00
OS Stop
Run-Time Critical Stop - Asserted

110
2019/01/22 18:10:03​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

111
2019/01/22 18:10:23​

OEM
CPLD
CATERR - Asserted

112
2019/01/22 18:15:06​

OS Boot #0x00
OS Boot
C: Boot Completed - Asserted

 

[BlueScreenTT]

i checked the pool with

-----------------------------------------------------------------------------------------------------------------

root@freenas[~]# zpool status -x Data21
pool: Data21
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 0 days 00:05:46 with 8 errors on Fri Jan 25 23:52:43 2019
config:

NAME STATE READ WRITE CKSUM
Data21 ONLINE 0 0 0
gptid/081e3b12-1f34-11e9-b141-0cc47a304374 ONLINE 0 0 0

-----------------------------------------------------------------------------------------------------------------

root@freenas[~]# zpool status -xv
pool: Data21
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 0 days 00:05:46 with 8 errors on Fri Jan 25 23:52:43 2019
config:

NAME STATE READ WRITE CKSUM
Data21 ONLINE 0 0 0
gptid/081e3b12-1f34-11e9-b141-0cc47a304374 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

-----------------------------------------------------------------------------------------------------------------


it looks to me like there is missing a point in the bottom
the Log line

 

[diedrichg]

Get a list of your disks

Code:

smartctl --scan

See all SMART information for xyz drive, post the output in this thread

Code:

smartctl -a /dev/xyz | more

 

[BlueScreenTT]

root@freenas[/]# smartctl --scan
/dev/ada0 -d atacam # /dev/ada0, ATA device
/dev/da0 -d scsi # /dev/da0, SCSI device
/dev/da1 -d scsi # /dev/da1, SCSI device
/dev/da2 -d scsi # /dev/da2, SCSI device

-------------------------------------------------------------------------------------------------------------------------------------------------

root@freenas[/]# smartctl -a /dev/da0 | more
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital RE4
Device Model: WDC WD1003FBYX-01Y7B1
Serial Number: WD-WCAW34360593
LU WWN Device Id: 5 0014ee 2b221cb62
Firmware Version: 01.01V02
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Jan 26 14:17:31 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (16800) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 165) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 10
3 Spin_Up_Time 0x0027 176 169 021 Pre-fail Always - 4175
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 642
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 12400
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 476
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 172
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 469
194 Temperature_Celsius 0x0022 113 093 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 2

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

----------------------------------------------------------------------------------------------------------------

root@freenas[/]# smartctl -a /dev/da1 | more
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Black
Device Model: WDC WD7501AALS-00J7B1
Serial Number: WD-WMATV1690404
LU WWN Device Id: 5 0014ee 00148d388
Firmware Version: 05.00K05
User Capacity: 750,155,292,160 bytes [750 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.5, 3.0 Gb/s
Local Time is: Sat Jan 26 14:14:47 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (15960) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 185) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 823
3 Spin_Up_Time 0x0027 243 224 021 Pre-fail Always - 7833
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1904
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 024 024 000 Old_age Always - 55665
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 1259
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 326
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1890
194 Temperature_Celsius 0x0022 116 096 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


-------------------------------------------------------------------------------------------------------------------------------------

root@freenas[/]# smartctl -a /dev/da2 | more
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD60EFRX-68MYMN1
Serial Number: WD-WX41DA40LP94
LU WWN Device Id: 5 0014ee 2b612f5c2
Firmware Version: 82.00A82
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5700 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jan 26 14:16:31 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 5564) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 709) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 217 188 021 Pre-fail Always - 8108
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 690
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 5924
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 245
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 97
193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 10930
194 Temperature_Celsius 0x0022 121 102 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 70% 231 -
# 2 Short offline Completed without error 00% 227 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

-------------------------------------------------------------------------------------------------------------------------------------------

 

[BlueScreenTT]

does this error have anything to do with the problem ??


Freenas Kernel: Failed to fully fault in a core file segment at VR 0X8008a4000 with size 0x3000 to be written at offset 0x32E000 for process win bind
Freenas Kernel: Failed to fully fault in a core file segment at VR 0X8008a7000 with size 0xd000 to be written at offset 0x331000 for process win bind

 

[joeschmuck]

Well if taken at face value, you are missing 5 hard drives. Note the hard drive serial numbers you listed above, figure out which ones are missing, your controller is likely the cause. You said you have three controllers for 8 drives, sounds a bit excessive but that is just me. If that is true then move all 8 drives to the controller that does work and remove the faulty controller. I know I'm being a bit simplistic here but that would be the fast way to restore operation.

 

[SweetAndLow]

Also your system needs to start running smart tests on those drives.

 

[BlueScreenTT]

after the first crach i removed all the drives and only attached 3 drives 700gig 1TB and a 6 TB.
i did this because i dont know where the problem is, if it is the controllers, Ram or something else.
i did a fresh install of Freenas and it chrashed again after 2 days i lost all data on the 6TB drive (test files only)

i have a total of
3 X 8TB drives
4 X 2TB drives
8 X 3TB drives
1 X 6TB drive
1 X 1TB drive
1 X 700/750Gig drive
3 X 1 TB drives

the reason i have only 3 drives now when testing is that I want to find the failty hardware before I install all drives again. I dont want to loose any more data

what i have done today is that i removed the 3 X LSI controllers and connected the drives to the onboard SAS and SATA controllers to test if i get the errors again.

also one thing i am wondering about.
does it have anything to do with UEFI or BIOS ?

i checked the logfiles in my BIOS and i got alot of EFI errors

Smbios event log

ERROR Code Severity
EFI 0300000A Minor
EFI 03058003 Major
EFI 03058003 Major
EFI 03005003 Major
EFI 01010003 Major
EFI 03058004 Major
EFI 03058004 Major
EFI 03058003 Major
EFI 03058003 Major

 

[BlueScreenTT]

Also your system needs to start running smart tests on those drives.

how do i set up a schedule for that ?

 

[joeschmuck]

how do i set up a schedule for that ?

Read the user guide, it has a full section on this.

does it have anything to do with UEFI or BIOS ?

Legacy/BIOS version of FreeNAS is likely to be less problematic than UEFI version of the boot loader. I can't say this is your problem.

Have you done any stability testing using MemTest86 or Prime95, or similar programs recently? If not, I'd run these to ensure that your system is rock solid. Also have you tried FreeNAS 11.1-U6? If not then try it out. You could have some kind of incompatibility going on and if that is true then you will need to submit a bug report.

You can't tell us that you have 8 hard drives, provide us data and then tell us that you really have 3 hard drives. Please be clear so we can provide you troubleshooting help without adding confusion. For the rest of this troubleshooting I'd recommend that you leave the LSI controllers removed/disconnected and use just the three test drives that you have. If you are able to make this work for say 2 weeks or longer then move on to the LSI controllers, if you really need them.

Lets go over a few more things you should be doing ro shouldn't be doing while troubleshooting a problem, and I only say this from a over 4 decades of experience, make only one change at a time and then test your system, ensure it's stable before moving forward. This even include little things like making another jail/plugin or any configuration change. Maybe it's stable with a basic setup and you are just sharing via SMB but then when you setup a jail it all fails. But first run the stress tests because right now I have no faith your system is stable until you have run these tests. MemTest86 should run several days, Prime95 should run at least 30 minutes, longer if you like.

 

[BlueScreenTT]

Got it

i will post later when i have some more information

Sorry but i thought i had explained about the discs in an earlier post :-(
again thank you i will give it a try