Hi All,
I have what appears to be some data corruption, and I'm trying to make sense of it. Server setup first, then some log data.
Server: Dell R720xd with 2x CPU & 384gb RAM (ECC)
Boot: from USB
2 pools -
jails - made up of a single set of mirrored drives
tank - made up of 2 vdevs with 12 drives each, raidz3
vdev0 - 12x3TB Disks locally attached to an HBA inside the server (NOT a raid controller, and these are the disks that are showing up)
vdev1 - 12x2TB Disks attached to an Netapp DS4246 via an X2065A (4 port QSFP HBA) [NOT SHOWING UP]
Yesterday something (power outage? HBA Fault?) occurred which caused a reboot of FreeNAS. The server was booting back up and was paused on one of the HBAs, saying the configuration of the HBA had changed. I did a cold boot and it came back up - still saying that same message however it went past it.
Upon booting into FreeNAS, two things occurred.
1) the GUI was corrupted, the main home page displayed HTML errors, several other pages just flat out failed to load.
2) both pools were in state "UNKNOWN"
After another reboot (swapped SAS ports to alternate ports on my HBA) the pool "Jails" came back online.
tank is still offline. With the command zpool import I get the following output: What sticks out to me is that one of my vdevs is not showing up. The disks show up in the "View Disks" however when I do
Questions
1) Why do disks show up in "View Disks" but not in cmd line?
2) If I trust the cmd line, then something is happening preventing my second vdev from showing up correct?
3) assuming it's just a failed HBA, cable, or IOM on my shelf I should be able to power down, swap the HBA / cable / IOM and bring it back up and be happy?
4) If I have errors on my GUI is it likely that something on my install is also corrupted?
5) where in the logs can I get some better clarity on what might have occurred. I think I neglected proper logging setup so can you point me into a good place to see what needs setup so when I recover or rebuild I'll have some sanity?
I have what appears to be some data corruption, and I'm trying to make sense of it. Server setup first, then some log data.
Server: Dell R720xd with 2x CPU & 384gb RAM (ECC)
Boot: from USB
2 pools -
jails - made up of a single set of mirrored drives
tank - made up of 2 vdevs with 12 drives each, raidz3
vdev0 - 12x3TB Disks locally attached to an HBA inside the server (NOT a raid controller, and these are the disks that are showing up)
vdev1 - 12x2TB Disks attached to an Netapp DS4246 via an X2065A (4 port QSFP HBA) [NOT SHOWING UP]
Yesterday something (power outage? HBA Fault?) occurred which caused a reboot of FreeNAS. The server was booting back up and was paused on one of the HBAs, saying the configuration of the HBA had changed. I did a cold boot and it came back up - still saying that same message however it went past it.
Upon booting into FreeNAS, two things occurred.
1) the GUI was corrupted, the main home page displayed HTML errors, several other pages just flat out failed to load.
2) both pools were in state "UNKNOWN"
After another reboot (swapped SAS ports to alternate ports on my HBA) the pool "Jails" came back online.
tank is still offline. With the command zpool import I get the following output: What sticks out to me is that one of my vdevs is not showing up. The disks show up in the "View Disks" however when I do
"geom disk list"
I only see the first half of the vdev.Code:
root@freenas:/ # zpool import pool: tank id: 907423887126384953 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. config: tank FAULTED corrupted data raidz3-0 ONLINE gptid/e6f85169-5136-11e8-b376-bc305bf48148 ONLINE gptid/f2bd171b-5136-11e8-b376-bc305bf48148 ONLINE gptid/181f4f1a-5137-11e8-b376-bc305bf48148 ONLINE gptid/24da255f-5137-11e8-b376-bc305bf48148 ONLINE gptid/325a8e49-5137-11e8-b376-bc305bf48148 ONLINE gptid/4a4df2c1-5137-11e8-b376-bc305bf48148 ONLINE gptid/6a52ee8d-5137-11e8-b376-bc305bf48148 ONLINE gptid/7bc6a747-5137-11e8-b376-bc305bf48148 ONLINE gptid/8d306d6b-5137-11e8-b376-bc305bf48148 ONLINE gptid/a82f0ce2-5137-11e8-b376-bc305bf48148 ONLINE gptid/bb5df1ae-5137-11e8-b376-bc305bf48148 ONLINE gptid/db7bb774-5137-11e8-b376-bc305bf48148 ONLINE
Questions
1) Why do disks show up in "View Disks" but not in cmd line?
2) If I trust the cmd line, then something is happening preventing my second vdev from showing up correct?
3) assuming it's just a failed HBA, cable, or IOM on my shelf I should be able to power down, swap the HBA / cable / IOM and bring it back up and be happy?
4) If I have errors on my GUI is it likely that something on my install is also corrupted?
5) where in the logs can I get some better clarity on what might have occurred. I think I neglected proper logging setup so can you point me into a good place to see what needs setup so when I recover or rebuild I'll have some sanity?