Howdy all!
Just recently moved from an older Supermicro X9 build (with one of the older 16 bay chassis) with an LSI2008 onboard SAS and an IBM M1015 HBA to a SuperStorage 5049P-E1CTR36L. I don't recall seeing any of these errors before (had prior system for 4+ years) and the pool the errors came on I've had for 2ish.
I migrated the pools from old to new system without issue. Everything is currently online and I do not believe at the time these errors occurred I was stressing the pool particularly. These disks are used an I will be eventually moving off of them for SSD instead. So far they have been serving as a pool for a bunch of VM's that I have been testing. It's been performing OK and doing its job but will eventually be retired (or kept as empty storage/spared for other things).
As mentioned, this particular pool is serving two ZVOL's via iSCSI. I ran the exact setup on the older box and didn't see any errors like this before. I would see some noise previously, but only if I was stressing the pool which did not suprise me due to the poor disk layout of this pool.
Attached Hypervisors are all VMware 6.7 U2. Setup has been rock solid the last few years no issues. Again disks are used, and they had a pretty high Load_Cycle_Count (see below). Perhaps those are the culprit
Here is the current Hardware list:
SuperStorage 5049P-E1CTR36L
X11SPH-nCTF
Onboard SAS LSI 3008 flashed to IT mode
128GB Samsung DDR4 2666MHz
Intel X520 (82599ES)
Intel Optane 900p (480GB) - SLOG
HBA Driver/Firmware
Disk List:
Entire Pool LIst, "vmware" is the pool in question:
Just recently moved from an older Supermicro X9 build (with one of the older 16 bay chassis) with an LSI2008 onboard SAS and an IBM M1015 HBA to a SuperStorage 5049P-E1CTR36L. I don't recall seeing any of these errors before (had prior system for 4+ years) and the pool the errors came on I've had for 2ish.
I migrated the pools from old to new system without issue. Everything is currently online and I do not believe at the time these errors occurred I was stressing the pool particularly. These disks are used an I will be eventually moving off of them for SSD instead. So far they have been serving as a pool for a bunch of VM's that I have been testing. It's been performing OK and doing its job but will eventually be retired (or kept as empty storage/spared for other things).
As mentioned, this particular pool is serving two ZVOL's via iSCSI. I ran the exact setup on the older box and didn't see any errors like this before. I would see some noise previously, but only if I was stressing the pool which did not suprise me due to the poor disk layout of this pool.
Attached Hypervisors are all VMware 6.7 U2. Setup has been rock solid the last few years no issues. Again disks are used, and they had a pretty high Load_Cycle_Count (see below). Perhaps those are the culprit
Here is the current Hardware list:
SuperStorage 5049P-E1CTR36L
X11SPH-nCTF
Onboard SAS LSI 3008 flashed to IT mode
128GB Samsung DDR4 2666MHz
Intel X520 (82599ES)
Intel Optane 900p (480GB) - SLOG
HBA Driver/Firmware
Code:
# sas3flash -list Avago Technologies SAS3 Flash Utility Version 16.00.00.00 (2017.05.02) Copyright 2008-2017 Avago Technologies. All rights reserved. Adapter Selected is a Avago SAS: SAS3008(C0) Controller Number : 0 Controller : SAS3008(C0) PCI Address : 00:19:00:00 SAS Address : 5003048-0-1cd2-dc03 NVDATA Version (Default) : 0e.01.30.28 NVDATA Version (Persistent) : 0e.01.30.28 Firmware Product ID : 0x2221 (IT) Firmware Version : 16.00.01.00 NVDATA Vendor : LSI NVDATA Product ID : LSI3008-IT BIOS Version : 08.37.00.00 UEFI BSD Version : 18.00.00.00 FCODE Version : N/A Board Name : LSI3008-IT Board Assembly : N/A Board Tracer Number : N/A Finished Processing Commands Successfully. Exiting SAS3Flash.
Disk List:
Code:
# camcontrol devlist -v scbus0 on ahcich0 bus 0: <SuperMicro SSD SOB20R> at scbus0 target 0 lun 0 (pass0,ada0) <> at scbus0 target -1 lun ffffffff () scbus1 on ahcich1 bus 0: <SuperMicro SSD SOB20R> at scbus1 target 0 lun 0 (pass1,ada1) <> at scbus1 target -1 lun ffffffff () scbus2 on ahcich2 bus 0: <> at scbus2 target -1 lun ffffffff () scbus3 on ahcich3 bus 0: <> at scbus3 target -1 lun ffffffff () scbus4 on ahcich4 bus 0: <> at scbus4 target -1 lun ffffffff () scbus5 on ahcich5 bus 0: <> at scbus5 target -1 lun ffffffff () scbus6 on ahcich6 bus 0: <> at scbus6 target -1 lun ffffffff () scbus7 on ahcich7 bus 0: <> at scbus7 target -1 lun ffffffff () scbus8 on ahcich8 bus 0: <> at scbus8 target -1 lun ffffffff () scbus9 on ahcich9 bus 0: <> at scbus9 target -1 lun ffffffff () scbus10 on mpr0 bus 0: <ATA CT2000MX500SSD1 023> at scbus10 target 11 lun 0 (da18,pass22) <ATA CT2000MX500SSD1 023> at scbus10 target 12 lun 0 (da14,pass18) <ATA CT2000MX500SSD1 023> at scbus10 target 13 lun 0 (da10,pass14) <ATA CT2000MX500SSD1 023> at scbus10 target 17 lun 0 (da19,pass23) <ATA CT2000MX500SSD1 023> at scbus10 target 18 lun 0 (da15,pass19) <ATA CT2000MX500SSD1 023> at scbus10 target 19 lun 0 (da11,pass15) <ATA CT2000MX500SSD1 023> at scbus10 target 23 lun 0 (da20,pass24) <ATA CT2000MX500SSD1 023> at scbus10 target 24 lun 0 (da16,pass20) <ATA CT2000MX500SSD1 023> at scbus10 target 25 lun 0 (da12,pass16) <ATA CT2000MX500SSD1 023> at scbus10 target 29 lun 0 (da21,pass25) <ATA CT2000MX500SSD1 023> at scbus10 target 30 lun 0 (da17,pass21) <ATA CT2000MX500SSD1 023> at scbus10 target 31 lun 0 (da13,pass17) <SMC SC846P 0c1f> at scbus10 target 32 lun 0 (pass2,ses0) <ATA WDC WD60EFRX-68M 0A82> at scbus10 target 51 lun 0 (pass3,da0) <ATA WDC WD40EFRX-68W 0A80> at scbus10 target 52 lun 0 (pass4,da1) <ATA WDC WD40EFRX-68W 0A80> at scbus10 target 53 lun 0 (pass5,da2) <ATA WDC WD60EFRX-68M 0A82> at scbus10 target 54 lun 0 (pass6,da3) <ATA WDC WD40EFRX-68W 0A82> at scbus10 target 55 lun 0 (pass7,da4) <ATA WDC WD40EFRX-68W 0A80> at scbus10 target 56 lun 0 (pass8,da5) <ATA WDC WD60EFRX-68M 0A82> at scbus10 target 58 lun 0 (pass9,da6) <ATA WDC WD40EFRX-68W 0A80> at scbus10 target 59 lun 0 (pass10,da7) <ATA WDC WD60EFRX-68M 0A82> at scbus10 target 61 lun 0 (pass11,da8) <ATA WDC WD40EFRX-68W 0A80> at scbus10 target 62 lun 0 (pass12,da9) <SMC SC826P 0c1f> at scbus10 target 63 lun 0 (pass13,ses1) <> at scbus10 target -1 lun ffffffff () scbus11 on camsim0 bus 0: <> at scbus11 target -1 lun ffffffff () scbus-1 on xpt0 bus 0: <> at scbus-1 target -1 lun ffffffff (xpt0)
Entire Pool LIst, "vmware" is the pool in question:
Code:
# zpool status pool: bucket state: ONLINE scan: scrub repaired 0 in 0 days 00:14:29 with 0 errors on Sun Sep 1 00:14:30 2019 config: NAME STATE READ WRITE CKSUM bucket ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/1a7255f7-a768-11e4-bb0d-0cc47a17532e ONLINE 0 0 0 gptid/1ad46006-a768-11e4-bb0d-0cc47a17532e ONLINE 0 0 0 gptid/1b37e43e-a768-11e4-bb0d-0cc47a17532e ONLINE 0 0 0 gptid/1bab5d4e-a768-11e4-bb0d-0cc47a17532e ONLINE 0 0 0 gptid/1c1432aa-a768-11e4-bb0d-0cc47a17532e ONLINE 0 0 0 gptid/1c76ebdc-a768-11e4-bb0d-0cc47a17532e ONLINE 0 0 0 errors: No known data errors pool: freenas-boot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 ada1p2 ONLINE 0 0 0 errors: No known data errors pool: vmware state: ONLINE scan: resilvered 1.27M in 0 days 00:00:01 with 0 errors on Thu Sep 5 22:09:47 2019 config: NAME STATE READ WRITE CKSUM vmware ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/a804ef0f-ff69-11e8-a676-0cc47a17532e ONLINE 0 0 0 gptid/a91db902-ff69-11e8-a676-0cc47a17532e ONLINE 0 0 0 gptid/aa30e266-ff69-11e8-a676-0cc47a17532e ONLINE 0 0 0 gptid/acd2fef9-ff69-11e8-a676-0cc47a17532e ONLINE 0 0 0 logs nvd0p1 ONLINE 0 0 0 errors: No known data errors pool: vmware-ssd state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM vmware-ssd ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/7e389fbd-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 gptid/7eb44778-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gptid/7f35d35e-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 gptid/7fb54f65-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 gptid/804952b3-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 gptid/80cc65c2-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 gptid/814bc69b-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 gptid/81d0e894-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 gptid/8259d4ec-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 gptid/82e05b85-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 gptid/8379784c-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 gptid/83ffc9d9-d99a-11e9-a453-ac1f6bbba0e8 ONLINE 0 0 0 logs nvd0p2 ONLINE 0 0 0 errors: No known data errors