Greetings folks,
Three questions, all directly related.
Question 1: I have an MSI E350IA-E44 motherboard in a Lian Li PC-Q25 case running Seagate NAS drives all connected to the backplane included in the case and am having an issue with SATA hot plug support. I have the SATA ports set to AHCI in the BIOS.
What seems to happen is when I attempt to add/remove a drive in one bay one of the other drives gets detached even though the new drive is recognized. The motherboard has 4 SATA 6Gbps ports and in the example that follows I had drives in bays 0, 2, and 3 (herein bays 0-3 refer to SATA ports 0-3 as well) all working fine. I then inserted a fresh drive into bay 1 and this is what I saw:
So it detached my perfectly fine ada0 and then added the new ada3 drive. After this camcontrol devlist, smartctl, and geom commands all showed all 4 drives fine but zpool status showed ada0 as REMOVED. I tried offlining/onlining ada0 to no avail, no matter what I did it wouldn't come back online.
I then tried re-seating ada0 (remember, this is bay 0 and was working just fine prior to insertion of a fresh drive into bay 1) and received this:
I did some reading and found it may simply mean ada0 was in use by a process (perhaps ZFS itself) so I offlined the drive (so it showed OFFLINE instead of REMOVED) and then tried try re-seating the drive again and this time not only did the above error repeat but another one of my previously working drives detached!
At that point my zpool with a single RAIDZ2 vdev went offline since it was already down to 3 disks at the start of all this, then the working-fine ada0 went into REMOVED during the first drive insertion above so I was down to two drives, and then upon re-seat another working drive failed leaving the pool/vdev with a single working drive so yeah, duh, it failed.
So next I rebooted (both cold and warm starts) and tried checking what was showing up in BIOS. No matter what I did I couldn't get any drive to show up in BIOS in bay 1 (the original bay I inserted a fresh drive into above). I tried multiple drives in bay 1 and none of them worked but they worked fine in bay 0.
Finally I decided to try completely disconnecting the ATX power supply (remember I already tried both cold and warm restarts) and wiggling the SATA data cables and then finally drives were recognized in bay 1 again. When I booted up all three of my working drives came online and I was able to start the resilvering process with the fourth brand new drive in bay 1.
My gut feeling is something somewhere in my HW build doesn't support hot plug SATA properly, but it seems odd that the new drive is recognized when it's inserted but at times (not every time, though) upon a drive insertion in one bay another drive will get detached (but still show up in OS commands as working just fine, except ZFS will refuse to use it). The drives support hot plug, the case backplane supports hot plug (it's just a simple backplane offering a physical connection mechanism, no electronics involved), and the chipset (AMD Hudson M1 aka A50M FCH) supports hot plug. The only thing I can think of is that the motherboard specification doesn't list hot plug support so I'm wondering if it's possible to have a chipset that supports hot plug but a motherboard that doesn't implement it. From the AMD manual in the description of the pin-outs:
What do you guys think?
Question 2: An amazing note: any drive I insert via hot plug gets recognized as a 600MB/sec drive whereas if you refer to my other recent thread the very same drives get recognized as 300MB/sec drives upon a cold/warm boot and stay that way forever. Is this indicative of an issue with FreeBSD or my HW do you think?
Question 3: Also, here's a related question. If I have drives in bays 0-3 they show up as ada0-3 in FreeNAS. If I shut down, remove say the drive from bay 1, then boot into FreeNAS again instead of the drives showing up as ada0 (then skipping ada1) ada2, ada3 they show up as: ada0, ada1, ada2. Is this normal? I guess the GUID and/or some other ZFS drive identifier works around this but just wanted to ask in case it's related to my issue above.
Thanks, folks.
Three questions, all directly related.
Question 1: I have an MSI E350IA-E44 motherboard in a Lian Li PC-Q25 case running Seagate NAS drives all connected to the backplane included in the case and am having an issue with SATA hot plug support. I have the SATA ports set to AHCI in the BIOS.
What seems to happen is when I attempt to add/remove a drive in one bay one of the other drives gets detached even though the new drive is recognized. The motherboard has 4 SATA 6Gbps ports and in the example that follows I had drives in bays 0, 2, and 3 (herein bays 0-3 refer to SATA ports 0-3 as well) all working fine. I then inserted a fresh drive into bay 1 and this is what I saw:
Code:
[root@freenas] ~# dmesg ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <ST3000DM001-9YN166 CC4H> s/n Z1F0KKMH detached ada3 at ahcich1 bus 0 scbus1 target 0 lun 0 ada3: <ST3000VN000-1H4167 SC43> ATA-9 SATA 3.x device ada3: Serial Number Z3101Y7C ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C) ada3: Previously was known as ad6
So it detached my perfectly fine ada0 and then added the new ada3 drive. After this camcontrol devlist, smartctl, and geom commands all showed all 4 drives fine but zpool status showed ada0 as REMOVED. I tried offlining/onlining ada0 to no avail, no matter what I did it wouldn't come back online.
I then tried re-seating ada0 (remember, this is bay 0 and was working just fine prior to insertion of a fresh drive into bay 1) and received this:
Code:
cam_periph_alloc: attempt to re-allocate valid device ada0 rejected flags 0x118 refcount 2 adaasync: Unable to attach to new device due to status 0x6
I did some reading and found it may simply mean ada0 was in use by a process (perhaps ZFS itself) so I offlined the drive (so it showed OFFLINE instead of REMOVED) and then tried try re-seating the drive again and this time not only did the above error repeat but another one of my previously working drives detached!
At that point my zpool with a single RAIDZ2 vdev went offline since it was already down to 3 disks at the start of all this, then the working-fine ada0 went into REMOVED during the first drive insertion above so I was down to two drives, and then upon re-seat another working drive failed leaving the pool/vdev with a single working drive so yeah, duh, it failed.
So next I rebooted (both cold and warm starts) and tried checking what was showing up in BIOS. No matter what I did I couldn't get any drive to show up in BIOS in bay 1 (the original bay I inserted a fresh drive into above). I tried multiple drives in bay 1 and none of them worked but they worked fine in bay 0.
Finally I decided to try completely disconnecting the ATX power supply (remember I already tried both cold and warm restarts) and wiggling the SATA data cables and then finally drives were recognized in bay 1 again. When I booted up all three of my working drives came online and I was able to start the resilvering process with the fourth brand new drive in bay 1.
My gut feeling is something somewhere in my HW build doesn't support hot plug SATA properly, but it seems odd that the new drive is recognized when it's inserted but at times (not every time, though) upon a drive insertion in one bay another drive will get detached (but still show up in OS commands as working just fine, except ZFS will refuse to use it). The drives support hot plug, the case backplane supports hot plug (it's just a simple backplane offering a physical connection mechanism, no electronics involved), and the chipset (AMD Hudson M1 aka A50M FCH) supports hot plug. The only thing I can think of is that the motherboard specification doesn't list hot plug support so I'm wondering if it's possible to have a chipset that supports hot plug but a motherboard that doesn't implement it. From the AMD manual in the description of the pin-outs:
Note: Each port has a pin (SATA_IS) for sensing the status of the external interlock switch. If the motherboard implements SATA interlock switches, it should connect the statuses of the switches to those pins. The FCH can sense such statuses and, when they change, generate a PME or interrupt. Normally, an inter-lock switch is required for supporting hot plug.
What do you guys think?
Question 2: An amazing note: any drive I insert via hot plug gets recognized as a 600MB/sec drive whereas if you refer to my other recent thread the very same drives get recognized as 300MB/sec drives upon a cold/warm boot and stay that way forever. Is this indicative of an issue with FreeBSD or my HW do you think?
Question 3: Also, here's a related question. If I have drives in bays 0-3 they show up as ada0-3 in FreeNAS. If I shut down, remove say the drive from bay 1, then boot into FreeNAS again instead of the drives showing up as ada0 (then skipping ada1) ada2, ada3 they show up as: ada0, ada1, ada2. Is this normal? I guess the GUID and/or some other ZFS drive identifier works around this but just wanted to ask in case it's related to my issue above.
Thanks, folks.