Existing drives failed when adding new drives

[rusalov]

Hi, I had a strange problem that noone else seems to have.

I have a system that is running 2 x 2tb drives, 1 x 1tb drive and 1 x 500gb drive. (each is an individual pool, so in case one fails, data is lost)

Everything worked fine for months, but today I bought 3 x 1tb drives, to build with them a raidz1. And move data from the 2tb drives into it, and then buy 2 more 2tb drives and also do a raidz1 with them.

Well, the moment I plugged the 3 new drives. All the existing drives failed. Making the system unable to go past the bios logo. After many hours of plugging and unplugging things the system is left like this:

· 1 of the 2tb drives failed completely, it makes a spinning noise, a clack, a system sound and then repeats this behaviour.
· the other 2tb drive works fine
· The 500 tb drive makes spinning and clacking noises but still works.
· The 1tb drive, shows up as a 4tb drive.

The pools on the 2tb, 1tb, and 500gb drive, show as detached.

Could anyone explain what happenned, and if there is a fix to this? Importing the pools even with zpool import -f, doesn´t work.

Thank you.

 

[HoneyBadger]

Hi @rusalov

Sorry to hear you're having problems with your system. Based on the description of the symptoms (clacking noises and drives not being recognized) this sounds like you may have experienced a hardware failure.

Can you please describe the hardware you are using, paying specific attention to the motherboard and power supply, as well as any additional hardware used to power your drives (power splitters, drive-bay adapter cages?)

 

[rusalov]

Hi @rusalov

Sorry to hear you're having problems with your system. Based on the description of the symptoms (clacking noises and drives not being recognized) this sounds like you may have experienced a hardware failure.

Can you please describe the hardware you are using, paying specific attention to the motherboard and power supply, as well as any additional hardware used to power your drives (power splitters, drive-bay adapter cages?)

Hello, yup sorry. Here are my specs:

· MOTHERBOARD Asus B150M-C
· PROCESSOR I5-6500A
· RAM 2x8Gb 2133 mhz
· CASE Thermaltake S31 insonorized
· POWERSUPPLY Mars MP700
· 3 x 120mm fans
· GTX 750 ti
· GeForce 8500 GT

· Power splitters: 2 x Molex to 3 Sata
· 2 x 2Tb
· 4 x 1Tb
· 1 x 500Gb
· 1 x 32 Gb SSD boot drive

I think this is all.

 

[HoneyBadger]

Power splitters: 2 x Molex to 3 Sata

These are likely to be the culprit. If a Molex-to-SATA adapter fails, or does not have proper isolation between the voltage and ground lines, it can cause a short-circuit that may result in permanent damage to the drive, cable, and under certain circumstances result in fire.

Remove the cables and drives, inspecting them for signs of physical or electrical damage such as burn marks. Check each drive one at a time - in a separate system, ideally, at the very least, or if not, then with the Molex-to-SATA cables removed - to ensure that each drive is recognized in your BIOS.

Unfortunately the symptoms you're describing here (clicking noises, being stuck at BIOS, incorrect capacity) are often indicative of hardware failure.

 

[rusalov]

These are likely to be the culprit. If a Molex-to-SATA adapter fails, or does not have proper isolation between the voltage and ground lines, it can cause a short-circuit that may result in permanent damage to the drive, cable, and under certain circumstances result in fire.

Remove the cables and drives, inspecting them for signs of physical or electrical damage such as burn marks. Check each drive one at a time - in a separate system, ideally, at the very least, or if not, then with the Molex-to-SATA cables removed - to ensure that each drive is recognized in your BIOS.

Unfortunately the symptoms you're describing here (clicking noises, being stuck at BIOS, incorrect capacity) are often indicative of hardware failure.

Ok, I will try this on friday (I´m working far from home) to see if they power up with the embedded sata power cable directly from the power supply and check all drives individually.
I hope the drives revive, because theres now 4tb of data thats not accessible anymore. Never imagined that plugging 3 blank drives would send the other drives into oblivion

 

[rusalov]

These are likely to be the culprit. If a Molex-to-SATA adapter fails, or does not have proper isolation between the voltage and ground lines, it can cause a short-circuit that may result in permanent damage to the drive, cable, and under certain circumstances result in fire.

Remove the cables and drives, inspecting them for signs of physical or electrical damage such as burn marks. Check each drive one at a time - in a separate system, ideally, at the very least, or if not, then with the Molex-to-SATA cables removed - to ensure that each drive is recognized in your BIOS.

Unfortunately the symptoms you're describing here (clicking noises, being stuck at BIOS, incorrect capacity) are often indicative of hardware failure.

You were absolutelly right. It swems that at startup, the drives peak starting to spin, and then maybe there is not enough current, or who knows, and drives fail to initialize.

I took off the splitters and all drives work when plugged directly.

Sad thing is that I have to find a psu with 10 sata connectors (8 hdd drives + 2 ssd drives).

Im also having now a ton of problems with the drives that failed when everything was plugged, because they got dettached. And when I import the pools back, it gives an I/O error. I tried "zpool import -f" and it also fails to import.

 

[HoneyBadger]

It's likely that the inrush current of trying to spin up 8 platter drives at once is overwhelming your PSU. A backplane with staggered spin-up might help, but a stronger PSU with sufficient individual SATA connectors will work as well.

Since you've already tried the -F (force) import, you may need to try the more aggressive -X flag. Connect only the drives from your previous pools and attempt to run zpool import -nFX against your pool - if it shows that it will succeed, you can remove the n to cause it to actually import at the provided time.

 

[joeschmuck]

Sad thing is that I have to find a psu with 10 sata connectors (8 hdd drives + 2 ssd drives).

You should be able to locate some good quality Molex to SATA adapters/splitters, I have a few laying around, don't recall what brand. But I agree with @HoneyBadger the PSU may be a bit light for running 10 drives.

Be specific, what are drive models you have for the ten drives? Why do I ask? Pins 1, 2, and 3 are now 3.3Volt lines and a 4 pin Molex to SATA adapter will not have those 3.3V lines. Most 3.5" hard drives will run fine without those three lines, even most SSD, but some 2.5" laptop drives will only run on 3.3V. While none of this may be impacting you, it's worth looking at to rule it out. If this is the issue, you might be able to move around the SATA power connectors on the splitters to drives that will run fine. Again, this might not be the issue but it should be examined if you think it could be the problem.

Also, You could fit 5 hard drives into a 3 bay area and using one or two power connectors. Here is an example.

 

[rusalov]

It's likely that the inrush current of trying to spin up 8 platter drives at once is overwhelming your PSU. A backplane with staggered spin-up might help, but a stronger PSU with sufficient individual SATA connectors will work as well.

Since you've already tried the -F (force) import, you may need to try the more aggressive -X flag. Connect only the drives from your previous pools and attempt to run zpool import -nFX against your pool - if it shows that it will succeed, you can remove the n to cause it to actually import at the provided time.

It worked! Thank you!!
The zpool import -FX worked like a charm. I imported one of the 3 faulty drives so far, the 1TB drive. It took about 3 hours to process. After that it showed in datasets but not in storage, and it kept throwing "unable to access group" errors, so I rebooted the system and it dissapeared again. Then I tried to import it again via the GUI and it worked! All files are untouched and accessible, it´s scrubing now.
Also i´m going now for the 2TB drive, that I expect to take 6 hours or so to complete.

 

[rusalov]

You should be able to locate some good quality Molex to SATA adapters/splitters, I have a few laying around, don't recall what brand. But I agree with @HoneyBadger the PSU may be a bit light for running 10 drives.

Be specific, what are drive models you have for the ten drives? Why do I ask? Pins 1, 2, and 3 are now 3.3Volt lines and a 4 pin Molex to SATA adapter will not have those 3.3V lines. Most 3.5" hard drives will run fine without those three lines, even most SSD, but some 2.5" laptop drives will only run on 3.3V. While none of this may be impacting you, it's worth looking at to rule it out. If this is the issue, you might be able to move around the SATA power connectors on the splitters to drives that will run fine. Again, this might not be the issue but it should be examined if you think it could be the problem.

Also, You could fit 5 hard drives into a 3 bay area and using one or two power connectors. Here is an example.

I tried to plug just 1 molex to x2 SATA splitter, aaaaaaand... drives started to crash again . However I don´t want to risk it, so i´m searching for a PSU that can withhold 10 drives + the system.

So, the 2x2TB HDD + 1x1TB HDD + 1x32GB SSD are plugged in to the main SATA connectors.

The other drives that would go with the Molex to SATA splitters are:
· 2x Western Digital - WD10EZEX 1TB
· 1x Seagate Barracuda ST1000DM010 1TB
· 1x Seagate Barracuda ST500DM002 500Gb

They all show that they use 5V and 12V, and noone has 3.3V. So... the splitters should have worked, but as soon as I plug only 1 drive, everything starts to fail.

In theory I calculated the Watts I´m using, and i´m between 500W - 600W. And my PSU is 750W. So, no idea why it is failing.

 

[NugentS]

Are those SMR drives - you probably shouldn't be using them if they are?

 

[rusalov]

Are those SMR drives - you probably shouldn't be using them if they are?

Does them failing have anything to do with them being SMR or CMR?

And I think some are CMR and some SMR. It´s the ones I had. This is my first server, and oh boy it has been a journey of learning. I am using the hardware I have, and I can´t toss 2000€-5000€ and get everything brand new.

My short term goal is to have a functioning server with 4x2TB + 4x1TB drives in a raid5 configuration.

 

[NugentS]

ZFS and SMR do not play well. Drives can and will be ejected from the pool because they don't respond in a timely manner.
Basically - don't use SMR with ZFS - its a recipe for disaster

 

[rusalov]

ZFS and SMR do not play well. Drives can and will be ejected from the pool because they don't respond in a timely manner.
Basically - don't use SMR with ZFS - its a recipe for disaster

Okay, thanks for the advice. I will focus for CMR drives from now on. In the mean time I will use the ones I already have. In the future I will upgrade the 4x1TB pool to a 4x4TB pool and will get NAS specific drives.

 

[joeschmuck]

They all show that they use 5V and 12V, and noone has 3.3V. So... the splitters should have worked, but as soon as I plug only 1 drive, everything starts to fail.

That would indicate to me that the splitter is faulty. While I understand the desire to locate a power supply with 10 SATA power connections, it may be difficult. You can try a different power splitter make/model to see if it works. I'm just curious if the Molex connector pins are in the correct location. See attached diagram, see if this matches. I've seen the red and yellow connections reversed before in Molex connectors. Maybe that is your issue? I hope so because swapping the pins are easy, or you can buy new splitters. But of course the issue could be a weak power supply.

 

[rusalov]

That would indicate to me that the splitter is faulty. While I understand the desire to locate a power supply with 10 SATA power connections, it may be difficult. You can try a different power splitter make/model to see if it works. I'm just curious if the Molex connector pins are in the correct location. See attached diagram, see if this matches. I've seen the red and yellow connections reversed before in Molex connectors. Maybe that is your issue? I hope so because swapping the pins are easy, or you can buy new splitters. But of course the issue could be a weak power supply.

View attachment 70417

Well, it ended up being a weak power supply. Even tho its rated at 700W (Mars MP700) somewhy it lacked enough power to spin all disks at once. I guess because it was manufactured around 2014, its very old.

I just bought and installed a second hand Mars Gaming Vulcano 750 W 80 plus Silver Semi Modular, it comes with 7 sata connectors and 4 molex. But you can swap the cables and plug into the 6-pin a cable with 3 sata connectors, so you can end up with 12 SATA. And everything works perfectly now.

All disks spin without a problem. And I still have room for some more.

Thank you all for your help. If it wasn´t for you I would have lost all data, and wouldn´t learn what was wrong, how to fix things, and how to avoid this problems next time.

 

[joeschmuck]

Glad you solved it.