Drives disappearing when making new volume

[KillerJupe]

I have a new clean install and I'm making a new volume with 10 disks.

I go to add all to the volume in a 5x2 array. After a few minutes it fails to create the volume and when i go back to the disk page, some disk are now not showing.
where before i saw disks 0-9 i now see 0,1,2,4,5,6,7,9.

I can reset the system, see all the disks and then try again and different disks might disappear.

I've tried two HBA's, differnt cables, and a sas expanded.

I'm thinking its these drives: ST4000LM016
https://www.amazon.com/dp/B016YFEYK6/?tag=ozlp-20

Any thoughts to see what is happening?

 

[DrKK]

That's a new one on me. Can we get a full report on precisely what hardware (CPU, mobo, RAM, HBA, PSU model, etc) that you're using?

 

[KillerJupe]

current version of FN running on a server directly from ix Systems.
Even they don't know whats going on.

SM- x9dri-f
2 xeon E5-2630
64gb ECC ddr3 ram
LSI 9211-81 x2 in IT mode (new fw/bios)
4 to intel min-isas slot on MB

20 ST4000LM016 hard drives directly wired.
Also tried going directly from the SAS2-216EL2 backplane to the LSI 9211.
They drives would disappear just like they do when wired directly.

I have two identical servers, but only one set of HD's. I've moved them from one server to the other w/ the same results.

I'm totally stumped and can only suspect the drives.
Can i pull FN logs to see whats happening, sorry I don't really know FN that well.
Thanks,
D

 

[DrKK]

INteresting.

These are, in fact, 2.5" (laptop) sized drives? That's mostly academic curiosity.

I suppose something could be...wrong? with the drives such that in this type of use-case something isn't right? But that seems like a stretch.

HBA's are frequent sources of problems. Can you directly wire some drives TO THE ACTUAL SATA PORTS OF THE MOTHERBOARD (I realize you won't have enough to do all 10), remove the HBA's, and see if you experience the problem? I think there's a good chance you do NOT experience the problem when you are bypassing the entire HBA. Let's at least rule that out.

But then it starts getting hard to find things to try. Maybe someone else will have other thoughts? @anodos @SweetAndLow @Ericloewe ?

For my part, my next two troubleshooting steps, after I confirm the HBA is not involved, would be to absolutely rule out a corrupted operating system (whatever my boot device is, I'd use a brand new boot device, with a brand new FreeNAS install). If that's not the problem, then I rule out a PSU with flagging voltages on the rails due to either defect or being over the capacity. You said you seem to have the problem precisely when all ten of the drives spin up and are put into maximum-load use (e.g., trying to format), so something like the power supply losing its rail cannot be ruled out.

 

[KillerJupe]

They are 15mm 2.5" drives running 5400 rpm, but yes, they are designed for low usage

I don't have any sata cables to directly wire 6 drives up with.
I tried removing the HBA's and using only the onboard Intel mini sas port. Same issue if i format a few times, one drive will fail at random.
I know its not as good as directly wired, but I figure a different vendor should give us a good idea.

I don't suspect the PSU because I pulled 22 10k sas drives to put these sata drives in. Each server has dual PSU's.
The 10k's use more power and never had an issue when being used hard. Also remember I tried in an identical server, with the same issues and i can't believe both psu's are failing.

I'm booting freenas of a 16gb sandisk usb, each server has its own clean install on different usb's hooked into the internal USB plug.
No changes, othe than setting an ip, were made to freenass prior to formatting.

I'll try re-downloading and reinstalling on a usb 2.0 device

 

[wblock]

Is it the same drives that disappear each time?

 

[KillerJupe]

I wish :/
Sometimes it will be drive 0, sometimes 7, or 12 or multiple ones together,... seems totally at random.

This time w/ the new USB and clean install 2 drives disappeared from the drive list: 7 and 11,

 

[KillerJupe]

If i now format the remaining disks, I am able to make a successful volume.
Out of 20 disks, do I have like 8 bad disks... maybe... but if i have that many, i honestly don't want any of these things.

I want to know what is happening to these missing disks.
Does freenas log why it "deleted" a disk from the list?

I wish :/
Sometimes it will be drive 0, sometimes 7, or 12 or multiple ones together,... seems totally at random.

This time w/ the new USB and clean install 2 drives disappeared from the drive list: 7 and 11,

 

[Ericloewe]

If i now format the remaining disks, I am able to make a successful volume.
Out of 20 disks, do I have like 8 bad disks... maybe... but if i have that many, i honestly don't want any of these things.

I want to know what is happening to these missing disks.
Does freenas log why it "deleted" a disk from the list?

Can we get a recap of what you've tried so far and what results you've obtained?

 

[BigDave]

I don't have any sata cables to directly wire 6 drives up with.

Are these new drives, or used? Have you run (badblocks) and burned them in,
any smart testing?

You can at least check them individually on another puter with Seagate's Seatools software.

Can we get a recap of what you've tried so far and what results you've obtained?

+1

 

[KillerJupe]

Sure.

All FW/BIOS update on mainboard and LSI.

All 20 drives in backplane, to LSI 9260 raid = All drives seen, random drives fail on format, sometimes the aray works for a bit then fails.
4 drives hooked directly to intel mini sas = random failures, sometimes it works for a bit then fails
8 drives hooked directly to LSI 9211-8i IT mode = All drives seen, random drives fail on format,
12 drives hooked to Intel mini sas and LSI 9211 = All drives seen, random drives fail on format,

I haven't put the last 9211 into the system as I haven't been able to figure out what the problem is. final design is for 20 drives.

Clean install, new usb, new download = same results as above
Totally different, but identical server = same results as above
Swearing and threats = All drives seen, random drives fail on format
Prayer = unanswered
Replaced sas expander and minisas to backplane cables w/ shortest that would fit 19" = no change

Samsung, supermicro, LSI all say everything should work...

I just tried using the wizard this time to see what happens and let it make all the decisions.
Failed w/ 4 disks down.... some of these disks were working in a previous try.

Can we get a recap of what you've tried so far and what results you've obtained?

 

[KillerJupe]

All new drives from SHI:
I haven't run any tests on the drives, but smart says they are all good.
I'd understand if the same drives kept failing, but they vary and I think each drive has indicated a fault at this point.
20/20 bad seems pretty unlikely.
But so does everything else... I still suspect Samsung has a FW problem.
I'm not the only idiot who tried this:


Is there a way to get an ouput to see where the missing drives when?
Before trying to create a 12 disk volume all 12 disks were showing up.
Am I being an idiot and doing something super wrong?

http://www.servethehome.com/seagate-4tb-2-5-sata-hdd-124-get-ready-density/
At least I can return mine!
D

Are these new drives, or used? Have you run (badblocks) and burned them in,
any smart testing?

You can at least check them individually on another puter with Seagate's Seatools software.


+1

 

[BigDave]

Try the ># glabel status< command from SSH,
it shows gptid and da/ada labels for all disks, including spares.
Then using View Disks as you posted above will give up the
serial numbers.

FreeNAS can sometimes change the ada/da
labels of drives between system boots, this becomes more likly
to occur when changing data cables/hot swap bays/SATA ports.

 

[BigDave]

I haven't run any tests on the drives, but smart says they are all good.

SMART testing is not a definitive way to determine drive health/condition/fitness for NAS use, etc.

 

[Ericloewe]

Sure.

All FW/BIOS update on mainboard and LSI.

All 20 drives in backplane, to LSI 9260 raid = All drives seen, random drives fail on format, sometimes the aray works for a bit then fails.
4 drives hooked directly to intel mini sas = random failures, sometimes it works for a bit then fails
8 drives hooked directly to LSI 9211-8i IT mode = All drives seen, random drives fail on format,
12 drives hooked to Intel mini sas and LSI 9211 = All drives seen, random drives fail on format,

I haven't put the last 9211 into the system as I haven't been able to figure out what the problem is. final design is for 20 drives.

Clean install, new usb, new download = same results as above
Totally different, but identical server = same results as above
Swearing and threats = All drives seen, random drives fail on format
Prayer = unanswered
Replaced sas expander and minisas to backplane cables w/ shortest that would fit 19" = no change

Samsung, supermicro, LSI all say everything should work...

I just tried using the wizard this time to see what happens and let it make all the decisions.
Failed w/ 4 disks down.... some of these disks were working in a previous try.

So, it's either a curse (ensure your workspace was not built on an ancient Indian burial ground) or a seriously nasty batch of drives.

Try running badblocks on these drives and let's see what happens.

 

[danb35]

ensure your workspace was not built on an ancient Indian burial ground

I expect you'll recall recommending an exorcist for me--sometimes weird stuff happens for no readily-apparent reason.