FreeNas going dead at random times and days.

[JustMichael]

I'm new to using FreeNas and BSD. Though I have been using FreeNas for about 6 months, I haven't done anything but what was needed to get things working. So far I am getting random events of my machine just going completely dead.

I went 4 months without a single issue, then all of a sudden it started dying on me, the machine itself would keep running, but there was no terminal, all my network drives where no longer accessible, the FreeNas UI was unreachable. My VM running my docker instance and all my services where unreachable.

This started happening more frequently, The Harddrives where brand new 8TB drives when I put them in, while I am slowly every 2 months starting to add more drives. Though I only have 2x 8TB right now, and 3 External Harddrives (I already had these from my previous setup before FreeNas) so I am currently moving all the files from those 3 external drives over to the internal ones. (Needs more drives to get the rest off, since one of the externals drives is 10TB).

This machine was an old machine I used to use as a desktop, so the specs are not too great.

CPU: AMD FX-6300
GPU: None/Integrated
RAM: 32GB DDR3
PSU: 650W Corsair
OS: FreeNas (On SSD, I had a spare one I wasn't using and didn't mind using it for now. Will copy to USB at some point)
HDD:
1x 500gb (projects and odd bits) (internal)
1x 3TB (backups of old stuff and VM data) (internal)
1x 2TB (backups of games and videos) (external) (self powered)
5TB (ripped bluerays, and music) (external) (self powered)
8TB (ripped bluerays) (external) (self powered)
8TB (ripped bluerays) (internal)

The drives as far as I can tell are all healthy, I have ran S.M.A.R.T tests on them, with no sign of issues. So I am at a lose here as to what I should do next. I hope I am posting in the right section and not missing any important details. But this is all I know I can help with unless someone can instruct me where I should be looking for additional information.

 

[Chris Moore]

More specifics about the hardware might be nice. Here is a guide:

Forum Guidelines
https://www.ixsystems.com/community/threads/forum-guidelines.45124/

Also, I am not quite sure what you mean by,

my machine just going completely dead.

Is it becoming unresponsive or does it power off? When this happens, is there anything on the screen / do you have a monitor connected to it?

HDD:
1x 500gb (projects and odd bits) (internal)
1x 3TB (backups of old stuff and VM data) (internal)
1x 2TB (backups of games and videos) (external) (self powered)
5TB (ripped bluerays, and music) (external) (self powered)
8TB (ripped bluerays) (external) (self powered)
8TB (ripped bluerays) (internal)

Are these all individual drives with no redundancy? When you say external, self powered, is that a USB drive in an external enclosure?
Did these drives get mounted with their existing file system?
How was this setup, because it doesn't sound at all standard?

 

[JustMichael]

I'm in no way an expert at this, I needed a way to store my data and be accessible across a number of devices on my network. So I pretty much just jumped right into it. When I started I couldn't afford to buy redundancy drive, so I'm probably not doing this correctly. I'm sure I can fix this in the future, but money is tight, so I am doing all I can to make this work as best I can in my situtation.

Is it becoming unresponsive or does it power off? When this happens, is there anything on the screen / do you have a monitor connected to it?

It's becoming unresponsive, like the whole system becomes inaccessible, however the machine itself is still powered on and running.

Are these all individual drives with no redundancy? When you say external, self powered, is that a USB drive in an external enclosure?

The drives I have shown are ones I have had for some time, that I have just put together, some are from other machines, some are from drives I bought for external data storage (so yes enclosures).

All the drives listed here are formatted into ZFS and a pool of their own. (Project Pool, Backup Pool, etc)
Only the 2x 8TB drives are in the same pool.

How was this setup, because it doesn't sound at all standard?

I whipped it all together, following a few guides and just bluffing my way through it. But it has worked for some time now, and I do keep it updated.
it's installed on an SSD at this time.

 

[JustMichael]

I should mention that the 2 8x drives, I bought is me attempting to fix the redundancy issue, I am hoping to buy 1x 8TB drive every 2 months and move all data from all other drives across, so there will only be a single pool.

Then one of the 8Tb drives can be a parity or something that will allow me to restore a drive if it dies.

 

[Chris Moore]

drives I bought for external data storage (so yes enclosures).

USB?

Only the 2x 8TB drives are in the same pool.

With one in a USB enclosure and the other on a internal (SATA?) connection?

it's installed on an SSD at this time.

That is actually a good thing that I wouldn't suggest changing.

I would say that the problem is between the two 8TB drives where one is on USB. If you have internal drive bays for them, it would be best to move all the drives internal because USB can create problems. Long ago, there was even a recommendation to use USB for the boot drive, but we don't suggest that any more as it can cause problems. It does still work, but not advisable. USB has never been suggested for primary storage.

The way you have described the situation with the two 8TB drives leaves me with questions. Is that configured as a stripe where half the data is on each drive?

 

[JustMichael]

USB?

yes

With one in a USB enclosure and the other on a internal (SATA?) connection?

yes, one is internal, and one is external

The way you have described the situation with the two 8TB drives leaves me with questions. Is that configured as a stripe where half the data is on each drive?

yes it is striped, so half the data is on one and the rest is on the other.

As for the external 8TB drive, I can't make this internal due to how Seagate have modified the drive to use their USB output. There are no SATA and Power ports accessible to move it internally. If I get another 8TB drive, is there a way for me to move specifically the data from the external drive to a new drive? that external drive is full for sure by now. Since I only recently bought the second internal 8TB drive to add to the pool.

 

[Chris Moore]

yes it is striped, so half the data is on one and the rest is on the other.

Depending on the model of the Seagate drive, it can be shucked from the external enclosure. Here is a video about it:

How to Remove / Shuck the Hard Drive from Seagate Backup Plus 5TB Drive

In this video, I show you how to shuck the internal SATA HDD as non-destructively as possible from a Seagate Backup Plus 5TB drive.Items in this video:Seagat...

www.youtube.com

The most likely cause of the system lockup is that part of the data is on one disk and the other part is on the USB disk. Due to the way USB has much higher latency than SATA, the system is likely getting into an error condition due to having received part of the data but the other part is not ready. USB drives are notoriously bad. I have seen people try to use laptop computers with all their storage on USB external drives. It can only work when each USB drive is a separate pool. They never deliver the data in time to prevent system errors when there is a USB drive in the same pool with another drive. With the situation you describe, I am surprised you didn't have problems sooner.

 

[JustMichael]

Depending on the model of the Seagate drive, it can be shucked from the external enclosure. Here is a video about it:

You must have missed the part at the bottom, where I stated the drive was modified by Seagate to not have the power/sata ports required for internal use. They are none existent. So it's not possible for me to make them internal. I have already tried to Shuck all 3 of the drives I own, and was only lucky with 1. The newer 8TB model I bought from WD was a Red drive and was very simple to shuck and make internal.

 

[JustMichael]

Thanks for the information, I will now attempt to move all the data over to a new pool with internal drives only. This going to be a long process, but if I can manage to get it done with no loss of data. I will be very happy.

 

[JustMichael]

this is the 8TB external drive. https://www.amazon.co.uk/gp/product/B07DQBFQ2D/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

 

[JustMichael]

Okay my other 5TB external drive was the modified one, so I actually just tried to re-shuck the 8TB one and found I was able to shuck it, it took a while to figure out how to get some screws out due to the way they where hidden. But I have successfully shucked it and am now ready to make it internal.

 

[JustMichael]

Moved it to internal and it has been picked up and it currently working so that's good. Do you have any links for a SATA expansion card? I've run out of ports and I should move the other 2 external drives internally until I move all the data and remove the need for them.

 

[JustMichael]

@Chris Moore After doing all of that, I left the machine on for about 2 hours without touching it, and then it once again become unresponsive. I also tried removing the 2 other external drives, to see if any of them where the issue, but the issue still continued.

I'm not entirely sure why this is now happening. The drives all show to be in good health, and the disk speed tests seems to be writing/reading at good speeds, so I can't imagine why the system is becoming unresponsive.

I don't have any jails running (since I use the VM for docker) and the only services I have is the standard ones and SMB.

 

[Chris Moore]

CPU: AMD FX-6300
GPU: None/Integrated

What model system board is it?

Do you have any links for a SATA expansion card?

This is what I use, because it is the most reliable solution for up to eight drives:

Genuine LSI 6Gbps SAS HBA LSI 9201-8i (=9211-8i) P20 IT Mode ZFS FreeNAS unRAID | eBay

I have fully tested them using all 8 SAS lanes. I've used ZFS for many years. I'm familiar with ZFS on Linux, FreeNAS, and many of the technologies commonly used by other enthusiasts. They have been flashed to Avago / LSI 9211-8i IT (Initiator Target) firmware version P20 (specifically 20.00.07.00).

www.ebay.com

Although it is a SAS controller, it can also be used to run SATA drives with a set of these cables:

2x Internal Mini-SAS SFF-8087 to 4xSATA Forward Breakout Cable Drive Cable US | eBay

Mini SAS SFF-8087 to 4 SATA Cable. 2 x Mini SAS to 4 SATA Cable. - 4 SATA 7 pin female connectors for drives.

www.ebay.com

Sorry for the US links, but I am sure you can find sources for these things in the UK.

 

[JustMichael]

What model system board is it?

It's an ASUS M5A78L-M

Amazon.co.uk

This is what I use, because it is the most reliable solution for up to eight drives:

Ah, thank you. I will find a UK source and definitively buy one.

Weird, I ran it all night and left it on when I went to work, and it's still running fine. It's really difficult to identify the issue here, especially when it seems to be a random event that is breaking it.