Disk Fail Simulation fails -> Pool completely gone

[ECC]

I wanted to test, how my TrueNAS Server handels a spontaneous disk failure. For this scenario, I created a mirrored pool from 2 USB HDDs. I copied a file via SMB Share to the server. During the transfer, I unplugged one of the drives.
What happened?
The file transfer stopped after a few seconds, but TrueNAS GUI did not report any failure or whatsoever. It just showed everything is normal, pool status was ok. After a few minutes, the GUI just froze: I was not able to open Dashboard or the storage tab. Well, wtf? Is this supposed to be normal?

My setup is in my signature.

Edit: yeah, the server totaly crashed after i tried to reboot. The server still runs but i don't have any access via browser
Edit2: I had to force to shutdown the server (pressing 5 sec the power button) to be able to start the server again. After that, TrueNAS showed that my mirrored test pool was completely gone...

 

[Chris Moore]

You can't rely on the behavior of USB drives. Sometimes when a USB drive is pulled while a transfer is in progress, it will reset the entire USB bus. That is why we don't even suggest them for boot drives any more, even though we used to, and you certainly can't rely on them for storage pool. You won't find that in the suggested hardware. You can't even rely on two USB drives to transfer data correctly when they are connected. Often times a pool on USB will become corrupted even when there is no disconnect. The drives don't work simultaneously. USB transfers data to one device at a time. You just can't do that. Been tried. It fails. Never heard of it working properly.

 

[ECC]

You can't rely on the behavior of USB drives.

OK that is sad, because my plan was to create USB Backup HDDs for easier cold/hot storage handling of my backups. As a result, I'll need 2 extra mobile backup servers instead, which have to be transported on a regular basis... Is there an easier option, like esata cases or something like this?

Today I tested it with unplugging sata drives. Fortunately, it worked as expected.

 

[ChrisRJ]

Why not connect the USB drive to a regular workstation (Windows or Linux) and copy the data via the client OS. Together with a proper verify operation that should work.

 

[Redcoat]

Forum member @Arwen has a Resouce covering backup that may help you https://www.truenas.com/community/resources/how-to-backup-to-local-disks.26/

 

[Arwen]

I have had generally good luck using ZFS on Linux, with USB attached backup drives. The source disk(s) are either SATA or NVMe. The backup drives are single drive ZFS pools, so it's not like the original post, where 2 drives were used in a mirror configuration.

Some of the enclosures support eSATA as well as USB. And 2 of the 4 computers I backup regularly can use eSATA connections. That seems to work as well.

But, these single drive backup disks are only connected as long as it takes for the initial scrub and backup. (They get scrubbed every backup, before the backup, because each disk is used once a month or less frequently.)



That does remind me to update the resource to include 2 new external single disk chassis that I now have.

 

[Chris Moore]

I have had generally good luck using ZFS on Linux, with USB attached backup drives

I have seen no problem with single drives on USB other than the occasional loss of connectivity. I have seen several instances where a pool connected by USB could not remain consistent. Data errors accumulated until the pool failed, or a loss of connectivity to a drive or two caused instant death of the pool. It usualy happens when the USB bus is overloaded, meaning that the pool is transfering a load of data.

 

[Arwen]

@Chris Moore, Yes, based on what we have seen, I'd guess that USB simply is less reliable that many other protocols. For example, I have seen minutes long network outages and all my connections survive. So I would guess that USB does not have many retries, does not have good error recovery procedures and just gives up too early for those of us wanting to use USB for disks.

 

[ECC]

Why not connect the USB drive to a regular workstation (Windows or Linux) and copy the data via the client OS. Together with a proper verify operation that should work.

Because I want to keep it as simple as possible and try to harvest most of my existing servers HW (RAM etc). And I want to test it with Snapshot replication task due to higher transfer speed, compared with file transfer speed.

Forum member @Arwen has a Resouce covering backup that may help you https://www.truenas.com/community/resources/how-to-backup-to-local-disks.26/

I read that. Maybe this is the right way to go. I was thinking about a (cheap) 2bay HDD enclosure with active cooling and esata connection, so my Backup HDDs are connected via SATA (for the obvious stability & SMART Info reasons). But I doubt this could be an easy "plug&play" solution, or?

But, these single drive backup disks are only connected as long as it takes for the initial scrub and backup. (They get scrubbed every backup, before the backup, because each disk is used once a month or less frequently.)

Maybe you can share me your current setup? HW & Setup Infos would be great.

Yes, based on what we have seen, I'd guess that USB simply is less reliable that many other protocols

The more I think about it, the more I prefer the external esata enclosure. But the standard products (ICYBOX etc.) are always expensive. If someone has got a cheap solution that would be great

 

[Arwen]

@ECC
My setup, both hardware and software, are in the resource.

One of my goals for the external enclosure was for active cooling, as a ZFS scrub basically hits the disk continuously. And I mean as fast as the disk can read the data.

1. The first enclosure was the only one I could find with eSATA & USB, trayless design, and included a fan. It however included 2 disk slots and a controller chip, for crappy RAID functions. So I did not get SMART out of my disks. It did work fine for single disk backups.
2. A few years later, I found a cheap single disk enclosure with USB only, but did have a fan. Did use it for a few backups. It was pretty cheap, so I kept looking.
3. Last, was the Cru-dataport. It cost a small fortune, but has trayless design, fan, both eSATA & USB, (& Firewire), and seems like it might last 5 to 10 years. It's heavy, certainly not something to carry to remote sites. On the other hand, it just works. This is the type of quality I would have no problem buying for a data center.

It does appear that eSATA external enclosures are a bit hard to find. I think my next NAS will have a dedicated 3.5", (and maybe 2.5"), disk bay for backups. The old FreeNAS Mini only had 4 x 3.5" slots, so it's not quite enough for a dedicated slot. However, the new TrueNAS Mini X has 5 x 3.5" and 2 x 2.5".

 

[jgreco]

CRU-Dataport has been making high quality stuff for many years. Back in the '90's their modules were our choice for both servers and disk shelves, back when it was attractive to make your own stuff because the vendor stuff was incompatible crapware. I am sure I've got a bunch of their old SCSI stuff sitting in the dusty bits of the inventory shelves.

 

[ECC]

My setup, both hardware and software, are in the resource.

Thanks. Unfortunately, these products aren't available atm, at least in my country (im from Germany). The only thing I found close to your HW was this, but without a fan. Although I found 2 bay enclosures, none of them had 2 eSATA Ports. Therefore they are using Port Multiplier, which seems to work badly, according to this post.

I'm seriously thinking about building my own HDD enclosure or at least searching for existing 3D printed solutions with fan mounting. If I'm right, I only need an esata to sata cable and a simple sata power connector (and a fan connection). Maybe something like this.

For the connection at the pc, I would use this esata to sata bracket from your resource

It does appear that eSATA external enclosures are a bit hard to find

True!