Copying a large file has random freezes on Windows??

[berengard]

I have seen many threads where during some kind of operation (resilver/scrub/other?), ZFS starts to kick SMR disks out of the pool because they are returning CAM STATUS timeouts...

Hmm that's good to know... Thanks!

I wonder if in the case, for my only SMR drive some of those operations can be limited or avoided altogether?
For example, since it's the only drive in the pool, there won't be any resilvering, right? If it dies, that's it.

As for scrubbing, how necessary is it in the case of zfs?
I understand that zfs is designed for some serious storage space where data is meant to be crucial, therefore scrubbing is meant to protect against worse case scenarios (like cosmic flares and such. Is that correct?). But what if my data on that SMR drive is just some movies and photos. If I can accept that one photo might get corrupted once a year, assuming that's how rare it would happen, can scrubs be avoided entirely to extend the drive's longevity?

I know I'm suggesting an unconventional use for zfs, but would that technically work better if all those conditions were met?

 

[Ericloewe]

I mean, data integrity is the selling point of ZFS... Foregoing scrubs is akin to replacing the seats in a Rolls-Royce with those out of a junked Fiat Punto. Technically not impossible, but why did you pay Rolls-Royce money for that?

 

[berengard]

I mean, data integrity is the selling point of ZFS... Foregoing scrubs is akin to replacing the seats in a Rolls-Royce with those out of a junked Fiat Punto. Technically not impossible, but why did you pay Rolls-Royce money for that?

It's because I needed an all in one OS for storage that I was familiar with.
If the Rolls-Royce was free but you only needed to drive to the grocery store, would you pay for a different car instead? Of course if the Rolls-Royce proved to be a liability, then it makes sense to replace it with something else...

But with that said, I just bought a CMR drive to replace the SMR one lol

 

[Arwen]

Performing ZFS scrubs on single disk pools has 2 benefits.

First, you find what data became bad. This allows you to find the backup, (or CD, DVD, Blue-ray), and restore just that file.

Second, by default all Metadata has is redundant, regardless of the vDev / Pool layout. So single disk pools have 2 copies of the standard Metadata which contain your directory entries. Any bad block in standard Metadata can be automatically repaired, thus avoiding the loss of a file or files due to bad directory entry blocks.

(Critical Metadata has 3 copies...)


As for stresses on SMR disks. Scrubs will take longer due to the fragmentation that SMR introduces. And may run longer. So keeping a SMR disk cool is a bit of a concern. But, that applies to SMR disks in general, not just ZFS scrubs on SMR disks. As for wear and tear on SMR disks caused by ZFS scrubs, I don't know. Perhaps running a scrub once a month instead of twice a month. User choice.


I personally run my Media pool on a low power, miniature desktop computer with only 1 x 2.5" bay and 1 mSATA bay. I have a 2TB laptop HDD and a 1TB mSATA in it at present. A portion is partitioned for the OS, Mirrored via ZFS. The rest is striped, no redundancy, (but multiple backups), for my Media.

Occasionally I loose a video file, which are larger and statistically more likely to have a bad block. ZFS tells me what file and I restore it from backups.

Once I had a automatically repaired fault that drove me crazy on how ZFS could repair it. After a few weeks I figured out it must have been a Metadata block that went bad.

 

[berengard]

Interesting. That's good to know!
I'm generally learning a lot from this thread. Thanks a lot everyone for the help!

Occasionally I loose a video file, which are larger and statistically more likely to have a bad block. ZFS tells me what file and I restore it from backups.

Yeah, sounds about like my case. I could use the original SMR drive to backup my media and the new CMR drive as the main one.
Will probably set scrubs to run once a month or so.

So my resulting setup will be 4 drives: 2 drives will form one pool with mirroring, that's where the more critical data will be.
3rd CMR drive will form the media pool with no inherent redundancy and have replication to the original SMR drive for backup.
Will probably be ok. :)

 

[berengard]

Update:
So the problem was finally rectified.
I've tried to pass the enclosure box to the TrueNAS VM by its USB device but that didn't work as it should because what happened is the box, when connected through USB, was passing all the drives with the same serial number.
So instead of passing the box as a USB device, I connected it through its eSATA interface and passed the on-board eSATA controller to it, which was appearing as an individual device.
That registered all the drives with their unique serial numbers.

On top of all that, I upgraded my one SMR drive to CMR and now am just using the SMR one for a cold backup and no scrubs.
One pool is setup with mirroring, so I have redundancy and both pools have replication tasks.

The freezing is no longer happening and files are being copied at 112MB/sec consistently!!

 

[Whattteva]

The freezing is no longer happening and files are being copied at 112MB/sec consistently!!

Glad to hear your success story. Now, I wish you the next good luck, which is that we (hopefully) won't see you again in a year or two from now with possible data loss so many other people have done in the past on these forums when running some less-tested configurations. Not saying your setup won't necessarily work, but do keep an eye on it and maintain a backup.

 

[berengard]

do keep an eye on it and maintain a backup.

Yup, I'll be backing up my data constantly and I have SMART checks now (cause it's working) and if anything happens I get an email
I will still probably be browsing around here a little. Just to see how others are doing.
Cheers!

 

[Arwen]

Update:
...
So instead of passing the box as a USB device, I connected it through its eSATA interface and passed the on-board eSATA controller to it, which was appearing as an individual device.
That registered all the drives with their unique serial numbers.
...

And now you are headed for the next problem;

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

Any eSATA enclosure supporting more than 1 disk will have either an eSATA Port Multiplier or lightweight RAID controller. Yours seems to have no RAID controller, so it must have an eSATA Port Multiplier.

Of course, an eSATA enclosure with multiple disks, that has 1 eSATA port per disk, is a different animal.

Now you might never have problems. It depends on a lot of factors, like SATA controller, eSATA Port Multiplier and the disks.

But, keep in mind that all your enclosures' disks are funneled through a singe SATA port. Probably 6Gbps... which will likely be fine with hard disk drives.

 

[berengard]

And now you are headed for the next problem;

Hmm, considering the post you linked, is the biggest issue with a port multiplier is not enough bandwidth for all the drives and the controller not being able to handle max available bandwith?
In my case there are 4 HDDs on a 6Gbps port. So if all HDDs run simultaneously (which shouldn't happen almost* ever), each is allowed 1.5Gbps, which divided by 8 is about 187MB/sec.

While these drives can technically read at that speed, this can't happen over the network as it's capped at 112-113MB/sec (1Gbps)
and locally there's only one replication task for one of the drives that could go that fast. It runs daily at 3AM when the other drives aren't working.

So with all that considered, I realize there could be unforeseen circumstances and with data, it's better safe than sorry...
but perhaps I'll look into a new server case that can host all the drives, each with individual SATA cable later on.
Though your point is definitely taken! :)

 

[sretalla]

You're looking at the problem backwards.

ZFS makes transaction groups which will push data to the controller for all disks at once every time (if they are in the same pool).

Those don't care about the speed the disks can handle, so time-outs will happen when writing doesn't report back fast enough.

Maybe it won't burn out the multiplier the same way it will a USB controller, but understand the problem that way around.

 

[Arwen]

Another aspect of SATA Port Multipliers is that it adds less tested hardware between the SATA controller and the SATA disk. This extra hardware can and does cause some people problems. It takes a combination of good OS driver support, good SATA ports, and disks that don't freak out, to have an enclosure with a SATA Port Multiplier work well. (Not to mention a good implementation of the SATA Port Multiplier standard...)

Though I suspect disks freaking out is going to be rare. However, their were some odd disks released in the last few years, that seemed to have a SATA Port Multiplier built in. I don't remember the details, but the SATA disks showed up as 2 separate disks. Was it that dual actuator ones? Or hybrid SSD & HDD? I don't recall... Just stay away from those odd disks, for your external enclosure that uses the SATA Port Multiplier.

Most OSes don't prioritize testing their SATA drivers with SATA Port Multipliers. So if they work, well, okay. If not, or it works erratically, their is nothing to do, except remove it from use.


Anyway, good luck and at least your main problem is solved.

 

[berengard]

ZFS makes transaction groups which will push data to the controller for all disks at once every time (if they are in the same pool).

I have 3 pools: 2 disks, 1 disk and 1 disk. Will a 6Gbps eSata controller really not be able to handle addressing 2 disks in one pool?
So far looks like it's working perfectly.

Those don't care about the speed the disks can handle, so time-outs will happen when writing doesn't report back fast enough.

What happens if I set the pools to synchronous writes? Is that related?

Maybe it won't burn out the multiplier the same way it will a USB controller

I really fail to see how it's possible that hardware in such a general sense doesn't handle a bunch of writes. Burn out the USB controller... Burn out the eSATA controller... What else?...
Sounds like zfs is a real danger to your server, or there should be precautions in place that allow it not to kill your hardware. ;)

However, their were some odd disks released in the last few years, that seemed to have a SATA Port Multiplier built in

Sounds like something you order on wish for 10$... Like those NES cartridges that claimed to have 10,000 games, but in reality it was 5 games with different color variations.

 

[Arwen]

...

However, their were some odd disks released in the last few years, that seemed to have a SATA Port Multiplier built in.

Sounds like something you order on wish for 10$... Like those NES cartridges that claimed to have 10,000 games, but in reality it was 5 games with different color variations.

Well, some would agree that Seagate HDDs are a joke... but this is one that I was thinking of:

Exos 2X18 Hard Drive | Seagate US

www.seagate.com

For the SATA version it does not appear to have a SATA Port Multiplier in it... just some funky software splitting first half to the first actuator, and the second half of the storage to the second actuator.

I could not find the details of the Western Digital hybred HDD / SSD easily.

 

[berengard]

For the SATA version it does not appear to have a SATA Port Multiplier in it... just some funky software splitting first half to the first actuator, and the second half of the storage to the second actuator.

I would imagine that even if there's no multiplier there, which should imply that there are 2 separate SATA ports,
that still means that 2 disks are somehow are working in one metal case, sharing the heat and vibration.
And if not, if they are actually logically separated disks, rather than physically, what's the difference between that and creating 2 virtual disks on one physical disk and passing them to the OS (unless of course you aren't using virtualization).

To take a page from this forum's book, that's a really bad idea! lol
Also, I can't seem to find this disk in any store, which makes it more sketchy.

 

[joeschmuck]

that still means that 2 disks are somehow are working in one metal case

Yes and No. It's one set of spinning platters and two sets of armatures and heads accessing the same physical drive space. "Available as two independently addressable, 9TB logical units for SAS or one 18TB logical device for SATA". Interesting concept but I'll never buy one unless they become the standard product.

 

[berengard]

Interesting concept but I'll never buy one unless they become the standard product.

yeah, this is both fascinating and concerning.

 

[berengard]

The adventure continues

I decided ditch the enclosure box and put the hard drives in a new case that supports them with a SAS controller card (I got a case from Facebook Marketplace for $10(!) and a Dell Perc H310 card for $40)
One deadly mistake I made was to order the wrong 6-pin to SATA power cables for my power supply to connect all the drives and... All my drives AND motherboard got fried when I turned it on!!
I was devastated, but the way forward was what I was trying to avoid the entire time, spend more money. So I got a new PSU and a new motherboard + 6 new SAS HDDs. I flashed the card to work in IT mode and now it looks like everything is working...

My new setup has a single zpool with 6 drives of 3TB each (4 data + 2 parity + 1 SSD for log)
All drives are plugged through the SAS card and passed to TrueNAS as a PCIe device.
Also, I have a backup of my critical files to S3 storage and all of the files to an external drive.

Again, thanks for all the help! This was a very educational experience, to say the least.

 

[joeschmuck]

One deadly mistake I made was to order the wrong 6-pin to SATA power cables for my power supply to connect all the drives and... All my drives AND motherboard got fried when I turned it on!!

Ouch! Sorry to hear about that. Very tragic.