Slow response and excessive reorganizing on 26TB system

[edatmpa]

I am new to using Freenas and have the following configuration

FreeNAS Build FreeNAS-8.0.3-RELEASE-p1-x64 (9591)
Platform AMD Athlon(tm) II X2 260 Processor
Memory 16113MB
System Time Tue Jan 8 15:27:35 2013
Uptime 3:27PM up 2 days, 11:22, 0 users
Load Average 0.21, 0.20, 0.10
OS Version FreeBSD 8.2-RELEASE-p6

with 5x2TB internal disks and 8x2TB in 2 external bays for a total available space as show in storage page below

jbod /mnt/jbod 15.9 TiB (76%) 4.8 TiB 20.6 TiB HEALTHY

The files are large usually 1GB up each. I am booting from a 4GB USB drive.

The system has been running fine for 6 months with very decent read/write speed between 30-60MB per sec.

However every so often (say, once every 2 weeks of usage as it is not always up and running) it goes into a self read/write (reorganizing?) mode indicated by constant red light on the disk activity for all disk units for about 3 days. That's 3x24 hours! I cannot determine what it''s doing because I do not know where to look except in the System Process screen which only says, 44 processes 1 running with no particular determinable process as it changes. During this time it becomes almost non-responsive so as to sometimes appear to Windows to be offline. The speed W/R speed drops down to less than 1MB per sec and become unusable. After 3 days it goes back to normal state and works great again.

I would appreciate any advice and understanding on what it's doing and how to improve. I would also like to hear from user with this large a storage in home environment.
Thanks.

 

[jgreco]

It's probably doing a pool scrub.

What does "zpool status" say from the shell when this is going on?

Code:

freenas# zpool status
  pool: yourpool
 state: ONLINE
  scan: scrub in progress since Tue Jan  8 18:05:44 2013
        83.5M scanned out of 9.77T at 5.97M/s, 476h44m to go
        0 repaired, 0.00% done

Check that likely possibility out.

 

[jgreco]

oh what the heck, that was pretty funny... I did actually start a scrub on a pool here so as to give you an example, but for some reason it was choosing to go Really Slow there for the first two or three minutes. "476h to go" my a**.

Code:

  scan: scrub in progress since Tue Jan  8 18:05:44 2013
        296G scanned out of 9.77T at 383M/s, 7h11m to go
        0 repaired, 2.96% done

It's a 4 drive configuration and usually pulls in from the individual components around 100MB/sec for scrubs.

I'm guessing that your storage topology is slow. Most 2TB disks are capable of at least 80MB/sec reads, but if you've done something like stuck them behind a SATA port multiplier (or worse a USB enclosure) ... you could have performance problems. A scrub will attempt to read all the pool data on all the disks simultaneously to proactively identify problems, but if your storage topology contains some straws rather than firehoses, then you can get immense I/O contention while the system tries to read all that.

 

[edatmpa]

Could you please explain pool scrub and how I might configure this more efficiently? I do have 2 external 4x2TB enclosures on USB 2.0 hubs. USB 3.0 is not yet supported.
Thanks.

 

[JaimieV]

Scrubbing is the process of ZFS checking the whole of the pool's disk surfaces to ensure each byte of your data can still be read. It's one of ZFS' major features.

You can change the schedule for this under Storage/ZFS Scrubs, but only on a per-volume basis - you can't set it to scrub one disk at a time, you have to just let ZFS go ahead.

/Edited to add: Your name for the pool is 'jbod' - please tell me you haven't set up the whole lot of disks as a stripe set with no redundancy? I'm hoping it's a RAIDZ3 from the disk space stats.
On the 'bright' side, if you did make a stripe set then you may as well turn off scrubbing because it won't be able to repair any damage it finds...

Even if it's a RAIDZ3, if one of those 4-disk USB devices or a hub loses power or connectivity for a moment, that's four disks gone and you'll lose the whole array. All data gone. I'm astonished that any USB-attached device has been stable and reliable enough to run for months already! At the very least take the USB hubs out of the equation... but really you should rearchitect the machine to have all the disks properly connected.

 

[ProtoSD]

It also looks like you're WAY below the recommended amount of RAM for your pool. 16GB of RAM for a 26TB pool??

 

[JaimieV]

16gig is probably ok for a domestic build - my 'backups of the main NAS' box only has 1gig RAM for 4Tb disk, no problem because it doesn't see much concurrent action.

 

[cyberjock]

He said he's using USB 2.0.... on a HUB! If you have a bunch of drives sharing a USB hub you can expect scrub time in days, weeks, or months. Even without a hub you shouldn't expect any reasonable scrub time. USB is just not able to handle the load. If you read the forum I've warned SO many people that attempting to use USB is borderline crazy for a bunch of reasons. USB3 support is flaky at best and not enabled by default on FreeNAS.

The really crappy part is that you should be doing a scrub at LEAST once a month. Of course, depending on how slow your drives are with USB 2 it could take 1/2 the month just to do the scrub.

You really need to rearchitect your system from the ground up. All those external drive need to be internal on SATA if you want this problem fixed. If you ignore it and choose not to do scrubs then don't ask me for help when you have problems later. The first time you have a disk fail you are sooooo screwed. You'll HAVE to do a scrub to recover the zpool back to full fault tolerance.

 

[jgreco]

So, "storage topology is slow"... nailed it!

JaimieV has correctly pointed out that there are severe risks to your pool if any of several various things happen, and if you lack redundancy, even more severely bad things can happen.

That having been said, if you enjoy living dangerously, to regain some of your performance, you could disable scrubs as someone else suggested or you could try lowering vfs.zfs.scrub_limit from 10 to something lower... probably take it one step at a time until the system becomes acceptably responsive... though it'll never be fast during a scrub given your setup.

 

[edatmpa]

Thank you very much for all the enlightened answers. When I started using Freenas, the objective was to find an inexpensive way to put up a lot of NAS at home as a media server. The cost of the disks 12x2TB was the main cost + a reasonable PC, AMD based as indicated. The idea was to have it up and running all the time. Space is also a small consideration. I have a feeling that I have been very luck up till now in not losing any data. The FreeNAS PC runs on a 1,000W power supply + 2 separate 2 4 bay USB external enclosures. So I am probably drawing down, say, 300-400 watts of power. Recently 4GB drives became available. Complete with raid (0-10) enclosures in a 4x4TB configuration, it draws a minuscule 43 watts! So I can get 2 units for 32 TB total with effective raid 5 available space of 24TB for a good price. And draws less than a 100W light bulb to keep my media online. Of course I still need a LAN server but I already have an old iMAC that is always on. I felt that I got good utility out of FreeNAS, but now it seems commercially available units have come down so much that FreeNAS does not have much of a price advantage. Am I missing something if I move back to commercial firmware raid units?

 

[Stephens]

I looked at commercial alternatives before going with FreeNAS. In my case, it was never about the rock bottom cheapest solution. There were other considerations

- I wanted ZFS. I store some media (mostly CD's and DVD's I've ripped) but I store other data. I'd like to avoid the silent bit rot if possible.
- I wanted decent speed. Most of the low end commercial units I looked at had terrible gigabit network speeds.
- I wanted to have something that would shut down automatically (and reliably) when a power outage occcurs.
- I wanted RAID6, which most simple commercial boxed didn't support. FreeNAS has the equivalent (but better) RAID-Z2.

and an item that applies only to me...

- I wanted something that'd force me to use *NIX on a semi-regular basis instead of learning commands for a day, then not using them again for a year or more, which has been my pattern until now.

There's something off about the power comparison of 300-400 watts vs 43 watts. If if the enclosure took 0 watts (fat chance), that suggests the 4 4tb drives take around 10 watts each. If that's the case, you're saying the rest of your computer system if you use FreeNAS takes 257-357 watts. Motherboard? RAM? CPU? Nah. Something's really wrong. My NAS with 6 drives takes 61 watts full blast (except when booting it can get up to around 100), and around 21 at idle.

 

[JaimieV]

You'd be losing out on the data-reliability features of ZFS, first and foremost. I know no domestic-class dedicated NAS boxes that do data scrubbing + repair, though some do snapshotting and rollback (but not live access to snaps via shadow copies). Plus when you're looking at more than four drive bays, dedicated NAS is *really* expensive for the chassis. And you may want two, because if the chassis develops a fault you've lost access to your data in most cases (not quite all, some are set up with easy-to-read Ext3 or 4 using mdadm and no stupid tricks, but mounting them on another system is not as simple as a ZFS array - you'll need to know your Linux disk admin). Also you need somewhere to backup to, which is a good use of the second device. Cheaper kit tends to have slow network throughput; more expensive kit with powerful enough CPU to keep up on the data rates raises the wattage up towards PC class again...

On the plus side they do have neat things like blinky LEDs to tell you which HDD has failed and needs pulling, and considerably simpler config - both of which are great!

On balance, I'll be sticking with low-power general purpose PCs with ZFS for some time yet.

 

[cyberjock]

Both Stephen and JamieV are right. For those that don't know all the nitty gritty those COTS look like a great deal. But dig down and your data is much safer with ZFS on FreeNAS. Its easier to get at your data if that COTS box fails, its easier to upgrade your whole server and move your data if you wish to, and MUCH cheaper if you start going to really large pools.

Not to mention some COTS products don't work well with hard drives except a few models of 1 or 2 brands.

 

[jgreco]

4 bay external RAID enclosures that can handle RAID5 will get you 12TB usable, yes. And, whee, you can tolerate a drive failure.

However, you must be particularly trusting that the RAID will be able to rebuild your data. Remember, while that's going on, you have no redundancy. The ability of a rebuild to complete may even be dependent on the error-free readability of all your other disks; some "RAID 5 rebuilds" actually STOP if they cannot read a block on one of the source disks (yikes! what a failure mode!)

The problem becomes more severe as drives get larger. This has been a known issue for many years now, and is nicely summarized in this article among many others.

So ZFS is pretty awesome because it's a proven system that is architected a bit differently. Why?

First, you can choose your array layout, rather than having "two 4x4TB=two 12TB virtual RAID5 devices" dictated at you by your enclosure choice. I'm kind of thinking I want to play with a 32TB array, 11 4TB drives in RAIDZ3. "Can do!"

Second, you can design your storage topology to reduce bottlenecks. Most posters here actually eliminate topology bottlenecks by dedicating a channel per drive.

Third, you can size your CPU to handle the maximum expected load. You can convert excess CPU to do things like on-the-fly compression as well.

Fourth, you can size your memory, and even add SSD, to increase access speeds.

All of these things work in your favor with ZFS, plus all the great points made by Stephens and JaimieV. And as noobsauce80 says, those USB connected RAID cans are nice until there's a problem. For example, right up to the point where the power supply in one fails, and if you've done something like stripe or concatenate them to get them into a single volume, then you lose the contents on both. Great.

Take a little time to check out what people have done. People have learned - often the hard way - that there are good ways and bad ways to design storage systems.

That having been said, there are less-than-ideal ways that you could use your existing hardware, but it's basically a question of whether or not it's wise.

 

[edatmpa]

I want to thank you everyone for these expert advice. As I am limited by the USB external hardware 4x2TB enclosure, I would like to start by doing something like changing the pool scrub parameter to take place less frequently. Could I get some specific instruction or webpage that will allow me to experiment with this? By necessity looks like I am stuck with the bottle neck on the USB hub devices. The hub present the disks as individual USB drives and are managed by FreeNAS individually. I assume that this is the most reliable (as opposed to presenting the whole device as one logical device). Therefore failure on any drive or power supply should not effect the recoverability. It would be too costly to move to a PC/server tower case with 13 disk slots to gain disk I/O speed.

 

[jgreco]

... to regain some of your performance, you could disable scrubs as someone else suggested or you could try lowering vfs.zfs.scrub_limit from 10 to something lower... probably take it one step at a time until the system becomes acceptably responsive...

Already been suggested...

 

[JaimieV]

By necessity looks like I am stuck with the bottle neck on the USB hub devices. The hub present the disks as individual USB drives and are managed by FreeNAS individually. I assume that this is the most reliable (as opposed to presenting the whole device as one logical device). Therefore failure on any drive or power supply should not effect the recoverability.

Oh, the 4-disk enclusures are cabled directly to the computer? That's better than I thought from your description. Those units aren't "USB hubs", they're just USB-connected enclosures.

You're correct when you say "Therefore failure on any drive or power supply should not effect the recoverability." - but probably not the way you mean. You have a RAIDZ3, and each enclusure holds 4 disks. This means that if either enclosure loses power or connection to the computer, you lose four disks at once and will instantly lose the whole array.

The closest to sane you can make this arrangement is actually to remove two disks: If you have just *3* disks in each enclosure, you can withstand losing either one of them - though you will be completely at risk with zero redundancy while you re-write the disks in the enclosure, which will take a looong time - about the same time as a scrub does now.

We're not suggesting you change the architecture for speed purposes - that's just a happy side consequence. We're suggesting it because for the absolute most likely problem you could have, loss of connection to a USB device, you will lose the whole array.

I shan't repeat it again, but I hope it's going in this time!

 

[edatmpa]

Thank you for pointing out the difference on the USB enclosure. But am I not losing speed by dividing up the 4 disks on a single USB port?

On the data integrity, I have lost connectivity to USB devices before without losing power. I do not know the reason. Rebooting brings the USB enclosure on-line without losing any data.

 

[cyberjock]

Thank you for pointing out the difference on the USB enclosure. But am I not losing speed by dividing up the 4 disks on a single USB port?

On the data integrity, I have lost connectivity to USB devices before without losing power. I do not know the reason. Rebooting brings the USB enclosure on-line without losing any data.

You mean without losing any data that you know of. Considering that you've never let it finish a scrub I'd bet there are errors you don't know about because you've never let a scrub complete.

 

[edatmpa]

Yes, no data that I know of. Is there a way to really check? Each time it goes into a scrubbing mode I always let it finish 3 days later. So I want to take just a little chance and let it scrub less frequently. Where can I set this schedule exactly? And where can I find the scrubbing report? Thanks.