Help. My zpool has slowed down to a crawl -- Solved

[jeffs]

Please help me to trouble shoot this problem. I'm pretty new to Freenas although I do have some experience with servers, in general (windows and linux). I installed this server with all recommended components (Supermicro motherboard, ECC RAM, etc.) about 4 months ago and things have been fine. About a week ago things slowed down to a crawl and streaming video became choppy at best. I can't even copy a large file from the server to my desktop without an error after about 40 Mb of data have been transferred. Freenas reports that my zpool is healthy but I am sure something is wrong.

Today I decided to start a scrub on the pool and it's progressing very slowly too, 206K/s. After an hour only 1Gb has been scanned out of 9.56Tb. I have a raidz3 setup with 8 2Tb disks.

All disks are reported to be online and the pool is marked as healthy.

Again, please help me to troubleshoot this problem.

Thank you in advance for your help.

jeffs

 

[depasseg]

I'm guessing it's because your pool is extremely close to being 100% full. Above 80%, performance starts to degrade due to fragmentation issues.

 

[Ericloewe]

I'm guessing it's because your pool is extremely close to being 100% full. Above 80%, performance starts to degrade due to fragmentation issues.

9.56Tb. I have a raidz3 setup with 8 2Tb disks.

There's your problem.

Add storage or delete files. Your choice, but choose quickly.

Edit: I somehow never realized that zpool status' scrub status referenced raw on-disk bytes, not usable space.
So yeah, pool is not insanely full.

 

[Darren Myers]

Well x8 2Tb hdds in RAIDz3 gives you a Usable data space 7.872TB .....with a Minimum recommended free space 1.968TB a combined space of 9.84TB....soooo you're over the 80%

 

[depasseg]

Add storage or delete files. Your choice, but choose quickly.

Do not let it fill completely, or there is a very high probability of losing your pool.

 

[jeffs]

Thank you for the responses!

The Storage/Volumes page states I'm at 68% (5.5Tb of 9.6 used).

 

[depasseg]

Can you post a screen capture of your storage page? In Windows use the snipping tool, and then you can just paste the image in the post.

And the CLI output of "zfs get all"

 

[cyberjock]

A debug file would be appropriate for this...

 

[depasseg]

A debug file would be appropriate for this...

Sure, go ahead and take all the fun out of this with your fancy tools.

 

[jeffs]

I'll post the CLI output in a few moments...but the screen capture should show now...

 

[jeffs]

I couldn't print out the CLI output due to the length. So I attached a file with the output instead.

 

[jeffs]

How do i do a debug file?

 

[depasseg]

System -> Advanced -> Save Debug

 

[Ericloewe]

A debug file would be appropriate for this...

Remember how I once mentioned compression on boot pools? Apparently they are compressed.

Code:

freenas-boot       compression      lz4         local

Which does explain why they suddenly take a lot more data than they did before 9.3.

Back on topic: Retracted statement above (need to read up on ZFS tools...). Next bet? Bad disk.

 

[rogerh]

Remember how I once mentioned compression on boot pools? Apparently they are compressed.

Code:

freenas-boot       compression      lz4         local

Which does explain why they suddenly take a lot more data than they did before 9.3.

Back on topic: Retracted statement above (need to read up on ZFS tools...). Next bet? Bad disk.

Back off-topic: compression isn't enabled on my boot pool - there doesn't seem to be any obvious choice about this.

 

[jeffs]

I just wanted to let you know that I've been trying to do:
System -> Advanced -> Save Debug

but I keep getting a message "waiting for 192.168.1....". I've waited about 30 minutes now.

I did stop the scrub. I think I'll reboot and try again.

 

[jeffs]

Well it finally gave me the option to save the debug file. I've attached it.

 

[jeffs]

I just wanted to check back to let anyone with a similar problem know that I solved the issue with my server. It was a failing hard drive. I ran some smart tests and that finally revealed the disk that was failing. After disk replacement and resilvering my issues all disappeared.

I want to thank everyone in this great Freenas community for all their help!!

 

[Robert Trevellyan]

This is a great example of the benefits of using proper NAS drives (or other drives that support TLER) in your FreeNAS. In your case, the failing drive was trying and trying and trying to return the data it was asked for, because if it were in a desktop computer, that's what you would want. In a NAS, assuming you have redundancy in the storage pool, what you want is for the drive to respond quickly with an error so the system can get on with delivering the data from another source, while reporting the fact that the drive has a problem.

 

[jeffs]

I replaced the failed drive with a WD Red which I believe does have TLER. The failed drive was a WD Green drive.