SOLVED All of a sudden everything running very slow

[mastercpt]

I am currently trying to migrate to a new server. My old server was an Asrock C2750 but was starting to have some issues with multiple plex transcoding. System would lock up then I would have to shut it down for a few hours before it would turn back on. Plan on reusing that server so will tackle that issue later. I was able to setup the new server and migrate all my data over. For about 2-3 days everything was running great. However now everything appears to be running very sluggish and locking up.

Example issues:
-- Accessing SMB share in windows will take time to populate file list
-- Play back video from SMB share in windows takes a few seconds to load, also longer than normal pause when skip ahead in video
-- Opening plex on different platforms (Xbox, Android, Browser) videos take a while to load.
-- sabnzbd takes a while to uncompress files and will sometimes lock up with Lost Connection message
-- sabnzbd will start to download at about 80 MB/s and then drop to 5 MB/s (confirmed not an issue with usenet provider or my isp)

New server specs:
Lenovo Thinkserver running Freenas 11.1-U5
Dual Xeon E5-2620 V3 CPUs
40GB ECC ram
freenas installed on a 32gb SD card
8x5TB drives in ZFS raidz1 about 75% full
All drives are Seagate Baracuda 2.5" 5400 drives
5 active Jails (Sabnzbd, Plex, Sonarr, Radarr and Tautulli)

Fixes so far:
-- Was having issues where smartd was not starting. Found out my LSI 9240-81 was in IR mode. I flashed it to a 9211-8i in IT mode with p20 firmware (I believe I read p20 is now supported). smartd now starts and I was able to run a short smart test on all my drives. No Errors

I am currently running a scrub but curious if there is anything else I should be looking at?

Thanks in advance.

 

[kdragon75]

8x5TB drives in ZFS about 75% full

In ZFS what? Also thats getting on the full side and if your using anything like torrents you are likely suffering from bad fragmentation.

is anything else I should be looking at?

How are your devices connected to the network? Have you tried running iperf tests to verify the underlying network? Looking at the reporting section in FreeNAS under Disks how does the Disk Busy look when under load but performing poorly?

 

[mastercpt]

In ZFS what? Also thats getting on the full side and if your using anything like torrents you are likely suffering from bad fragmentation.

Sorry left that out, RaidZ-1 (Edited post to add that). No torrents, just usenet. However I was testing out zoneminder for about a day. Didn't really like it so the jail is off and will be deleted. If I recall reading, I think zoneminder uses images instead of video so maybe that could have caused similar bad fragmentation.

I know it is getting on the full side. I plan on removing some data off it and split it up between my old freenas and this new one but need to redo the old one first. My old server also averaged 70-80% full at any given time. Hopefully once I get things setup it should average about 50%.

How are your devices connected to the network? Have you tried running iperf tests to verify the underlying network? Looking at the reporting section in FreeNAS under Disks how does the Disk Busy look when under load but performing poorly?

I have the server connected with 2 cables to my gigabit switch but NOT in link aggregation (not supported). I did run a few iperf tests from my old server to my new server prior to transferring the data and everything looked great. I will run it again shortly to test again. Other servers and computers in my house all appear to be running fine. I tested video and copying data from my old freenas it it was quick as normal.

Looks to be averaging 60-80% under disk busy

*EDIT* Just checked fragmentation and it says 6%

 

[kdragon75]

Looks to be averaging 60-80% under disk busy

So it's doing what it can.

*EDIT* Just checked fragmentation and it says 6%

ZFS fragmentation is free space fragmentation. Its not what most people are used to.

8x5TB drives in ZFS raidz1 about 75% full

That's a risky configuration especially that full. The rebuild time will be extremely long and if a second disk fails during that heavy workload, your SOL. You should consider splitting that into 2 RAIDz2 pools or at least 2 RAIDz1 pools. Really anything with more redundancy.

 

[Chris Moore]

There are several problems with the configuration you have going here, but the most important is probably that the pool is 75% full, and if you are using torrents to download files, your fragmentation is probably very high.
When you download a torrent, the data comes in fragmented to begin with and if you are downloading multiple files at once, it only makes the fragmentation worse. The thing that you must do to is download the file to a temporary location, then copy it to the permanent storage, and then delete the original download. The copy of the file reduces the fragmentation, but the best thing is to use a physically separate pool (even a single drive) for downloading the torrent, then copy it to permanent storage.
As to the storage pool, RAIDz1 is not a suggested storage topology for drives larger than 1TB, and you need more capacity, start planning now because your pool is getting close to the 80% point.

It would probably help you if you review this resource:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[Chris Moore]

I have the server connected with 2 cables to my gigabit switch but NOT in link aggregation (not supported).

This is an utter waste of effort. There is nothing to be gained and it can actually cause problems. One cable is all you need.

 

[mastercpt]

So it's doing what it can.

So possibly my disks are the bottle neck. I will have to compare to heavy load on my old server. I should have checked that reporting tab. Forgot about it. Thank you.

ZFS fragmentation is free space fragmentation. Its not what most people are used to.

Thank you. Did not know that.

That's a risky configuration especially that full. The rebuild time will be extremely long and if a second disk fails during that heavy workload, your SOL. You should consider splitting that into 2 RAIDz2 pools or at least 2 RAIDz1 pools. Really anything with more redundancy.

I understand it is a risky config and hopefully not a permanent one. Thankfully I am not concerned with loss of data. It is all movie/tv and I have a file list backed up. I could download/rip most of it again. I have had failed disks in my old server with similar full level and it did take a while to rebuild but time did not bother me. I will look into 2 RAIDz1 pools though that is a good idea. I will also be moving about half of the data off onto another freenas server and create a share between the two. This way I should be at about 35% on both.

 

[kdragon75]

This is an utter waste of effort. There is nothing to be gained and it can actually cause problems. One cable is all you need.

You mean like ARP issues? Switch loops? Things like that?;)

 

[mastercpt]

There are several problems with the configuration you have going here, but the most important is probably that the pool is 75% full, and if you are using torrents to download files, your fragmentation is probably very high.
When you download a torrent, the data comes in fragmented to begin with and if you are downloading multiple files at once, it only makes the fragmentation worse. The thing that you must do to is download the file to a temporary location, then copy it to the permanent storage, and then delete the original download. The copy of the file reduces the fragmentation, but the best thing is to use a physically separate pool (even a single drive) for downloading the torrent, then copy it to permanent storage.
As to the storage pool, RAIDz1 is not a suggested storage topology for drives larger than 1TB, and you need more capacity, start planning now because your pool is getting close to the 80% point.

It would probably help you if you review this resource:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

I understand the 75% full is approaching full however it has never been an issue on my older server. However one of the main reason I am setting up this second server is to split up the data so I should be at about 30-35% on each server going forward. But for now everything has to remain on the single server.

I know RAIDz1 is not the suggested way of handling things but I am not concerned with data loss. Knock on wood I have not had any issues on my old server (had to replace a drive or 2 over the last few years but nothing major). When I start splitting data up between 2 servers, I will look into 2 RAIDz1 pools instead of 1.

I am not using torrents however I am using usenet but that is a good idea about a different pool/drive for downloads.

Thank you for the links. I will check them out.

 

[mastercpt]

This is an utter waste of effort. There is nothing to be gained and it can actually cause problems. One cable is all you need.

Agree on there is nothing to be gained. I only used the 2nd network cable for rsync as I was copying data and setting things up. As it was just plugged in not being used I didn't think it could cause issues since they each have a separate static ip. But I will disconnect one anyway.

 

[mastercpt]

So it's doing what it can.

My old server is showing about 20-30% so possibly my drives are a bottleneck.

 

[toadman]

Agree on there is nothing to be gained. I only used the 2nd network cable for rsync as I was copying data and setting things up. As it was just plugged in not being used I didn't think it could cause issues since they each have a separate static IP. But I will disconnect one anyway.

It would make sense if the nics were on separate subnets and did not support vlans I suppose. i.e. two separate networks configured in the switch. I had to wire my Win 10 box that way as it didn't support vlans at all.

 

[mastercpt]

So I removed about 40% of the data on the drives and down to 30%. Ran SMART long and SMART short tests on all 8 drives. All passed. Running a scrub now. I did see the following errors in the smartctl logs (smart long tests were run after the errors seen in log):

https://pastebin.com/FN6JV6fj

Getting mixed information online. Do you think the drives should be replaced?