Massive CPU/Disk load while SABNZBD unpacking...

[sYndax]

Hey, I've been using my new NAS for about a month or so and I've been fairly satisfied so far.

However while sabnzbd is repairing/unpacking completed downloads i'm unable to
"smoothly" use the pool. If i try and stream something to my OPPO 103 using SAMBA
i get stutters and the content is un-playble.

If i try and use windows explorer it is lagging massively, (i just try browsing directories).

Also after the repair/unpack process has finished i can still hear the drive preforming
what i assume is the last "write" functions from the sabnzbd file, (the drive is "ticking").
Generally the array seems busy for quite some time after sabnzbd reports the unpacking as "finished".

I run a Xeon E3-1230v3 with 16gb on x10sll-f board and 6X3TB RaidZ2.

Should i add more ram? install a l2arc? replace sabnzbd with nzbget? switch to NFS?

Thanks!

 

[ProblematicLlama]

I think sabnzbd's post-processing is greedy in that it'll try to do as much as it can given available resources, meaning unless you have a really powerful system you're going to get a bottleneck somewhere (and my guess is if you have a powerful CPU and lots of RAM, you'll max out your IO capacity). I had a look at the "switches" section in sabnzbd config and the post-processing section has something called "nice parameters". See "Nice to reduce CPU priority" and "IONice to reduce disk priority" here:

http://wiki.sabnzbd.org/configure-switches-0-7#nice

FreeNAS by default appears to have "nice", not sure about "IONice" though. Either way just limiting CPU usage may be all you need.

 

[Joshua Parker Ruehlig]

Yup nice is probably where you want to start.

When sabnzbd is unpacking it's reading and writing to the pool at the same time so it'll use a ton of IOPs switching between the two things. If you have multiple zpools you could take advantage of having your Incomplete and Complete downloads on different disks, but this isn't necessary.

 

[sYndax]

My Incomplete downloads are on a different pool, (stripped ssd), from the finished downloads, (6x3tb raidz2)...

 

[sYndax]

I think sabnzbd's post-processing is greedy in that it'll try to do as much as it can given available resources, meaning unless you have a really powerful system you're going to get a bottleneck somewhere (and my guess is if you have a powerful CPU and lots of RAM, you'll max out your IO capacity). I had a look at the "switches" section in sabnzbd config and the post-processing section has something called "nice parameters". See "Nice to reduce CPU priority" and "IONice to reduce disk priority" here:

http://wiki.sabnzbd.org/configure-switches-0-7#nice

FreeNAS by default appears to have "nice", not sure about "IONice" though. Either way just limiting CPU usage may be all you need.

I've looked in "switches" and the Nice IO option is grayed out...

 

[Joshua Parker Ruehlig]

My Incomplete downloads are on a different pool, (stripped ssd), from the finished downloads, (6x3tb raidz2)...

so IO shouldn't be to much of an issue cause your writes should be sequential. maybe your system is so fast IO wise it eats all your CPU? but, that's enough speculating from me.

 

[sYndax]

If I place my incomplete folder on the pool... i cannot max my 200mbit line, sabnzbd halts from time to time in order to give the pool time to write the data...
That's why I choose SSD for a temporary drive.

 

[Joshua Parker Ruehlig]

hmm, the ticking doesn't sound good. maybe post in the performance section, and get some benchmarks of your pool during and not during processing. also do some network tests over a wired connection.

I don't experience these issues with a much weaker system(CPU wise and only a single zpool), using NFS to xbmc.

 

[sYndax]

hmm, the ticking doesn't sound good. maybe post in the performance section, and get some benchmarks of your pool during and not during processing. also do some network tests over a wired connection.

I don't experience these issues with a much weaker system(CPU wise and only a single zpool), using NFS to xbmc.

Well... it's not a ticking sound which indicated the read head bumping the outer frame... you simply hear the drives "crunching" data...

 

[Joshua Parker Ruehlig]

hmm maybe ask on the sabnzbd forum why the nice io option is greyed out. im not sure if freebsd includes ionice, but we can try finding a way to present this tool to sabnzbd.

 

[Joshua Parker Ruehlig]

looks like ionice is included with linux_base-c6 which requires the Linux kernel module be installed.
seems like a lot of bloat for this functionality.

mind installing this, see if it works for you. I may include it if or doesn't make the plugin gigantic and works well.

 

[sYndax]

tried tweaking the sabnzbd... the zpool still chokes while unpacking...

 

[flower]

systemd resource quotas solved it for me
this settings allow sabnzbd to use 2GB RAM and no swap. It can read and write 20M/s (you may need to change your disk device)

create file /etc/systemd/system/nzb.slice

Code:

[Unit]
Description=Resource Limits for sabnzbd
DefaultDependencies=no
Before=slices.target

[Slice]
CPUAccounting=true
CPUWeight=50
CPUQuota=50%
MemorySwapMax=0
MemoryAccounting=true
MemoryHigh=1G
MemoryMax=2G
IOAccounting=true
IOWeight=50
IOReadBandwidthMax=/dev/vg/lv_vola 20M
IOWriteBandwidthMax=/dev/vg/lv_vola 20M

run:
systemctl daemon-reload
systemctl edit sabnzbd

Code:

[Service]
Slice=nzb.slice

run:
systemctl restart sabnzbd

 

[SweetAndLow]

Are your drives healthy? What is the smart output for each of them.