l2arc_feed_thread using high cpu?

[aghadjip]

Hello,

I seem to have an issue with the l2arc _feed_thread using a high percentage of cpu. Below is my system information

FreeNAS Build FreeNAS-8.0.4-RELEASE-x64 (10351)
Platform AMD Athlon(tm) II X2 270 Processor
Memory 16367MB
System Time Wed Apr 4 21:12:23 2012
Uptime 9:12PM up 26 days, 22:12, 1 user
Load Average 0.85, 0.85, 0.81
OS Version FreeBSD 8.2-RELEASE-p6

Here is the top snapshot.


PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 32K RUN 1 623.8H 95.36% {idle: cpu1}
178 root -8 - 0K 60K CPU0 0 49.5H 79.05% {l2arc_feed_threa}
11 root 171 ki31 0K 32K RUN 0 598.3H 29.20% {idle: cpu0}
0 root -16 0 0K 928K - 0 129:27 0.00% {zio_read_intr}
2363 root 44 0 31004K 18184K kqread 1 52:10 0.00% {connthread #0}
12 root -32 - 0K 224K WAIT 1 52:09 0.00% {swi4: clock}


You can see process 178 is using 79% of the cpu and has been doing so for the past 2 days. Ive searched for this error on other sites, some of them reference the clock_t overflow. Is there a fix for this issue?

-Andrew

 

[aghadjip]

Ill have to reboot soon, its affecting iscsi performance and chron jobs. No one has experienced the above?

 

[nixcz]

This is a known bug that was discovered in the late 8.1, early 8.2 time frame. See this thread:
http://lists.freebsd.org/pipermail/freebsd-fs/2011-January/010558.html

I don't remember now if it was available in 8.2 release (it might have been too late to make the release) or only in 8.2 stable. Maybe someone with SVN-fu can dig out the commit logs. This has obvious implications for any FreeNAS release based on 8.2. My workaround was to remove all cache vdevs from the pool.

 

[aghadjip]

Yea thats similar to what i found. And this continues to be a pain. Im wondering what iXsystems does to re mediate this in their production enterprise appliances. They surely are running cache vdevs.... Do they ever post here?

 

[nixcz]

As they have several well known developers on staff, I think it's a relatively simple matter for them to backport the fix to their enterprise products. It's a mistake to think what you see in FreeNAS reflects the features/stability of their storage products. They need some way to differentiate their products after all, and truly enterprise class features are one method. Support is another.

I've been told for example, that their enterprise storage products have mirrored boot devices, which FreeNAS does not support out of the box. This is one of the showstoppers for my deployment scenario, so at least for now, I'm sticking to vanilla FreeBSD (8.3 or 9). This has the additional advantage of having those bug fixes and also ZFS v28.

I'm hopeful that they'll get over this 8.2-beta slump any day now and start working on a version based on 8.3 or the upcoming 9.1.

 

[delphij]

Whoops, this should have been fixed earlier. I've committed a fix as r11998, should there be a new 8.2 series release, this will be fixed. (Note that we are going to test and release 8.3-RELEASE soon so there may be no more 8.2 releases, though).

 

[nixcz]

Good to hear. Thanks for your (and iX's) continued support.