kmem_malloc: kmem_map too small and number of processes

[BuddyButterfly]

Hi,

yesterday I experienced system crash because of

Code:

kmem_malloc(1318912): kmem_map too small: 2208272384 total allocated

I was able to recover from it but found the root cause of this problem.
Therefore I would like to set focus on fixing the issue with the ever increasing number of processes
(threads) which in fact are a thread leak (though systemically caused). Please see this discussion

http://forums.freenas.org/index.php?threads/number-of-processes-continuesly-rising.21852/

As the number of snapshots are continuesly increasing (normally), the number of threads reach hights of thousands of threads. As we all know that threads consume resources especially thread stack size, this also exhausts memory! If you see the chart in the discussion and see 5k threads and lets assume a default threads stack size of 1MB, you should know where the memory is going to. I do not know the backgrounds on how thread stack size in freebsd is handled but this discussion thread also gave some background

http://lists.freebsd.org/pipermail/freebsd-threads/2005-January/002793.html

So let's hope that this thread leak is solved in version 9.2.1.6 as stated in the forum discussion. If not
then please put the reduction of threads at high prio.

P.S. Please do not give any advices to increase RAM as this would not solve the systemic cause - it would only delay the problem!

 

[mav@]

Kernel thread stack for FreeBSD amd64 architecture consumes 16KB, not an 1MB. But any way, I am now running today's 9.2.16-RC2 snapshot and I see no of those threads at all.

 

[BuddyButterfly]

Kernel thread stack for FreeBSD amd64 architecture consumes 16KB, not an 1MB. But any way, I am now running today's 9.2.16-RC2 snapshot and I see no of those threads at all.

Hi, very cool... I am looking forward to having less threads zvols and snapshots in 9.2.1.6!

Are you sure about the 16kB? If a kthread is created with a stack size of "0" then the default kernel stack size is allocated. Isn't the default kernel stack size "kern.dflssiz"? If so, then this is the value on my 9.2.1.5 amd64:

Code:

kern.dflssiz: 8388608

This is a bit more than 16kB ;-)

 

[mav@]

Are you sure about the 16kB? If a kthread is created with a stack size of "0" then the default kernel stack size is allocated. Isn't the default kernel stack size "kern.dflssiz"? If so, then this is the value on my 9.2.1.5 amd64:

Code:

kern.dflssiz: 8388608

This is a bit more than 16kB ;-)

That is not exactly my area, but I think that tunable controls user-level stack size. For kernel it is hard-coded in kernel sources. For amd64 that is:

Code:

#define KSTACK_PAGES    4       /* pages of kstack (with pcb) */

, where "page" is 4KB.

 

[BuddyButterfly]

Ok, thanks for this info. I was referring to http://www.freebsd.org/cgi/man.cgi?query=kthread&sektion=9

Btw issue has definitely been solved in 9.2.1.6. It has been verified in bug 5426!

This thread can then be set to "solved". I am not able to do this... :-(

 

[BuddyButterfly]

That is not exactly my area, but I think that tunable controls user-level stack size. For kernel it is hard-coded in kernel sources. For amd64 that is:

Code:

#define KSTACK_PAGES    4      /* pages of kstack (with pcb) */

, where "page" is 4KB.

The numbers are a bit differente here:

From this

Code:

PID      RSS    TT    STAT        TIME    COMMAND
267    41184    ??    DL        0:00.00    [zfskern/zvol mwpool]

I have 2586. And from this

Code:

PID        RSS    TT    STAT          TIME    COMMAND
4731    128376    ??    Ss        0:00.92    /usr/local/bin/istgt -c /usr/local/etc/istgt/istgt.conf

I have 61.

This means, the zvol threads reserve 41MB and the istgt 127MB. This is where my whole memory
is going to. I think this will be solved in 9.2.1.6. But how to explain the 41MB compared to 16kB?