kernel panic error

[rvaldeza]

i got this error yesterday...

panic: kmem_malloc(16384): kmem_map too small

and on every reboot its the same.

i lost everything??.. how can i recover my 500GB backups??
if i reinstall the usb drive? could i see the info again?


i think,,,, maybe,,, because i got only 512MB,

the strange thing is:
System Requirements
To run FreeNAS, you will need, at a minimum, the following:

Motherboard with x86 processor
128 MB RAM
32 MB free disk space
Network card

 

[warri]

Do you refer to FreeNAS 7 with UFS or FreeNAS 8 with ZFS. If the later is the case, you should use a minumum of 4 GB RAM. If you are refering to FreeNAS 7, use the FreeNAS 7 Forum, this one is only for FreeNAS 8.

 

[rvaldeza]

I thought freenas 8 it was a lighter system, what can i disable to can work for a while,, and make a backup on my windows pc??

 

[warri]

So you are using freenas8 and UFS file system? ZFS needs a lot of RAM, UFS should be fine with less RAM.
Try backing up your complete usb stick and doing a fresh re-install (or install a fresh freenas on a new usb stick), and then try to import your existing pool. If FreeNAS still does not start, try to import your pool on another freenas system (it can also be a virtual machine on your windows pc).

Can you give us some more hardware specifications for your system, e.g. how many disks, which CPU and RAM, etc.?

 

[rvaldeza]

ok,thanks, i use freenas8 and ZFS file system
i will try to reinstall freenas 7 and ufs

System Information
Hostname Respaldos.local
FreeNAS Build FreeNAS-8.0.3-RELEASE-p1-x86 (9591)
Platform Intel(R) Pentium(R) CPU E5200 @ 2.50GHz
Memory 512MB
System Time Tue Mar 27 11:59:53 2012
Uptime 11:59AM up 5 mins, 0 users ----------> 5-15 min. then shows kernel panic
Load Average 0.02, 0.08, 0.07
OS Version FreeBSD 8.2-RELEASE-p6

 

[warri]

IN this case you definitely do not have enough RAM. Also I'm not sure if freenas 7 can import a freenas 8 created ZFS pool - you might need to upgrade RAM or use another computer to import the pool and backup your data.

 

[ilium007]

Same issue - kmem_map too small FreeNAS 8.0.4

Same problems here. Brand new install (moved to FreeNAS from Solaris 11 and NappIt) with the following hardware specs:

System Information

Hostname freenas.harland.local
FreeNAS Build FreeNAS-8.0.4-RELEASE-x64 (10351)
Platform Intel(R) Core(TM) i3 CPU 550 @ 3.20GHz
Memory 8065MB
System Time Mon Apr 9 02:31:12 2012
Uptime 2:31AM up 32 mins, 1 user
Load Average 0.00, 0.00, 0.00
OS Version FreeBSD 8.2-RELEASE-p6


Here is some more info:

[root@freenas] ~# /usr/local/www/freenasUI/tools/arc_summary.py | grep zfs
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 0
vfs.zfs.mfu_ghost_metadata_lsize 0
vfs.zfs.mfu_ghost_size 0
vfs.zfs.mfu_data_lsize 5525504
vfs.zfs.mfu_metadata_lsize 42189824
vfs.zfs.mfu_size 47944704
vfs.zfs.mru_ghost_data_lsize 39583744
vfs.zfs.mru_ghost_metadata_lsize 81920
vfs.zfs.mru_ghost_size 39665664
vfs.zfs.mru_data_lsize 35334144
vfs.zfs.mru_metadata_lsize 871424
vfs.zfs.mru_size 37452800
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 131072
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 0
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 1778839552
vfs.zfs.arc_meta_used 45469104
vfs.zfs.mdcomp_disable 0
vfs.zfs.arc_min 889419776
vfs.zfs.arc_max 7115358208
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.write_limit_override 0
vfs.zfs.txg.synctime 5
vfs.zfs.txg.timeout 30
vfs.zfs.scrub_limit 10
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 10485760
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.version.zpl 4
vfs.zfs.version.spa 15
vfs.zfs.version.dmu_backup_stream 1
vfs.zfs.version.dmu_backup_header 2
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
[root@freenas] ~#


And mt zpools:

[root@freenas] ~# zpool status
pool: vol0
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
vol0 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
ada2p2 ONLINE 0 0 0
ada3p2 ONLINE 0 0 0

errors: No known data errors

pool: vol1
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
vol1 ONLINE 0 0 0
mirror ONLINE 0 0 0
ada4p2 ONLINE 0 0 0
ada5p2 ONLINE 0 0 0

errors: No known data errors






I have 8GB RAM - where do I begin to start tuning memory ? This happened as I was doing an rsync of 1.2TB of data back onto the NAS from backup.

Cheers

 

[ilium007]

So I kick off the file copy again and see this:

root@freenas] ~# /usr/local/www/freenasUI/tools/arcstat.py 1 5
Time read miss miss% dmis dm% pmis pm% mmis mm% size tsize
02:45:23 3600 375 10 321 9 53 60 253 9 973M 7115M
02:45:28 0 0 0 0 0 0 0 0 0 1157M 7115M
02:45:33 0 0 0 0 0 0 0 0 0 1366M 7115M
02:45:38 0 0 0 0 0 0 0 0 0 1567M 7115M
02:45:43 14 0 0 0 0 0 0 0 0 1662M 7115M
02:45:48 0 0 0 0 0 0 0 0 0 1871M 7115M
02:45:53 0 0 0 0 0 0 0 0 0 2053M 7115M
02:45:58 0 0 0 0 0 0 0 0 0 2261M 7115M
02:46:03 0 0 0 0 0 0 0 0 0 2469M 7115M
02:46:08 0 0 0 0 0 0 0 0 0 2666M 7115M
02:46:13 2 0 0 0 0 0 0 0 0 2850M 7115M
02:46:18 0 0 0 0 0 0 0 0 0 3031M 7115M
02:46:23 0 0 0 0 0 0 0 0 0 3213M 7115M
02:46:28 18 0 0 0 0 0 0 0 0 3330M 7115M
02:46:33 0 0 0 0 0 0 0 0 0 3510M 7115M
02:46:38 0 0 0 0 0 0 0 0 0 3692M 7115M
02:46:43 3 0 0 0 0 0 0 0 0 3891M 7115M
02:46:48 0 0 0 0 0 0 0 0 0 4101M 7115M
02:46:53 0 0 0 0 0 0 0 0 0 4310M 7115M
02:46:58 0 0 0 0 0 0 0 0 0 4500M 7115M
02:47:03 0 0 0 0 0 0 0 0 0 4710M 7115M
Time read miss miss% dmis dm% pmis pm% mmis mm% size tsize
02:47:08 16 0 0 0 0 0 0 0 0 4883M 7115M
02:47:13 0 0 0 0 0 0 0 0 0 5091M 7115M
02:47:18 0 0 0 0 0 0 0 0 0 5301M 7115M
02:47:23 0 0 0 0 0 0 0 0 0 5510M 7115M
02:47:28 0 0 0 0 0 0 0 0 0 5185M 5185M
02:47:33 0 0 0 0 0 0 0 0 0 5185M 5185M
02:47:38 1 0 22 0 22 0 0 0 22 5185M 5185M
02:47:43 0 0 0 0 0 0 0 0 0 5185M 5185M
02:47:48 13 0 1 0 1 0 0 0 3 5185M 5185M
02:47:53 0 0 0 0 0 0 0 0 0 5185M 5185M
02:47:58 0 0 0 0 0 0 0 0 0 5022M 5023M
02:48:03 0 0 0 0 0 0 0 0 0 4740M 5023M
02:48:08 3 0 5 0 5 0 0 0 5 4791M 5023M
02:48:13 0 0 0 0 0 0 0 0 0 5001M 5023M

Whats with the drop in Target ARC cache size as the file copy progresses ?

I also found this in the ZFS tuning guide:

amd64

NOTE (gcooper): this blanket statement is far from true 100% of the time, depending on how the system with ZFS is being used.

FreeBSD 7.2 has improved kernel memory allocation strategy and no tuning may be necessary on systems with more than 2 GB of RAM.

Generic ARC discussion

The value for vfs.zfs.arc_max needs to be smaller than the value for vm.kmem_size (not only ZFS is using the kmem).

Looking at my settings (out of the box) I see that my vfs.zfs.arc_max is actually LARGER than the value for vm.kmem_size. Should I start fiddling with this ??

Any help appreciated.

 

[William Grzybowski]

I don't remember whether 8.0.4-p1 has the autotune script or not...

Do you guys have anything in System->{Tunables,Loaders} for zfs tunning? or /boot/loader.conf?

If so I recommend removing it...

 

[ilium007]

I don't remember whether 8.0.4-p1 has the autotune script or not...

Do you guys have anything in System->{Tunables,Loaders} for zfs tunning? or /boot/loader.conf?

If so I recommend removing it...

No - mine was an out of the box install. 7mins after installing I was copy file back across. I checked /boot/loader.conf but there was nothing relevant in there.