Panic: kmem_malloc: kmem_map too small

[Will Dormann]

Hi folks,

I've got a FreeNAS 9.3 (latest from December) machine that still occasionally panics. I ran into the panic in the past when the system only had 32GB of RAM. And based on some other threads I saw, I chalked it up as not having enough RAM. However, lately the system has had 256GB of RAM, and it just saw a panic yesterday (after being up for a month or so).

[root@HOSTNAME] /data/crash# less info.0
Dump header from device /dev/dumpdev
Architecture: amd64
Architecture Version: 1
Dump Length: 122368B (0 MB)
Blocksize: 512
Dumptime: Thu Jan 7 23:30:34 2016
Hostname: HOSTNAME
Magic: FreeBSD Text Dump
Version String: FreeBSD 9.3-RELEASE-p28 #0 r288272+f229c79: Sat Dec 12 11:58:01 PST 2015
root@build3.ixsystems.com:/tank/home/stable-builds/FN/objs/os-base/amd64/tank/home/stable-builds/FN/FreeBSD/src/syskmem_malloc(16777216): kmem_map too small: 158405312512 total allocated Panic String: kmem_malloc(16777216): kmem_map too small: 158405312512 total allocated
Dump Parity: 523755048
Bounds: 0
Dump Status: good

From the textdump:

panic: kmem_malloc(16777216): kmem_map too small: 158405312512 total allocated
cpuid = 13
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xfffffe3ffa9010e0
kdb_backtrace() at kdb_backtrace+0x37/frame 0xfffffe3ffa9011a0
panic() at panic+0x1ce/frame 0xfffffe3ffa9012a0
kmem_malloc() at kmem_malloc+0x23e/frame 0xfffffe3ffa901300
uma_large_malloc() at uma_large_malloc+0x4a/frame 0xfffffe3ffa901340
malloc() at malloc+0xd9/frame 0xfffffe3ffa901380
dmu_recv_stream() at dmu_recv_stream+0xc0/frame 0xfffffe3ffa901550
zfs_ioc_recv() at zfs_ioc_recv+0x381/frame 0xfffffe3ffa901790
zfsdev_ioctl() at zfsdev_ioctl+0x7c8/frame 0xfffffe3ffa901830
devfs_ioctl_f() at devfs_ioctl_f+0x7b/frame 0xfffffe3ffa9018a0
kern_ioctl() at kern_ioctl+0x106/frame 0xfffffe3ffa9018f0
sys_ioctl() at sys_ioctl+0xfd/frame 0xfffffe3ffa901950
amd64_syscall() at amd64_syscall+0x5ea/frame 0xfffffe3ffa901a70
Xfast_syscall() at Xfast_syscall+0xf7/frame 0xfffffe3ffa901a70
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019bb97c, rsp = 0x7fffffffa788, rbp = 0x1b ---
KDB: enter: panic


Is there some sort of tunable that I can set to prevent this from happening? Though to be honest, I would have expected the system to avoid this situation by default.

Thanks!

 

[Sakuru]

Please post the full specs of your system. Also, are you running deduplication?

 

[Will Dormann]

I've filed a ticket here that has a bit more info, FWIW: https://bugs.freenas.org/issues/13061

But FTR:
Platform: AMD Opteron(tm) Processor 4274 HE
Memory: 262092MB
34TB primary zpool
38TB backup zpool
10TB vsphere zpool (not backed up)

No deduplication happens. Backups happen every 30 minutes on the hour and half-hour, using snapxfer. Based on the timestamp of the panic, my suspicion is that the extra zfs operations that were happening then may have caused the memory exhaustion. However, on a system with 256GB of RAM, it's sort of odd that that could happen.

 

[SweetAndLow]

Post full hardware and read the rules. We need motherboard, controller, memory, CPU, disk layout, nic and network configuration.

 

[Will Dormann]

Dell R515 with latest BIOS.
Dual opteron 4274 HE (16 cores total)
256GB ECC RAM
Primary array is a Jetstor 716JH SAS2 array plugged into an LSI 9200-8e HBA
16 drives are configured as a 3-vdev RAIDZ2 array
Backup array are the internal Dell SAS drives plugged into an LSI 9207-8i HBA
8 drives are configured as a 1-vdev RAIDZ1 array
Third array is a Jetstor 816JH array plugged into the second port of the LSI 9200-8e
4 SSDs make up a 2-vdev MIRROR array, 4 SAS drives make up a 2-vdev MIRROR array
Primary network is a single gigabit link using the onboard bce0 NIC.
High-speed 10Gbit network is provided by add-on ix1 NIC.

If that's not enough:
# pciconf -lv
hostb0@pci0:0:0:0: class=0x060000 card=0x04891028 chip=0x5a121002 rev=0x02 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'RD890 Northbridge only dual slot (2x8) PCI-e GFX Hydra part'
class = bridge
subclass = HOST-PCI
pcib1@pci0:0:2:0: class=0x060400 card=0x04891028 chip=0x5a161002 rev=0x00 hdr=0x01
vendor = 'ATI Technologies Inc'
device = 'RD890 PCI to PCI bridge (PCI express gpp port B)'
class = bridge
subclass = PCI-PCI
pcib2@pci0:0:3:0: class=0x060400 card=0x04891028 chip=0x5a171002 rev=0x00 hdr=0x01
vendor = 'ATI Technologies Inc'
device = 'RD890 PCI to PCI bridge (PCI express gpp port C)'
class = bridge
subclass = PCI-PCI
pcib3@pci0:0:9:0: class=0x060400 card=0x04891028 chip=0x5a1c1002 rev=0x00 hdr=0x01
vendor = 'ATI Technologies Inc'
device = 'RD890 PCI to PCI bridge (PCI express gpp port H)'
class = bridge
subclass = PCI-PCI
pcib4@pci0:0:11:0: class=0x060400 card=0x04891028 chip=0x5a1f1002 rev=0x00 hdr=0x01
vendor = 'ATI Technologies Inc'
device = 'RD890 PCI to PCI bridge (NB-SB link)'
class = bridge
subclass = PCI-PCI
ahci0@pci0:0:17:0: class=0x01018f card=0x04891028 chip=0x43901002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]'
class = mass storage
subclass = ATA
ohci0@pci0:0:18:0: class=0x0c0310 card=0x04891028 chip=0x43971002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 USB OHCI0 Controller'
class = serial bus
subclass = USB
ohci1@pci0:0:18:1: class=0x0c0310 card=0x04891028 chip=0x43981002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0 USB OHCI1 Controller'
class = serial bus
subclass = USB
ehci0@pci0:0:18:2: class=0x0c0320 card=0x04891028 chip=0x43961002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 USB EHCI Controller'
class = serial bus
subclass = USB
ohci2@pci0:0:19:0: class=0x0c0310 card=0x04891028 chip=0x43971002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 USB OHCI0 Controller'
class = serial bus
subclass = USB
ohci3@pci0:0:19:1: class=0x0c0310 card=0x04891028 chip=0x43981002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0 USB OHCI1 Controller'
class = serial bus
subclass = USB
ehci1@pci0:0:19:2: class=0x0c0320 card=0x04891028 chip=0x43961002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 USB EHCI Controller'
class = serial bus
subclass = USB
none0@pci0:0:20:0: class=0x0c0500 card=0x04891028 chip=0x43851002 rev=0x3d hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SBx00 SMBus Controller'
class = serial bus
subclass = SMBus
atapci0@pci0:0:20:1: class=0x01018f card=0x04891028 chip=0x439c1002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 IDE Controller'
class = mass storage
subclass = ATA
isab0@pci0:0:20:3: class=0x060100 card=0x04891028 chip=0x439d1002 rev=0x00 hdr=0x00
vendor = 'ATI Technologies Inc'
device = 'SB7x0/SB8x0/SB9x0 LPC host controller'
class = bridge
subclass = PCI-ISA
pcib5@pci0:0:20:4: class=0x060401 card=0x00000000 chip=0x43841002 rev=0x00 hdr=0x01
vendor = 'ATI Technologies Inc'
device = 'SBx00 PCI to PCI Bridge'
class = bridge
subclass = PCI-PCI
hostb1@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x16001022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 0'
class = bridge
subclass = HOST-PCI
hostb2@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x16011022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 1'
class = bridge
subclass = HOST-PCI
hostb3@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x16021022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 2'
class = bridge
subclass = HOST-PCI
hostb4@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x16031022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 3'
class = bridge
subclass = HOST-PCI
hostb5@pci0:0:24:4: class=0x060000 card=0x00000000 chip=0x16041022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 4'
class = bridge
subclass = HOST-PCI
hostb6@pci0:0:24:5: class=0x060000 card=0x00000000 chip=0x16051022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 5'
class = bridge
subclass = HOST-PCI
hostb7@pci0:0:25:0: class=0x060000 card=0x00000000 chip=0x16001022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 0'
class = bridge
subclass = HOST-PCI
hostb8@pci0:0:25:1: class=0x060000 card=0x00000000 chip=0x16011022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 1'
class = bridge
subclass = HOST-PCI
hostb9@pci0:0:25:2: class=0x060000 card=0x00000000 chip=0x16021022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 2'
class = bridge
subclass = HOST-PCI
hostb10@pci0:0:25:3: class=0x060000 card=0x00000000 chip=0x16031022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 3'
class = bridge
subclass = HOST-PCI
hostb11@pci0:0:25:4: class=0x060000 card=0x00000000 chip=0x16041022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 4'
class = bridge
subclass = HOST-PCI
hostb12@pci0:0:25:5: class=0x060000 card=0x00000000 chip=0x16051022 rev=0x00 hdr=0x00
vendor = 'Advanced Micro Devices [AMD]'
device = 'Family 15h Processor Function 5'
class = bridge
subclass = HOST-PCI
mps0@pci0:1:0:0: class=0x010700 card=0x30801000 chip=0x00721000 rev=0x03 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
class = mass storage
subclass = SAS
ix0@pci0:2:0:0: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82599EB 10-Gigabit SFI/SFP+ Network Connection'
class = network
subclass = ethernet
ix1@pci0:2:0:1: class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82599EB 10-Gigabit SFI/SFP+ Network Connection'
class = network
subclass = ethernet
bce0@pci0:3:0:0: class=0x020000 card=0x04891028 chip=0x163b14e4 rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II BCM5716 Gigabit Ethernet'
class = network
subclass = ethernet
bce1@pci0:3:0:1: class=0x020000 card=0x04891028 chip=0x163b14e4 rev=0x20 hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II BCM5716 Gigabit Ethernet'
class = network
subclass = ethernet
mps1@pci0:4:0:0: class=0x010700 card=0x30201000 chip=0x00871000 rev=0x05 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS2308 PCI-Express Fusion-MPT SAS-2'
class = mass storage
subclass = SAS
vgapci0@pci0:5:4:0: class=0x030000 card=0x04891028 chip=0x0532102b rev=0x0a hdr=0x00
vendor = 'Matrox Graphics, Inc.'
device = 'MGA G200eW WPCM450'
class = display
subclass = VGA

Thanks!

 

[cyberjock]

Is "autotune" enabled in the WebGUI? My guess is it isn't, and for a system of this size, it should be.

To use it, enable it, then reboot to apply the settings. If this is like typical problems I've seen, the issue will not come back after using autotune. ;)

 

[Will Dormann]

Autotune is disabled in the WebGUI, and I've never explicitly enabled it.

I'll enable Autotune to see if that prevents the panic. Is there a reason why the default configuration doesn't use it? Especially if a panic can be the outcome?

 

[cyberjock]

Yes, autotune is supposed to prevent the kernel from causing panics. It's a problem that gets more likely the more RAM you have. Its a weird scenario because that panic generally means "add more RAM". The problem is that if you add more RAM you actually increase the chances of having the panic occur again. So either you use the autotune or you buy infinite amounts of RAM. If you have that kind of money to afford that kind of RAM, please send some my way. ;)

 

[Will Dormann]

Is there some scenario where having autotune enabled is actually worse than having it disabled? If that's not the case, then I suspect that's it's an oversight that FreeNAS doesn't enable it by default. If that is the case, then what is the scenario? And perhaps its default setting could be conditional depending on it?

 

[Will Dormann]

Related: The FreeNAS documentation indicates that autotune "should only be used as a temporary measure on a system that hangs until the underlying hardware issue is addressed by adding more RAM. Autotune will always slow the system down as it caps the ARC."

Which guidance is correct?

 

[anodos]

Is there some scenario where having autotune enabled is actually worse than having it disabled? If that's not the case, then I suspect that's it's an oversight that FreeNAS doesn't enable it by default. If that is the case, then what is the scenario? And perhaps its default setting could be conditional depending on it?

It's more like the defaults are what a typical FreeNAS user needs. Most people around here have 8GB-32GB RAM. Autotune shouldn't be enabled in that case. It can also be used in situations like described in the docs.

 

[tobiasbp]

Was OP's problem resolved by turning on autotune?

 

[mav@]

FreeNAS 9.10 has completely different kmem allocator, so after update this problem should no longer appear unless there is really some memory leak in the system.

 

[Will Dormann]

I haven't seen the crash with autotune enabled on 9.3. I'm also running 9.10 now.

 

[tobiasbp]

Thank you for sharing.