I have the following system:
The preceding operation that seemingly triggered the panic was a zfs recv of a snapshot from an older solaris box over ssh:
The system had worked fine under normal load for 35 days at that point, and had previously been heavily benchmarked without issues.
Looking around the forums and bsd mailing lists etc it seems this type of panic is not unheard of. The underlying cause seems unclear, though I've seen memory fragmentation suggested as the main culprit (But the kernel should be able to handle that?). A common fix suggestion seems to be increasing vm.kmem_size. I've had it at default and am now in the process of switching to the autotuner values for zfs.arc_max, and setting kmem_size to 128GB (2x phys mem, which seems to be the max limit). As kmem_map seems to be directly related to the size of kmem_size this seems reasonable. Several posts regarding this type of panic on the forums however suggests that the autotune values are actually the culprit (For example: https://forums.freenas.org/index.php?threads/kmem_map-too-small.6702/#post-26153)
If anyone is familiar with the intricacies of these tunables I'd love some input.
- Supermicro X9DRH-iF
- Dual Xeon E5-2620v2
- 64GB DDR3 ECC RAM
- 14x4TB Drives
- 7x450GB Drives
- 2x200GB SSD
- FreeNAS-9.3-STABLE-201506292130
The preceding operation that seemingly triggered the panic was a zfs recv of a snapshot from an older solaris box over ssh:
Code:
db:0:kdb.enter.default> bt Tracing pid 81211 tid 102758 td 0xfffffe011d0ea490 kdb_enter() at kdb_enter+0x3b/frame 0xffffff90c487d230 panic() at panic+0x1c7/frame 0xffffff90c487d330 kmem_malloc() at kmem_malloc+0x23e/frame 0xffffff90c487d390 uma_large_malloc() at uma_large_malloc+0x4a/frame 0xffffff90c487d3d0 malloc() at malloc+0xd9/frame 0xffffff90c487d410 dmu_recv_stream() at dmu_recv_stream+0xc0/frame 0xffffff90c487d5e0 zfs_ioc_recv() at zfs_ioc_recv+0x326/frame 0xffffff90c487d810 zfsdev_ioctl() at zfsdev_ioctl+0x7c8/frame 0xffffff90c487d8b0 devfs_ioctl_f() at devfs_ioctl_f+0x7b/frame 0xffffff90c487d920 kern_ioctl() at kern_ioctl+0x106/frame 0xffffff90c487d970 sys_ioctl() at sys_ioctl+0xfd/frame 0xffffff90c487d9d0 amd64_syscall() at amd64_syscall+0x5ea/frame 0xffffff90c487daf0 Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff90c487daf0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019b6dcc, rsp = 0x7fffffffa748, rbp = 0x1b ---
The system had worked fine under normal load for 35 days at that point, and had previously been heavily benchmarked without issues.
Looking around the forums and bsd mailing lists etc it seems this type of panic is not unheard of. The underlying cause seems unclear, though I've seen memory fragmentation suggested as the main culprit (But the kernel should be able to handle that?). A common fix suggestion seems to be increasing vm.kmem_size. I've had it at default and am now in the process of switching to the autotuner values for zfs.arc_max, and setting kmem_size to 128GB (2x phys mem, which seems to be the max limit). As kmem_map seems to be directly related to the size of kmem_size this seems reasonable. Several posts regarding this type of panic on the forums however suggests that the autotune values are actually the culprit (For example: https://forums.freenas.org/index.php?threads/kmem_map-too-small.6702/#post-26153)
If anyone is familiar with the intricacies of these tunables I'd love some input.
Last edited:
