Weird issue I have with Bhyve or Freenas?

[blathierry]

So my issue is very specific but its driving me insane.

First of all I know Freenas isn't the best for VMS but want to know if this is fixable anyway or know where the issue lies.

I run a Ubuntu 18.04 Server VM where I am trying to use https://pterodactyl.io/ on, this is a server manager for game servers (I use it for minecraft and such).

After installing everything in the panel and shooting up a minecraft server I quickly noticed that the whole VM freezes from time to time randomly for 5 - 20 seconds.
This also makes it impossible to log in via SSH until the lag spike of the VM is over.

I have reinstalled Ubuntu and everything a million times but issue presists.

The VM is safed on an SSD Pool, I disabled compression completely on the VM dataset and the ZVOL, I have both tried AHCI and Virtio disk.
I assign 12 VCPUS to the VM and 32768MB RAM.

I also enabled logging in my minecraft server and it displays this (might be helpful maybe not):

[22:17:08 ERROR]: The server has not responded for 15 seconds! Creating thread dump
> [22:17:08 ERROR]: ------------------------------
> [22:17:08 ERROR]: Server thread dump:
> [22:17:08 ERROR]: ------------------------------
> [22:17:08 ERROR]: Current Thread: Server thread
> [22:17:08 ERROR]: PID: 25 | Suspended: false | Native: true | State: RUNNABLE
> [22:17:08 ERROR]: Stack:
> [22:17:08 ERROR]: java.io.RandomAccessFile.read0(Native Method)
> [22:17:08 ERROR]: java.io.RandomAccessFile.read(RandomAccessFile.java:337)
> [22:17:08 ERROR]: java.io.RandomAccessFile.readInt(RandomAccessFile.java:798)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.RegionFile.a(RegionFile.java:146)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.RegionFileCache.read(RegionFileCache.java:248)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.ChunkRegionLoader.a(ChunkRegionLoader.java:170)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.ChunkRegionLoader.a(ChunkRegionLoader.java:116)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.ChunkRegionLoader.loadChunk(ChunkRegionLoader.java:244)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PaperAsyncChunkProvider$PendingChunk.loadChunk(PaperAsyncChunkProvider.java:387)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PaperAsyncChunkProvider$PendingChunk.run(PaperAsyncChunkProvider.java:579)
> [22:17:08 ERROR]: com.destroystokyo.paper.util.PriorityQueuedExecutor.lambda$createPendingTask$1(PriorityQueuedExecutor.java:133)
> [22:17:08 ERROR]: com.destroystokyo.paper.util.PriorityQueuedExecutor$$Lambda$1800/1883645863.get(Unknown Source)
> [22:17:08 ERROR]: com.destroystokyo.paper.util.PriorityQueuedExecutor$PendingTask.run(PriorityQueuedExecutor.java:273)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PaperAsyncChunkProvider.requestChunk(PaperAsyncChunkProvider.java:197)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PaperAsyncChunkProvider.loadOrGenerateChunk(PaperAsyncChunkProvider.java:154)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PaperAsyncChunkProvider.getChunkAt(PaperAsyncChunkProvider.java:150)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.ChunkProviderServer.getChunkAt(ChunkProviderServer.java:111)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PaperAsyncChunkProvider.getChunkAt(PaperAsyncChunkProvider.java:136)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.World.getChunkAt(World.java:385)
> [22:17:08 ERROR]: com.destroystokyo.paper.antixray.ChunkPacketBlockControllerAntiXray.onChunkPacketCreate(ChunkPacketBlockControllerAntiXray.java:162)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerChunk.sendChunk(PlayerChunk.java:182)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerChunk.a(PlayerChunk.java:117)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerChunkMap.movePlayer(PlayerChunkMap.java:364)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerList.updateChunks(PlayerList.java:417)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerConnection.a(PlayerConnection.java:1094)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PacketPlayInFlying.a(SourceFile:126)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PacketPlayInFlying$PacketPlayInPositionLook.a(SourceFile:18)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerConnectionUtils.lambda$ensureMainThread$0(PlayerConnectionUtils.java:15)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.PlayerConnectionUtils$$Lambda$2092/1256873683.run(Unknown Source)
> [22:17:08 ERROR]: java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [22:17:08 ERROR]: java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.SystemUtils.a(SystemUtils.java:108)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.MinecraftServer.b(MinecraftServer.java:1018)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.DedicatedServer.b(DedicatedServer.java:439)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.MinecraftServer.a(MinecraftServer.java:940)
> [22:17:08 ERROR]: net.minecraft.server.v1_13_R2.MinecraftServer.run(MinecraftServer.java:837)
> [22:17:08 ERROR]: java.lang.Thread.run(Thread.java:748)
> [22:17:08 ERROR]: ------------------------------
> [22:17:08 ERROR]: --- DO NOT REPORT THIS - THIS IS NOT A BUG OR A CRASH ---
> [22:17:08 ERROR]: ------------------------------

I ran this exact panel on the exact same machine on Windows Server 2019 HyperV and don't have this issue (to make sure it isn't ubuntu, panel or the server)

I really love Freenas and have been super happy with everything about it but this is driving me insane almost to the point of just giving up and moving to a different OS.
:(

I run FreeNAS-11.2-U3

Here are my hardware specs:
2X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (40 cores)
64GB of ECC Ram
supermicro X9DRL-IF
3x kingston 500 GB SSD

If any more information is needed please let me know I happily attach it.

Thanks in advance!

 

[blathierry]

I have figured out by a lot of trial and error that it has to do with the way ZFS works the VMs randomly freeze on ZFS formated disks even on other OSes.

 

[Apollo]

You should use Netdata to give you some performance observations.
I think the issue maybe related to the VNET used in iocage

 

[blathierry]

You should use Netdata to give you some performance observations.
I think the issue maybe related to the VNET used in iocage

I tested this on a different OS aswell (Proxmox) and the issue only happend when the VM disk was located on an ZFS formated disk.