VM slows down over time, then mysteriously work with repackaged zvol.

[Anthony Chastain]

I have a 28TB FreeNAS 11.2 with 64GB RAM, a 256GB L2ARC and a 16-core processor.

I am running FreePBX (based on CentOS) on a VM with 4GB RAM and 4 cores.
I'm also running a few other VMs.

I've noticed that all my VMs get slower and slower as time goes on. It gets to the point that it takes 20 minutes to start up a simple linux VM that originally took 30 seconds to start. The guest OS doesn't seem matter. Also, this has been happening since FreeNAS 9.10, so it doesn't seem related to updates. It takes about 6 months to get to where it's unbearably slow.

I've tried:
• Restarting the VMs
• Restarting the NAS server
• Recreating the VM based on the same info
• Doubling the memory, processor cores, etc.

None of these things worked.

Here's what did work though:
I took a snapshot of the ZVOL, saved it to a file, reimported it to a new ZVOL and connected the VM to that.

This sped things up instantly back to the original speed. And it stays that way for about 6 months again.

So what exactly is going on here?

 

[dlavigne]

That is worth reporting at bugs.freenas.org. Please post the issue number here.

 

[bigphil]

Sounds like fragmentation to me. Hard to confirm now that you have it on a new zvol, but I'd bet money on it. When it starts getting slow, take a look at the output of "zpool list" and report back on the frag %.

 

[kdragon75]

Sounds like fragmentation to me. Hard to confirm now that you have it on a new zvol, but I'd bet money on it. When it starts getting slow, take a look at the output of "zpool list" and report back on the frag %.

That's what I thought at first too. The issue with that is that a zol is not a fixed span of blocks on the pool. Therefore space while reserved, is not limited to a specific area. It would be no more prone to fragmentation than any other files. As he is snapshotting, sending to a file (I read this as on the same pool) and then re-importing (ZFS send...) as a zvol, I would expect overall pool fragmentation to only get worse. Also keep in mind that ZFS reported fragmentation is the fragmentation of the freespace, not the files/data.