Very slow NFS when performing VM snapshot

[francisaugusto]

Ok, hear me out because this is a bit weird...

I have two pools: /mnt/mainpool and /mnt/backup

The `/mnt/backup` is mounted as a datastore on an ESXI host so that I perform backups. I started to use ghettoVCB to backup my VMs.

I noticed recently that out of the blue my nfs shares get painfully slow. They are mounted from `/mnt/mainpool`. Even booting a Raspberry Pi from an NFS share gets incredibly slow.

I couldn't find anything on my logs, but I realized today that, while I experienced this, ghettoVCB was backing up a VM. The backup should be saved on `/mnt/backup`, but still ALL my shares on other disks were super slow. I killed the job, and bang, shares started to be responsive again.

What could be the cause of this? I mean, nothing is being written to my main pool, or read. Why suddenly my whole NFS gets slow? And are there any remediation for this?

I have another ESXI host and I experienced painfully slow VMs there, when they were on NFS, even if they have always run fine. I suspect it was the same problem. The way ghettoVCB works - if I got that right - was to snapshot a VM to my backup pool. Could it be that snapshotting a working VM to another NFS share makes things slow? I see that some of those operations take 6-7 hours, as the machine is very active.

Any tips?

 

[HoneyBadger]

Hey @francisaugusto

Is your system spec in the your signature accurate as shown below?

Supermicro X11SDV-4C-TP8F
96GB DRAM
Supermicro SC825 TQC-600LPB chassis
2x 8TB WD RED HDD (for FreeNAS pool)
1x 4TB Seagate Barracuda (for TimeMachine pool)
Intel SSD 660 NVMe 1TB
ESXi 7
TrueNAS Scale 22.12 (VM)

I suspect your 4TB drive is SMR, and your 8TB HDDs might also be.

 

[francisaugusto]

Hey @francisaugusto

Is your system spec in the your signature accurate as shown below?

Yes, except for the memory

I suspect your 4TB drive is SMR, and your 8TB HDDs might also be.

And why would it matter when the drives being affected are those without a lot of read/write?

 

[HoneyBadger]

And why would it matter when the drives being affected are those without a lot of read/write?

SMR drives can be extraordinarily slow, to the point that even a minor workload will cause them to be unresponsive. If your WD drives are SMR, there is also a known issue with that vendor specifically; however, the 8TB drives may or may not be SMR.

Can you identify which drives belong to which pool, and how you have passed them through to TrueNAS - as you have indicated that you are running TrueNAS SCALE as a VM.

 

[Whattteva]

Which drives are involved in mainpool and which drives are involved for backup? and which drive hosts the hypervisor?