VMware (coordinated) snapshot not working properly

[Ender117]

Hi all, I am using FreeNAS to serve iSCSI storage to my ESXi host, and configured the VMware-snapshot(BTW I feel it needs a better name so it won't get confused with the VM snapshot) in GUI. It appears working fine that it instructed vSphere to snapshot the VMs before a ZFS snapshot was made. However, when I clone and re-mount a previous snapshot, it appears that the VM was snapshotted w/o any state preservation

Code:

Name
15b9e477-e55a-46c9-8b56-fdd7aababbd5
Description
2019-05-19 11:41:01 FreeNAS Created Snapshot
Created
05/19/2019, 11:41:02 AM
Disk usage
40 GB
Snapshot the virtual machine's memory
No
Quiesce guest file system
No

It is my understanding that the unless either of the "Snapshot the virtual machine's memory" or "Quiesce guest file system" option is set it is going to be a crash-consistent snapshot, which is all these coordination aim to avoid.

Is there any further configuration that I need to make? Have anyone of you verify that this feature is working properly? I am running FreeNAS 11.2, ESXi 6.5 (Dell customized image) and vCenter 6.7.

 

[Ender117]

OK, looking at source code (https://github.com/freenas/freenas/blob/master/gui/tools/autosnap.py ) line 452-456

Code:

            VimTask.WaitForTask(vm.CreateSnapshot_Task(
               name=vmsnapname,
               description=vmsnapdescription,
               memory=False, quiesce=False,
             ))

It appears to snapshot with both memory/quiesce false? Is there any misunderstanding on my part, or it's actually a bug?

 

[HoneyBadger]

You could probably just edit the .py script to set quiesce=true - I thought those were the defaults in an older version as well.

Snapping memory can result in a long VM stun/suspend under certain circumstances, so it's usually left disabled.

Try a manual snapshot with quiesce enabled first and validate that it was successful (check Windows event logs for VSS activity, the manifest .zip in the VMFS directory, etc) and then make the change to the Python script perhaps?

With regards to app-consistent vs crash-consistent, most users are satisfied with the latter which is much faster, so perhaps the defaults were changed; but ideally I'd like to see the option exposed through the UI so that it can be decided on a per-VM basis.

 

[Ender117]

Try a manual snapshot with quiesce enabled first and validate that it was successful (check Windows event logs for VSS activity, the manifest .zip in the VMFS directory, etc) and then make the change to the Python script perhaps?

Not sure what you mean here, just making sure that the VM snaphot in vSphere (ahhhhh I hate the naming confusion) works properly?

With regards to app-consistent vs crash-consistent, most users are satisfied with the latter which is much faster, so perhaps the defaults were changed; but ideally I'd like to see the option exposed through the UI so that it can be decided on a per-VM basis.

If people are fine with the latter they can just snapshot the storage w/o letting hypervisor knowing it lol, but I agree that being able to set it as needed would be helpful.

 

[HoneyBadger]

Not sure what you mean here, just making sure that the VM snaphot in vSphere (ahhhhh I hate the naming confusion) works properly?

Correct. I'm just suggesting doing a manual VMware snapshot first to ensure there aren't any problems at that level, before introducing the scripted approach with FreeNAS.

 

[Ender117]

Correct. I'm just suggesting doing a manual VMware snapshot first to ensure there aren't any problems at that level, before introducing the scripted approach with FreeNAS.

Tried to edit the file and snapshot, a few findings:

1. For some reason manual snapshot won't get synced with vSphere in new GUI
2. In old GUI there is a checkbox for this but it just halt on error (I got a PCI pass-throughed VM that cannot be snapped)

Will wait for next scheduled snapshot and see what happens

 

[Ender117]

Correct. I'm just suggesting doing a manual VMware snapshot first to ensure there aren't any problems at that level, before introducing the scripted approach with FreeNAS.

Yep, it appears to be working as it should be, all VMs are snapshot with Quiesced disk. The only problem I have now is when FreeNAS trys to snap an appliance (which cannot be snapped by design), it will send me an email. I now also got my inbox flooded with DC connection lost alerts, is there any way to suppress certain alarms in FreeNAS?

 

[dlavigne]

Yep, it appears to be working as it should be, all VMs are snapshot with Quiesced disk. The only problem I have now is when FreeNAS trys to snap an appliance (which cannot be snapped by design), it will send me an email. I now also got my inbox flooded with DC connection lost alerts, is there any way to suppress certain alarms in FreeNAS?

Refer to Alert Settings: https://www.ixsystems.com/documentation/freenas/11.2/system.html#alert-settings.

 

[Ender117]

Refer to Alert Settings: https://www.ixsystems.com/documentation/freenas/11.2/system.html#alert-settings.

I tried setting to daily but still get email from every snapshot. I ended up move those VM to a separate zvol and call it a day.

 

[Ender117]

https://jira.ixsystems.com/browse/NAS-101944

created a bug report, let's see how it goes.

 

[SMnasMAN]

Ender, does FN vmware aware snapshots only work when the entire VM + all of its disks, are on a FN dataset (ie via nfs).

IE if i have a VM which has disk1 on a FN dataset (via nfs), but disk2 is on a host Direct attached drive (directly attached to the esxi host, that is) - should this work?

i ask as i did a quick test, and when I cloned and re-mounted (via nfs) a previous snapshot, esxi could not restore the snapshot (nor could it power up the vm). (esxi errors were related to invalid snapshot config + missing disk)
thanks

 

[blanchet]

According to my experience, using snapshots to backup VM is error-prone. Even if everything goes as expected, the restoration procedure is picky, and can be done only by seasoned sysadmin. Therefore, in most cases, a specialized product, like Veeam Backup, is prefeable: this is easier for both backup and restoration.

 

[SMnasMAN]

thanks blanchet, i am using veeam, but as im also using zfs snapshots on freenas (on my nfs to esxi shared datastores) i also wanted to use the vmware-snapshot function of freenas (to get vm app-consistent snapshots vs vm crash-consistent snapshots with regular zfs snaps).

Are you saying that the freenas vmware-snapshot function is error-prone?
(so far in my tests im seeing the same thing, see this post of mine: https://www.ixsystems.com/community...hen-i-try-to-revert-to-them-on-a-clone.80748/)

 

[blanchet]

The FreeNAS vmware-snapshot function is not specially error-prone by itself, but the full backup-and-restoration procedure is error-prone, especially with iSCSI datastores.

Do the test yourself:

• snapshot your iSCSI datastore
• try to restore a single VM from the snapshot while the other virtual machines are running

Then do the same with Veeam

• backup your virtual machines with Veeam
• try to restore a single VM from the snapshot while the other virtual machines are running

Then compare the complexity of each procedure

 

[HoneyBadger]

Comparing taking SAN-level snapshots to the functionality of a bespoke backup product is hardly a fair comparison. ;)