Windows VM after Failover/Failover test/Move is showing up with an incorrect time

KB Number:
000001349

Symptoms:
After Failover a Windows virtual machine (VM) has the wrong time. Performing a Live Failover, Test Failover, Move, or other recovery operation on a VM running a Microsoft Windows operating system results in the recovered VM reporting a time offset of UTC or "UTC + 0" instead of the expected time offset.

Cause:
Zerto causes: The time offset configuration, also known as the "Local time setting", for any Windows VM is stored in the vSphere NVRAM file. Zerto Virtual Replication does not replicate the NVRAM file of VMware vSphere virtual machines. The time offset will be applied based on the timekeeping configuration of and/or time services available to your recovery site VMware vSphere infrastructure.

Non-Zerto causes: The recovery site vSphere timekeeping configuration and\or timekeeping services are not configured to communicate with the recovered VM, or the recovered VM is not configured to communicate with those systems or services, resulting in the recovered VM being unable to receive any time, timekeeping, or other time offset configuration data

Solution:

Generally speaking when a Failover VM boots the following time keeping steps are performed:

  1. Since Zerto will sanitize the NVRAM File on DR, ESXi by default will set the BIOS time of the VM to UTC. This option cannot be modified (As the ESxi CMOS is always set to UTC  - cf. https://kb.vmware.com/kb/1436 ). Shortly after, offsetting the time from the NVRAM file will commence (if there are any offset values). 
  2. When the OS is loading, If the VM has a time keeping configuration with WIN32TM (Windows Time service) from within the Guest OS – it will continue with the W32TM configuration and will sync up to the defined time source. If the Guest OS  W32TM registry entries are set to use an NTP server for time keeping it will proceed to do so.
  3. If W32TM has been disabled and the VM was set to sync with an ESXi hosts time (which is usually set to sync to an NTP server) VMware tools will be employed to sync the VM guest clock to its underlying Host time.
  4. If none of the above are configured, if some prerequisites for the above are missing or if the ESXi has a hardware-oriented issue the Guest OS will continue to use the BIOS-configured time which was set during BIOS boot up (Which, is always, UTC time).

When failing over some discrepancies pertaining to the above might occur that result in the Windows VM loading with an incorrect time. This depends on the time keeping configuration on the Production site and how the Guest OS is now interacting with the DR sites time keeping policies. As Zerto does not replicate the content of the VMs NVRAM file - which contains the local time for a VM as an offset relative to UTC - a VM will load with BIOS UTC time  and all subsequent time syncing will need to occur thereafter.  When the Guest OS is set to sync to an NTP server, a certain delay for syncing the default UTC to the correct time is expected. All of the mentioned issues should be regarded as a migration artifact arising from a mismatch between the sites. 

VMWare's time keeping best practices for Windows can be found here:
https://kb.vmware.com/kb/1318

Do note that time keeping practices have no barring on the time zone as is shown in the OS and vise versa. 

Using W32TM
 
Common issues may arise from the fact that W32TM was configured to sync time to a Domain Controller. A common conflict occurs when the VM has no access to a domain controller or when the group policy was put in place but is configured incorrectly. Inspecting W32TMs logs can provide insight and proof for these failures (cf. https://blogs.msdn.microsoft.com/w32time/2008/02/28/configuring-the-time-service-enabling-the-debug-log/)
 
As Zerto replicates the protected volumes in their entirety – the registry entries for W32TM are kept as they were when performing a Failover/Failover Test/Move. It is advised to check the WIN32 configuration as is explained here:
https://technet.microsoft.com/en-us/windows-server-docs/identity/ad-ds/get-started/windows-time-service/windows-time-service-tools-and-settings#w2k3tr_times_tools_vwtt
W32TM. Should you see that W32TM cannot configure the correct time due to an offset which is too big, i.e., W32Time private log file a “*TOO BIG* message, please confer to:
https://support.microsoft.com/en-us/kb/884776

Additionally, there are a few queries that can be performed from CMD too that can one you to see the status and peer list for this VMs NTP sync settings on the DR VM (and source too): 

w32tm /query /status - Status viz. peers 
w32tm /query /peers - this will give you the peers configured for NTP sync 
w32tm /query /source - if NTP connection fails it will usually show CMOS as a time authority 

The main place that one should look at is at the peers - if the Peer cannot be resolved by a DNS sever, or cannot be pinged - this might explain why the issue is happening. 

In some cases the W32TM service will not start at all on the recovered VM. One can check the service startup trigger for w32tm by issuing the following command: 
sc qtriggerinfo w32time 

The following MS KB offers two solutions in case the service triggering is not being met on DR (usually when it set with domain or work-group conditions. See method 1 and 2) : 
https://support.microsoft.com/he-il/help/2385818/windows-time-service-doesn-t-start-automatically-on-a-workgroup-computer-that-s-running-windows-7-or-windows-server-2008-r2 

If non of the above applies enabling debugging logs is needed so that one might check why W32TM is failing to sync time. For debugging instructions see:
https://blogs.msdn.microsoft.com/w32time/2008/02/28/configuring-the-time-service-enabling-the-debug-log/ 

Sync VM to ESXi
 
If you choose to sync the Guest OS to the host time please note that it will sync up to the time presented in the hosts Time Configuration  value, as is shown in the configuration tab of vSphere client. It is also recommended to check the ESXi time via an SSH session to rule out any inconsistencies with vCenter.
 
When using Vmware tools for time keeping please perform the following:

  1. In vSphere, and from the VM's configuration, click on Edit Settings > Options > VMware tools.
  2. Mark the synchronise guest time with host box.

For more info see here:
https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vmtools.install.doc%2FGUID-C0D8326A-B6E7-4E61-8470-6C173FDDF656.html
For syncing an ESXi host with an NTP server please see:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2012069
Do note that VMware's prerequisite for ESXi based time keeping is that all other methods are disabled on the Guest OS.

If the VM is not set to use W32TM and no evident causes are observed it is recommended to inspect the local VMware tools logs. For enabling VMware tools logs please see: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007873
 
Do note that if the host is set to use NTP but is not working as expected the VM will sync on initial boot to BIOS time. Moreover, if no outside sources are working for time keeping – when the VMs time is changed manually and the VM reboots, the newly configured time is kept (which is not the case when external time keeping is occurring). This means that when installing a new OS on an empty VM with no time keeping settings - it should get a proper time when configured with the correct time during the installation. 
 


Affected Versions:
All versions

Hypervisor:
VMWare

1 Star2 Stars3 Stars4 Stars5 Stars (3 votes, average: 5.00 out of 5)
Loading...