How to Troubleshoot Zerto Virtual Manager (ZVM) services crash
Viewed 2432 times
There are 4 different ZERTO services running on a ZVM. This article will focus on the ‘Zerto Virtual Manager’ service.
- Zerto UI will be unavailable
- We can see the following alert:
- "The Zerto Virtual Manager is not connected to site <Peer Site> (<Peer ZVM IP>:9081)"
- "The Zerto Virtual Manager is not connected to site <Peer Site> (<Peer ZVM IP>:9071)"
In order to take these actions, we must have administrator credentials entered in the Wizard.
In addition, after the vCenter connection authentication the ZVM services will be restarted.
- Navigate to the ZVM VM and open the Zerto Diagnostics utility.Choose the option to “Reconfigure Zerto Virtual Manager”
- Follow the Wizard and verify all the details are correct, as shown below:
In figure A please make sure to enter the correct IP addressHost name
The username should have an administrator credentials in vCenter.
If, for some reason, you would like to avoid using the administrator credentials, please review this document for the necessary permissions for the Zerto user.
Once this is done, click on Next.
- The drop-down list(B) will contain a list of the network interfaces present on the ZVM. Select the one you would like to use for replication.
- Below, you will see a list of ports that will be used for Zerto replication. We recommend not changing those unless mandatory.
- If you wish to use a custom certificate, please check the ‘Replace SSL certificate’ box and browse for the replacement certificate(C).
- The next screen will allow you to use online services. Those are not mandatory for this troubleshooting guide.
- The validation step will run now. If everything comes back green, please go ahead and click on ‘Run’ to start the reconfiguration process. The ZVM service will be restarted as a part of the process.
- Verify that you can ping and Telnet over port 443 to the vCenter IPHostname from the ZVM VM.
- Log in to vCenter and verify that the user registered in the Reconfiguration tool has Administrator permissions.
- While reconfiguring, try to replace the vCenter Hostname by IP and vice versa.
- Try to disable the proxy settings, if there are any.
- Verify that you can access to vCenter from the ZVM using the same IPHostname registered in the Reconfigure process and the same credentials registered in the Reconfigure process.
Usually, the addition of the ‘Object reference not set to an instance of an object’ string will indicate that there is a host in maintenance mode in vCenter, or that there is an underlying stuck task for entering/exiting MM in vCenter.
If this is the case, follow these steps:
- Navigate to %ProgramData% > Zerto > ConfigurationLog. You will see 2 files.
- Open the Zerto_Hlogfile.csv
- Search for “host-“(without the quotation marks). You will find a host MoRef ID.
- It is likely that Zerto caught that host in ‘Entering maintenance mode’ state.
- Get the host into maintenance mode, take it out of it and try to run the tool again.
- While in the ZVM VM itself, open Event Viewer
- Navigate to Windows Logs -> Application
- Look for the Zerto service crash events. Those will usually come in pairs of an Application Error coupled with a .Net Runtime error, as shown below:
We will always look at the .Net Runtime errors, as they provide more information related to the crash. In the example, we can see the highlighted “AddressAlreadyInUseException”.
This indicates a network issue. More specifically, it is likely that there is another software that uses the ports that the ZVM assigned to itself (9080 and 9081 by default).
Here are some common .Net Runtime errors that you might encounter:
- System.ServiceModel.AddressAlreadyInUseException– Likely to have another software using the ports that the ZVM intended to use.
- System.Security.Cryptography.CryptographicException– Likely to have an issue with a custom certificate that was installed. Running the reconfiguration tool and double checking the certificate password is recommeneded. Also, make sure to review this KB.
- System.ServiceModel.AddressAccessDeniedException – Usually an issue with the user running the Zerto service. Open the Services window and change the logon user for the Zerto ZVM service to a local account as the first troubleshooting step as shown here. If that fails, try to reconfigure. Otherwise, contact Zerto support.
- System.InvalidOperationException at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(System.Data.Common.DbConnection, System.Threading.Tasks.TaskCompletionSource`1<System.Data.ProviderBase.DbConnectionInternal>, System.Data.Common.DbConnectionOptions, System.Data.ProviderBase.DbConnectionInternal, System.Data.ProviderBase.DbConnectionInternal ByRef)– Common when an external SQL is being used and the login credentials might have been changed.
- System.Data.SqlServerCe.SqlCeException at System.Data.SqlServerCe.SqlCeConnection.ProcessResults(Int32)– Common when there’s an Antivirus software running on the ZVM. Make sure that it does not scan the ZVM folder and that no Zerto-related files have been quarantined.
Navigate to the Settings > Internet options > Advanced > Scroll down to Security and enable All SSL and TLS communications.
Figure G Interoperability Matrix .
You can find all the supported ZVM Operating systems according to the Zerto Version.
- Navigate to the ZVM VM and search for Programs and Features.
- Click Change and run through the repair Wizard.
- Verify you are running the Zerto for the correct underlying hypervisor technology.
- Open the Zerto diagnostic utility and choose the third option ‘ collect local Zerto logs’.
Important : if the Zerto service is crashing the first option of collecting logs will fail
- Gather the Event viewer logs from: Event Viewer > Windows logs > Application, save the logs, and initiate a log bundle collection of 2 hours (if you fail to collect, choose option 3 to collect local logs.)
and contact Zerto Support.