Article number
000004678
Affected Versions
7.5
8.0
8.5
Source Hypervisor
VMware
Target Hypervisor
VMware

Problem with Mismatched Owner IDs of Protected and Recovery Sites resulting in Stuck Bitmap Syncs

Viewed 514 times

Summary

This article covers a scenario in which a partial unpairing of sites may leave behind stale entries in the recovery database impacting VPG syncs when the sites are re-paired.
The Owners ID represents a pairing between a Protected and Recovery Site.

Root Cause

To summarize, the mismatched OwnersID left from a previous pairing wasn't cleared properly during an unpair task.
The VRA therefore sets the wrong Owner ID causing the VPG to be in a stuck bitmap sync.

In further detail:
1. VPG is created, the protected VRA will log a task to create the protected volume (logical). Here we can see the OwnersIdentifierGuid set to: e83762e8-6f22-4c2a-b502-1c1901d485f1. The IsOwner=1 means that it is claiming to be the protected owner of this volume. Afterwards it sends that IsOwner command to all its peers.
,Control,791,VRA,CreateProtectedVolumesRsp_SR VraImpl,createProtectedVolumes,<VolumeIdentifier=0000000000000000000 OwnersIdentifierGuid=e83762e8-6f22-4c2a-b502-1c1901d485f1 HypervisorDlp=<VmUuids=<VmUuids=<VmUuid=42 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 VmInstanceUuid=50 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 > Controller=0 UnitNumber=2 > CloudDlp=NA IsRdm=0 SizeInLb=18876416 IsOwner=1 IsNew=1 >

2. Moving onto the recovery VRA, the corresponding VRA create the mirrors and same ownersIdentifier=e83762e8-6f22-4c2a-b502-1c1901d485f1:
,Info,903,VRA,MirrorStatus_SR Mirror,getStatus, m_mirId=1111111111111111111 Adding mirror. status=mirror status <1111111111111111111> = targetDlp=<VmUuids=<VmUuid=42 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 VmInstanceUuid=50 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 > Controller=2 UnitNumber=5 > rdm=0 settings=<IsUseCompression=1 IsSwap=0 Priority=1 IsMd5Enabled=0 LogBacklogInMinutes=60 AdditionalFlags=0 > ownersIdentifier=e83762e8-6f22-4c2a-b502-1c1901d485f1 prtVol=0000000000000000000 state=OBJECT_STATE_ACTIVE isCreatingMirror=0 isCreatingView=0 criticalCp=NA isOnCriticalCp=0 latestCpId=0 latestCpAge=26884471 volumeStatus=RCSUCCESS incrementalReadCp=NA occupiedByLog=1 occupiedByResync=0 occupiedByMisc=0 totalLogOccupied=1 historyInSec=0 totalMiniFifoOccupied=0 SYNC_STATESYNC syncDisableReasons=0000000000 syncCause=SYNC_CAUSEACTIVATION totalBlocks=0 alreadySynced=0 startTime=2021-01-01 00:00:00 VIEW_TYPENONE BACKUP_STATENONE CLONE_STATE_NONE <HardenLbsCounter=0 IncommingBytesCount=0 AppliedLbsCount=0 >

3. Upon any restart of the VRA, it will perform a task to set the peer VRA, locally and remote:
, ----------------Log Start--------------------- branch: glenlivet_u3 build: 100 commit: 8eabe7249654e96704034cc1814df4813d1f02d2 ----------------------
,Control,973,VRA,GeneralRsp_SR VraImpl,setPeerVra,<PeerRcmIdentifier=888888888888888888 OwnersIdentifierGuid=983ac18d-8445-4a22-8c7e-3a162d04faa7 PeerNetworkSettings=<PeerIp=x.x.x.x PeerDataPort=4008 PeerControlPort=4007 ZcpPort=4009 ZvmPort=4006 > >

4. Due to the OwnersIdentifierGuid mismatch, the replication becomes stuck

Symptoms

  • Upon creation of a VPG, the VPG will immediately enter a bitmap sync.
  • The bitmap sync will not progress, only to report '0 out of 0 GB' remaining with no movement.
NOTE: This symptom is also similar to when VRA to VRA communication is blocked. Please ensure VRA ports are properly opened and reachable.

Solution

  • Issue will be resolved in ZVR 9.0
  • If on previous version Zerto, the steps to resolve include a Zerto database edit and Recovery VRA restart. Please contact Zerto Support to confirm issue and to assist with resolution.