VPG Initial Sync from AWS is Not Progressing
Viewed 728 times
An administrator may notice newly created VPGs to replicate from AWS are not progressing their first sync with the DR site.
ZCA AMIs are created by default as size 'm4.xlarge' and due to performance issues with the default 'm4.xlarge' instance size, the minimum required ZCA AMI instance size for Zerto running on version 7.0 and later is now 'm5.xlarge'. See the Zerto Virtual Replication AWS Enterprise Guidelines document for further details.
Please see the following link for more information on all AWS Instances:
Another potential cause is the KMS IAM policy is not set accurately to allow for EBS encrypt/decrypt when the customer has replaced the default AWS-provided KMS keys with their own generated keys. Due to this, when the volume is attempted to be encrypted automatically after successful creation (expected in Zerto 7.0+), the encryption fails due to permission issues and the volume is effectively never fully created. Yet, since a vol-xxx ID is returned to Zerto via the API call, the AttachVolume API call continues in a loop of trying to attach said volume yet it does not exist.
As for the "Cannot attach volume 'vol-xxxxx' with Marketplace codes as the instance 'i-xxxxx' is not in the 'stopped' state" error, this is an AWS limitation. AWS specifically states:
"This limitation is related to the need for bringing the billing code of the original instance launched from an AWS Marketplace AMI to the new instance. Product codes are copied from the root volume to the instance, and this process happens when the instance is started. That way, the instance being in stopped state ensures that the process of carrying the billing code happen. The requirement above is listed in our documentation of prerequisites for attaching an EBS Volume to an Instance: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html"
Additionally, according to the documentation linked above, Zerto cannot attach Marketplace Windows volume(s) to Linux instances (ZSAT). Therefore, mitigation will need to come from the Zerto side to allow for supportability of such Marketplace AMIs.
After creating a VPG from AWS, VPG initial-sync may not progress past 0% and could potentially revert back and forth between x% and 100%.
Another potential scenario is a ZSAT continuously fails to "AttachVolume" as the volume ID is not valid per the Cloud Trail logs.
The ZSAT may also fail to "AttachVolume" due to "Cannot attach volume 'vol-xxxxx' with Marketplace codes as the instance 'i-xxxxx' is not in the 'stopped' state."
To resolve a VPG that is stuck in an initial sync state due to the ZCA instance size being 'm4.xlarge':
Shutdown the ZCA.
Change the instance size of the ZCA from m4.xlarge to an m5 class instance size, minimum 'm5.xlarge'.
Power on the ZCA.
Monitor the initial sync via the Zerto GUI to ensure the initial-sync completes.
To resolve a VPG initial sync issue where a volume is clearly failing to attach to a ZSAT per AWS Cloud Trail logs and the customer is utilizing their own KMS keys:
1. Add the KMS permissions below to the IAM Policy attached to the ZCA as such:
2. Or, you can disable EBS encryption for ZCA(s) running 7.0 by engaging Zerto Support to assist with the following Zerto KB . This way encryption/decryption will not be attempted and avoid this issue entirely.
NOTE: You can find more information regarding the AWS KMS IAM policy via here and here.
With regards to the "Cannot attach volume 'vol-xxxxx' with Marketplace codes as the instance 'i-xxxxx' is not in the 'stopped' state" error, the only workaround is to clone the current instance to a new EC2 instance with a different AMI that is not from the AWS Marketplace. Otherwise, Marketplace AMIs are not supported for Zerto replication out of AWS at this time.