Article number
000003168

VPG initial sync from AWS not progressing

Root Cause

ZCA AMIs are created by default as size 'm4.xlarge' and due to performance issues with the default 'm4.xlarge' instance size, the minimum required ZCA AMI instance size for Zerto running on version 7.0 is now 'm5.xlarge'.  (http://s3.amazonaws.com/zertodownload_docs/Latest/Zerto%20Virtual%20Replication%20AWS%20Enterprise%20Guidelines.pdf),

ZCA performance can be further improved by converting the default 'm4.xlarge' ZCA instance size to 'm5.xlarge' as 'm5.xlarge' is less expensive and provides a newer hardware generation than 'm4x.large'.

Please see the following link for more information on all AWS Instances:

https://www.ec2instances.info/


Another potential cause is the KMS IAM policy is not set accurately to allow for EBS encrypt/decrypt. Due to this, when the volume is attempted to be encrypted automatically after successful creation (expected in 7.0+), the encryption fails due to permission issues and the volume is effectively never fully created. Yet, since a vol-xxx ID is returned to Zerto via the API call, the AttachVolume API call continues in a loop of trying to attach said volume yet it does not exist.


As for the "Cannot attach volume 'vol-xxxxx' with Marketplace codes as the instance 'i-xxxxx' is not in the 'stopped' state" error, this is an AWS limitation. AWS specifically states:

"This limitation is related to the need for bringing the billing code of the original instance launched from an AWS Marketplace AMI to the new instance. Product codes are copied from the root volume to the instance, and this process happens when the instance is started. That way, the instance being in stopped state ensures that the process of carrying the billing code happen. The requirement above is listed in our documentation of prerequisites for attaching an EBS Volume to an Instance: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html"

Additionally, according to the documentation linked above, we cannot attach Marketplace Windows volume to Linux instances (ZSAT). Therefore, mitigation will need to come from the Zerto side to allow for supportability of such Marketplace AMIs.

Symptoms

After creating a VPG from AWS, VPG initial-sync may not progress past 0% and could potentially revert back and forth between x% and 100%.

Another potential scenario is a ZSAT continuously fails to "AttachVolume" as the volume ID is not valid per the Cloud Trail logs.

The ZSAT may also fail to "AttachVolume" due to "Cannot attach volume 'vol-xxxxx' with Marketplace codes as the instance 'i-xxxxx' is not in the 'stopped' state."

Solution

To resolve a VPG that is stuck in an initial-sync state due to the ZCA instance size being 'm4.xlarge':

1) Shutdown the ZCA
2) Change the instance size of the ZCA from m4.xlarge to an m5 class instance size, such as 'm5.xlarge'
3) Power on the ZCA
4) Monitor the initial-sync via the Zerto GUI to ensure the initial-sync completes

NOTE: Should the initial-sync still not progress after the instance size was migrated to an m5 class instance size, please contact Zerto Support to assist with additional troubleshooting.


To resolve a VPG initial sync issue where a volume is clearly failing to attach to a ZSAT per AWS Cloud Trail logs,

1) Set the KMS IAM Policy as such

{
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey",
"kms:CreateGrant",
"kms:ListAliases"
],
"Resource": [
"*"
]
},

2) Or, you can disable EBS encryption for ZCA(s) running 7.0 by following the following Zerto KB. This way encryption/decryption will not be attempted and avoid this issue entirely.

NOTE: You can find more information regarding the AWS KMS IAM policy via here and here.


With regards to the "Cannot attach volume 'vol-xxxxx' with Marketplace codes as the instance 'i-xxxxx' is not in the 'stopped' state" error, the only workaround is to clone the current instance to a new EC2 instance with a different AMI that is not from the AWS Marketplace. Otherwise, Marketplace AMIs are not supported for Zerto replication out of AWS at this time.