- This topic has 46 replies, 33 voices, and was last updated May 10, 2022 by Carlos C.
Driver I/O cache ran out of memory on host
Just upgraded from 8.5.3 to 9.0 Update one and now one of our host is getting a alert message – : Driver I/O cache ran out of memory on host
. This may impair replication performance. Current I/O cache size: 96 MB. Last full I/O cache memory occurrence: 9 seconds ago. Full I/O cache memory has occurred 1 times in the last 9 seconds.
Not sure what to do
Same issue after 9.0 U1 upgrade a week ago, power cycle of VRA stopped it for a bit but came back on 60% of my VRA’s. My RPO has not increased so this is a bug IMO.Fernando GOctober 1, 2021 02:26:31 PM
Same issue here after upgrading to 9.0 U1 from 8.5 U3Duane WOctober 5, 2021 12:33:50 PM
I am having the same issues as well after upgrading to 9.0 U1Allen GOctober 5, 2021 01:35:59 PM
I am having the same issue after upgrading to 9.0 U1.Ahillan TOctober 5, 2021 08:47:06 PM
This error is a new error that was introduced in Zerto 9.0 U1. Basically, the alerts were triggered by write bursts in greater values than the cache configured for the VRA.
For disabling this new warning.. perform the following steps:
Download the Tweak Editor tool from the following link into both machines:
2. Extract the archive to any Temp folder and run the tool (ZVM O/S);
3. Navigate to “VRA tweaks” section.
4. Choose Z-VRA (You can apply this tweak for one or more VRAs).
5. Use the “+” button to add a new tweak;
6. In the Tweak name write the current tweak name: t_queryDriverAlerts
7. In the Tweak value field specify tweak value: false
8. Click “apply” and close the tool.William COctober 7, 2021 03:40:46 PM
I am having the same issues as well after upgrading to 9.0 U1 I tried the tweak t_queryDriverAlerts , but it says it does not exist any other names available for the tweak ?Travis BOctober 7, 2021 04:58:01 PM
You can add me to this list as well. Upgraded to 9u1 yesterday and now this morning I am seeing this error. As long as it isn’t service affecting I may just live with it till Zerto gets it fixed.
This error also appeared to me after the update. I am waiting for the problem to be solved.Matt MOctober 9, 2021 01:52:23 AM
Same thing here 9.0 to 9.0 U1. If this is a known issue, it would be nice to see a KBDuane WOctober 11, 2021 01:48:34 PM
Same issue with me here also. Just upgraded to 9.0 u1. Looks like Zerto has removed the tweaks tool download option. I get the following error when trying to download and update my current tweaks tool: “Only Zerto employee’s can download this tool”Adam HOctober 11, 2021 07:39:32 PM
Add me to the list! This started happening after upgrading from 8.5U3 to 9U1.Sushil KOctober 12, 2021 03:40:28 PM
I am also in the same boat. It doesn;t help that Synch is not going beyond 3% . Any resolution ?Aaron COctober 20, 2021 05:07:17 PM
The resolution of alert will depend on the frequency of the full I/O cache events. In the most severe case, the VPG may end up in a long bitmap sync.
1. As some I/O bursts are expected, the VPG could enter a bitmap sync and catch up once the system levels out and this alert allows you to identify on which host this issue occurred. The identification of the host could be useful in troubleshooting a stuck/slow bitmap sync, performance troubleshooting and load balancing your protected workloads.
Refer to the following KB for more info on “How to Troubleshoot Stuck Bitmap Syncs ”
2. These alerts can be disabled at the VRA level. Due to the exposure of this event in 9.0u1, if the expected I/O bursts line up with the alerts in a consistent manner, then you can safely ignore/acknowledge or disable the alerts. Contact Zerto Support for the required steps to disable the alerts at the VRA level or to confirm and investigate this alert.
3. Open a support case to assist in increasing the I/O cache size. The current methods to increase the IO cache size involves reloading the Zerto driver at the host level rather than increasing VRA VM memory configuration. When opening the support case, to expedite and validate the issue, please include at a minimum a 1 hour log bundle and include the associated VRA/host in the alert.Mike COctober 21, 2021 02:49:40 PM
Also seeing this issue. Zerto support error code shows “DRV0001” and suggest to spread VMs across multiple hosts. Nothing in our environment has changes as far as workload is concerned. Possible bug or new feature?
Error pulled from ZVM:
Driver I/O cache ran out of memory on host [ESXi host]. This may impair replication performance. Current I/O cache size: 96 MB. Last full I/O cache memory occurrence: 10 hours, 2 minutes and 59 seconds ago. Full I/O cache memory has occurred 33 times in the last 1 day.Carlos COctober 24, 2021 02:18:52 AM
Tweak Editor tool is not longer available has some the link?