Recently I visited a client who was having failing Exchange 2010 backups with VEEAM Backup & Replication 7.0 Patch 4.
The error was: “Processing Error: VSSControl: Failed to freeze guest, wait timeout”
This short article describes the problem and how we resolved the problem in the end.
The VSS Exchange writer has a hard-coded freeze time-out of 20 sec, so it can freeze for a maximum of 2o sec. before it automatically unfreezes and subsequently causes a VSS Application-aware backup to fail. Logging showed that VEEAM was taking to long before unfreezing the VSS writers.
VEEAM has 2 ways of communicating with VSS in the guest:
- VMware VIX API
VIX is an API that lets you programmatically control the products that host VMware VMs, and control the VMs themselves without having direct network access to the VM itself. The VIX method is primary used by VEEAM in multi-tenancy environments where no direct communication is possible from the VEEAM Backup Proxy to the VEEAM guest, so there the communication has to be handled by VIX API.
In this case VEEAM was using VMware VIX API communication with the guest to freeze/unfreeze the VSS writers:
This communication was too slow (command for unfreeze happend >20 sec. after the freeze, which caused the VSS Exchange writer to fail and subsequently the backup job to fail. But why was VEEAM B&R using VIX API instead of RPC, since there was nothing blocking the network communication between VEEAM & the vm and by default it should try to use RPC first.
After some troubleshooting with Jan van Leuken (SE @ VEEAM) we discovered that the following registry key was set, which caused the order in which the VEEAM tries to communicate with VSS in the guest to be reversed (so it first tried VIX instead of RPC):
[HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication]
Value = 1 (First VIX, then RPC)
Default Value = 0 (First RPC, then VIX)
After setting this registry setting back to the default value (or removing the setting completely) and restarting the VEEAM services, the backups completed successfully within the 20 seconds freeze.
Improvement in VEEAM B&R v8
To make further troubleshooting more easy in VEEAM v8 there’s a button to test if RPC/VIX is working for credentials used in a backup job, but I guess we still need to look at the logging which communication method is used.