Monday, January 10, 2011

Tips: Backup Fault Tolerant VMs in vSphere 4

If you deploy VMware’s vSphere 4, it will offer you business continuity and disaster recovery like Virtual Machine level backups and high availability features. The editions vSphere 4 Essentials Plus, Advanced, and Enterprise come with VMware Data Recovery, and a number of third party applications (e.g. Veeam's Backup & Replication v5 , "Best of Show" @ VMworld 2010) provide even more powerful features. However, we are going to discuss about the potential of vSphere based DR solutions.
Fault Tolerance is one of the best features of vSphere 4. It is intended to eliminate VM downtime in the event of a host server failure by creating a live shadow instance of the VM on another host and keeping them in “lockstep” synchronization. As it operates at the hypervisor level, it doesn’t need any special features at the VM software level. Moreover, this effect is similar to clustering and has some advantages and disadvantages. The main thing you’ve note about this is the FT enabled VMs can’t be snapshotted.

What is the problem with backing up FT enabled VMs?
As we’ve discussed, the vSphere based VM backup solutions rely on snapshot technology to image live VMs. Most of the users use this when trying to run their first backup of a FT VM. Depending on the application you’re using, you will be allowed either to create backup in the first place or the process will get failed after a short period. Sometimes, the situation will be aggravated due to the conflict information when the VMware 4 is released. At a time, the Fault Tolerant VMs would be allowed a single snapshot for backup purposes. This was not included in the released version of vSphere 4.0. In order to establish that your backup application is able to image an FT VM, it is required to turn OFF the Fault Tolerance for the VM. Please make sure that you’re turning OFF the feature and not disabling it. Otherwise, the snapshot will not be created.

Now, let’s try troubleshooting the issue.
•    It is required to right click on the FT VM in the vSphere Client and select “Fault Tolerance”.
•    Then, select “Turn OFF Fault Tolerance” from the sub-menu. Make sure that the "Disable Fault Tolerance" option will only stop the lockstep synchronisation.
•    The “Turn OFF” option will automatically disable the lockstep sync and then remove the secondary VM completely.
Now, you can start your backup process and if you watch the bottom "Tasks" section in the vSphere Client, you should see the snapshot being created by the backup application. The snapshot should be removed when you finish backup process. Then, right click the VM and it is required to select the "Turn ON Fault Tolerance". This process may take a few minutes as it is required to create a secondary VM and bring it into lockstep sync with the primary. This process is as same as when you originally enabled FT.
With this step, you can get an idea of the potential hazards involved. As already mentioned, the VM will not be FT protected during the backup process.

Now, let’s discuss about using any scripts.
1.    In order to use any scripts, it is required to install the vSphere CLI (Command Line Interface). It is available for download from the VMware website. Then, browse to the "Downloads" section and select vSphere4, then click the "Drivers & Tools" tab. Now, expand the “Automation Tools & SDKs" section then download the version of the vSphere CLI that matches your vCenter installation, note that you want the standard CLI not the "PowerCLI" (the PowerCLI has similar functionality but integrates with the Windows PowerShell):
2.    After finishing the download process, run the file to install it. You can install the downloaded application in the same computer as the VM backup application. For this instruction, it is required to install it in the default recommended folder location.
3.    Now, it is required to download the script file here. Then, save it to the C:\Program Files (x86)\VMware\VMware vSphere CLI\perl\bin folder.
4.    You can see an entry for the "VMware vSphere CLI" in the Start menu, program and VMware menu. If you click on this, you can get a command prompt. You can select the C:\Program Files (x86)\VMware\VMware vSphere CLI\perl\bin folder to the default folder paths.

5.    Now, you can execute your scripts from a standard Command Prompt with the path set. Then, you can try entering the /? in order to see the online help listing all the options for this script. You will see that you can specify explicit or passthrough authentication.
6.    Now, it is required to test running the script with the required options to first turn OFF FT and then to turn it back ON again. Then, it is required to turn in back ON again. You can use the command --server vcenter.domain.local --passthroughauth --operation stop --vmname MyTestFTvm to turn OFF FT. Here, vcenter.domain.local is the FQDN (or IP) of the vCenter Server. Then, it is required to enter the command and run it. This should return some progress information.
7.    If the script fails to turn OFF FT, you can get a better indication from output in the Command Prompt Window or from the Task Status in the vSphere Client. If you wish you can also add the –verbose option to the command that should bring more error messages out.
8.    For turning ON FT, the command should be identical to the turn OFF command except with --operation create. Now, you can make a test VM Fault Tolerant and then remove FT again afterwards.
9.    If you wish to use this new script commands effectively, it is required to co-ordinate them with the backup application.
10.    The CD C:\Program Files (x86)\.... line should not be necessary if you’ve already added the folder to your default system path list. This example is for turning ON FT as it has the --operation create option.
11.    Then, you can save the file to a suitable location and make sure that you’re changing the "Save as type" to "All files" and include a .bat extension at the end of the file name.

12.    Now, you can follow the same steps to replace create with stop in order to create a batch file to turn OFF FT for your VM.
13.    Now, you have two batch files. Subsequently, all it is required to do is changing the --vmname MyTestFTvm option to match the name of your FT VM as shown in the vSphere Client.
Scheduling your Fault Tolerant VM Backups
How to schedule your Fault Tolerant VM Backups?
The first step for every scheduled FT VM backup needs to be turning OFF Fault Tolerance. Therefore, you can use the Windows Task Scheduler to create scheduled task to run your batch file at the appropriate time. However, you can use the Windows Task Scheduler to create a scheduled task to run the batch file at the appropriate time. The Windows Task Scheduler in Windows 2008 Server is different from the old Windows 2003 Server version.

Please make sure that when you create your task, you can specify what Windows user account it should run under. You should specify an account that has sufficient rights on your vCenter Server to change the Fault Tolerance settings for the VM. You can configure the task to run at a suitable time and frequency for your backup schedule.
Now, it is required to create a backup job for the FT VM like other VM backup. However, it is required to schedule it to run at a suitable period after the “Turn OFF FT” task and ensure that it has time to finish the process; otherwise the process will be failed. Usually the process takes 15 minutes, but you can confirm the best for your system with some testing.

B y