You can, but maybe you do not want to test them all at the same time. But don’t worry, there’s a way to do completely automated tests even for huge environments.
Surebackup
UPDATE 11-03-2016: one of our customers, Hans Leysen, took this script as the starting point and improved it by adding an hash table, saved in an xml file, to check only VMs that are yet tested. So, no more randomness, but a precise procedure to test them all over the days. The great thing is that this script is running in his production environment. Thanks Hans!
SureBackup in Veeam is a great solution. With it, you can take any backup and then run automated tests against each protected VM. By powering on the VM in a virtual lab, isolated from the original VM in production, you can first of all be sure that the virtual machine can be powered on when needed from a backup, but you can also configure SureBackup to tests different things like vmware tools heartbeat (and again, it would mean the OS is up and the services are running, as VMware tools is one of those services), network ping (so you can verify the network stack is up), and you can also run scripts of any kind against the applications. I’ve seen customers doing amazing things like a sql query that goes into a MS SQL database and retrieves a table, so they can be sure that the database is up nd running, it’s answering on the correct port, accepts the user used into the script, and return the correct data.
Obviously, this is a configuration that can and should be done for those critical VMs, your “preferred pets” that you care about the most. You go and configure a dedicated Application Group, a dedicated SureBackup job, probably you also have configured a dedicated Backup job to protect those VMs in the first place, as they may require specific settings like VSS, credentials, encryption and so on.
But out there, there are many other VMs in your environment, and managing all of them one by one is crazy.
On the backup side, you can automated all the backup activities by using tags for example. I’ve explained how to do this in a whitepaper I wrote: Using Veeam and VMware vSphere Tags for Advanced Policy-driven Data Protection.
But what about SureBackup? You can easily backup 1000 VMs per day, actually we have customer protecting environments at least with one more zero in the number. SureBackup requires just a spare hypervisor server to power-on the virtual machines from the backups, but then if you have a dynamic environment, with VMs coming and going, you would have to re-program surebackup jobs each time to be sure that each VM is tested. It’s not a problem of available resources, but more of automating the tests as much as possible, even if the environment changes frequently.
Dynamic SureBackup
I developed this idea few months ago when discussing a Veeam deployment with a large customer (yes, I work on the field from time to time :)). He loved since the very beginning the idea of SureBackup, because it would have been a completely automated solution for the internal audit that requires, among other things, that backups and replicas are tested periodically, and the results verified.
Hey, SureBackup can do it! But for 1000 VMs with different settings, and many of them new every month?
Here’s where the idea took form. After removing from the list those 30-40 “special VMs” that I talked about before, we ended up with around 1000 “regular” VMs. For those VMs, we designed something like this:
– each day, every VM is protected with a daily backup job
– after backups are complete, a new SureBackup job is created via powershell
– the job randomly select a certain amount of VMs that don’t have the “special” tag applied to them
– with those VMs, the script creates a new Application group
– the Application Group is tested with a SureBackup Job
– after the test is over and the report has been sent, Surebackup Job and Application Group are deleted
– on day 2, the same procedure is repeated
You probably see where I’m going. Each day some random VMs are tested. On the short run, it may happen that a given VM is tested 3 times, and some are never tested. But here is where the improvement I’ve received from Hans comes into play: an hash table is created on the first execution, or updated in the following days. Once a VM is tested, it receives a value of 1, every other non tested VM has a value of 0. At each execution, the script only selects the VMs to test among those having a value of 0, that are those still non tested. If you let the script run for a long enough period of time, each VM will be tested, and then the hash table will be reset, and the tests will start again from scratch. The minimum required time to rotate in the job 1000 VMs is 50 days if you run 20 VMs per day, so less than two months; if you have an internal audit happening every 6 months like in my use case, we are now sure that by the deadline, all VMs would have been tested.
So, here is the script. As usual, people with better Powershell skills than me can go and improve it at will as Hans did, and maybe post here their changes so I can integrate them in my original script:
# Variables for script ------------------------
$AppGroupName = "Dynamic App Group"
$SbJobName = "Dynamic Surebackup Job"
$SbJobDesc = "Dynamic App Testing"
$Date = (get-date).AddDays(-30)
$VirtualLab = "Virtual Lab 1"
#Variables for function selectUntestedVMs
[string]$VeeamBackupCounterFile = ".\hashtable.xml"
[int]$NumberofVMs = 20
###############################################
# Functions ------------------------------
Function selectUntestedVMs
{
param([string]$fVeeamBackupCounterFile,[int]$fNumberofVMs,$fVbrObjs)
$fHashtable = @{}
$fTestVMs = [System.Collections.ArrayList]@()
$fDeletedVMs = [System.Collections.ArrayList]@()
# Import VeeamBackupCounterfile if exists from a previous iteration
if(Test-Path $fVeeamBackupCounterFile)
{
$fHashtable = import-clixml $fVeeamBackupCounterFile
}
# Check if all VM's were tested
# if so the hashtable is cleared
if(!($fHashtable.Values -contains 0))
{
$fHashtable = @{}
}
# Add newly created VM's from backup
Foreach($fVbrObj in $fVbrObjs)
{
if(!($fHashtable.Contains($fVbrObj.name)))
{
$fHashtable.Add($fVbrObj.name, "0")
}
}
# Remove old VM's from hashtable
$fHashtable.getEnumerator() | %{ if($fVbrObjs.name -notcontains $_.name) {$fDeletedVMs += $_.name}}
$fDeletedVMs | foreach{ $fHashtable.Remove($_)}
# Sort hashtable by Value
# Used new object because sorting the hashtable converts it to dictionary entry
$fHashtableOrdered = $fHashtable.GetEnumerator() | sort -Property "Value", "Name"
# Select least tested VMs and increment their value to 1 (tested)
for ($i = 0; $i -lt $fNumberofVMs; $i++)
{
$fTestVMs += $fHashtableOrdered[$i].Name
$fHashtable.Set_Item($fHashtableOrdered[$i].Name, 1)
}
#Save hashtable to file for the next iteration
$fHashtable | export-clixml $fVeeamBackupCounterFile
Return $fTestVMs
}
##########################################
asnp "VeeamPSSnapIn" -ErrorAction SilentlyContinue
# Check if there is no dynamic surebackup job running
if(!(get-vsbjob -Name "Dynamic Surebackup Job"))
{
# Find all VM objest successfully backed up in last 1 days
$VbrObjs = (Get-VBRBackupSession | ?{$_.JobType -eq "Backup" -and $_.EndTime -ge $Date}).GetTaskSessions() | ?{$_.Status -eq "Success" -or $_.Status -eq "Warning" }
# Call function selectUntestedVMs
$TestVMs = selectUntestedVMs -fVeeamBackupCounterFile $VeeamBackupCounterFile -fNumberofVMs $NumberofVMs -fVbrObjs $VbrObjs
# Create the new App Group, SureBackup Job, Start the Job
$VirtualLab = Get-VSBVirtualLab -Name $VirtualLab
$AppGroup = Add-VSBViApplicationGroup -Name $AppGroupName -VmFromBackup (Find-VBRViEntity -Name $TestVMs)
$VsbJob = Add-VSBJob -Name $SbJobName -VirtualLab $VirtualLab -AppGroup $AppGroup -Description $SbJobDesc
Start-VSBJob -Job $VsbJob
# Remove the old App Group
Remove-VSBJob -Job (Get-VSBJob -Name $SbJobName) -Confirm:$false
Remove-VSBApplicationGroup -AppGroup (Get-VSBApplicationGroup -Name $AppGroupName) -Confirm:$false
}
Let’s explain what the script does:
– first of all, the job looks for an existing and configured virtual lab, that you have to configure in advance. The lab needs to be called “Virtual Lab 1”, or you can change its name
– the variable “NumberofVMs” defines how many VMs you want to test in each run. Change the number at will
– then, it retrieves the list of VMs that were successfully backed up during the last 30 days (adddays -30, you can change this) and add them to the hashtable with a value of 0. This is because we do not want to test a VM that doesn’t have a consistent restore point, but also avoid to reset the hashtable if for any reason a backup is not executed for one day. Also, with this value you can test VMs that are only saved once a week for example
– then, out of the list of untested VMs, it select the “NumberofVMs” virtual machines
– the job rebuilds each time a new application group and inserts the selected VMs in it
– the job is executed, and by default it only tests VMware tools. This is because each VM can be different, so it cannot do specific advanced tests like TCP ports or services. For those VMs, better to configure a custom SureBackup job
– if Veeam is properly configured for email notifications, each job will send out the results, so the results can be parsed over a long period of time and obtain statistics
– each tested VM is modified in the hash table with a value of 1. So, as long as there are VMs with value 0, a VM will not be tested twice before each other VM is tested
– if no more VMs exist in the hashtable with a value of 0 the hashtable is reset. This also happens if there’s no xml file, so if you delete the xml, the process will start from scratch
Finally, The hashtable also performs a few checks in order to keep itself up to date with the restore points available in Veeam. New VM entries are automatically added and old ones are removed from the hashtable. For example when a VM is deleted, there will be a point in time when there are no more “recent restore points” available in Veeam from that VM. The hashtable will need to be updated to the new status of Veeam. If a new VM is deployed and backed up by Veeam after the hashtable was created, it will need to be added to the hashtable.