Azure: Recovery Services VM Backups Failing

This morning I received a number of emails telling me that the backups of some of my Azure VMs had failed with the following message;

Error Code	UserErrorGuestAgentStatusUnavailable
Error Message	VM agent is unable to communicate with the Azure Backup Service.
Recommended Action	Ensure the VM has network connectivity and the VM agent is latest and running. For more information, please refer to http://go.microsoft.com/fwlink/?LinkID=823919

When I have encountered this message previously, one of the following three actions has resolved the error and allowed backups to work again;

1. Connect to the VM and check the "IaasVmProvider" service. This should be running with the startup type set to Automatic and logging on with the local system account. If this is stopped, start the service.

2. Update the agent on the VM by reinstalling the VMAgent binary as outlined at https://docs.microsoft.com/en-gb/azure/backup/backup-azure-vms-troubleshoot#vm-agent. NOTE: Make sure no backups are running on your VM when you do this.

3. Reinstall the VMSnapshot extension by browsing to the VM in the Azure portal and clicking on "Extensions". Uninstall the "VMSnaphot" extension then try to run the backup again. This will reinstall the extension and automatically start the IaasVMProvider service in the OS.

vSphere: Snapshot Error - Change Tracking Target File Already Exists

I ran into an issue when attempting to consolidate and create snapshots of VMs being backed up using NetBackup. The snapshot creation would start but fail at 10% with the following error;

An error occurred while saving the snapshot: Change tracking target file already exists.

 This is quite a easy fix and simply involves moving the ctk files out of the root folder.

1. SSH to ESXi host that the VM is running on.
2. Navigate to the virtual machine directory using the following command;

cd /vmfs/volumes/datastore/virtual_machine/

3. List the contents of the directory with "ls" and look for file including "-ctk.vmdk".
4. Create a temporary directory using the following command;

mkdir temp

5. Move the ctk files into this directory using the following command;

 mv *-ctk.vmdk temp/

6. Attempt to take another snapshot to confirm this is now working.

Windows: Wrongly Hidden "Protected Operating System" Files and Folders

I came across an issue where a folder of SQL backups had been wrongly marked as hidden. They had been classed as "Protected Operating System Files" and so the "hide" box in the properties was greyed out.

Running the below from the command line removes the Hidden and System properties from the folder as well as from all sub-folders inside the top level.

attrib -H -S C:\path\to\your\folder /S /D 

or 

attrib -H -S C:\path\to\your\file.doc /S /D

VMware SRM 6.0: Unable To Protect VMs - "Device not found: CD/DVD drive 1"

When trying to protect a number of VMware Virtual Machines with SRM 6.0 we ran into the following alert on some of the VMs;

"Device not found: CD/DVD drive 1"

This is due to an ISO image being mounted to the Virtual Machine (sometimes VMware tools or an OS disk). If you have a large number of VMs to set back to "Client Device" for CD settings you may want to do this by running the following script;

$vms = Get-Content C:\Scripts\In\cd-vms.txt

foreach($vm in $vms){
    Get-VM -Name $vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$false
}

This will set the CD settings to "Client Device" and confirm silently rather than needing to receive user input for each VM. 

vSphere: List Network Details For All Hosts

I had a requirement to view detailed network information for all host NICs under a vCenter server. The following command generates the required information (including MTU, IP and MAC addresses) and exports this to a .CSV file;

Get-VMHostNetworkAdapter | select VMhost, Name, IP, SubnetMask, Mac, PortGroupName, vMotionEnabled, mtu, FullDuplex, BitRatePerSec | Export-Csv C:\VMHostNetworkDetails.csv

AWS: Enabling Federation With ADFS

Following the guide found in the blog post below;

https://blogs.aws.amazon.com/security/post/Tx71TWXXJ3UI14/Enabling-Federation-to-AWS-using-Windows-Active-Directory-ADFS-and-SAML-2-0

This guide allowed us to set up federation using ADFS 3.0. The only thing to note which does not appear to be in the blog post, is that the Active Directory Security Groups that you create (starting AWS-) must be Global or Universal. This will not work if the group has been created as Domain Local.

If you have created the group at this level you can change this without needing to recreate the Security Group. Simply set the Group Scope to "Universal" as this will then allow the "Global" box to be selected again.

This will also accept nested groups when granting access so you do not need to add all users individually.

Azure: Using Azure AzCopy

When attempting to upload files to a storage account in Azure, if you upload using the portal no Check-MD5 value appears to be generated. In order to have a value generated you need to use the AzCopy tool.

Once you have this installed, open a command prompt and run the tool (default installation location will be C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy\AzCopy.exe)

To upload a single file to a location, use the following command;

AzCopy /Source:A:\Folder\On\My\Computer /Dest:https://storageaccount.file.core.windows.net/filescontainer/folder /DestKey:Storage-Account-Access-Key /Pattern:FileName.ext

Replace the bold text with your parameters

/Source: needs to be the folder location only - do not include the file name or the command will fail. The file name is specified in /Pattern:

Update:

Alternatively, if you have several files to upload to a single location and so want to copy the entire contents of a folder, you can run the command below. This copies all content in a directory including structure;

AzCopy /Source:A:\Folder\On\My\Computer /Dest:https://storageaccount.file.core.windows.net/filescontainer/folder /DestKey:Storage-Account-Access-Key /S