Microsoft Azure virtual machines are based on some of the same virtualization technologies in Windows Server Hyper-V. From a storage perspective, this means that the underlying disks and images are based on Microsoft Virtual Hard Disk (VHD) format. As you have seen in Chapter 3, virtual machine disks are stored in Microsoft Azure Storage, which itself is a highly scalable, durable, and available service. In this chapter we will dig deeper into how you can use PowerShell to manage these disks and configure storage for your virtual machines.
To start with, let’s explore how to upload and download VHD files with Microsoft Azure storage. As you have already seen, it is possible to create VHD files in the cloud without the need to create them locally and upload them. However, I feel that starting at this point makes some of the concepts later in the chapter easier to explain.
The Microsoft Azure PowerShell cmdlets provide two cmdlets for uploading and downloading VHD files to a storage account: Add-AzureVHD
and Save-AzureVHD
. Let’s explore each of these cmdlets using a simple walk-through.
Using the Microsoft Azure PowerShell cmdlets over most generic storage tools has its advantages. In addition to easy scriptability, the cmdlets do not treat the VHD files as plain old blobs of data. Instead, the cmdlets have knowledge of the native VHD file format. This means that the cmdlets can optimize how they upload or download VHDs.
The Add-AzureVHD
cmdlet supports an optimized upload. When you upload a VHD, the cmdlet evaluates which bytes of the file have data and transmits only the written bytes to storage while keeping the overall structure of the file intact.
Microsoft Azure currently supports only VHD files in fixed format. That does not mean that you have to convert your dynamic disks to fixed before uploading. The Add-AzureVHD
cmdlet can dynamically convert from dynamic to fixed on upload. Unfortunately, at this time it does not support converting from VHDX to VHD, but that scenario can easily be accomplished with the Convert-VHD
cmdlet that comes with Server 2012 R2 and Windows 8.1.
The Add-AzureVHD
cmdlet supports three optional parameters shown in Table 5-1.
-OverWrite | If the VHD exists on the destination, you can overwrite it by passing this parameter. |
-NumberOfThreads | The default value is 8. Change if you feel that more or less parallelization would improve your upload. |
-BaseImageUriToPatch | When uploading a differencing disk, specify the image name to patch. |
Authentication with the Add-AzureVHD
and Save-AzureVHD
cmdlets works the same way as the other Microsoft Azure cmdlets. You can call Select-AzureSubscription
prior to their usage, and they will use the selected subscription and authentication method. As an alternative, shared access signatures are also fully supported. You can use a shared access signature by specifying it as part of the destination URL.
To try uploading a VHD on your own, create a new PowerShell script in the PowerShell ISE named chapter5upload.ps1. Add the code shown in Example 5-1 to the file. Ensure that you replace the placeholder values with real values. Do not execute the script yet because you still have to create the mydatadisk.vhd file using the disk management tool in Windows.
This example assumes that you have the virtual machine named ps-vm1 from Chapter 3 (feel free to substitute another virtual machine in its place).
$subscription
=
"[subscription name]"
$storageAccount
=
"[storage account name]"
$serviceName
=
"[cloud service name]"
$vmName
=
"ps-vm1"
Select-AzureSubscription
$subscription
$source
=
"C:VHDFilesmydatadisk.vhd"
$destination
=
"https://$storageAccount.blob.core.windows.net/upload/mydatadisk.vhd"
Add-AzureVHD
-LocalFilePath
$source
-Destination
$destination
Add-AzureDisk
-DiskName
"mydatadisk"
-MediaLocation
$destination
Get-AzureVM
-ServiceName
$serviceName
-Name
$vmName
|
Add-AzureDataDisk
-Import
-DiskName
"mydatadisk"
-LUN
1
|
Update-AzureVM
The call to the Add-AzureDisk
cmdlet associates the VHD file with the mydatadisk disk name. Once the name is registered, you can easily attach or detach the disk by name using the Add-AzureDataDisk
and Remove-AzureDataDisk
cmdlets.
These cmdlets work by modifying the returned virtual machine configuration from Get-AzureVM
by either adding or removing the referenced disk name. The modified configuration is then passed to Update-AzureVM
, which in turn calls the appropriate API for passing the configuration to Azure to perform the update.
Before you can execute the script, you need to create the VHDFiles folder on the C: drive and the mydatadisk.vhd file. Using the built-in functionality in Windows 7 and above, you can create the VHD file locally, mount it, add data to it, and then upload it using the Azure cmdlets.
To do this on your own, open Control Panel→Administrative Tools, and then open Computer Management (see Figure 5-1).
When Computer Management is open, expand Storage, right-click Disk Configuration, and click Create VHD (see Figure 5-2).
When the dialog opens, specify C:VHDFilesmydatadisk.vhd in the path (make sure you have created the VHDFiles folder first!) and specify 50 MB to keep the disk small but still usable. You should also ensure that VHD is selected and not VHDX before creating the VHD (see Figure 5-3).
After the disk is created, you will then need to intialize and format it before adding data. To initialize the VHD, right-click on the left side (where it says Disk 1) and click Initialize Disk (see Figure 5-4). A new dialog will open; change the partition table format to MBR and click OK.
After the disk is initialized, right-click on the right side of the disk and select New Simple Volume. Accept the defaults for the New Simple Volume Wizard and let Windows format the drive (see Figure 5-5).
When the disk formatting has completed, open the new drive in File Explorer and put data of some kind on it (such as a simple text file). See Figure 5-6.
Finally, detach the disk from Windows by going back to Computer Management→Storage→Disk Configuration, and then right-clicking on the new disk and selecting Detach VHD (see Figure 5-7).
You are now ready to upload the VHD and attach it to the virtual machine!
Execute the script by pressing F5 or by highlighting the script and pressing F8 to start the upload (see Figure 5-8).
If you are following the exercise, the next step is to validate that the disk was actually uploaded and attached to the virtual machine (see Example 5-2). This step just requires you to log in using remote desktop via the Microsoft Azure management portal, or by using the Get-AzureRemoteDesktopFile
cmdlet.
Get-AzureRemoteDesktopFile
-ServiceName
$serviceName
-Name
"ps-vm1"
-Launch
When you are logged in, launch File Explorer and browse your disks. You should have a 49 to 50 MB disk attached with a simple file in it (see Figure 5-9).
At this point I like to modify the file with some changes. I do this because the next step is to download the VHD again, and I can validate the changes locally after the download.
Downloading a VHD from Microsoft Azure with PowerShell is very similar to uploading, since the the parameters are reversed from Add-AzureVHD
. The source parameter is now the location of the VHD in storage, and the destination is the location on the local file system. To try this out on your own, create a new PowerShell script in the PowerShell ISE named chapter5download.ps1 and add the code shown in Example 5-3.
$subscription
=
"[subscription name]"
$storageAccount
=
"[storage account name]"
Select-AzureSubscription
$subscription
$destination
=
"C:VHDFilesmydatadisk_downloaded.vhd"
$source
=
"https://$storageAccount.blob.core.windows.net/upload/mydatadisk.vhd"
Save-AzureVhd
-Source
$source
-LocalFilePath
$destination
Press F5, or highlight the script and press F8 to execute the script (see Figure 5-10).
When the VHD file is downloaded, you can double-click the downloaded file to mount it in Windows and examine the file for your changes.
There are some things you should know about downloading using the Save-AzureVHD
cmdlet. First, if you attempt to download a VHD that is actively being written to, chances are very high that the operation will fail due to conflict between the new I/O from the virtual machine and the cmdlet downloading the written bytes.
The second thing you should know is that download is also optimized like the upload. The cmdlet will download only the written bytes of the VHD. However, the cmdlet currently does not support converting to a dynamic VHD on save and saves only to fixed disks. If the disk you are downloading is 1 TB in size but has only 50 GB written to it, the cmdlet will download only 50 GB but will require the full 1 TB of local storage to hold the file.
Similar to Add-AzureVHD
, the Save-AzureVHD
cmdlet supports the -Overwrite
and -NumberOfThreads
parameters. This cmdlet also supports a -StorageKey
parameter that allows you to specify the authentication key for the storage account (see Table 5-2). If the key is not specified, the currently selected subscription is used to attempt authentication.
-OverWrite | If the VHD exists on the destination, you can overwrite it by passing this parameter. |
-NumberOfThreads | Default value is 8. Change if you feel that more or less parallelization would improve your download. |
-StorageKey | The authentication key to the storage account. |
Virtual machine VHD files are referenced indirectly through disks and images in Microsoft Azure. Disks and images are named entities that contain metadata and a link to the underlying VHD.
In Microsoft Azure an image is a named entity that is mapped to an operating system disk and optionally a set of data disks. Images are used to instantiate virtual machines with a specific operating system with or without customizations applied. Images can reside in the Microsoft Azure image gallery if they are part of the platform, or they can reside in a storage account in your own subscription if they are your own custom images. Images are designed to be a customized version of what a virtual machine should look like.
There are two types of images that can be created. The original image type is called an OS image. This image type can reference only the operating system disk and works only with images that have been generalized with Sysprep or the Linux Waagent. The second image type is called a virtual machine (VM) image. This image type can reference data disks in addition to the operating system disk and can be generalized or specialized.
A VM image that has been generalized is used for creating multiple virtual machine instances with the same starting point. These virtual machines will be customized with their own computer names at provision time. A specialized VM image is not generalized. This means that it will retain its own identity such as a computer name and even domain-join information. This image type is especially useful for taking backups of a specific virtual machine for development and test, or as a general-purpose backup solution at the VHD level. It is very similar in concept to exporting a virtual machine in Hyper-V. A key difference is that you have to take special care to shut the virtual machine down, because currently the image creation process does not freeze the state of your applications before capture.
To register a VHD file as an image, the VHD file must first be in a Microsoft Azure storage account. Whether it was uploaded there or copied from another location (we’ll talk about this capability a bit later) doesn’t really matter. To register a VHD file located in a storage account, simply call the Add-AzureVMImage
cmdlet.
For example, let’s assume that you have previously used the Add-AzureVHD
cmdlet to upload the myWindowsImage.vhd file to your storage account in the upload container. The call to Add-AzureVMImage
creates the image named MyWinImage that is backed by the file you uploaded, and it is marked as being a Windows-based virtual machine (see Example 5-4). Specifying the OS on the image is important because Linux and Windows have distinct provisioning processes for creating a new virtual machine instance from a generalized image.
A Microsoft Azure disk is another entity mapped to a VHD file in Microsoft Azure. Unlike an image, it is never generalized and it is not required to have an operating system on it, as it could just hold data. A disk with an operating system is called an OS disk, and one is created each time you create a virtual machine from an image. A disk without an operating system in Microsoft Azure is a data disk. These are meant to hold exactly what you would expect to be on a data disk: data.
Example 5-5 is a simple example of how you could register two previously created or uploaded VHD files in your subscription as usable disks. The first call to Add-AzureDisk
registers the myosdrive.vhd as an OS disk by specifying the -OS Windows
parameter to the cmdlet.
The second call to Add-AzureDisk
registers a simple data disk. When execution is complete, this disk could then be attached to a virtual machine at provision time or to a virtual machine that already exists.
$storageAccount
=
"[storage account name]"
$osdisk
=
"https://$storageAccount.blob.core.windows.net/upload/myosdrive.vhd"
$osdiskName
=
"os disk"
$datadisk
=
"https://$storageAccount.blob.core.windows.net/upload/mydatadrive.vhd"
$datadiskname
=
"data disk"
# Register the operating system disk
Add-AzureDisk
-DiskName
$osdiskName
-MediaLocation
$osdisk
-OS
Windows
# Register the data disk
Add-AzureDisk
-DiskName
$datadiskname
-MediaLocation
$datadisk
Creating a virtual machine from disks is similar to creating a virtual machine from an image, with just a few differences.
The first difference is you need to specify the OS disk name to the -DiskName
parameter on New-AzureVMConfig
instead of an ImageName
. The second difference is when creating a virtual machine from a disk, you do not need to use the Add-AzureProvisioningConfig
cmdlet to populate the provisioning configuration. The provisioning configuration data is used only with a generalized image. The third change is regarding attaching the data disk. Instead of using the -CreateNew
parameter with Add-AzureDataDisk
, you specify -Import
and the name of the data disk with the -DiskName
parameter (see Example 5-6).
$serviceName
=
"[cloud service name]"
$location
=
"[region name]"
$vmName
=
"migratedVM"
$vmSize
=
"Small"
$osdiskName
=
"os disk"
$datadiskname
=
"data disk"
# Create the configuration specifying the disk name instead of an image
$vmConfig
=
New-AzureVMConfig
-Name
$vmName
`
-InstanceSize
$vmSize
`
-DiskName
$osdiskName
# Import the data disk to the first LUN
$vmConfig
|
Add-AzureDataDisk
-Import
-DiskName
$datadiskname
-LUN
0
# Create the virtual machine
$vmConfig
|
New-AzureVM
-ServiceName
$serviceName
-Location
$location
Now that you understand the basics of images and disks, I want to dig deeper into each of these entities, starting with images.
There are cmdlets to help you manage images. We have seen one of them in Chapter 3 (Get-AzureVMImage
), and we have also just seen how to register an image based on a VHD using the Add-AzureVMImage
cmdlet. Let’s investigate some of the additional options you have with managing virtual machine images.
Running Get-AzureVMImage
on its own will give you a list of all of the platform images and user images available to your subscription.
Each image has a list of properties that can be filtered on using PowerShell (see Table 5-3). For custom images that you create, some of these properties can also be set with the Set-AzureVMImage
cmdlet.
Category | Public (Microsoft Azure images) or User (your images). |
Location | The location(s) where the image is usable. |
LogicalSizeInGB | The size of the image (and the size your OS disk will be). |
Label | Descriptive label. |
MediaLink | Full URI to the underlying VHD. Ths is applicable only to User images. |
ImageName | The name of the image. This is the value passed to create a virtual machine. |
OS | Whether the OS is Windows or Linux. |
OSDiskConfiguration | Configuration of the operating system disk. |
DataDiskConfigurations | Configuration of the captured data disks (if any). |
Eula | End-user license agreement for the image. |
Description | Descriptive text. |
ImageFamily | A logical grouping of images (SQL, Server 2012 R2, etc.) |
PublishedDate | When the image was made available. |
IsPremium | If True, there will be an additional usage charge on top of compute. |
IconUri | URI to the icon for the image that will show up in the portal. |
PrivacyUri | URI to the privacy policy for the image. |
RecommendedVMSize | Recommended virtual machine size for the image. |
PublisherName | Name of the group or company that published the image. |
Filtering images by using these properties is simple with PowerShell. For instance, if you would like to see all of the images that charge a premium on top of the regular compute hours, run the command shown in Example 5-7.
With PowerShell, you can combine these filters to make more-advanced queries. Example 5-8 shows filtering on only premium images that recommend the A6 instance size.
Get-AzureVMImage
|
where
{
$_
.
IsPremium
-eq
$true
-and
$_
.
RecommendedVMSize
-eq
"A6"
}
The only value you will use from a virtual machine image to create a virtual machine is the ImageName
property. It is useful to know how to select only that property when querying images, as shown in Example 5-9.
Get-AzureVMImage
|
where
{
$_
.
IsPremium
-eq
$true
}
|
select
ImageName
In Chapter 3 I showed how you can use the filtering capabilities to return the image by publish date so you always get the latest version with the latest patches (see Example 5-10).
$imageName
=
Get-AzureVMImage
|
where
{
$_
.
ImageFamily
-eq
$imageFamily
}
|
sort
PublishedDate
-Descending
|
select
-ExpandProperty
ImageName
-First
1
If you would like to view images that are specialized, you can filter on the OSState
property of the OSDiskConfiguration
property. This value can be Specialized
or Generalized
(see Example 5-11).
Now that you know how to view and filter images, how do you create one of your own? To create a custom image, let’s walk through the steps from the very beginning by creating a new virtual machine, customizing it, and then capturing it as an image.
The image capture process works by starting with a base virtual machine. When the virtual machine is booted, you customize it (install software, make configuration changes, and so on) to make the virtual machine exactly how you want new virtual machines to start.
When the virtual machine is customized, you then run the tool to generalize the operating system. In Windows, you use the Sysprep tool. In Linux, you can use the Microsoft Azure agent (Waagent) to accomplish a similar task. This step prepares the operating system in the same way that OEMs do after installing software on a new computer. When the virtual machine boots up, it will go through the provisioning process just like a new install of Windows would, except your changes will already be on the virtual machine. As you can imagine, this is a fairly destructive process, so read the documentation for any applications you are considering imaging. SQL Server and SharePoint are just two examples of applications that support Sysprep but have very specific instructions on how to accomplish it in a supported manner.
Finally, after you have successfully run Sysprep and the virtual machine is shut down, you can then capture the virtual machine as an image. Capturing a generalized image also deletes the virtual machine instance that was captured, as it no longer has a machine name and requires going through the full provisioning process to be useful again. You now have the saved image stored in a storage account to allow you to easily provision one or more instances of the virtual machine that you captured.
Hopefully, the imaging process is a little more clear to you now if it was not already. With that in mind, let’s use PowerShell to create an image (we’ll cheat ever so slightly by using remote desktop to customize the virtual machine, but I’ll show how you can script that in Chapter 7).
Using the PowerShell ISE, create a new script named chapter5vmimage.ps1 and add the code shown in Example 5-12. Make sure to replace all of the placeholder values with correct values. For this example, ensure that your cloud service name is globally unique.
$subscription
=
"[subscription name]"
Select-AzureSubscription
$subscription
# Specify the admin credentials
$adminUser
=
"[admin username]"
$password
=
"[admin password]"
$location
=
"[region name]"
$serviceName
=
"[cloud service name]"
$vmName
=
"ps-vmImage"
$vmSize
=
"Small"
$imageFamily
=
"Windows Server 2012 R2 Datacenter"
$imageName
=
Get-AzureVMImage
|
where
{
$_
.
ImageFamily
-eq
$imageFamily
}
|
sort
PublishedDate
-Descending
|
select
-ExpandProperty
ImageName
-First
1
New-AzureQuickVM
-Windows
`
-ServiceName
$serviceName
`
-Name
$vmName
`
-ImageName
$imageName
`
-Location
$location
`
-AdminUsername
$adminUser
`
-Password
$password
`
-InstanceSize
$vmSize
`
-WaitForBoot
In this example I have also introduced the -WaitForBoot
flag. This flag ensures that New-AzureQuickVM
(or New-AzureVM
) does not return until the virtual machine is in the ReadyRole
state. This makes it very convenient to know when you can log in to the virtual machine or execute a script remotely to customize the virtual machine.
Next, run the code shown in Example 5-13 to launch a remote desktop session into the virtual machine.
Get-AzureRemoteDesktopFile
-ServiceName
$serviceName
-Name
$vmName
-Launch
When prompted, enter the username and password specified in your script to log in. When you are logged in to the newly created virtual machine, click the Windows PowerShell icon on the task bar.
The whole point of creating an image is that you would like the ability to create multiple instances of a virtual machine that has a preexisting configuration on it. In this case, we can install IIS so that we can easily deploy a web server using the custom image.
Launch PowerShell, and at the PowerShell command line on the virtual machine, execute the command shown in Example 5-14 to install IIS (see Figure 5-11).
Add-WindowsFeature
-Name
"Web-Server"
-IncludeAllSubFeature
-IncludeManagementTools
When all customizations for your image have been made (which in this case is just installing the Web-Server feature), the next step is to run Sysprep.
Within the virtual machine, press Windows Key + R and enter C:WindowsSystem32Sysprepsysprep.exe
as the command to run, and press Enter. See Figure 5-12.
When the Sysprep wizard is displayed on the screen, as shown in Figure 5-13, check the Generalize checkbox and change Shutdown Options to Shutdown, and then click OK.
The system preparation process can take several minutes to complete. Eventually, your remote desktop session will be stopped (this is supposed to happen), and the next step is to wait until the virtual machine is in the StoppedVM
state. You can check this by running the Get-AzureVM
cmdlet to view the virtual machine’s status.
When the virtual machine is stopped, you can capture the image by using the Microsoft Azure management portal or with PowerShell.
Run the code shown in Example 5-15 in the console of the PowerShell ISE. I have split the call to Save-AzureVMImage
across multiple lines for clarity, but it can be executed on one line by removing the line continuation character (`). This example creates a new virtual machine image and saves it in the storage account where the virtual machine disks are currently located.
To create a virtual machine using this new image, create a new script using the PowerShell ISE named chapter5createfromimage.ps1 and enter the code shown in Example 5-16. Ensure that you replace the placeholder values with real ones from your subscription.
$subscription
=
"[subscription name]"
$location
=
"[region name]"
$serviceName
=
"[cloud service name]"
# Specify the admin credentials
$adminUser
=
"[admin username]"
$password
=
"[admin password]"
Select-AzureSubscription
$subscription
$vmSize
=
"Small"
$vmName
=
"ps-webserver1"
$imageName
=
"WEBSERVERIMAGE"
New-AzureQuickVM
-Windows
`
-ServiceName
$serviceName
`
-Name
$vmName
`
-ImageName
$imageName
`
-Location
$location
`
-AdminUsername
$adminUser
`
-Password
$password
`
-InstanceSize
$vmSize
When the image is created, there are two pieces of the image that you may want to modify in the future.
The first is the properties of the virtual machine image entity in Microsoft Azure. To update the properties, use the Update-AzureVMImage
cmdlet along with the image name.
To try this on your own, create a new script called chapter5updateimage.ps1 and add the code shown in Example 5-17.
$subscription
=
"[subscription name]"
Select-AzureSubscription
$subscription
$imageName
=
"WEBSERVERIMAGE"
Update-AzureVMImage
-ImageName
$imageName
`
-Description
"A Pre-Built web server"
`
-ImageFamily
"My Company Custom Images - IIS"
`
-Label
"Image with IIS Pre-Installed"
Execute the script by pressing F5, or by highlighting the text and pressing F8 to update the image.
Execute the Get-AzureVMImage
cmdlet to verify your changes (see Example 5-18).
$imageName
=
"WEBSERVERIMAGE"
Get-AzureVMImage
-ImageName
$imageName
The second type of update is to update the underlying image bits themselves. This can be accomplished in one of two ways. The first and likely the simplest way is to create a new virtual machine instance from the image, and then update the virtual machine by applying patches and software updates. When the virtual machine instance is updated, run Sysprep again and capture the image with a new name.
The alternative method assumes that you have created the image offline and uploaded it using the Add-AzureVHD
cmdlet. If you have, then you can create differencing disks offline with the updates and upload only the differencing disks and have them applied to the base image using the Add-AzureVHD
cmdlet and specifying the image URL with the -BaseImageUriToPatch
parameter.
Deleting a custom image is straightforward with the Remove-AzureVMImage
cmdlet. Remove-AzureVMImage
accepts two parameters: the -ImageName
and, optionally, the -DeleteVHD
parameter.
Example 5-19 will delete the custom virtual machine image just created and, with the -DeleteVHD
parameter added, will also delete the underlying VHD in your storage account.
The original imaging technology that was released when Microsoft shipped virtual machines is known as OS images. Since then, a new imaging architecture has been released and this new architecture is known as VM images. The biggest difference between OS images and VM images is that VM images can also capture data disks. The second difference is that VM images also support capturing specialized images (see Table 5-4). This support for specialized images means you can create a virtual machine and capture it without running Sysprep or Waagent (Linux), and the OS disk and data disks will be captured as is, including the unique identity of the VM.
OS image | VM image |
OS disk only | OS and data disks |
Generalized only | Generalized or specialized |
The Save-AzureVMImage
cmdlet can be used to create each image type.
To create an OS image, simply omit the -OSState
parameter, and the image will be created as an OS image, as shown in Example 5-20. This is an important point to remember, because if you were anticipating capturing data disks, you will be disappointed if you forget to specify the -OSState
parameter.
$serviceName
=
"[cloud service name]"
$vmName
=
"[virtual machine name]"
$imageName
=
"[image name]"
$imageLabel
=
"[image label]"
Save-AzureVMImage
-ServiceName
$serviceName
`
-Name
$vmName
`
-NewImageName
$imageName
`
-NewImageLabel
$imageLabel
Calling Save-AzureVMImage
without the -OSState
parameter reverts to using the OS image type, which does not support data disks.
Specifying the -OSState Generalized
parameter creates a VM image, and if there are data disks associated with the virtual machine, they will be captured as well (see Example 5-21). Using this parameter assumes that the virtual machine has been generalized with Sysprep on Windows or the Microsoft Azure Linux agent (Waagent).
$serviceName
=
"[cloud service name]"
$vmName
=
"[virtual machine name]"
$imageName
=
"[image name]"
$imageLabel
=
"[image label]"
Save-AzureVMImage
-ServiceName
$serviceName
`
-Name
$vmName
`
-NewImageName
$imageName
`
-NewImageLabel
$imageLabel
`
-OSState
Generalized
Specifying the -OSState Specialized
parameter creates a VM image, and if there are data disks associated with the virtual machine, they will be captured as well (see Example 5-22). Using this parameter assumes that the virtual machine has not been generalized with Sysprep on Windows or the Azure Linux agent (Waagent). This scenario is great for capturing a virtual machine exactly how it should be when deployed. An example is creating an image of the virtual machine before applying updates or an application deployment. If the update or deployment fails, you can delete the virtual machine and re-create it from the image, and the state will be exactly as it was before provisioning.
$serviceName
=
"[cloud service name]"
$vmName
=
"[virtual machine name]"
$imageName
=
"[image name]"
$imageLabel
=
"[image label]"
Save-AzureVMImage
-ServiceName
$serviceName
`
-Name
$vmName
`
-NewImageName
$imageName
`
-NewImageLabel
$imageLabel
`
-OSState
Specialized
Before creating a specialized image, it is a good idea to shut the virtual machine down first to ensure that all writes to the disks are flushed before capture. When creating a generalized image, the VM is already shut down due to the generalization process. With the specialization option, it is up to you whether you want to shut down the virtual machine first.
As you now know, a disk is similar in concept to an image. It is an entity in Microsoft Azure that is backed by a VHD file in a storage account. The VHD file can be a bootable OS disk with a supported version of Windows or Linux on it or it can be a simple data disk that contains only data.
When you create a virtual machine from an image, a copy of that image is made to the target location you specified for your virtual machine, and this becomes the OS disk.
Just like images, there are additional cmdlets to help you manage your disks. You have already seen how to attach a new disk to a virtual machine by using Add-AzureDataDisk
with the -CreateNew
parameter, and you have seen how to register a VHD file to a Microsoft Azure disk by using the Add-AzureDisk
cmdlet. In this section we will dig deeper and show more scenarios for how disks can be managed using PowerShell.
An OS disk is simply a disk that contains a bootable operating system on it, and the OS property of the disk entity is set to Windows or Linux. An OS disk is created each time you create a virtual machine from an image or when you add an existing VHD to storage and register it by calling the Add-AzureDisk
cmdlet with the -OS
parameter specified.
In this partial example, shown in Example 5-23, the $mediaLocation
variable points to a previously uploaded VHD in a storage account. To make this VHD usable to a virtual machine, the Add-AzureDisk
cmdlet is used to create the disk entity that is mapped to the underlying VHD in storage. The -OS Windows
parameter is also specified to register the disk as a Windows OS disk.
A data disk is similar to an OS disk except that it does not have an OS property specified. Microsoft Azure uses this property to decide whether the disk can be booted or just attached to a virtual machine.
As we have seen in Chapter 3, using the PowerShell cmdlets makes it easy to create additional empty disks to a virtual machine with up to 1023 GB in storage.
In this partial example, shown in Example 5-24, the $vmConfig
variable is a virtual machine configuration object. It is piped to the Add-AzureDataDisk
cmdlet, which has the -CreateNew
parameter specified. The -CreateNew
parameter makes the Add-AzureDataDisk
cmdlet populate the configuration object with properties that, when sent to the Microsoft Azure API, will create new empty disks on the virtual machine.
$vmConfig
|
Add-AzureDataDisk
-CreateNew
`
-DiskSizeInGB
50
`
-DiskLabel
"data 1"
`
-LUN
0
$vmConfig
|
Add-AzureDataDisk
-CreateNew
`
-DiskSizeInGB
50
`
-DiskLabel
"data 2"
`
-LUN
1
Besides creating a new disk with the -CreateNew
parameter, you can also attach existing disks by using the -Import
or -ImportFrom
parameters. This can be a disk that you have uploaded previously or other data disks that were created in Azure on other virtual machines. One thing to remember is that a disk cannot be attached to multiple virtual machines at the same time.
In this partial example, shown in Example 5-25, a data disk is created with the Add-AzureDisk
cmdlet using the MyDataDisk name and the URL to an existing VHD in an Azure storage account. The -Import
parameter of Add-AzureDataDisk
allows you to just specify the disk name and attach it to a virtual machine.
$mediaLocation
=
"https://$storageAccount.blob.windows.net/upload/mydatadisk.vhd"
Add-AzureDisk
-DiskName
"MyDataDisk"
-MediaLocation
$mediaLocation
$vmConfig
|
Add-AzureDataDisk
-Import
`
-DiskName
"MyDataDisk"
`
-LUN
0
Example 5-26 shows that it is also possible to specify an existing VHD in a storage account that is in the same subscription and same region by using the Add-AzureDataDisk
cmdlet with the -ImportFrom parameters
parameter. What does -ImportFrom
actually mean? When you specify the -ImportFrom
parameter and a storage account URI to a VHD, the cmdlet will automatically register the VHD as a disk and add it to your virtual machine configuration. This is a shortcut that is the same as calling Add-AzureDisk
to register the disk and using the -Import
parameter to add it to the virtual machine configuration at the same time.
To view disks in your Azure subsription or to view the properties of a specific disk (listed in Table 5-5), you can use the Get-AzureDisk
cmdlet. Running Get-AzureDisk
with no parameters will return all of the disks in your subscription (see Example 5-27).
AffinityGroup | The affinity group (if any) of the parent storage account. |
AttachedTo | The virtual machine that the disk is attached to (null if not attached). |
IsCorrupted | Whether corruption has been detected on the VHD. |
Label | Descriptive label. |
Location | The Microsoft Azure region (if not affinity group) of the parent storage account. |
DiskSizeInGB | The size, in gigabytes, of the disk. |
MediaLink | Full URI to the underlying VHD. |
DiskName | The name of the disk. This is the value passed to any disk-related API. |
SourceImageName | Source image name if the disk was created by an image. |
OS | Windows or Linux if an OS disk or null if a data disk. |
Get-AzureDisk
also supports specifying just the -DiskName
parameter if you would like to return only properties of a specific disk.
As with images, you can filter on the properties of disks. Example 5-28 calls Get-AzureDisk
to return all of the disks from your subscription. The output of Get-AzureDisk
is piped to the where
command, which filters the results further by including only disks where the AttachedTo
property equals null
(not mounted to a virtual machine) and OS equals null
(not an operating system disk).
Get-AzureDisk
|
where
{
$_
.
AttachedTo
-eq
$null
-and
$_
.
OS
-eq
$null
}
Example 5-29 is another example of using the PowerShell filtering abilities. The commands are almost identical, except for filtering by a different value of the operating system. These commands may or may not return any data, depending on whether you have unattached OS disks for Windows or Linux in your current subscription.
Under the covers, the PowerShell cmdlets automatically set the location of the disk to be in the storage account you specified with the -CurrentStorageAccountName
parameter of the Set-AzureSubscription
cmdlet and the default container vhds
. Using the PowerShell cmdlets, you have the power to override this default behavior and have full control over the location and filenames of the VHDs, including the OS disk, by specifying the -MediaLocation
parameter.
Example 5-30 is a full example. I am using the -MediaLocation
parameter to specify the location of the underlying disks of a virtual machine when it is created. The -MediaLocation
parameter is specified in the call to New-AzureVMConfig
to override the location of the operating system disk, and each call to Add-AzureDataDisk
specifies the location of each data disk.
If you would like to try this example on your own, create a new PowerShell script using the PowerShell ISE named chapter5medialocation.ps1.
$subscription
=
"[subscription name]"
Select-AzureSubscription
$subscription
$location
=
"[region name]"
$serviceName
=
"[cloud service name]"
# Specify the admin credentials
$adminUser
=
"[admin username]"
$password
=
"[admin password]"
$storageAccount
=
(
Get-AzureSubscription
-Current
).
CurrentStorageAccountName
$vmName
=
"ps-vmDisks"
$vmSize
=
"Small"
$imageFamily
=
"Windows Server 2012 R2 Datacenter"
$imageName
=
Get-AzureVMImage
|
where
{
$_
.
ImageFamily
-eq
$imageFamily
}
|
sort
PublishedDate
-Descending
|
select
-ExpandProperty
ImageName
-First
1
# Custom location and filenames
$oslocation
=
"https://$storageAccount.blob.core.windows.net/custom/osdisk.vhd"
$d1Location
=
"https://$storageAccount.blob.core.windows.net/custom/data1.vhd"
$d2Location
=
"https://$storageAccount.blob.core.windows.net/custom/data2.vhd"
$vmConfig
=
New-AzureVMConfig
-Name
$vmName
`
-InstanceSize
$vmSize
`
-ImageName
$imageName
`
-MediaLocation
$oslocation
$vmConfig
|
Add-AzureProvisioningConfig
-Windows
`
-AdminUsername
$adminUser
`
-Password
$password
$vmConfig
|
Add-AzureDataDisk
-CreateNew
`
-DiskSizeInGB
50
`
-DiskLabel
"data 1"
`
-LUN
0
`
-MediaLocation
$d1Location
$vmConfig
|
Add-AzureDataDisk
-CreateNew
`
-DiskSizeInGB
50
`
-DiskLabel
"data 2"
`
-LUN
1
`
-MediaLocation
$d2Location
$vmConfig
|
New-AzureVM
-ServiceName
$serviceName
-Location
$location
Press F5, or highlight the script and press F8 to create the virtual machine. You can then use the Get-AzureVM
cmdlet to return the virtual machine configuration and pipe it to Get-AzureOSDisk
or Get-AzureDataDisk
to view the custom media location (see Figures 5-14 and 5-15).
By default, ReadWrite
caching is enabled on the OS disk of a virtual machine for performance. This means that most reads and writes to the C: drive are cached on the physical disk that the virtual machine is hosted on before being committed to the disk backed in Microsoft Azure storage. You can change the setting of the OS disk during disk creation by specifying the -HostCaching
parameter to New-AzureVMConfig
, or use the Set-AzureOSDisk
cmdlet for changing the cache setting on an existing virtual machine’s OS disk.
Supported -HostCaching
values for the OS disk are ReadOnly
and ReadWrite
.
Up to four data disks per virtual machine can be enabled for disk caching. The default cache setting of a data disk is None
and is most likely the safest bet if you are actively writing data to the disk. However, some workloads can be optimized by enabling local read or write cache on the disk. During creation, you can specify the cache settings of a data disk with the -HostCaching
flag of the Add-AzureDataDisk
cmdlet, or use the Set-AzureDataDisk
cmdlet to change the cache configuration on an existing virtual machine.
Supported -HostCaching
flag values for the OS disk are None
, ReadOnly
, and ReadWrite
.
Example 5-31 is a partial example that shows how you can specify the -HostCaching
parameter during creation to change the default cache behavior of OS and data disks.
# -HostCaching on New-AzureVMConfig refers to the OS disk setting
$vmConfig
=
New-AzureVMConfig
-Name
$vmName
`
-InstanceSize
$vmSize
`
-ImageName
$imageName
`
-HostCaching
ReadOnly
$vmConfig
|
Add-AzureProvisioningConfig
-Windows
`
-AdminUsername
$adminUser
`
-Password
$password
$vmConfig
|
Add-AzureDataDisk
-CreateNew
`
-DiskSizeInGB
50
`
-DiskLabel
"data 1"
`
-LUN
0
`
-HostCaching
ReadWrite
$vmConfig
|
Add-AzureDataDisk
-CreateNew
`
-DiskSizeInGB
50
`
-DiskLabel
"data 2"
`
-HostCaching
ReadWrite
-LUN
1
Example 5-32 is a partial example that shows changing the -HostCaching
property of an existing virtual machine’s disks using the Set-AzureOSDisk
and Set-AzureDataDisk
cmdlets.
$vmConfig
=
Get-AzureVM
-ServiceName
$serviceName
-Name
$vmName
$vmConfig
|
Set-AzureOSDisk
-HostCaching
ReadOnly
$vmConfig
|
Set-AzureDataDisk
-LUN
0
-HostCaching
ReadWrite
$vmConfig
|
Set-AzureDataDisk
-LUN
1
-HostCaching
ReadWrite
$vmConfig
|
Update-AzureVM
Before completing this section of the chapter, I want to highlight a few common pitfalls that occur with cloud services, images, disks, and storage accounts.
When creating a virtual machine using your own custom OS image, it is a requirement that the CurrentStorageAccountName
property of your subscription is set to the same storage account where the image is located. This applies even if they are in the same region. Remember, you set this property by using the Set-AzureSubscription
cmdlet passing the -CurrentStorageAccountName
property.
If you do not follow this rule, you will receive the error:
BadRequest: The disk’s VHD must be in the same account as the VHD of the source image (source account: storageaccount1.blob.core.windows.net, target account: storageaccount2.blob.core.windows.net).
If you are using a VM image as the source, the behavior is slightly different. Even though the CurrentStorageAccountName
is not the same location as the VM image location, the cmdlet will succeed. However, the newly created virtual machine’s disks will be created in the storage account that the user image is in and not the storage account specified in CurrentStorageAccountName
.
Another common error that you will receive is when provisioning virtual machines where the image or disk is in one region and the location of the cloud service is in another.
The error for this common problem is as follows:
BadRequest: The location or affinity group North Europe of the storage account where the source image CUSTOMIMAGE resides is not in the same location or affinity group as the specified cloud service. The source image must reside in a storage account that has the same affinity group or location as the cloud service West US.
The resolution to all of these problems is to ensure that all of your assets are in the same location. The cloud service container must always be in the same region as your storage. If you are provisioning from a custom image, the CurrentStorageAccountName
should match the same region and storage account as your source image.
You have already seen several of the cmdlets for managing blob storage in Microsoft Azure. While this book is mainly focused on managing infrastructure services, there are some key things to know about managing the underlying system where the virtual machine disks are actually stored.
In addition to the storage account, container, and blob cmdlets, there are also cmdlets that allow you to configure Microsoft Azure storage metrics to monitor storage usage. There are also cmdlets to directly manage tables, queues, and Azure File Services (which is in preview at the time of this writing).
If you would like to see all of the Microsoft Azure storage-related cmdlets, you can use the PowerShell Get-Command
cmdlet and filter the Name
property for AzureStorage
, as shown in Example 5-33.
Get-Command
|
Where
{
$_
.
Name
-like
"*AzureStorage*"
}
In the remainder of this chapter I want to focus on some of these cmdlets that specifically impact running and managing virtual machines.
In Chapter 3 we discussed the basics of creating a Microsoft Azure storage account by using the New-AzureStorageAccount
cmdlet. There are some items that merit further review.
Enabling geo-replication on a storage account tells Microsoft Azure to replicate all of the data in the storage account to a remote Microsoft Azure region. Each Microsoft Azure region has a failover region that is used for such purposes. For example, a storage account in the East US region with geo-replication enabled will asynchronously copy all of its data to the West US region.
This is meant to ensure a higher level of durability. Microsoft Azure storage already makes three copies of each blob within the same region. Enabling geo-replication enables an additional three copies of the blob to be replicated to the failover region for additional durability. A storage account with geo-replication enabled gives you six copies of every blob in the storage account. This means there is very little chance that you will have significant data loss due to a failure in the data center.
To enable or disable geo-replication from PowerShell, you can use the Set-AzureStorageAccount
cmdlet (see Example 5-34).
Set-AzureStorageAccount
-StorageAccountName
$storageAccount
`
-GeoReplicationEnabled
$true
There are a few reasons why you would want to disable geo-replication on a storage account.
The first should be obvious, and that is cost. Having data stored in multiple regions is going to cost more than storing the data in just a single region. If you are storing non-critical data that does not need the assurance of being replicated to a remote region, then you should disable this option.
The second reason is that not all workloads support geo-replication. For instance, SQL Server does not support having data and transaction logs on separate disks on a geo-replicated storage account. Geo-replication is completely asynchronous. This means that the writes are not guaranteed to be written in order, which makes a transactionally consistent database difficult to restore in the event of a data center failover.
So far, from a security perspective, we have only discussed using the Select-AzureSubscription
cmdlet and a brief mention of shared access signatures. The Select-AzureSubscription
cmdlet tells the other cmdlets which subscription to use and, more important, which credentials to use to perform the operation. The current implementation of this approach implies that you are either an administrator or a co-administrator of the Microsoft Azure subscription.
However, there are times when you need to directly authenticate against storage instead of using authentication at the subscription level. A good example is when you need to access a storage account, container, or just a file in a container and you are not an administrator on the Microsoft Azure subscription itself.
There are two methods for authenticating access to storage.
The first method is to use a combination of the storage account name and one of the storage account authentication keys. The second method is generating a shared access signature and passing that to a lower privilege client to use for authentication.
The differences between the two methods is significant. With the storage account name and key, you have full access to everything in the storage account and can perform all operations on its contents. With a shared access signature, the level of access and even the duration of access is defined during the creation of the shared access signature itself. Using a shared access signature is a two-step process. The first step is to generate the SAS token, and this does require full rights on the storage account. The second step is to actually use the SAS token.
Storage context objects
Many of the operations regarding storage accept a storage context parameter. The storage context is simply a data type that stores the authentication method and the credentials used for whatever storage operation you wish to perform. This can either be the storage account name and key or a generated SAS token.
In Example 5-35 I’m using the Get-AzureStorageKey
cmdlet to retrieve the primary authentication key value and store it in the $storageKey
variable. I then pass it along with the name of the storage account itself to the New-AzureStorageContext
cmdlet. This cmdlet creates the context object that can be used by other storage cmdlets to authenticate their operations.
Accessing the storage account key by using the Get-AzureStorageKey
cmdlet requires Microsoft Azure administrative access. It is possible to retrieve this key as an administrator and pass it to less-elevated code or users. Just remember that whoever has access to the name and key can perform any operation on the contents of the storage account.
$storageAccount
=
"[storage account name]"
$storageKey
=
(
Get-AzureStorageKey
-StorageAccountName
$storageAccount
).
Primary
$context
=
New-AzureStorageContext
-StorageAccountName
$storageAccount
`
-StorageAccountKey
$storageKey
Passing the $context
object to the Get-AzureStorageContainer
cmdlet tells the cmdlet to use the specified storage account and key, whether your local PowerShell configuration has access to the subscription or not (see Example 5-36). You can think of it as an authentication override for storage. If you do not specify a context object to the storage cmdlets, they will assume you want to work on the storage account specified in the CurrentStorageAccountName
property of your subscription (remember, you can change this setting with the Set-AzureSubscription
cmdlet).
$container
=
"vhds"
Get-AzureStorageBlob
-Context
$context
-Container
$container
The alternative to passing out the storage account name and key (and full permissions to your storage account) is to generate a shared access signature instead (see Example 5-37). This type of access allows for much finer-grained permissions, duration of permissions, and is also revokable.
To create the shared access signature in the first place, you do need full access to the storage account (see Table 5-6).
$sas
=
New-AzureStorageContainerSASToken
-Name
$container
`
-Permission
rwdl
`
-Context
$context
Permission | Symbol | Description |
Read | r | Read the content, properties, metadata, or block list of any blob in the container. Use any blob in the container as the source of a copy operation. |
Write | w | For any blob in the container, create or write content, properties, metadata, or block list. Snapshot or lease the blob. Resize the blob (page blob only). Use the blob as the destination of a copy operation within the same account. |
Delete | d | Delete any blob in the container. |
List | l | List blobs in the container. |
For more details on blob and container permissions, see the following article in MSDN: http://bit.ly/shared_access_sig_URI.
The New-AzureStorageContainerSASToken
cmdlet creates the actual shared access signature token. You can specify a start time and an end time for how long the token is valid and, optionally, you can specify a previously created Shared Access Signature Policy from which to base the token settings.
At the time of this writing, you can only reference an existing shared access signature policy but you cannot create one using the PowerShell cmdlets. Of course, you can call .NET, which would allow you to call in the Storage Client Libraries where you have full access to create policies.
When the token has been created, it can be passed to a client and used based on the access that has been granted. To use this token from the Microsoft Azure PowerShell cmdlets, you need to create a storage context object. This time, instead of passing in the the storage account name and key, you will pass in the storage account name and SAS token (see Example 5-38).
When the context object is created, you pass it as a parameter or through the pipeline to any Azure storage cmdlet that accepts a Context
object as a parameter.
The final security topic I want to mention is setting the public access policy for a storage container. Each container within a Microsoft Azure storage account can have an access policy for nonauthenticated (public) requests (see Table 5-7). By default this policy is Off
on each container but can be changed using the Set-AzureStorageContainerAcl
cmdlet (see Example 5-39).
Off | No anonymous access. |
Container | Allows read-only access to the blobs in the container and allows the blobs to be enumerated. |
Blob | Allows read-only access to the blobs in the container. The requestor needs to know the full path to the blob in question. |
$storageAccount
=
"[storage account name]"
$storageKey
=
(
Get-AzureStorageKey
-StorageAccountName
$storageAccount
).
Primary
$context
=
New-AzureStorageContext
-StorageAccountName
$storageAccount
`
-StorageAccountKey
$storageKey
# Create a new storage container with Blob public access
New-AzureStorageContainer
-Name
"newcontainer"
-Permission
Blob
-Context
$context
# Modify the container to have the Container public access instead
Set-AzureStorageContainerAcl
-Name
"newcontainer"
`
-Permission
Container
`
-Context
$context
So far I have skirted around managing blob data directly. Let us tackle it head-on in this section.
You have already seen how to create a container by using the New-AzureStorageContainer
cmdlet. Let’s work off of that knowledge and fill in some of the gaps.
For instance, how do you enumerate all of the existing containers in your storage account? Before creating a new container, how do you test whether the container exists already? How do you delete a container? These are all very good questions that we can quickly address with some examples.
Enumerating containers (see Example 5-40) will show you the current public access policy and the modification date of the container (see Figure 5-16). You can pipe the container output to the Get-AzureStorageBlob
container and enumerate all of the files in the storage account as well (see Example 5-41 and Figure 5-17).
One of the more useful storage-related tasks is scripting the upload or download of blob data to or from a Microsoft Azure storage account. The cmdlets make this a fairly painless process while at the same time providing you all of the power you would expect as a PowerShell user by supporting the PowerShell pipeline operator. This allows you to enumerate multiple files locally or remotely, and process them as a batch.
You can, of course, take the output of Get-AzureStorageBlob
and process it via the pipeline too. Passing the output to Remove-AzureStorageBlob
to delete files or to the Get-AzureStorageBlobContent
file to download the files locally are two examples of processing you could do with the pipeline output of Get-AzureStorageBlob
.
In Example 5-42 I am using the Get-AzureStorageContainer
output and piping it to Get-AzureStorageBlob
. From there, the output is passed to the PowerShell for-each
command that allows me to execute a script block for each blob returned. The script block is simple; it creates a local path using the Join-Path
cmdlet, the C:Temp directory, and the name of the file (see Figure 5-18).
The returned blob information $_
is piped to the Get-AzureStorageBlobContent
cmdlet, which does the heavy lifting of actually downloading the files from the videoexample container. Without specifying the videoexample container, this same code could download all of the files in the storage account. It would need a little help to account for creating local folders, but the code change would be minimal.
Get-AzureStorageContainer
-Name
"videoexample"
|
Get-AzureStorageBlob
|
foreach
{
$localPath
=
Join-Path
"C:Temp"
$_
.
Name
$_
|
Get-AzureStorageBlobContent
-Destination
$localPath
}
Uploading files from the local hard drive to storage is just as simple. In Example 5-43 there are two things going on in order to upload files.
The call to Get-AzureStorageContainer
is also passed the -ErrorAction
SilentlyContinue
in order to test whether the storage container exists or not. If the specified container does not exist, the cmdlet returns $null
and the container is created in the New-AzureStorageContainer
call.
When the container has been verified to exist, the script uses the Get-ChildItem
cmdlet to enumerate all of the files in the C:AdventureWorksDB folder on the C: drive. The path of each file and the name of the container is then passed to the Set-AzureStorageBlobContent
cmdlet, which does the heavy lifting of uploading the files. Figure 5-19 shows the upload status using PowerShell ISE.
$localPath
=
"C:AdventureWorksDB"
$container
=
"adventureworks"
$existingContainer
=
Get-AzureStorageContainer
-Name
$container
`
-ErrorAction
SilentlyContinue
If
(
$existingContainer
-eq
$null
)
{
New-AzureStorageContainer
-Name
$container
}
Get-ChildItem
$localPath
|
foreach
{
Set-AzureStorageBlobContent
-File
$_
.
FullName
-Container
$container
}
You have probably noticed in the previous examples that I have not passed a storage context to any of these calls to storage. When the storage cmdlets are called and the -Context
parameter is not specified, the cmdlets will use the currently selected subscription and the CurrentStorageAccountName
assocated with it. This property is set by using the Set-AzureSubscription
cmdlet.
The final subject regarding Microsoft Azure storage that I want to cover in this chapter is using the asynchronous blob copy cmdlets. From the perspective of an infrastructure services administrator, this is one of the more critical features of the Microsoft Azure PowerShell cmdlets.
The asynchronous blob copy service allows you to initiate a copy from a source location to a destination location. The beauty is that the source location does not have to be local. You can specify any URL that the copy service itself can access. This could be another Microsoft Azure storage account or just a file on a website. When the copy is initiated, the blob copy service will copy the file from the source to the destination without using your local machine as a middle man.
This opens several scenarios for infrastructure services. You can copy a virtual machine’s underlying disks between regions and even between subscriptions. For example, if you have a functioning environment running in the West US region and you would like to copy or move it to East US or even East Asia, you can, all through PowerShell!
Before I dive into the practical aspects of how to do these types of copies in PowerShell, there are some things that you should know about how Azure storage works. For the examples in the table below, I am using four storage accounts.
If I ping each storage account using the full Domain Name System (DNS) name, I can see that opsgilitywest1 and opsgilitywest2 share the same IP address, and that opsgilitywest0 and opsgilityeast1 do not match opsgilitywest1 or opsgilitywest2.
Storage account | FQDN | IP address |
opsgilitywest0 | opsgilitywest0.blob.core.windows.net | 168.62.0.14 |
opsgilitywest1 | opsgilitywest1.blob.core.windows.net | 168.63.89.142 |
opsgilitywest2 | opsgilitywest2.blob.core.windows.net | 168.63.89.142 |
opsgilityeast1 | opsgilityeast1.blob.core.windows.net | 138.91.96.142 |
Within each Azure region, storage is further subdivided into storage stamps. When you create a storage account, Azure places that storage account in one of the stamps for that region.
How do you know if two storage accounts in the same region are actually in the same stamp? Simply ping the name, and if the IP address returned for each storage account is the same, then they are in the same stamp. In Figure 5-20, the storage accounts opsgilitywest1 and opsgilitywest2 are in the same stamp, but not in the same stamp as opsgilitywest0, even though they are all in the same region.
Storage stamps do not span regions, so opsgilityeast1 is in a completely separate region and stamp than the storage accounts in the West US region (see Figure 5-21).
Why does all of this matter? If you just want to copy files from PowerShell, it shouldn’t matter, right? It matters because copying blob data between storage accounts in the same stamp is ridiculously fast when copying files between storage accounts even in the same region, but copying between different stamps is not so fast. Of course, copying files between regions is going to be slowest of all and will vary based on the distance the regions are from each other.
Table 5-8 shows some example copies from testing the copy of a 127 GB OS disk between storage accounts in Microsoft Azure.
Source | Destination | Result |
opsgilitywest1 | opsgilitywest1 | Shadow Copy—instantaneous |
opsgilitywest1 | opsgilitywest2 | Shadow Copy—instantaneous |
opsgilitywest0 | opsgilitywest1 | Cross-Stamp Copy (12 minutes and 3 seconds) |
opsgilitywest1 | opsgilityeast1 | Cross-Region Copy (22 minutes and 54 seconds) |
As you can see, the location of your storage accounts can have a dramatic impact on the performance of a copy operation using the asynchronous blob copy cmdlets. Unfortunately, now that you know this critical information, I do have bad news for you. The bad news is that you cannot directly control the placement of which stamp your storage account will be created in. When you create a new storage account, the stamp is automatically selected for you.
Now that you understand how storage is architected and how the asynchronous blob copy works behind the scenes, let’s look at triggering a copy operation between two storage accounts in the same subscription.
Example 5-44 starts by using the Select-AzureSubscription
cmdlet to set the PowerShell context to the correct subscription. From there it creates a variable $vhdName
that references the name of the VHD file to copy in the storage account, and the source and destination container names. The next step is to use the Get-AzureStorageKey
cmdlet to retrieve the primary authentication key for the source and destination storage accounts.
The code then creates two storage context objects for both storage accounts using the New-AzureStorageContext
cmdlet. This cmdlet is passed the storage account name and key and generates the context object that will be used by the asynchronous blob copy API to authenticate at the source and destination.
The New-AzureStorageContainer
cmdlet is used with the destination context to create a container on the destination storage account that will hold the copied VHD file.
Finally, the code calls the Start-AzureStorageBlobCopy
cmdlet, passing the appropriate information for the VHD name on the source and destination, along with the container names and the storage security context objects for both storage accounts.
This cmdlet executes a request to the async blob copy API, and the blob copy state is returned and stored in the $blobCopyState
variable to use for polling the copy status.
# Select the subscription
Select-AzureSubscription
"[subscription name]"
$vhdName
=
"[filename.vhd]"
$srcContainer
=
"[source container]"
$destContainer
=
"[destination container]"
# Source storage account
$srcStorage
=
"[source storage]"
# Destination storage account
$destStorage
=
"[dest storage]"
$srcStorageKey
=
(
Get-AzureStorageKey
-StorageAccountName
$srcStorage
).
Primary
# If the destination storage account is in a separate subscription, switch to the
# destination subscription first to retrieve the storage account key.
# Select-AzureSubscription "[destination subscription name]"
$destStorageKey
=
(
Get-AzureStorageKey
-StorageAccountName
$destStorage
).
Primary
# If the destinaton is in a separate subscription,
# switch back to the source subscription here
# Select-AzureSubscription "[subscription name]"
# Create the source storage account context
$srcContext
=
New-AzureStorageContext
–
StorageAccountName
$srcStorage
`
-StorageAccountKey
$srcStorageKey
# Create the destination storage account context
$destContext
=
New-AzureStorageContext
–
StorageAccountName
$destStorage
`
-StorageAccountKey
$destStorageKey
# Create the container on the destination
New-AzureStorageContainer
-Name
$destContainer
-Context
$destContext
# Start the asynchronous copy - specify the source authentication with -Context
$blobCopyState
=
Start-AzureStorageBlobCopy
-srcBlob
$vhdName
`
-srcContainer
$srcContainer
`
-Context
$srcContext
`
-DestContainer
$destContainer
`
-DestBlob
$vhdName
`
-DestContext
$destContext
The Get-AzureStorageBlobCopyState
cmdlet can then be used to validate whether the copy has completed or not. Simply pipe the returned value of a call to Start-AzureStorageBlobCopy
to the cmdlet, and it will return the current status of the copy operation. You can use this to monitor the status of the async copy to know when it is complete (see Example 5-45).
# Retrieve the current status of the copy operation
$status
=
$blobCopyState
|
Get-AzureStorageBlobCopyState
# Print out status
$status
# Loop until complete
While
(
$status
.
Status
-eq
"Pending"
){
$status
=
$blobCopyState
|
Get-AzureStorageBlobCopyState
Start-Sleep
10
# Print out status
$status
}
Example 5-46 is a working example with values from my subscription added to show a practical aspect to the sample.
# Select the subscription
Select-AzureSubscription
"opsgilitytraining"
# Retrieved the OS disk name using:
# Get-AzureVM -ServiceName "psdeploysvc" -Name "psdeploy" | Get-AzureOSDisk
$vhdName
=
"psdeploysvc-psdeploy-2014-03-17.vhd"
$srcContainer
=
"vhds"
$destContainer
=
"copiedvhds"
# Source storage account
$srcStorage
=
"opsgilitywest"
# Destination storage account
$destStorage
=
"opsgilityeast1"
$srcStorageKey
=
(
Get-AzureStorageKey
-StorageAccountName
$srcStorage
).
Primary
$destStorageKey
=
(
Get-AzureStorageKey
-StorageAccountName
$destStorage
).
Primary
# Create the source storage account context
$srcContext
=
New-AzureStorageContext
–
StorageAccountName
$srcStorage
`
-StorageAccountKey
$srcStorageKey
# Create the destination storage account context
$destContext
=
New-AzureStorageContext
–
StorageAccountName
$destStorage
`
-StorageAccountKey
$destStorageKey
# Create the container on the destination
New-AzureStorageContainer
-Name
$destContainer
-Context
$destContext
# Start the asynchronous copy - specify the source authentication with -Context
$blobCopyState
=
Start-AzureStorageBlobCopy
-srcBlob
$vhdName
`
-srcContainer
$srcContainer
`
-Context
$srcContext
`
-DestContainer
$destContainer
`
-DestBlob
$vhdName
`
-DestContext
$destContext
# Retrieve the current status of the copy operation
$status
=
$blobCopyState
|
Get-AzureStorageBlobCopyState
# Print out status
$status
# Loop until complete
While
(
$status
.
Status
-eq
"Pending"
){
$status
=
$blobCopyState
|
Get-AzureStorageBlobCopyState
Start-Sleep
10
# Print out status
$status
}
To retrieve the OS disk name, I piped the results of Get-AzureVM
to the Get-AzureOSDisk
cmdlet to extract the OS disk information (see Figure 5-22).
The storage accounts are in separate regions, so the copy operation takes anywhere from 7 to 12 minutes. The simple loop at the end of the script will print out the status of the copy operation every 10 seconds, updating the progress (see Figure 5-23).
In this chapter, we explored Microsoft Azure storage in the context of Microsoft Azure Virtual Machines. While this chapter is not comprehensive, I do hope it helps you understand the capabilities of the platform and hits on the key tasks that a Microsoft Azure administrator will likely be given with automating. In the next chapter we will discuss automation with Microsoft Azure Virtual Networks, including features such as the internal load balancer, and hybrid technologies such as site-to-site, point-to-site, and ExpressRoute.