The quickest and easiest way to start running a Windows Container in Azure is using Azure Container Instances (ACI).
The problem is that they currently (as of 03/21) don’t support running Windows Containers inside a VNET. This blog is about how I worked around this limitation by automating the deployment and management of a Windows Containers with PowerShell DSC and Terraform.
As we needed VNET integration for the sensitive data handled on the project, I set out to build an ACI like experience on a VM and have that connected to a VNET in Azure.
First up it must:
- Handle restarting the container if things go wrong
- Give an easy way to retrieve the container logs from the commandline
- Connect reliably to the VNET
- Support updating easily (ie. When I push a new image tag the container is restarted running the new version or when new environment variables are applied handle restarting the container to pick these up)
- Support authentication to an Azure Container Repository
- Be runnable as part of a Terraform Deployment
Seems like a pretty long list right? At this point I reached out to a friend, Marcus Robison, who’d done more Windows Admin than me in his time. He suggested looking at Powershell DSC.
So what is Powershell DSC, what does it give us?
- Desired State Configuration for the VM. “I want a VM that looks like x” and it makes that happen. Much like Terraform or a K8s operator. It queries the current state and takes actions that move the current state closer to the desired state
- Integration with Azure VMs. There is a nice extension in Azure which allows you to submit a DSC config and Azure manages starting it on the VM for you.
- Handling of sensitive variables securely, with the Azure extension variables are encrypted
What does this all look like when you have it finished?
dsc_config.ps1
This script is responsible for configuring the machine, logging into ACR and ensuring the container is running. This runs periodically and handles things like restarting the container if a new deployment has been made with update environment variables.
Each Script (think resource in terraform) has a Get, Set and Test method. Test checks the current state of things, if they’re not how they’re meant to be Set is responsible for getting them configured correctly and lastly Get returns an identifier for the item.
module.tf
This is the terraform responsible for creating the VM and pushing up the DSC script for it to run.
It takes the dsc_config.ps1 and creates a zip file, this zip is then passed to the PowerShell DSC extension for the Azure VM which is responsible for applying the configuration to the VM.
As well as this the module also takes the environment variables you want set for your container. These are provided as a map and converted to a base64 encoded .env file. The DSC config on the VM decodes them and provides the .env file to the docker run command used to start the container.
*Worth nothing env.tpl is used in the process of creating the env file.
usage.tf
This is an example of using the terraform module from module.tf to create a VM which runs a container image on a VNET with a set of environment variables.
getlogs.ps1
Once deployed this little script demonstrates how you can get the logs out from the container running in the VM. It requires the Azure CLI to be installed and you to provide the VM’s Azure ID.
You can also hook this up to the outputs of your Terraform to automate it further.
All together now!
| <# | |
| .SYNOPSIS | |
| Uses PowerShell DSC to configure the machine to run the container | |
| .PARAMETER Image | |
| Docker Image to run complete with tag | |
| .PARAMETER Command | |
| Command to run in the docker image | |
| .PARAMETER RegistryUrl | |
| Azure container registry url | |
| .PARAMETER RegistryUsername | |
| .PARAMETER RegistryPassword | |
| .PARAMETER EnvironmentVariables | |
| A base64 encoded string of a .env file to use when running the container | |
| .PARAMETER InstanceName | |
| The name to give the running docker container | |
| .PARAMETER DockerConfigLocation | |
| [Optional defaults to '"C:\ProgramData\Docker\config\daemon.json"'] The location on disk of the docker daemon.json config file | |
| .PARAMETER DockerDataDir | |
| [Optional defaults to 'D:\\'] Location on disk for docker to store volumes and images | |
| #> | |
| Configuration DockerImageStart { | |
| [CmdletBinding()] | |
| param | |
| ( | |
| [Parameter(Mandatory = $true)] | |
| [ValidateNotNullOrEmpty()] | |
| [String] | |
| $Image, | |
| [Parameter(Mandatory = $true)] | |
| [String] | |
| $Command, | |
| [Parameter(Mandatory = $false)] | |
| [String] | |
| $ContainerName = "dscManagedContainerInstance", | |
| [Parameter(Mandatory = $true)] | |
| [ValidateNotNullOrEmpty()] | |
| [String] | |
| $RegistryUrl, | |
| [Parameter(Mandatory = $true)] | |
| [String] | |
| $RegistryUsername, | |
| [Parameter(Mandatory = $true)] | |
| [string] | |
| $RegistryPassword, | |
| [Parameter(Mandatory = $true)] | |
| [ValidateNotNullOrEmpty()] | |
| [string] | |
| $EnvironmentVariables, | |
| [Parameter(Mandatory = $false)] | |
| [string] | |
| $DockerConfigLocation = "C:\ProgramData\Docker\config\daemon.json", | |
| [Parameter(Mandatory = $false)] | |
| [string] | |
| $DockerDataDir = "D:\\" | |
| ) | |
| Import-DscResource –ModuleName 'PSDesiredStateConfiguration' | |
| Node localhost | |
| { | |
| # Have the machine check every 15 mins that config is good. | |
| # See details here: https://devblogs.microsoft.com/powershell/understanding-meta-configuration-in-windows-powershell-desired-state-configuration/ | |
| LocalConfigurationManager { | |
| ConfigurationMode = "ApplyAndAutoCorrect" | |
| RefreshFrequencyMins = 30 | |
| ConfigurationModeFrequencyMins = 15 | |
| RefreshMode = "PUSH" | |
| RebootNodeIfNeeded = $true | |
| } | |
| # Docs: Each 'Script' resource ensures a configruation is setup correctly on the VM | |
| # Get, test and set: https://docs.microsoft.com/en-us/powershell/scripting/dsc/resources/get-test-set?view=powershell-7.1 | |
| # Script module: https://docs.microsoft.com/en-us/powershell/scripting/dsc/reference/resources/windows/scriptResource?view=powershell-7.1 | |
| # Responsible for configuring the storage to use the ephemeral locally attached disk for docker storage | |
| Script DockerStorageLocation { | |
| SetScript = { | |
| Set-Content –Path $using:DockerConfigLocation –Value "{ `"data-root`": `"$using:DockerDataDir`" }" | |
| # Restart the Daemon so it picks up the new config | |
| Restart-Service –Force Docker | |
| } | |
| TestScript = { | |
| if (!(Test-Path $using:DockerConfigLocation)) { | |
| return $false | |
| } | |
| $dataroot = (Get-Content $using:DockerConfigLocation | ConvertFrom-Json)."data-root" | |
| if ($dataroot -ne $using:DockerDataDir) { | |
| return $false | |
| } | |
| if ((Get-Service Docker).Status -ne "Running") { | |
| return $false | |
| } | |
| return $true | |
| } | |
| GetScript = { | |
| # Return the ID of the current container | |
| @{ Result = (Get-Content $using:DockerConfigLocation | ConvertFrom-Json)."data-root" } | |
| } | |
| } | |
| # Responsible for ensuring that docker is logged into the ACR | |
| Script AzureContainerRepositoryLogin { | |
| DependsOn = "[Script]DockerStorageLocation" | |
| SetScript = { | |
| # Handle running the paramaterised docker commands: https://stackoverflow.com/questions/6338015/how-do-you-execute-an-arbitrary-native-command-from-a-string | |
| function Invoke-Login($command) { | |
| Write-Verbose "Running command $command" | |
| $output = $using:RegistryPassword | & 'docker' $command.Split(" ") 2>&1 | |
| if (!$? -and -not ($output -like "Login Succeeded")) | |
| { | |
| throw "Docker command failed, err: $output" | |
| } | |
| Write-Output $output | |
| } | |
| # Login to the ACR | |
| try { | |
| $output = Invoke-Login "login -u $using:RegistryUsername –password-stdin $using:RegistryUrl" | |
| } | |
| catch { | |
| Write-Error "Failed running login command $_ $output" | |
| } | |
| } | |
| TestScript = { | |
| # Check we're logged into the ACR | |
| if (!(Test-Path ~/.docker/config.json)) | |
| { | |
| Write-Verbose "No docker config file found so can't be logged in" | |
| return $false | |
| } | |
| $ConfigSettings = Get-Content ~/.docker/config.json | ConvertFrom-Json | |
| $LoginDetailsBase64 = $ConfigSettings.auth."$using:RegistryUrl" | |
| if (!$LoginDetailsBase64) { | |
| Write-Verbose "Didn't find login details for the repo $using:RegistryUrl" | |
| return $false | |
| } | |
| $LoginDetailsRaw = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($LoginDetailsBase64)) | |
| if ($LoginDetailsRaw -ne "$($using:RegistryUsername):$using:RegistryPassword") { | |
| Write-Verbose "Current credentials don't match expected creds for $using:RegistryUrl" | |
| return $false | |
| } | |
| return $true | |
| } | |
| GetScript = { | |
| @{ Result = (Get-Content ~/.docker/config.json | ConvertFrom-Json).auth."$using:RegistryUrl" } | |
| } | |
| } | |
| # Responsible for starting the container and keeping it running | |
| Script ContainerInstance { | |
| DependsOn = '[Script]AzureContainerRepositoryLogin', '[Script]DockerStorageLocation' | |
| SetScript = { | |
| # Handle running the paramaterised docker commands: https://stackoverflow.com/questions/6338015/how-do-you-execute-an-arbitrary-native-command-from-a-string | |
| function Invoke-Executable($command) { | |
| Write-Verbose "Running command $command" | |
| $output = & 'docker' $command.Split(" ") 2>&1 | |
| if (!$?) | |
| { | |
| throw "Docker command failed, err: $output" | |
| } | |
| Write-Output $output | |
| } | |
| # Attempt to remove the container if it exists | |
| try { | |
| Write-Verbose "Attempting to remove existing container" | |
| $output = Invoke-Executable "container rm -f $using:ContainerName" | |
| } | |
| catch { | |
| Write-Warning "Failed to remove existing container Error: $_ $output" | |
| # This is allowed to fail, for example the container might not be present | |
| } | |
| # Pull image | |
| try { | |
| $output = Invoke-Executable "pull $using:Image" | |
| } | |
| catch { | |
| Write-Error "An error occurred pulling image: $_ stdOut: $output" | |
| } | |
| # Start the container | |
| try { | |
| $EnvFileContent = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($using:EnvironmentVariables)) | |
| $EnvFileLocation = "C:\$($using:ContainerName).env" | |
| Set-Content –Path $EnvFileLocation –Value $EnvFileContent | |
| $EnvFileHash = (Get-FileHash $EnvFileLocation).Hash | |
| $output = Invoke-Executable "container run -d –restart=always –name=$using:ContainerName –env-file=$EnvFileLocation –label EnvFileHash=$EnvFileHash $using:Image $using:Command" | |
| } | |
| catch { | |
| Write-Error "An error occurred starting the container: $_ stdOut: $output" | |
| } | |
| } | |
| TestScript = { | |
| # Track errors from external commands: https://stackoverflow.com/questions/12359427/try-catch-on-executable-exe-in-powershell | |
| $ErrorActionPreference = 'Stop' | |
| # Retrieve all running conatiners | |
| $RunningContainers = iex 'docker ps –format "{{json . }}"' | ConvertFrom-Json | Where-Object { $_.Names -eq $using:ContainerName } | |
| # Write a "-check" version of the env file provided | |
| $EnvFileContent = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($using:EnvironmentVariables)) | |
| $EnvFileLocation = "C:\$($using:ContainerName)-check.env" | |
| Set-Content –Path $EnvFileLocation –Value $EnvFileContent | |
| # Get the files hash | |
| $EnvFileHash = (Get-FileHash $EnvFileLocation).Hash | |
| if ( | |
| ($RunningContainers | Measure-Object).Count -eq 1 ` | |
| -and $RunningContainers[0].Image -eq $using:Image ` | |
| -and $RunningContainers[0].Status -like "up *" ` | |
| ) | |
| { | |
| # Cool so it's running, is it running the right version of the environment variables? | |
| # Compare the hash to the currently set 'envhash' label on the running instance | |
| $DockerInspecResult = iex "docker inspect $($RunningContainers[0].ID)" | ConvertFrom-Json | |
| Write-Verbose "Env hash $($DockerInspecResult.Config.Labels.EnvFileHash)" | |
| if ($DockerInspecResult.Config.Labels.EnvFileHash -eq $EnvFileHash) { | |
| Write-Verbose "Container is in running state with correct image and env file" | |
| return $true | |
| } | |
| } | |
| Write-Verbose "Container does not exist, is not in a running state or has the incorrect image or env file" | |
| return $false | |
| } | |
| GetScript = { | |
| # Return the ID of the current container | |
| @{ Result = (iex 'docker ps –format "{{json . }}"' | ConvertFrom-Json | Where-Object { $_.Names -eq $using:ContainerName }).ID} | |
| } | |
| } | |
| } | |
| } |
| %{ for config_key, config_value in config ~} | |
| ${config_key}=${config_value} | |
| %{ endfor ~} |
| $output = az vm run–command invoke —ids <machine_id_here> —command–id RunPowershellScript —scripts "docker inspect dscManagedContainerInstance" | ConvertFrom-Json | |
| Write-Host $output.value[0].message |
| variable shared_env { | |
| type = any | |
| } | |
| variable subnet_id { | |
| type = string | |
| } | |
| variable name { | |
| type = string | |
| } | |
| variable image { | |
| type = string | |
| } | |
| variable command { | |
| type = string | |
| } | |
| variable environment_variables { | |
| description = "The environment variables to be set in the container" | |
| default = {} | |
| } | |
| variable docker_registry_url {} | |
| variable docker_registry_username {} | |
| variable docker_registry_password {} | |
| variable releases_storage_account_name { | |
| type = string | |
| } | |
| variable releases_storage_account_key { | |
| type = string | |
| } | |
| variable releases_container_name { | |
| type = string | |
| } | |
| variable releases_storage_sas { | |
| type = string | |
| } | |
| locals { | |
| env_file = base64encode(templatefile( | |
| "${path.module}/env.tpl", | |
| { | |
| config = var.environment_variables | |
| } | |
| )) | |
| script_name = "dsc_config.ps1" | |
| script_zip = "dsc_config.zip" | |
| script_hash = filemd5("${path.module}/dsc_config.ps1") | |
| } | |
| resource "random_string" "random" { | |
| length = 5 | |
| special = false | |
| upper = false | |
| number = false | |
| } | |
| resource "random_string" "adminpw" { | |
| length = 18 | |
| special = true | |
| upper = true | |
| number = true | |
| } | |
| resource "azurerm_network_interface" "nic" { | |
| name = "${var.name}${random_string.random.result}" | |
| resource_group_name = var.shared_env.rg.name | |
| location = var.shared_env.rg.location | |
| ip_configuration { | |
| name = "internal" | |
| subnet_id = var.subnet_id | |
| private_ip_address_allocation = "Dynamic" | |
| } | |
| } | |
| resource "azurerm_windows_virtual_machine" "vm" { | |
| name = "${var.name}${random_string.random.result}-vm" | |
| resource_group_name = var.shared_env.rg.name | |
| location = var.shared_env.rg.location | |
| computer_name = "relimporter" | |
| // 8 cores, 16GB ram and 128GB temp disk for import data to live on | |
| size = "Standard_F8" | |
| admin_username = "adminuser" | |
| admin_password = random_string.adminpw.result | |
| network_interface_ids = [ | |
| azurerm_network_interface.nic.id, | |
| ] | |
| os_disk { | |
| caching = "ReadWrite" | |
| storage_account_type = "Standard_LRS" | |
| } | |
| source_image_reference { | |
| publisher = "MicrosoftWindowsServer" | |
| offer = "WindowsServer" | |
| sku = "2019-Datacenter-Core-with-Containers" | |
| version = "latest" | |
| } | |
| patch_mode = "AutomaticByOS" | |
| } | |
| data "archive_file" "script_zip" { | |
| type = "zip" | |
| source_file = "${path.module}/${local.script_name}" | |
| output_path = "${path.module}/${local.script_zip}" | |
| } | |
| resource "azurerm_storage_blob" "dscps1" { | |
| name = "dsc${local.script_hash}.zip" | |
| storage_account_name = var.releases_storage_account_name | |
| storage_container_name = var.releases_container_name | |
| type = "Block" | |
| source = "${path.module}/${local.script_zip}" | |
| depends_on = [data.archive_file.script_zip] | |
| } | |
| // Using this https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/dsc-windows | |
| // to run a DSC configuration with will make sure the VM is running the container and | |
| // periodically check for any issue and correct them. | |
| // See: https://docs.microsoft.com/en-us/powershell/scripting/dsc/overview/overview?view=powershell-7.1 | |
| resource "azurerm_virtual_machine_extension" "dscconfig" { | |
| name = "dscconfig${local.script_hash}" | |
| virtual_machine_id = azurerm_windows_virtual_machine.vm.id | |
| publisher = "Microsoft.Powershell" | |
| type = "DSC" | |
| type_handler_version = "2.77" | |
| auto_upgrade_minor_version = true | |
| depends_on = [azurerm_storage_blob.dscps1] | |
| settings = jsonencode(jsondecode(<<SETTINGS | |
| { | |
| "configuration": { | |
| "url": "https://${var.releases_storage_account_name}.blob.core.windows.net/${var.releases_container_name}/dsc${local.script_hash}.zip", | |
| "script": "dsc_config.ps1", | |
| "function": "DockerImageStart" | |
| } | |
| } | |
| SETTINGS | |
| )) | |
| protected_settings = jsonencode(jsondecode(<<JSON | |
| { | |
| "configurationArguments": { | |
| "Image": "${var.image}", | |
| "Command": "${var.command}", | |
| "RegistryUrl": "${var.docker_registry_url}", | |
| "RegistryUsername": "${var.docker_registry_username}", | |
| "RegistryPassword": "${var.docker_registry_password}", | |
| "EnvironmentVariables": "${local.env_file}" | |
| }, | |
| "configurationUrlSasToken": "${var.releases_storage_sas}" | |
| } | |
| JSON | |
| )) | |
| } |
| # An example of using the above module | |
| module "container_vm" { | |
| source = "./docker_vm" | |
| shared_env = local.shared_env | |
| subnet_id = var.subnet_id | |
| docker_registry_username = var.docker_registry_username | |
| docker_registry_password = var.docker_registry_password | |
| docker_registry_url = var.docker_registry_url | |
| releases_storage_account_name = module.core.releases_storage_account_name | |
| releases_storage_account_key = module.core.releases_storage_account_key | |
| releases_storage_sas = module.core.releases_account_sas | |
| releases_container_name = module.core.releases_container_name | |
| name = "consolecontainer" | |
| image = "myregistry.azurecr.io/thingy:imagetag" | |
| command = "myConsoleApp.exe" | |
| environment_variables = { | |
| COSMOS_KEY = module.core.cosmos_account_key, | |
| COSMOS_ENDPOINT = module.core.cosmos_account_endpoint, | |
| COSMOS_DB_NAME = module.core.cosmos_db_name, | |
| COSMOS_CONTAINER_NAME = module.core.cosmos_container_name, | |
| } | |
| } |
