Category Archives: Uncategorized

Uncategorized

Running Pester Tests in Parallel

I’m working on a project which uses pester tests to validate our deployment and system health.

We’ve accumulated a lot of tests. Nearly all of these sit and wait on things to happen in the deployment and we’re running them sequentially which takes around 20mins. Each of our tests is in it’s own file.

What I wanted to do was, as these are IO bound tests waiting on things to trip in the environment or polling http endpoints, run them all in parallel. One file per thead or something similar.

This issue discusses this topic. I took a lead from the Invoke-Parallel snipped and started playing, unfortunately the output from the tests was mangled and overlapping.

Then I realised I could use Powershell Jobs to schedule the work and poll for the job to be completed then receive each jobs to have to output displayed nicely.

So now the output looks good:



Note: We’re using Pester v4 you’ll have to do a little bit of fiddling to port to Pester 5

Install-Module –Name Pester –RequiredVersion 4.6.0 –force
$testFilePath = "./tests"
# Start a jobs running each of the test files
$testFiles = Get-ChildItem $testFilePath
$resultFileNumber = 0
foreach ($testFile in $testFiles)
{
$resultFileNumber++
$testName = Split-Path $testFile –leaf
# Create the job, be sure to pass argument in from the ArgumentList which
# are needed for inside the script block, they are NOT automatically passed.
Start-Job `
–ArgumentList $testFile, $resultFileNumber `
–Name $testName `
–ScriptBlock {
param($testFile, $resultFileNumber)
# Start trace for local debugging if TEST_LOG=true
# the traces will show you output in the ./testlogs folder and the files
# are updated as the tests run so you can follow along
if ($env:TEST_LOGS -eq "true") {
Start-Transcript –Path "./testlogs/$(Split-Path $testFile –leaf).integrationtest.log"
}
# Run the test file
Write-Host "$testFile to result file #$resultFileNumber"
$result = Invoke-Pester "$testFile"
if ($result.FailedCount -gt 0) {
throw "1 or more assertions failed"
}
}
}
# Poll to give insight into which jobs are still running so you can spot long running ones
do {
Write-Host ">> Still running tests @ $(Get-Date –Format "HH:mm:ss")" –ForegroundColor Blue
Get-Job | Where-Object { $_.State -eq "Running" } | Format-Table –AutoSize
Start-Sleep –Seconds 15
} while ((get-job | Where-Object { $_.State -eq "Running" } | Measure-Object).Count -gt 1)
# Catch edge cases by wait for all of them to finish
Get-Job | Wait-Job
$failedJobs = Get-Job | Where-Object { -not ($_.State -eq "Completed")}
# Receive the results of all the jobs, don't stop at errors
Get-Job | Receive-Job –AutoRemoveJob –Wait –ErrorAction 'Continue'
if ($failedJobs.Count -gt 0) {
Write-Host "Failed Jobs" –ForegroundColor Red
$failedJobs
throw "One or more tests failed"
}
view raw run.ps1 hosted with ❤ by GitHub

Here is the full repro: https://github.com/lawrencegripper/hack-parallelpester

Standard
Uncategorized

Azure Devops: How to run a Task if files have changed since last build

Shout out to the awesome work here from Alex Yates! This post builds on that work and updates a few bits.

What is the the aim? I have a file called IMAGETAG.txt which contains a simple version v1.0.1. It is used to build and push a Docker container as part of the build. If the file is changed in a commit, I want to build and push the docker image.

Now normally you could use the Path filtering stuff build into the Azure Devops Triggers but in this case I have lots of other tasks which I DO want to run and I don’t want multiple builds.

So how does it work? Well first up we’re going to create a script based off Alex’s work and update the params, API versions used and update the way it matches files.

The result is something we can call like this,

      changesSinceLastBuild.ps1` 
            -outputVariableName 'ML_IMAGE_TAG_CHANGED'`
            -fileMatchExpression 'containers/IMAGETAG.txt'`
            -branch 'refs/heads/main'`
            -buildDefinitionId '20'

This goes and gets the latest successful build from main for the specified build definition (in case you have multiple builds) and compares the changes done between then and the current HEAD commit.

In then uses powershell -match to find out if the file containers/IMAGETAG.txt was changed.

If it was it sets the Azure Devops Build variable ML_IMAGE_TAG_CHANGED to true.

We can then use this as a condition on another task in the Job.

So all together it looks like this, in my case the tasks invoke psake targets for ciml-docker.

– job: ciml
displayName: "Machine Learning CI"
pool:
vmImage: 'Ubuntu 20.04'
steps:
– task: PowerShell@2
displayName: 'Run CI Task from make.ps1 in Devcontainer'
inputs:
targetType: 'inline'
script: 'Install-Module -Name PSake -Force && Invoke-psake ./make.ps1 ciml'
errorActionPreference: 'stop'
pwsh: true
workingDirectory: $(Build.SourcesDirectory)
– task: PowerShell@2
displayName: 'Check if IMAGETAG.txt has changed since last build on main'
env:
System_AccessToken: $(System.AccessToken)
inputs:
targetType: 'filePath'
filePath: ./scripts/build_scripts/changesSinceLastBuild.ps1
arguments: >-
-outputVariableName 'ML_IMAGE_TAG_CHANGED'
-fileMatchExpression 'containers/IMAGETAG.txt'
-branch 'refs/heads/main'
-buildDefinitionId '20'
errorActionPreference: 'stop'
pwsh: true
workingDirectory: $(Build.SourcesDirectory)
– task: AzurePowerShell@5
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables.ML_IMAGE_TAG_CHANGED, 'true'))
displayName: 'Build ML Container and Push to ACR'
inputs:
azureSubscription: $(azure_connection)
ScriptType: 'inlineScript'
pwsh: true
inline: 'Install-Module -Name PSake -Force && Invoke-psake ./make.ps1 ciml-docker; exit (!$psake.build_success)'
errorActionPreference: 'stop'
workingDirectory: $(Build.SourcesDirectory)
azurePowerShellVersion: 'LatestVersion'
view raw buildyamlsnippet.yaml hosted with ❤ by GitHub
# This is an updated and tweaked version of a script designed to find if changes
# have occurred since the last build. We've moved to the latest API and tweaked to spot changes in a file.
# Original Source: http://workingwithdevs.com/azure-devops-services-api-powershell-hosted-build-agents/
param(
# The variable in AZDO to set when changes are found
[string]$outputVariableName = "changesFound",
# This can be 'containers/IMAGETAG.txt' or '.*/IMAGETAG.txt'. Expression is passed to powershell '-match' so valid regex NOT minimatch/glob
[string]$fileMatchExpression = $env:FILE_TO_CHECK,
# The build definition ID. In this case it's 20, look at the URL when editing a build to find it.
[string]$buildDefinitionId = "20",
[string]$branch = "refs/heads/main"
)
# Configuring this PS script to use the Azure DevOps Service Rest API
# Attempting to follow steps at: http://donovanbrown.com/post/how-to-call-team-services-rest-api-from-powershell
$pat = "Bearer $env:System_AccessToken"
# Set $env:LocalTest to test things out locally
if ($env:LocalTest) {
# When testing locally use the below to authenticate
$pat = $env:PAT_TOKEN # <- set this env to your pat token
$pat = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$pat"))
$env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI = "https://dev.azure.com/{ORGNAME}/"
$env:SYSTEM_TEAMPROJECTID = "{PROJECT NAME}"
$env:BUILD_BUILDID = 1163 # <- pick the build you'd like to pretend to be
}
Write-Host $env:System_AccessToken
# Setting the script to authenticate using the system access token on the Azure DevOps Build Agent
# Note: Remember to set the agent job to "Allow scripts to access OAuth token" (under "Additional options")
# FINDING THE PREVIOUS SUCCESSFUL BUILD
# Note: Combining double quotes and single quotes below because double quotes allow variable substitution
# (desired in first part) but single quotes do not (not desired in second part)
$env:SYSTEM_TEAMPROJECTID = [uri]::EscapeUriString($env:SYSTEM_TEAMPROJECTID)
$lastGreenBuildUrl = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/builds?branchName=$branch&definitions=$buildDefinitionId&queryOrder=finishTimeDescending&resultFilter=succeeded&api-version=6.0"
Write-Host "url: $lastGreenBuildUrl"
$data = Invoke-RestMethod –Uri "$lastGreenBuildUrl" –Headers @{Authorization = $pat}
Write-Host "Raw data returned from Green Build API call: $data"
$lastGreenBuild = $data.value.id | Select-Object –First 1
Write-Host "Last successfult build was $lastGreenBuild"
# FINDING ALL COMMITS SINCE THE LAST SUCCESSFUL COMMIT
$from = $lastGreenBuild
$to = $env:BUILD_BUILDID
Write-Host "from: $from"
Write-Host "to: $to"
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/changes?fromBuildId=$from&toBuildId=$to&api-version=5.0-preview.2"
Write-Host "url: $url"
$commits = Invoke-RestMethod –Uri $url –Headers @{Authorization = $pat }
Write-Host "Raw data returned from API call: $commits"
# saving all commit ids into a list
$commitHashes = @()
foreach ($id in $commits.value) {
$rawData = Get-Member –inputObject $id –name "id" | Out-String
# $rawData is in format "string id=<hash> ". We need to clean it up.
$label, $hash = $rawData.Split("=")
$hash = $hash.trim()
# Adding $hash to $commitHashes
$commitHashes += $hash
}
Write-Host "Commit hashes associated with this build:"
foreach ($hash in $commitHashes) {
Write-Host $hash
}
# We can use this function to check if a given directory has been updated since last build
$lastCommit = $commitHashes | Select-Object –Last 1
$currentCommit = git rev–parse HEAD
$currentCommit = $currentCommit.Trim()
Write-Host "Finding files changed between $lastCommit and $currentCommit"
# Show diff between
# See: https://stackoverflow.com/questions/3368590/show-diff-between-commits
$changedFiles = git diff $lastCommit^..$currentCommit —name–only
Write-Host "Changed files"
$changedFiles
# Check if the file we're interested in changed
$relevantChanges = $changedFiles -match $fileMatchExpression
if ($relevantChanges) {
Write-Host ">>>>> CHANGED FOUND in file $fileMatchExpression since last successful build $from. Build var $outputVariableName set to 'true'"
# Updating the build variables accordingly
Write-Output ("##vso[task.setvariable variable=$outputVariableName;]true")
} else {
Write-Host "No relevant changes found.. variable set to $changesFound"
}
view raw changesSinceLastBuild.ps1 hosted with ❤ by GitHub

Standard
Uncategorized

Terraform, Azure VPN Gateway and OpenVPN Config

Recently I needed a quick way to spin-up a VPN Gateway and generate the openvpn config clients could use to connect.

There is a good guide to generating the necessary certificates and manually editing the openvpn config you can download from the portal in the official docs.

Being a sucker for punishment I wondered if I could automate the process (mainly because I always forget the openssl commands) and secondly it was to be run by someone else (not me) so I wanted it to be as simple as possible.

Warning: Please take time to understand the limitations of this certificate generation before production use (TF State file containing CA private keys) and review the limitations in the TLS provider for Terraform.

So how would this work? First choice is Terraform for the automation. Luckily that has a provider for generating certs! So the flow goes like this:

1. Create the Root CA
2. Use that to generate a client cert
3. Output the client cert for use in the openvpn config file
4. Inject the CA into the Azure VPN configuration and create it
5. Run a script to fetch the Azure VPN OpenVPN configuration file (as this contains the Pre-shared key we don’t set) then inject the client cert we outputted from the Terraform.
6. Connect

One gotcha – The TLS provider only outputs standard pems. Azure VPN requires the CA to be in a specific format outputted by the following command.

openssl x509 -in caCert.pem -outform der | base64 -w0 > caCert.der

As a result there is a null_resource and local_file resource to handle this translation.

So all you should need is Terraform, Bash, OpenSSL and Azure CLI – not perfect but it’s the best I could do! (This is currently a working draft – please use with caution).

resource "random_string" "random" {
length = 8
special = false
upper = false
number = false
}
resource "azurerm_public_ip" "vpn_ip" {
name = "vpn-ip"
location = var.region
resource_group_name = var.resource_group_name
domain_name_label = random_string.random.result
allocation_method = "Dynamic"
tags = var.tags
}
resource "tls_private_key" "example" {
algorithm = "RSA"
rsa_bits = "2048"
}
# Create the root certificate
resource "tls_self_signed_cert" "ca" {
key_algorithm = tls_private_key.example.algorithm
private_key_pem = tls_private_key.example.private_key_pem
# Certificate expires after 1 year
validity_period_hours = 8766
# Generate a new certificate if Terraform is run within three
# hours of the certificate's expiration time.
early_renewal_hours = 200
# Allow to be used as a CA
is_ca_certificate = true
allowed_uses = [
"key_encipherment",
"digital_signature",
"server_auth",
"client_auth",
"cert_signing"
]
dns_names = [ azurerm_public_ip.vpn_ip.domain_name_label ]
subject {
common_name = "CAOpenVPN"
organization = "dev env"
}
}
resource "local_file" "ca_pem" {
filename = "caCert.pem"
content = tls_self_signed_cert.ca.cert_pem
}
resource "null_resource" "cert_encode" {
provisioner "local-exec" {
# Bootstrap script called with private_ip of each node in the clutser
command = "openssl x509 -in caCert.pem -outform der | base64 -w0 > caCert.der"
}
depends_on = [ local_file.ca_pem ]
}
data "local_file" "ca_der" {
filename = "caCert.der"
depends_on = [
null_resource.cert_encode
]
}
resource "tls_private_key" "client_cert" {
algorithm = "RSA"
rsa_bits = "2048"
}
resource "tls_cert_request" "client_cert" {
key_algorithm = tls_private_key.client_cert.algorithm
private_key_pem = tls_private_key.client_cert.private_key_pem
# dns_names = [ azurerm_public_ip.vpn_ip.domain_name_label ]
subject {
common_name = "ClientOpenVPN"
organization = "dev env"
}
}
resource "tls_locally_signed_cert" "client_cert" {
cert_request_pem = tls_cert_request.client_cert.cert_request_pem
ca_key_algorithm = tls_private_key.client_cert.algorithm
ca_private_key_pem = tls_private_key.client_cert.private_key_pem
ca_cert_pem = tls_self_signed_cert.ca.cert_pem
validity_period_hours = 43800
allowed_uses = [
"key_encipherment",
"digital_signature",
"server_auth",
"key_encipherment",
"client_auth",
]
}
resource "azurerm_virtual_network_gateway" "vpn-gateway" {
name = "vpn-gateway"
location = var.region
resource_group_name = var.resource_group_name
type = "Vpn"
active_active = false
enable_bgp = false
sku = "VpnGw1"
ip_configuration {
name = "vnetGatewayConfig"
public_ip_address_id = azurerm_public_ip.vpn_ip.id
private_ip_address_allocation = "Dynamic"
subnet_id = azurerm_subnet.yoursubnethere.id
}
vpn_client_configuration {
address_space = ["10.1.0.0/16"]
vpn_client_protocols = ["OpenVPN"]
root_certificate {
name = "terraformselfsignedder"
public_cert_data = data.local_file.ca_der.content
}
}
}
output "client_cert" {
value = tls_locally_signed_cert.client_cert.cert_pem
}
output "client_key" {
value = tls_private_key.client_cert.private_key_pem
}
output "vpn_id" {
value = azurerm_virtual_network_gateway.vpn-gateway.id
}
view raw main.tf hosted with ❤ by GitHub
#!/bin/bash
set -e
# Get vars from TF State
VPN_ID=`terraform output vpn_id`
VPN_CLIENT_CERT=`terraform output client_cert`
VPN_CLIENT_KEY=`terraform output client_key`
# Replace newlines with \n so sed doesn't break
VPN_CLIENT_CERT="${VPN_CLIENT_CERT//$'\n'/\\n}"
VPN_CLIENT_KEY="${VPN_CLIENT_KEY//$'\n'/\\n}"
CONFIG_URL=`az network vnet-gateway vpn-client generate –ids $VPN_ID -o tsv`
wget $CONFIG_URL -O "vpnconfig.zip"
# Ignore complaint about backslash in filepaths
unzip -o "vpnconfig.zip" -d "./vpnconftemp"|| true
OPENVPN_CONFIG_FILE="./vpnconftemp/OpenVPN/vpnconfig.ovpn"
echo "Updating file $OPENVPN_CONFIG_FILE"
sed -i "s~\$CLIENTCERTIFICATE~$VPN_CLIENT_CERT~" $OPENVPN_CONFIG_FILE
sed -i "s~\$PRIVATEKEY~$VPN_CLIENT_KEY~g" $OPENVPN_CONFIG_FILE
cp $OPENVPN_CONFIG_FILE openvpn.ovpn
rm -r ./vpnconftemp
rm vpnconfig.zip
view raw openvpn_gen.sh hosted with ❤ by GitHub
Standard