Uncategorized

Azure Devops: How to run a Task if files have changed since last build

Shout out to the awesome work here from Alex Yates! This post builds on that work and updates a few bits.

What is the the aim? I have a file called IMAGETAG.txt which contains a simple version v1.0.1. It is used to build and push a Docker container as part of the build. If the file is changed in a commit, I want to build and push the docker image.

Now normally you could use the Path filtering stuff build into the Azure Devops Triggers but in this case I have lots of other tasks which I DO want to run and I don’t want multiple builds.

So how does it work? Well first up we’re going to create a script based off Alex’s work and update the params, API versions used and update the way it matches files.

The result is something we can call like this,

      changesSinceLastBuild.ps1` 
            -outputVariableName 'ML_IMAGE_TAG_CHANGED'`
            -fileMatchExpression 'containers/IMAGETAG.txt'`
            -branch 'refs/heads/main'`
            -buildDefinitionId '20'

This goes and gets the latest successful build from main for the specified build definition (in case you have multiple builds) and compares the changes done between then and the current HEAD commit.

In then uses powershell -match to find out if the file containers/IMAGETAG.txt was changed.

If it was it sets the Azure Devops Build variable ML_IMAGE_TAG_CHANGED to true.

We can then use this as a condition on another task in the Job.

So all together it looks like this, in my case the tasks invoke psake targets for ciml-docker.

job: ciml
displayName: "Machine Learning CI"
pool:
vmImage: 'Ubuntu 20.04'
steps:
task: PowerShell@2
displayName: 'Run CI Task from make.ps1 in Devcontainer'
inputs:
targetType: 'inline'
script: 'Install-Module -Name PSake -Force && Invoke-psake ./make.ps1 ciml'
errorActionPreference: 'stop'
pwsh: true
workingDirectory: $(Build.SourcesDirectory)
task: PowerShell@2
displayName: 'Check if IMAGETAG.txt has changed since last build on main'
env:
System_AccessToken: $(System.AccessToken)
inputs:
targetType: 'filePath'
filePath: ./scripts/build_scripts/changesSinceLastBuild.ps1
arguments: >-
-outputVariableName 'ML_IMAGE_TAG_CHANGED'
-fileMatchExpression 'containers/IMAGETAG.txt'
-branch 'refs/heads/main'
-buildDefinitionId '20'
errorActionPreference: 'stop'
pwsh: true
workingDirectory: $(Build.SourcesDirectory)
task: AzurePowerShell@5
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables.ML_IMAGE_TAG_CHANGED, 'true'))
displayName: 'Build ML Container and Push to ACR'
inputs:
azureSubscription: $(azure_connection)
ScriptType: 'inlineScript'
pwsh: true
inline: 'Install-Module -Name PSake -Force && Invoke-psake ./make.ps1 ciml-docker; exit (!$psake.build_success)'
errorActionPreference: 'stop'
workingDirectory: $(Build.SourcesDirectory)
azurePowerShellVersion: 'LatestVersion'
# This is an updated and tweaked version of a script designed to find if changes
# have occurred since the last build. We've moved to the latest API and tweaked to spot changes in a file.
# Original Source: http://workingwithdevs.com/azure-devops-services-api-powershell-hosted-build-agents/
param(
# The variable in AZDO to set when changes are found
[string]$outputVariableName = "changesFound",
# This can be 'containers/IMAGETAG.txt' or '.*/IMAGETAG.txt'. Expression is passed to powershell '-match' so valid regex NOT minimatch/glob
[string]$fileMatchExpression = $env:FILE_TO_CHECK,
# The build definition ID. In this case it's 20, look at the URL when editing a build to find it.
[string]$buildDefinitionId = "20",
[string]$branch = "refs/heads/main"
)
# Configuring this PS script to use the Azure DevOps Service Rest API
# Attempting to follow steps at: http://donovanbrown.com/post/how-to-call-team-services-rest-api-from-powershell
$pat = "Bearer $env:System_AccessToken"
# Set $env:LocalTest to test things out locally
if ($env:LocalTest) {
# When testing locally use the below to authenticate
$pat = $env:PAT_TOKEN # <- set this env to your pat token
$pat = "Basic " + [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$pat"))
$env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI = "https://dev.azure.com/{ORGNAME}/"
$env:SYSTEM_TEAMPROJECTID = "{PROJECT NAME}"
$env:BUILD_BUILDID = 1163 # <- pick the build you'd like to pretend to be
}
Write-Host $env:System_AccessToken
# Setting the script to authenticate using the system access token on the Azure DevOps Build Agent
# Note: Remember to set the agent job to "Allow scripts to access OAuth token" (under "Additional options")
# FINDING THE PREVIOUS SUCCESSFUL BUILD
# Note: Combining double quotes and single quotes below because double quotes allow variable substitution
# (desired in first part) but single quotes do not (not desired in second part)
$env:SYSTEM_TEAMPROJECTID = [uri]::EscapeUriString($env:SYSTEM_TEAMPROJECTID)
$lastGreenBuildUrl = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/builds?branchName=$branch&definitions=$buildDefinitionId&queryOrder=finishTimeDescending&resultFilter=succeeded&api-version=6.0"
Write-Host "url: $lastGreenBuildUrl"
$data = Invoke-RestMethod Uri "$lastGreenBuildUrl" Headers @{Authorization = $pat}
Write-Host "Raw data returned from Green Build API call: $data"
$lastGreenBuild = $data.value.id | Select-Object First 1
Write-Host "Last successfult build was $lastGreenBuild"
# FINDING ALL COMMITS SINCE THE LAST SUCCESSFUL COMMIT
$from = $lastGreenBuild
$to = $env:BUILD_BUILDID
Write-Host "from: $from"
Write-Host "to: $to"
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/changes?fromBuildId=$from&toBuildId=$to&api-version=5.0-preview.2"
Write-Host "url: $url"
$commits = Invoke-RestMethod Uri $url Headers @{Authorization = $pat }
Write-Host "Raw data returned from API call: $commits"
# saving all commit ids into a list
$commitHashes = @()
foreach ($id in $commits.value) {
$rawData = Get-Member inputObject $id name "id" | Out-String
# $rawData is in format "string id=<hash> ". We need to clean it up.
$label, $hash = $rawData.Split("=")
$hash = $hash.trim()
# Adding $hash to $commitHashes
$commitHashes += $hash
}
Write-Host "Commit hashes associated with this build:"
foreach ($hash in $commitHashes) {
Write-Host $hash
}
# We can use this function to check if a given directory has been updated since last build
$lastCommit = $commitHashes | Select-Object Last 1
$currentCommit = git revparse HEAD
$currentCommit = $currentCommit.Trim()
Write-Host "Finding files changed between $lastCommit and $currentCommit"
# Show diff between
# See: https://stackoverflow.com/questions/3368590/show-diff-between-commits
$changedFiles = git diff $lastCommit^..$currentCommit nameonly
Write-Host "Changed files"
$changedFiles
# Check if the file we're interested in changed
$relevantChanges = $changedFiles -match $fileMatchExpression
if ($relevantChanges) {
Write-Host ">>>>> CHANGED FOUND in file $fileMatchExpression since last successful build $from. Build var $outputVariableName set to 'true'"
# Updating the build variables accordingly
Write-Output ("##vso[task.setvariable variable=$outputVariableName;]true")
} else {
Write-Host "No relevant changes found.. variable set to $changesFound"
}

Standard

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s