Author Archives: lawrencegripper

#terraform, Apps, Azure, Coding, How to

Azure Functions Get Key from Terraform without InternalServerError

So you’re trying to use the Terraform azurerm_function_app_host_keys resource to get the keys from an Azure function after deployment. Sadly, as of 03/2021, this can fail intermittently šŸ˜¢ (See issue 1 and 2).+

[Edit: Hopefully this issue is resolved by this PR once released so worth reviewing once the change is released]

These errors can look something like these below:

Error making Read request on AzureRM Function App Hostkeys “***”: web.AppsClient#ListHostKeys: Failure responding to request: StatusCode=400 — Original Error: autorest/azure: Service returned an error. Status=400 Code=”BadRequest” Message=”Encountered an error (ServiceUnavailable) from host runtime”

Error: Error making Read request on AzureRM Function App Hostkeys “somefunx”: web.AppsClient#ListHostKeys: Failure responding to request: StatusCode=400

You can work around this by using my previous workaround with ARM templates but it’s a bit clunky so I was looking at another way to do it.

There is an AWESOME project by Scott Winkler called Shell Provider, it lets you write a custom Terraform provider using scripts. You can implement data types and full resources with CRUD support.

Looking into the errors returned by the azurerm_function_app_host_keys resource they’re intermittent and look like they’re related to a timing issue. Did you know the curl command support retrying out of the box?

So using the Shell provider we can create a simple script to make the REST request to the Azure API and use curls inbuilt retry support to have the request retried with an exponential back-off until it succeeds or 5mins is up!

Warning: This script uses –retry-all-errors which is only available in v7.71 and above. The version shipped with the distro your using might not be up-to-date user curl --version to check.

Here is a rough example of what you end up with:

terraform {
required_providers {
shell = {
source = "scottwinkler/shell"
version = "1.7.7"
}
}
}
resource "azurerm_function_app" "functions" {
name = "${var.function_name}–${var.random_string}-premium"
location = var.resource_group_location
resource_group_name = var.resource_group_name
app_service_plan_id = var.app_service_plan_id
version = "~3"
storage_account_name = var.storage_account_name
storage_account_access_key = var.storage_account_key
identity {
type = "SystemAssigned"
}
site_config {
# Ensure we use all the mem on the box and not only 3.5GB of it!
use_32_bit_worker_process = false
pre_warmed_instance_count = 1
}
app_settings = merge({
StorageContainerName = var.test_storage_container_name
https_only = true
FUNCTIONS_WORKER_RUNTIME = "dotnet"
HASH = base64encode(filesha256(local.func_zip_path))
WEBSITE_RUN_FROM_PACKAGE = "https://${var.storage_account_name}.blob.core.windows.net/${var.deployment_container_name}/${azurerm_storage_blob.appcode.name}${var.storage_sas}"
# Route outbound requests over VNET see: https://docs.microsoft.com/en-us/azure/azure-functions/functions-networking-options#regional-virtual-network-integration
WEBSITE_DNS_SERVER = "168.63.129.16"
WEBSITE_VNET_ROUTE_ALL = 1
}, var.app_settings)
}
data "azurerm_subscription" "current" {
}
data "shell_script" "functions_key" {
lifecycle_commands {
read = file("${path.module}/readkey.sh")
}
environment = {
FUNC_NAME = azurerm_function_app.functions.name
RG_NAME = var.resource_group_name
SUB_ID = data.azurerm_subscription.current.subscription_id
}
depends_on = [azurerm_function_app.functions]
}
view raw main.tf hosted with ❤ by GitHub
output "function_master_key" {
# Try is used here to ensure destroy works as expected. On destroy the map will be
# empty so try instead returns an empty string
# See: https://www.terraform.io/docs/language/functions/try.html
value = try(data.shell_script.functions_key.output["masterKey"], "")
}
output "function_hostname" {
value = azurerm_function_app.functions.default_hostname
}
output "function_name" {
value = azurerm_function_app.functions.name
}
view raw output.tf hosted with ❤ by GitHub
#!/bin/bash
set -e
# Get a token so we can call the ARM api
TOKEN=$(az account get-access-token -o json | jq -r .accessToken)
# Attempt to list the keys with exponential backoff and do this for 5mins max
# –fail required see https://github.com/curl/curl/issues/6712
curl "https://management.azure.com/subscriptions/$SUB_ID/resourceGroups/$RG_NAME/providers/Microsoft.Web/sites/$FUNC_NAME/host/default/listkeys?api-version=2018-11-01" \
–compressed -H 'Content-Type: application/json;charset=utf-8' \
-H "Authorization: Bearer $TOKEN" -d "{}" \
–retry 8 –retry-max-time 360 –retry-all-errors –fail –silent
view raw readkeys.sh hosted with ❤ by GitHub

Standard
Uncategorized

Add Diagnostic Settings to Azure Storage account via Terraform

So you want to add a diagnostic setting to your Azure storage account via Terraform and you pass the storage account ID to target_resource_id only to get the following error:

Status=400 Code=”BadRequest” Message=”Category ‘StorageWrite’ is not supported.”

Here is the fix, the diagnostic target resource actually needs to be a sub-resource of the storage account, the ID for that is constructed as:

<StorageAccountId>/blobServices/default/

Using this you can create a terraform file similar to the below and it will create the diagnostic setting for you on the blob account.

For a more detailed explanation the issue here goes into more detail.

resource "random_string" "random" {
length = 5
special = false
upper = false
number = false
}
resource "azurerm_log_analytics_workspace" "core" {
name = "corelaw${random_string.random.result}"
location = "westeurope"
resource_group_name = "test1"
sku = "PerGB2018"
retention_in_days = 30
}
resource "azurerm_storage_account" "core" {
location = "westeurope"
resource_group_name = "test1"
name = "corestor${random_string.random.result}"
account_tier = "Standard"
account_replication_type = "LRS"
allow_blob_public_access = "false"
is_hns_enabled = true
enable_https_traffic_only = true
}
resource "azurerm_monitor_diagnostic_setting" "core-diagnostic" {
name = "readwritecore${random_string.random.result}"
# See workaround details: https://github.com/terraform-providers/terraform-provider-azurerm/issues/8275#issuecomment-755222989
target_resource_id = "${azurerm_storage_account.core.id}/blobServices/default/"
log_analytics_workspace_id = azurerm_log_analytics_workspace.core.id
log {
category = "StorageRead"
enabled = true
}
log {
category = "StorageWrite"
enabled = true
}
metric {
category = "Transaction"
enabled = true
retention_policy {
days = 5
enabled = true
}
}
}
view raw storagediagnosticlogs.tf hosted with ❤ by GitHub
Standard
Uncategorized

Running Pester Tests in Parallel

I’m working on a project which uses pester tests to validate our deployment and system health.

We’ve accumulated a lot of tests. Nearly all of these sit and wait on things to happen in the deployment and we’re running them sequentially which takes around 20mins. Each of our tests is in it’s own file.

What I wanted to do was, as these are IO bound tests waiting on things to trip in the environment or polling http endpoints, run them all in parallel. One file per thead or something similar.

This issue discusses this topic. I took a lead from the Invoke-Parallel snipped and started playing, unfortunately the output from the tests was mangled and overlapping.

Then I realised I could use Powershell Jobs to schedule the work and poll for the job to be completed then receive each jobs to have to output displayed nicely.

So now the output looks good:



Note: We’re using Pester v4 you’ll have to do a little bit of fiddling to port to Pester 5

Install-Module –Name Pester –RequiredVersion 4.6.0 –force
$testFilePath = "./tests"
# Start a jobs running each of the test files
$testFiles = Get-ChildItem $testFilePath
$resultFileNumber = 0
foreach ($testFile in $testFiles)
{
$resultFileNumber++
$testName = Split-Path $testFile –leaf
# Create the job, be sure to pass argument in from the ArgumentList which
# are needed for inside the script block, they are NOT automatically passed.
Start-Job `
–ArgumentList $testFile, $resultFileNumber `
–Name $testName `
–ScriptBlock {
param($testFile, $resultFileNumber)
# Start trace for local debugging if TEST_LOG=true
# the traces will show you output in the ./testlogs folder and the files
# are updated as the tests run so you can follow along
if ($env:TEST_LOGS -eq "true") {
Start-Transcript –Path "./testlogs/$(Split-Path $testFile –leaf).integrationtest.log"
}
# Run the test file
Write-Host "$testFile to result file #$resultFileNumber"
$result = Invoke-Pester "$testFile"
if ($result.FailedCount -gt 0) {
throw "1 or more assertions failed"
}
}
}
# Poll to give insight into which jobs are still running so you can spot long running ones
do {
Write-Host ">> Still running tests @ $(Get-Date –Format "HH:mm:ss")" –ForegroundColor Blue
Get-Job | Where-Object { $_.State -eq "Running" } | Format-Table –AutoSize
Start-Sleep –Seconds 15
} while ((get-job | Where-Object { $_.State -eq "Running" } | Measure-Object).Count -gt 1)
# Catch edge cases by wait for all of them to finish
Get-Job | Wait-Job
$failedJobs = Get-Job | Where-Object { -not ($_.State -eq "Completed")}
# Receive the results of all the jobs, don't stop at errors
Get-Job | Receive-Job –AutoRemoveJob –Wait –ErrorAction 'Continue'
if ($failedJobs.Count -gt 0) {
Write-Host "Failed Jobs" –ForegroundColor Red
$failedJobs
throw "One or more tests failed"
}
view raw run.ps1 hosted with ❤ by GitHub

Here is the full repro: https://github.com/lawrencegripper/hack-parallelpester

Standard