The case of the inconsistent Azure Functions host key

I recently ran into a situation using the Azure Functions default host key where I did not understand the behavior I was observing. Thanks to the help of some fantastic colleagues, we figured out what was going on. I understand what is happening now. I want to share here in hopes that my experience will help others that may run into a similar challenge.

Scenario

I needed to create an Azure Function app via an ARM template. The ARM template should create the Function app resource and set the necessary application settings. One of those app settings should be a Key vault reference to a secret which contains the default host key for the function app. My function’s application code would retrieve that and invoke another function in the app.

Also, please see Benjamin Perkin’s “Azure Function keys, keys and more keys, regenerated and synced” blog post for more observations on Azure Function key generation.

It all seems straight-forward enough. I’m very comfortable with Azure Functions and feel that I’ve got a decent grasp on ARM templates. Famous last words.

How it started

The snippet below shows my starting ARM template. Can you spot the error?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        // Omitted for brevity.
    },
    "variables": {
        // Omitted for brevity.
    },
    "resources": [
        {
            "type": "Microsoft.KeyVault/vaults",
            "apiVersion": "2019-09-01",
            "name": "[variables('keyVaultName')]",
            "location": "[parameters('location')]",
            "dependsOn":[
                "[resourceId('Microsoft.Web/sites',variables('functionAppName'))]"
            ],
            "properties": {
                // Omitted for brevity.
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('keyVaultName'), '/', 'functionKey')]",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[resourceId('Microsoft.KeyVault/vaults', variables('keyVaultName'))]",
                "[resourceId('Microsoft.Web/sites',variables('functionAppName'))]"
            ],
            "properties": {
                "value": "[listKeys(concat(resourceId('Microsoft.Web/sites', variables('functionAppName')), '/host/default'), '2016-08-01').functionKeys.default]"
            }
        },
        {
            "type": "Microsoft.Web/serverfarms",
            "apiVersion": "2020-06-01",
            "name": "[variables('appServicePlanName')]",
            "location": "[parameters('location')]",
            "sku": {
                "name": "Y1",
                "tier": "Dynamic",
                "size": "Y1",
                "family": "Y",
                "capacity": 0
            },
            "properties": {
               // Omitted for brevity.
            },
            "kind": "functionapp"
        },
        {
            "type": "Microsoft.Web/sites",
            "apiVersion": "2020-06-01",
            "name": "[variables('functionAppName')]",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]"
            ],
            "kind": "functionapp",
            "identity": {
                "type": "SystemAssigned"
            },
            "properties": {
                "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]",
                "httpsOnly": true
            },
            "resources": []
        },
        {
            "type": "Microsoft.Web/sites/config",
            "apiVersion": "2020-06-01",
            "name": "[concat(variables('functionAppName'), '/appsettings')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/sites', variables('functionAppName'))]",
                "[resourceId('Microsoft.KeyVault/vaults/secrets', variables('keyVaultName'),'functionKey')]"
            ],
            "properties": {
                "AzureWebJobsStorage": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'), ';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]",
                "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'), ';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]",
                "WEBSITE_CONTENTSHARE": "[variables('functionAppName')]",
                "FUNCTIONS_EXTENSION_VERSION": "~3",
                "FUNCTIONS_WORKER_RUNTIME": "dotnet",
                "APPINSIGHTS_INSTRUMENTATIONKEY": "[reference(resourceId('Microsoft.Insights/components', variables('appInsightsName')), '2018-05-01-preview').instrumentationKey]",
                "WEBSITE_RUN_FROM_PACKAGE": "1",
                "MyFunctionHostKey":"[concat('@Microsoft.KeyVault(SecretUri=', reference(resourceId('Microsoft.KeyVault/vaults/secrets', variables('keyVaultName'), 'functionKey')).secretUriWithVersion, ')')]"
            }
        }
    ]
}

You can find the full problematic ARM template on my GitHub repo here.

The problem

After running the template, I noticed that the Key Vault secret value for the default host key was not the same value as the Azure Function portal showed.

keys do not match

I had two different host keys! How is this possible?

The analysis

The problem is where the FUNCTIONS_EXTENSION_VERSION application setting was set. By default, Azure Functions apps are set to use the v1 runtime. The Azure Functions v1 runtime persists its key in Azure Files (see also this).

When creating the function app via the ARM template, I did not explicitly set the FUNCTIONS_EXTENSION_VERSION app setting. Thus, the runtime was set to the v1 runtime. When retrieving the key in the ARM template (setting to a Key Vault secret), the Azure Functions runtime returned the key from Azure Files (the v1 key).

In my template, another ARM resource block set the function’s application settings, and it is there where the FUNCTIONS_EXTENSION_VERSION was set to ~3 (the v3 runtime). By setting the application setting in a separate step, that forced a restart of the function. Any change to the application settings triggers an application restart. A consequence of this restart was the Azure Functions’ runtime changing to v3.

The v3 runtime uses Azure Blob storage for persisting the keys. Thus, when inspecting at the host key via the Azure portal after the template finished executing, the key returned was the key from Azure Blob storage (the v3 key).

To summarize, the Key Vault secret value was set, and then the function app restarted. Meaning, the default host key that was retrieved and set in the Key Vault secret was a v1 key, not a v3 key. The sequence was as follows:

  1. Create a function app with the default runtime (~1).
  2. Retrieve the function’s default host key and set as a Key Vault secret.
  3. Set the function app’s application settings.
  4. The function app restarts.
  5. Function is running a v3 (~3) runtime.

The solution

The “fix” is to set the FUNCTIONS_EXTENSION_VERSION at the same time as the function app is created. Which, is exactly what this documentation indicates. Figures.

If I set the FUNCTIONS_EXTENSION_VERSION to ~3 at app creation time, I get the expected default host key value in Key Vault, and in the portal. They keys are the same!

It all seems so simple . . . until it is not.

The downside is that there is now a need to define the application settings twice! The majority of the application settings should be set during function creation. The site resource needs to be created before the Key Vault resource in order to set the Key Vault’s access policy allowing the site access to the Key Vault’s secrets. Thus, I need to define, again, the app settings in a separate resource block, just like I had originally! The template’s Microsoft.Web/Sites resource looks as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

{
            "type": "Microsoft.Web/sites",
            "apiVersion": "2020-06-01",
            "name": "[variables('functionAppName')]",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/serverfarms',variables('appServicePlanName'))]",
                "[resourceId('Microsoft.Storage/storageAccounts',variables('storageAccountName'))]",
                "[resourceId('Microsoft.Insights/components', variables('appInsightsName'))]"
            ],
            "kind": "functionapp",
            "identity": {
                "type": "SystemAssigned"
            },
            "properties": {
                "name": "[variables('functionAppName')]",
                "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]",
                "httpsOnly": true,
                "siteConfig": {
                    "appSettings": [
                        {
                            "name": "AzureWebJobsStorage",
                            "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'), ';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]"
                        },
                        {
                            "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
                            "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'), ';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]"
                        },
                        {
                            "name": "WEBSITE_CONTENTSHARE",
                            "value": "[variables('functionAppName')]"
                        },
                        {
                            "name": "FUNCTIONS_EXTENSION_VERSION",
                            "value": "~3"
                        },
                        {
                            "name": "FUNCTIONS_WORKER_RUNTIME",
                            "value": "dotnet"
                        },
                        {
                            "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
                            "value": "[reference(resourceId('Microsoft.Insights/components', variables('appInsightsName')), '2018-05-01-preview').instrumentationKey]"
                        },
                        {
                            "name": "WEBSITE_RUN_FROM_PACKAGE",
                            "value": "1"
                        }
                    ]
                }
            },
            "resources": [
                {
                    "type": "config",
                    "name": "appsettings",
                    "apiVersion": "2020-06-01",
                    "dependsOn": [
                        "[resourceId('Microsoft.Web/sites', variables('functionAppName'))]",
                        "[resourceId('Microsoft.KeyVault/vaults/', variables('keyVaultName'))]",
                        "[resourceId('Microsoft.KeyVault/vaults/secrets', variables('keyVaultName'), 'functionKey')]"
                    ],
                    "properties": {
                        // Need to set the application settings seperate from the web site due to the Key Vault reference dependency.
                        "AzureWebJobsStorage": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'), ';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]",
                        "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'), ';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2019-06-01').keys[0].value, ';EndpointSuffix=', environment().suffixes.storage)]",
                        "WEBSITE_CONTENTSHARE": "[variables('functionAppName')]",
                        "FUNCTIONS_EXTENSION_VERSION": "~3",
                        "FUNCTIONS_WORKER_RUNTIME": "dotnet",
                        "APPINSIGHTS_INSTRUMENTATIONKEY": "[reference(resourceId('Microsoft.Insights/components', variables('appInsightsName')), '2018-05-01-preview').instrumentationKey]",
                        "WEBSITE_RUN_FROM_PACKAGE": "1",
                        "MyFunctionHostKey": "[concat('@Microsoft.KeyVault(SecretUri=', reference(resourceId('Microsoft.KeyVault/vaults/secrets', variables('keyVaultName'), 'functionKey')).secretUriWithVersion, ')')]"
                    }
                }
            ]
        }

You can find the full ARM template on my GitHub repo.

Application settings are defined as a block. Meaning, I cannot set just the MyFunctionHostKey. All the settings need to be defined again. I end up duplicating the settings, which I do not like.

Summary

I learned the key is to explicitly set the FUNCTIONS_EXTENSION_VERSION to the desired runtime at creation time. Setting it afterwards will result in an app restart and may yield unexpected results.

Resources