Azure Function Secretless Extensions - First Experience

Mon, Mar 15, 2021 (Last Modified: Thu, Jan 4, 2024)

I recently started experimenting with the beta versions of the new Azure Storage and Event Hub extensions for Azure Functions. The new extensions use the new Azure SDK, and as a result, include support for using Azure AD to authenticate to specific Azure resources (a.k.a., managed identities). I’m a fan of having fewer secrets to manage. Less secrets . . . more better. 😉

This intent of this blog post is to share my initial experiences with the extensions. It’s still early, and thus I expect a few bumps in along the way. Jerry Seinfeld - Giddy Up

Getting Started

I’m going to start by describing how I was able to get a very simple Azure Storage queue-triggered function to use an identity-based connection. Later, I’ll discuss how I was able to apply these learnings to work with Event Hubs.

Eager to get to it? Skip to the code samples in my GitHub repo.

Create the resources

I first need to create an Azure Storage account. This storage account will host the queue from which the function will receive messages. I’ll also use this storage account when provisioning my function app in Azure.

I know I’m going to eventually deploy this function to Azure. Thus, I’ll create a new Azure Functions Premium plan (an App Service plan should also work, but I haven’t tried yet). At the time I’m writing this, Azure Function Consumption plans are not yet supported for use with identity-based connections. Since the function will connect to Azure Storage using an identity-based connection, the Azure Function will need to be set up with a managed identity.

Azure Portal - enable managed identity

While creating the Azure Function app, I’m also going to create an Application Insights instance. I like Application Insights available with my Azure Functions. The Live Metrics and Logs are incredibly helpful in figuring out what may be going wrong.

My first function using managed identity

As previously mentioned, my first function to use an identity-based connection is an Azure Storage queue-triggered function. My primary objective is to establish the connection, using an Azure AD identity, to an Azure Storage queue so that the queue trigger executes. Since I’m starting locally, I want to use my local identity (on my dev laptop). Once I deploy to Azure, I want the function to use the identity of the function app.

I start by creating an Azure Storage queue-triggered function. In order to use the new Storage extension, I need to add the new extension to my project. This upgrades the extension to use the beta version of the v5.x extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.Storage --version 5.0.0-beta.3

Additionally, instead of using a CloudQueueMessage as the input type, I change to use the new QueueMessage type. This change is due to the use of the v5.x extension. My code now looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


using Azure.Storage.Queues.Models;

using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;

namespace Company.Function
{
    public static class QueueTriggerCSharp1
    {
        [FunctionName("QueueTriggerCSharp1")]
        public static void Run(
          [QueueTrigger("%QueueName%", Connection="MyStorageConnection")] QueueMessage myQueueItem,
          ILogger log)
        {
            log.LogInformation($"C# Queue trigger function processed: {myQueueItem}");
        }
    }
}

Making a connection

With my initial Azure Storage queue-triggered function created locally in Visual Studio Code, I need to set up an identity-based connection to my Azure Storage account. I need to set a local setting of name “MyStorageConnection__serviceUri” and value of the URI for my Azure Storage queue. My function code set the QueueTrigger attribute’s Connection property to “MyStorageConnection”. The key here is to have the value of the “Connection” property be the prefix of the application setting, with “__serviceUri” being the suffix. The double underscore (__) can be used to override host.json values.

Cat surprised

Thus, my local.settings.json file looks as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
  "IsEncrypted": false,
  "Values": {
    "APPINSIGHTS_INSTRUMENTATIONKEY":"[APPLICATION-INSIGHTS-KEY]",
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "QueueName":"[AZURE-STORAGE-QUEUE-NAME]",
    "MyStorageConnection__serviceUri": "https://[AZURE-STORAGE-ACCOUNT-NAME].queue.core.windows.net/"
  }
}

When starting the function, I notice this unsettling warning message in my console:

Warning: Cannot find value named 'MyStorageConnection' in local.settings.json that matches 'connection' property
set on 'queueTrigger' in 'C:\Users\mcollier\OneDrive - Microsoft\src\mcollier\blog-az-func-managed-identity\bin\output\QueueTriggerCSharp1\function.json'.
You can run 'func azure functionapp fetch-app-settings <functionAppName>' or specify a connection string in local.settings.json.

Azure Function Core Tools - Unable to find matching connection string setting

It is true . . . there is no “MyStorageConnection” setting in my local.settings.json file. But there is a “MyStorageConnection__serviceUri”! This seems to be a false warning message from the tooling. Presumably because the identity-based connection feature is new, and still preview, the core tools have not yet been updated to handle identity-based connections.

Steve Martin - It’s Fine. Let’s Just Move Past It

Access denied

With that connection string right, the next step is to try to debug my function locally from Visual Studio Code, connecting to my Azure Storage account (not the storage emulator). I try and get this error:

This request is not authorized to perform this operation using this permission.
RequestId:[REDACTED]
Time:2021-03-08T22:57:07.9749740Z
Status: 403 (This request is not authorized to perform this operation using this permission.)
ErrorCode: AuthorizationPermissionMismatch

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
RequestId:[REDACTED]
Time:2021-03-08T22:57:07.9749740Z</Message></Error>

Headers:
Server: Windows-Azure-Queue/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id: [REDACTED]
x-ms-version: 2018-11-09
x-ms-error-code: AuthorizationPermissionMismatch
Date: Mon, 08 Mar 2021 22:57:07 GMT
Content-Length: 279
Content-Type: application/xml

Right . . . I can work with that. My local identity doesn’t have the necessary permissions to work with the designated Azure Storage queue. I need to set the right permissions for my local identity to work with the Azure Storage queue. Through a bit of trial and error, I learn that I need to put myself in the Storage Queue Data Contributor role. I often set that up via the Azure portal, but I’ve included an Azure CLI example below.

1
2
3
4
5
6


#!/bin/bash

assigneeId=$(az ad user list --filter "displayName eq 'Michael Collier'" --query []."objectId" -o tsv)
roleId=$(az role definition list --name "Storage Queue Data Contributor" --query []."name" -o tsv)
storageAccountResourceId=$(az storage account show --name blogazfuncmanagedid -g blog-azure-func-managed-identity --query "id" -o tsv)
az role assignment create --assignee $assigneeId --role $roleId --scope $storageAccountResourceId

Azure portal - view assigned Azure Storage RBAC roles

Once my local identity is in the right Azure AD role, and thus has the correct permissions to work with the Azure Storage queue, my local function is able to process messages!!

Local function processing messages from Azure Storage queue

Macho Man - Oh Yeah

Deploy to Azure

Publishing my function to Azure is relatively straightforward with Visual Studio Code. Like I did for my personal identity, I also need to set the correct permissions for my Azure Function app. I need to add the function’s identity to the Storage Queue Data Contributor role

Azure Portal - set the function identity to the needed role

I use Azure Storage Explorer to add a few test messages to the Azure Storage queue to make sure the function is picking up the messages. I can use the Application Insights Logs to see my highly verbose trace statements. 😉

Application Insights log statements

Star Wars - It’s Working

Event Hubs

Now that I know out how to use an identity-based connection with an Azure Storage queue, I want to try doing the same with Event Hubs. Before I get started with the code, I’m going to need to create a new Event Hubs namespace and an event hub.

I’m going to start with the default Event Hub-triggered function generated by the Visual Studio Code template. Like with the Azure Storage extension, I need to update my project to use the new Event Hub extension.

dotnet add package Microsoft.Azure.WebJobs.Extensions.EventHubs --version 5.0.0-beta.2

The new Azure SDK uses a few different data types when working with Event Hubs. The sample GitHub code is very helpful in figuring out the needed changes. I need to make a few minor changes:

Instead of using the Microsoft.Azure.EventHubs namespace, use Azure.Messaging.EventHubs
Instead of eventData.Body.Array, use eventData.EventBody

No other changes were needed, and thus my sample code looks very similar to that generated by the Visual Studio Code template.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


[FunctionName("EventHubTriggerCSharp1")]
public static async Task Run(
    [EventHubTrigger("%EventHubName%", Connection = "MyEventHubConnection")] EventData[] events,
    ILogger log)
{
    var exceptions = new List<Exception>();

    foreach (EventData eventData in events)
    {
        try
        {
            string messageBody = Encoding.UTF8.GetString(eventData.EventBody);

            // Replace these two lines with your processing logic.
            log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
            await Task.Yield();
        }
        catch (Exception e)
        {
            // We need to keep processing the rest of the batch - capture this exception and continue.
            // Also, consider capturing details of the message that failed processing so it can be processed again later.
            exceptions.Add(e);
        }
    }

    // Once processing of the batch is complete, if any messages in the batch failed processing
    // throw an exception so that there is a record of the failure.

    if (exceptions.Count > 1)
        throw new AggregateException(exceptions);

    if (exceptions.Count == 1)
        throw exceptions.Single();
}

The GitHub Azure SDK page for the new extension also shows me the connection string details. For Event Hubs, the connection string suffix isn’t “serviceUri”, but instead is “fullyQualifiedNamespace”.

Guy with beard nodding head

Thus, my local.settings.json now appears as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
  "IsEncrypted": false,
  "Values": {
    "APPINSIGHTS_INSTRUMENTATIONKEY": "[APPLICATION-INSIGHTS-KEY]",
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "QueueName": "[AZURE-STORAGE-QUEUE-NAME]",
    "MyStorageConnection__serviceUri": "https://[AZURE-STORAGE-ACCOUNT-NAME].queue.core.windows.net/",
    "EventHubName":"[EVENT-HUB-NAME]",
    "MyEventHubConnection__fullyQualifiedNamespace":"[EVENT-HUB-NAMESPACE].servicebus.windows.net"
  }
}

Trying to run the function locally now results in the following error:

EventProcessorHost error (Action='Retrieving list of partition identifiers from a Consumer Client.', HostName='[REDACTED]', PartitionId='').
System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.
	at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response, String eventHubName)
	at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
EventProcessorHost error (Action='Retrieving list of partition identifiers from a Consumer Client.', HostName='[REDACTED]', PartitionId='').
	at Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.
	at Azure.Messaging.EventHubs.EventHubConnection.GetPartitionIdsAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.AmqpError.ThrowIfErrorResponse(AmqpMessage response, String eventHubName)
	at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.RunProcessingAsync(CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.Amqp.AmqpClient.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.EventHubConnection.GetPropertiesAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.EventHubConnection.GetPartitionIdsAsync(EventHubsRetryPolicy retryPolicy, CancellationToken cancellationToken)
	at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.RunProcessingAsync(CancellationToken cancellationToken)
EventProcessorHost error (Action='Retrieving list of partition identifiers from a Consumer Client.', HostName='[REDACTED]', PartitionId='').

Got it . . . I again don’t have the right permissions. I need to get my local identity into the Azure Event Hubs Data Receiver role. While I’m getting myself right, I might as well do the same for my Azure Function.

Azure Portal- setting RBAC assignment for Event Hub namespace

I can use Service Bus Explorer to add a few events to my newly created Event Hub. Run it again. No errors! Ship it!!

WreckIt Ralph - I’m Gonna Ship It

Important Considerations

Before fully adopting identity-based connections with Azure Functions, there a few important things to consider. First and foremost, the extensions are in a preview state. They may not work as expected (submit feedback). Functionality may still change (expect it).

No Consumption plan . . . yet

Identity-based connections are new, and thus there is not yet full support across all Azure Functions hosting options. In general, identity-based connections are not supported on the Azure Functions Consumption plan. As of this writing, support is for Azure Functions Premium plan and an App Service/Dedicated plan. It’d be good to keep an eye on the official docs if this should change (overall, or for specific extensions).

Not all extensions . . . yet

Not all Azure Functions extensions support identity-based connections. As of this writing, support is limited to Azure Storage (blobs and queues) and Event Hubs. I assume Service Bus will be coming soon, as Service Bus is in the same (generally speaking) family of messaging/eventing services.

Conclusion

I’m excited to see how the new extensions develop and to be able to use the extensions, along with managed identity, for Azure Functions triggers and bindings. Fewer secrets floating is good. Using identity-based authentication (via managed identity), combined with virtual network-based restrictions such as private endpoints, should provide for two avenues to control access to valuable resources.

So far, I’ve only experimented with using identity-based connections for Azure Storage queues and Event Hubs. I have not yet started doing the same with Azure Storage blobs. They’re next. As I keep working with Azure Functions and identity-based extensions, I’ll keep my samples in my GitHub repo.

Thor - Another