Azure Functions with Private Endpoints

As enterprises continue to adopt serverless (and Platform-as-a-Service, or PaaS) solutions, they often need a way to integrate with existing resources on a virtual network. These existing resources could be databases, file storage, message queues or event streams, or REST APIs. In doing so, those interactions need to take place within the virtual network. Until relatively recently, combining serverless/PaaS offerings with traditional network access restrictions was complex, if not nearly impossible.

I feel the story is changing. With the introduction of Azure Virtual Network service endpoints and private endpoints, it’s becoming easier for enterprises to realize the benefits of serverless, while also complying with necessary virtual network access controls.

If you would like to learn more about virtual networking with Azure Functions, please check out my previous post where I show how to restrict access to an Azure Function using private site access restrictions. A slightly modified version is also now available in the official Azure docs as a tutorial.

This post will detail how I am able to configure an Azure Function to work with Azure resources using private endpoints. By using private endpoints, I ensure that the resources are accessible only via my virtual network. The Azure Function app will communicate with designated resources using a resource-specific private IP address (e.g. 10.100.0/24 address space). This gives me an additional level of network-based security and control.

The sample shown in this post, and accompanying GitHub repository, discusses the following key concepts necessary to use private endpoints with Azure Functions:

• Azure Function with blob trigger and CosmosDB output binding
• Azure Function Premium plan with Virtual Network Integration enabled
• Virtual network
• Configuring private endpoints for Azure resources
  • Azure Storage private endpoints
  • Azure Cosmos DB private endpoint
• Using private Azure DNS zones

Additionally, the sample uses an Azure VM and Azure Bastion in order to access Azure resources within the virtual network. The VM and Azure Bastion setup is not discussed in this post. If you want to learn more about using Azure Bastion, please refer to https://docs.microsoft.com/azure/bastion/bastion-overview

Architecture Overview

The following diagram shows the high-level architecture of the solution to be created:

Deployment

In order to get started with this sample, you’ll need an Azure subscription. If you don’t have one, you can get a free Azure account at https://azure.microsoft.com/free/.

Prerequisites

• Azure CLI
• Azure Function Core Tools

Deploy the Resource Manager template

I’ve created an Azure Resource Manager (ARM) template to provision all the necessary Azure resources. The template will also create the application settings needed by the Azure Function sample code. The Azure CLI can be used to deploy the template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

resourceGroupName="functions-private-endpoints"
location="eastus"
now=`date +%Y%m%d-%H%M%S`
deploymentName="azuredeploy-$now"

echo "Creating resource group '$resourceGroupName' in region '$location' . . ."
az group create --name $resourceGroupName --location $location

echo "Deploying main template . . ."
az deployment group create -g $resourceGroupName --template-file azuredeploy.json --parameters azuredeploy.parameters.json --name $deploymentName

Deploy the code

The function can be published manually by using the Azure Function Core Tools:

func azure functionapp publish <function-app-name>

Virtual Network

One of the first components to set up is the virtual network. Nearly all other Azure services in this sample are either provisioned into the virtual network, or integrated with, the virtual network. After all, this sample is about using private endpoints, and private endpoints go along with a virtual network (can’t have one without the other).

The sample uses four subnets:

1. Subnet for Azure Function virtual network integration. This subnet is delegated to the function.
2. Subnet for private endpoints. Private IP addresses are allocated from this subnet.
3. Subnet for the virtual machine. The template creates a VM which is placed within this subnet.
4. Subnet for the Azure Bastion host.

Virtual Network (VNet) Integration

In order for the function to access resources within the virtual network, VNet Integration is needed. The matrix of Azure Functions networking features shows that there are currently three options for VNet Integration:

1. Azure Functions Premium plan
2. Azure App Service Plan
3. Azure App Service Environment

As I like the purely serverless approach, I use an Azure Function Premium plan. By using an Azure Function Premium plan with VNet Integration enabled, the function is able to access Azure Storage and CosmosDB via the configured private endpoints.

Please refer to the official documentation for more information on using Azure Functions with virtual network integration.

Azure Function

The function used in this sample is based on a simplified concept of processing census data. Many of the Azure resources used in this sample will be related to census data. At a high level, the function logic is as follows:

1. Trigger on a new CSV-formatted file being available in a specific Azure Storage blob container
2. Convert the CSV file to JSON
3. Save the JSON document to CosmosDB via an output binding

The function is invoked via an Azure Storage blob trigger. The storage account used by the blob trigger is configured with a private endpoint.

The function assumes the file is in a CSV format, and then converts the CSV content to JSON. The resulting JSON document is saved to an Azure CosmosDB collection via an output binding.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

[FunctionName("CensusDataFilesFunction")]
public static async Task ProcessCensusDataFiles(
    [BlobTrigger("%ContainerName%/{blobName}", Connection = "CensusResultsAzureStorageConnection")] Stream myBlobStream,
    string blobName,
    [CosmosDB(
        databaseName: "%CosmosDbName%",
        collectionName: "%CosmosDbCollectionName%",
        ConnectionStringSetting = "CosmosDBConnection")] IAsyncCollector<string> items,
    ILogger logger)
{
    logger.LogInformation($"C# Blob trigger function processed blob of name '{blobName}' with size of {myBlobStream.Length} bytes");

    var jsonObject = await ConvertCsvToJsonAsync(myBlobStream);

    foreach (var item in jsonObject)
    {
        await items.AddAsync(JsonConvert.SerializeObject(item));
    }
}

Azure Function Configuration

There are a few important details about the configuration of the function.

Run from Package

The function is configured to run from a deployment package. As such, the package is persisted in an Azure File share referenced by the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING application setting. Please review the section below on Azure Storage Private Endpoints for why this is important in this scenario.

Virtual Network Triggers

Virtual network trigger support must be enabled in order for the function to trigger against resources using a private endpoint. Virtual network trigger support can be enabled via the Azure portal, the Azure CLI, or via an ARM template (as done in this sample).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
    "type": "config",
    "name": "web",
    "apiVersion": "2019-08-01",
    "dependsOn": [
        "[resourceId('Microsoft.Web/sites', variables('functionAppName'))]",
        "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]"
    ],
    "properties": {
        "functionsRuntimeScaleMonitoringEnabled": true
    }
}

View the full ARM template here.

Azure DNS Private Zones

When using VNet Integration, the function app uses the same DNS server that is configured for the virtual network. To work with a private endpoint, the default configuration needs to be overridden. In order to make calls to a resource using a private endpoint, it is necessary to integrate with Azure DNS Private Zones.

Private endpoints automatically create Azure DNS Private Zones. The Azure DNS Private Zone contains the details on how to route requests to the private IP address for the designated Azure service. Therefore, it is necessary to configure the app to use a specific Azure DNS server, and also route all network traffic into the virtual network. This is accomplished by setting the following application settings:

Azure Storage Private Endpoints

Azure Functions requires an Azure Storage account for persisting runtime metadata and metadata related to various triggers. The official Microsoft documentation indicates that it is currently not possible to use Azure Functions with a storage account which uses virtual network restrictions. While that’s mostly true, there is a workaround.

The workaround being that it is possible to put virtual network restrictions on the Azure storage account referenced via the AzureWebJobsStorage application setting. However, if that is done, then a separate storage account - one without network restrictions - is needed.

The other (without network restrictions) storage account needs to be referenced via the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING application setting. It is this storage account that will contain an Azure File share used to persist the function’s application code.

I expect the need to use two separate storage accounts, one with vnet restrictions and one without, will change relatively soon. It was mentioned during the April 2020 Azure Functions Live webcast that the team is working to remove this restriction. Once that is done, you’ll be able to keep everything confined to the virtual network.

Furthermore, for this sample, a third storage account is used. This third storage account is used by the sample application code - it’s where the CSV blob file will be placed. The Function blob trigger will pick up this file and the function will do work against it. This storage account will also use a private endpoint.

The sample will use three Azure storage related application settings:

When using private endpoints for Azure Storage, it is necessary to create a private endpoint for each Azure Storage service (table, blob, queue, or file). Therefore, this samples sets up 5 private endpoints related to Azure Storage.

• Four private endpoints related to each of the services referenced by the AzureWebJobsStorage application setting.
• One private endpoint for blob storage referenced by the CensusResultsAzureStorageConnection application setting. This is the only private endpoint related to the CensusResultsAzureStorageConnection application setting.

Azure Cosmos DB Private Endpoints

As mentioned previously, a CosmosDB output binding is used to save data to a CosmosDB collection. A CosmosDB private endpoint is created, and the function communicates with CosmosDB via the private endpoint.

CosmosDB supports different API (Sql, Cassandra, Mongo, Table, etc.) types, and a private endpoint is needed for each. Meaning, there is a private endpoint for the SQL protocol, and another private endpoint for the Mongo protocol, etc. Since I’m using the Sql API type in this sample, it is only necessary to configure a private endpoint for the Sql API.

For this sample, I set up everything with an ARM template. It is important to note the value for the groupIds in the ARM template below is case sensitive. In most cases, ARM templates are not case sensitive. In this case, since I’m using the Sql API, the value must be “Sql”. If it is “sql”, you’ll receive an “InternalServerError” error that leaves you guessing as to what went wrong. I learned this the hard way.

Don’t be like Mike - use “Sql”. However, if you want to see this changes, so that both “Sql” and “sql” are valid, or that a descriptive error messages is returned, please vote up the Azure Feedback I’ve submitted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
    "type": "Microsoft.Network/privateEndpoints",
    "name": "[variables('privateEndpointCosmosDbName')]",
    "apiVersion": "2019-11-01",
    "location": "[parameters('location')]",
    "dependsOn": [
        "[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('privateCosmosDbAccountName'))]",
        "[resourceId('Microsoft.Network/virtualNetworks', variables('vnetName'))]"
    ],
    "properties": {
        "subnet": {
            "id": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('vnetName'), variables('privateEndpointSubnetName') )]"
        },
        "privateLinkServiceConnections": [
            {
                "name": "MyCosmosDbPrivateLinkConnection",
                "properties": {
                    "privateLinkServiceId": "[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('privateCosmosDbAccountName'))]",
                    "groupIds": [
                        "Sql"
                    ]
                }
            }
        ]
    }
}

Private Azure DNS Zones

When working with private endpoints, it is necessary to make changes your DNS configuration. You can either use a host file on a VM within the virtual network, a private DNS zone, or your own DNS server hosted within the virtual network. I decided to use private DNS zones (because I want to manage as little infrastructure as possible).

More information on Private Endpoint DNS configuration can be found in the official documentation.

Azure services have DNS configuration to know how to connect to other Azure services over a public endpoint. However, when using a private endpoint, the connection isn’t made over the public endpoint. It’s made using a private IP address allocated specifically for that Azure resource. So, the default DNS configuration will need to be overridden.

One of the nice things about working with private endpoints is that the connection string used by the calling service doesn’t need to change. In other words, I can use contoso.blob.core.windows.net to connect to either the public endpoint or the private endpoint (for blob storage in the Contoso storage account).

This is made possible by using private DNS zones. The private endpoint creates an alias in a subdomain prefixed with “privatelink”. For example, blobs in an Azure Storage account may have a public DNS name of contoso.blob.core.windows.net. A private DNS zone is created which corresponds to contoso.privatelink.blob.core.windows.net. A DNS A record is created for each private IP address associated with the private endpoint. Clients within the virtual network resolve the connection to the storage account as follows:

Clients external to the virtual network continue to resolve to the public IP address of the service.

The Zones

This sample uses private endpoints for Azure Storage and CosmosDB. As such, private DNS zones are needed for each Azure storage service, as well as the Sql API for CosmosDB. Meaning, five DNS zones are needed to support this sample:

1. privatelink.queue.core.windows.net
2. privatelink.blob.core.windows.net
3. privatelink.table.core.windows.net
4. privatelink.file.core.windows.net
5. privatelink.documents.azure.com

When creating the zones, I use recommended zone names.

Setting it up

For me, setting up the Azure Private DNS Zones was one of the more challenging parts of this exercise. Fortunately, there is new API available that makes this process much easier! The privateDnsZoneGroups sub-type handles configuring the DNS zone, obtaining the private IP address for the configured service, and setting up the corresponding DNS A record.

Below is a snippet from the ARM template which shows using the Microsoft.Network/privateEndpoints/privateDnsZoneGroups type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

{
    "type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups",
    "apiVersion": "2020-03-01",
    "location": "[parameters('location')]",
    "name": "[concat(variables('privateEndpointCosmosDbName'), '/default')]",
    "dependsOn": [
        "[resourceId('Microsoft.Network/privateDnsZones', variables('privateCosmosDbDnsZoneName'))]",
        "[resourceId('Microsoft.Network/privateEndpoints', variables('privateEndpointCosmosDbName'))]"
    ],
    "properties": {
        "privateDnsZoneConfigs": [
            {
                "name": "config1",
                "properties": {
                    "privateDnsZoneId": "[resourceId('Microsoft.Network/privateDnsZones', variables('privateCosmosDbDnsZoneName'))]"
                }
            }
        ]
    }
}

Summary

This post outline the key components that are necessary to use private endpoints with Azure Functions. Private endpoints allow for interaction with designated Azure resources via private IP address, thus keeping network traffic between the function and the resource confined to the virtual network. As enterprises continue to adopt serverless, I’m currently of the opinion that using private endpoints will become increasingly common.

Using private endpoints with Azure Functions requires there to be a virtual network (with a few subnets), an Azure Functions Premium plan with VNet Integration enabled, Azure resources to connect to which support private endpoints, and modifications to DNS configuration. There is quite a bit of setup to get right to make it all work. I hope that this post, along with the sample provided in my GitHub repository, make it easier for someone else to get started.

What’s Next

I hope to spend some time to enhance this sample. My initial idea list includes:

• Moving secrets to Azure Key Vault, and configuring Key Vault with a private endpoint
• Creating a Terraform script to complement the existing ARM template

What would you like to see? Please let me know in the comments below or file an issue on the GitHub repo.

Resources

• GitHub repository