Troubleshooting Key Vault References in Azure App Service

Key Vault references are a great way to secure your configuration values when using Azure app service. Unfortunately, it doesn't always work perfectly first time round.

To try and help with the troubleshooting process, here are some common errors to check for, broken down into different sections.

Remember to also check the error message for the reference in the portal as they are generally quite descriptive of what the issue is.

Syntax

  • Check that the syntax follows exactly as described in the documentation
  • Have you got semi-colons in the correct place?
  • Is the name of the key vault spelt correctly and does it exist?
  • Is the name of the secret spelt correctly and does it exist?
  • Does the secret version exist?

Private Networking

If you're using private networking there are multiple places that issues can occur with key vault references. The sections below highlight some key areas to check.

Troubleshooting commands

To help with troubleshooting, you can use the Kudu advanced tools on the app service.

App service Kudu blade

From here you can select Debug Console > CMD

Kudu CMD link

This provides a console where you can run commands to check connectivity from your app to your key vault's private endpoint.

Here are two useful commands that can help to pinpoint the problem:

nameresolver

nameresolver my-key-vault-name.vault.azure.net

If the output shows the private IP address of your key vault, then DNS is resolving correctly, and it's likely there is an issue with your app connecting to the key vault's private endpoint instead.

Kudu successful nameresolver output

Note you should also check which DNS server the app is using if it's not resolving the private IP correctly.

tcpping

tcpping my-key-vault-name.vault.azure.net:443

This tries to connect to your key vault on port 443 (HTTPS), which is the same way the key vault references work. Note the :443 at the end of the command - it won't work without this.

Kudu tcpping output

If this shows as successful, then the connectivity from your app service to your key vault is all good, and it's likely that your app service managed identity does not have the correct permissions over the key vault.

If this shows as unsuccessful, then it's possible that you need to allow traffic from your app service to your key vault. Alternatively, that connectivity could be fine, and it could be your DNS configuration that needs changing. If this does fail, then make sure that the nameresolver command above succeeds so that you can rule out any DNS issues.

Quick note that may catch some out as well: if this command succeeds, but you are still having issues, make sure that you haven't got public access enabled on your key vault (unless it needs to be).

If you do have public access enabled, then this command will succeed as it will be able to connect to the public URL, rather than your private one - again, check that the nameresolver command correctly returns the private IP address associated with your key vault's private endpoint.

Private Endpoint

  • Have you got a private endpoint for your key vault?
  • Is the private endpoint in an Approved state?

Key vault private endpoint

DNS

  • Do you have a DNS record for the private endpoint?

If using private DNS zones in Azure:

  • Is the private endpoint registered with an Azure private DNS zone?

Key vault private endpoint DNS config

If using DNS servers (on-prem or in the cloud):

  • Do you have a DNS record for the key vault in your DNS server?
  • Is the DNS record pointing to the privatelink.vaultcore.azure.net address?

App Service VNET Integration

  • Does your app service have VNET integration enabled?

App service VNET integration

  • Is the VNET your app is connected to peered to the VNET your key vault private endpoint exists in (if they are not the same VNET)?

App Service Network Security Group

  • Does the subnet your app is connected to have a network security group (NSG)?
  • Does the NSG for your app allow outbound traffic on port 443 to the IP address/ASG/VNET of the key vault's private endpoint?

App service NSG outbound rule to key vault

  • Does your app have connectivity to DNS to be able to resolve the key vault's private endpoint?

If using private DNS zones in Azure:

  • Is the VNET for your app service registered with the privatelink.vaultcore.azure.net private DNS zone containing your key vault's private endpoint?

App service VNET link to key vault private DNS zone

If using DNS servers (on-prem or in the cloud):

  • Are the DNS server IPs added to the VNET that the app service is connected to?
  • Does the NSG for your app allow traffic on port 53 to the DNS servers containing the DNS record for your key vault's private endpoint?

Key Vault Private Endpoint Network Security Group

  • Does the subnet your key vault private endpoint is connected to have a network security group (NSG)?
  • Does the NSG for your key vault private endpoint allow inbound traffic on port 443 from the app service subnet/VNET?

Key vault inbound NSG rule from app service

Firewall

  • Does your firewall allow inbound traffic on port 53 from the app service subnet/VNET to your DNS servers (if using custom DNS servers)?
  • Does your firewall allow traffic on port 443 from the app service subnet/VNET to your key vault private endpoint IP address/subnet/VNET?