09: Azure Key Vault Integration

09: Azure Key Vault Integration

Objective

• Understand why Azure Key Vault is preferred over Kubernetes Secrets for production workloads
• Create an Azure Key Vault and add secrets
• Set up Workload Identity (Managed Identity + federation) to allow Pods to authenticate to Key Vault
• Create a SecretProviderClass and mount Key Vault secrets in a Pod

Theory

Why Azure Key Vault?

Kubernetes Secrets have several limitations in production:

Azure Key Vault solves these problems by acting as an external, centralized secret store:

• Secrets are stored outside the cluster in a FIPS 140-2 compliant vault
• Fine-grained access policies and Azure RBAC control who can read or manage secrets
• Full audit logging via Azure Monitor and Microsoft Entra ID
• Built-in secret rotation and versioning
• Supports secrets, keys, and certificates

How It Works: Azure Key Vault Secrets Provider

The Azure Key Vault Provider for Secrets Store CSI Driver is an AKS add-on that makes Key Vault secrets available to Pods as mounted volumes.

flowchart LR
    KV["Azure Key Vault<br/>Stores secrets, keys, certs"]
    CSI["Secrets Store<br/>CSI Driver<br/>(runs on each node)"]
    SPC["SecretProviderClass<br/>(CRD in your namespace)"]
    POD["Pod<br/>mounts the volume"]
    FS["Container Filesystem<br/>/mnt/secrets/"]

    KV -->|"authenticated read"| CSI
    SPC -->|"defines which secrets"| CSI
    CSI -->|"provides volume"| POD
    POD -->|"volumeMount"| FS

Authentication Flow: Workload Identity

Workload Identity allows a Pod to authenticate to Azure services without stored credentials:

1. You create an Azure Managed Identity
2. You grant that identity access to Key Vault
3. You create a Federated Identity Credential that links a Kubernetes ServiceAccount to the Managed Identity
4. You create a Kubernetes ServiceAccount annotated with the Managed Identity client ID
5. The Pod uses this ServiceAccount — the CSI driver authenticates to Key Vault automatically

flowchart LR
    SA["K8s ServiceAccount<br/>(annotated with client ID)"]
    FIC["Federated Identity<br/>Credential"]
    MI["Managed Identity"]
    KV["Key Vault"]
    Pod["Pod"]

    Pod -->|uses| SA
    SA -->|OIDC token| FIC
    FIC -->|validates| MI
    MI -->|authenticates| KV

Practical Tasks

Replace XX with your student number throughout (e.g., 01, 02, …).

Task 1: Verify the CSI Driver Is Installed

The Secrets Store CSI Driver runs as a DaemonSet in kube-system. Check that it is running:

# Check CSI driver and Azure Key Vault provider pods
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)'

You should see one CSI driver pod and one provider pod per node.

Verify the SecretProviderClass CRD is available:

kubectl api-resources | grep SecretProviderClass

If the CSI driver pods are not running or the CRD does not exist, inform the instructor — the AKS add-on may not be enabled.

Task 2: Create an Azure Key Vault

Create your own Key Vault. Key Vault names must be globally unique, so include your student number:

# Set variables (replace XX with your number)
export RESOURCE_GROUP="k8s-aks-training-rg"
export LOCATION="swedencentral"
export KV_NAME="kv-student-XX"

# Create the Key Vault
az keyvault create \
  --name $KV_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --enable-rbac-authorization true

Note: We use --enable-rbac-authorization true so that access is controlled via Azure RBAC roles (not the older access policies).

Now add some secrets:

# You need the Key Vault Secrets Officer role to set secrets
# First, get your own user object ID
MY_OBJECT_ID=$(az ad signed-in-user show --query id -o tsv)

# Assign yourself Key Vault Secrets Officer on the vault
az role assignment create \
  --role "Key Vault Secrets Officer" \
  --assignee-object-id $MY_OBJECT_ID \
  --assignee-principal-type User \
  --scope $(az keyvault show --name $KV_NAME --query id -o tsv)

# Wait a moment for the role to propagate, then add secrets
sleep 30

az keyvault secret set --vault-name $KV_NAME --name "db-password" --value "SuperSecret123!"
az keyvault secret set --vault-name $KV_NAME --name "api-key" --value "ak-training-key-2026"

Verify the secrets exist:

az keyvault secret list --vault-name $KV_NAME -o table

Task 3: Create a Managed Identity

Create an Azure Managed Identity that your Pod will use to authenticate to Key Vault:

export IDENTITY_NAME="id-student-XX"

az identity create \
  --name $IDENTITY_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION

# Save the client ID and principal ID — you will need them
export IDENTITY_CLIENT_ID=$(az identity show --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP --query clientId -o tsv)
export IDENTITY_PRINCIPAL_ID=$(az identity show --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP --query principalId -o tsv)

echo "Client ID: $IDENTITY_CLIENT_ID"

Now grant this identity read access to your Key Vault secrets:

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee-object-id $IDENTITY_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --scope $(az keyvault show --name $KV_NAME --query id -o tsv)

Task 4: Set Up Workload Identity Federation

Link the Managed Identity to a Kubernetes ServiceAccount using federated credentials.

First, get the AKS cluster’s OIDC issuer URL:

export CLUSTER_NAME="k8s-aks-training"

export OIDC_ISSUER=$(az aks show \
  --name $CLUSTER_NAME \
  --resource-group $RESOURCE_GROUP \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

echo "OIDC Issuer: $OIDC_ISSUER"

Create the federated credential. This tells Entra ID: “trust tokens from this Kubernetes ServiceAccount”:

export SERVICE_ACCOUNT_NAME="sa-keyvault-XX"
export NAMESPACE="student-XX"

az identity federated-credential create \
  --name "fed-student-XX" \
  --identity-name $IDENTITY_NAME \
  --resource-group $RESOURCE_GROUP \
  --issuer $OIDC_ISSUER \
  --subject "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}" \
  --audiences "api://AzureADTokenExchange"

What this does: When a Pod in namespace student-XX uses ServiceAccount sa-keyvault-XX, Kubernetes issues an OIDC token. Azure Entra ID trusts this token (because of the federated credential) and grants the Pod the Managed Identity’s permissions.

Task 5: Create the Kubernetes ServiceAccount

Create a ServiceAccount in your namespace, annotated with the Managed Identity client ID:

export TENANT_ID=$(az account show --query tenantId -o tsv)

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: $SERVICE_ACCOUNT_NAME
  namespace: $NAMESPACE
  annotations:
    azure.workload.identity/client-id: "$IDENTITY_CLIENT_ID"
  labels:
    azure.workload.identity/use: "true"
EOF

Verify:

kubectl get sa $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o yaml

You should see the azure.workload.identity/client-id annotation with your Managed Identity’s client ID.

Task 6: Create a SecretProviderClass

The SecretProviderClass tells the CSI driver which Key Vault to connect to and which secrets to fetch.

Create the file secret-provider-class.yaml:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-secrets
  namespace: student-XX
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "<IDENTITY_CLIENT_ID>"          # paste your $IDENTITY_CLIENT_ID
    keyvaultName: "kv-student-XX"
    tenantId: "<TENANT_ID>"                   # paste your $TENANT_ID
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
        - |
          objectName: api-key
          objectType: secret

Tip: Run echo $IDENTITY_CLIENT_ID and echo $TENANT_ID to get the values to paste.

kubectl apply -f secret-provider-class.yaml
kubectl get secretproviderclass -n student-XX

Task 7: Create a Pod That Mounts Key Vault Secrets

Create the file pod-keyvault.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-keyvault
  namespace: student-XX
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: sa-keyvault-XX
  containers:
    - name: app
      image: busybox
      command:
        - sh
        - -c
        - |
          echo "Secrets mounted:"
          ls /mnt/secrets/
          echo "---"
          echo "db-password: $(cat /mnt/secrets/db-password)"
          echo "api-key: $(cat /mnt/secrets/api-key)"
          sleep 3600
      volumeMounts:
        - name: secrets-store
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets-store
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: azure-kv-secrets
  restartPolicy: Never

Important: The label azure.workload.identity/use: "true" on the Pod is required — the Workload Identity webhook uses it to inject the token volume.

kubectl apply -f pod-keyvault.yaml

# Watch the Pod start
kubectl get pod pod-keyvault -n student-XX -w

If the Pod is stuck in ContainerCreating, check events:

kubectl describe pod pod-keyvault -n student-XX

Task 8: Verify Secrets Are Available

Once the Pod is running:

# Check the logs — secrets should be printed
kubectl logs pod-keyvault -n student-XX

# Or exec in and read them directly
kubectl exec pod-keyvault -n student-XX -- ls /mnt/secrets/
kubectl exec pod-keyvault -n student-XX -- cat /mnt/secrets/db-password
kubectl exec pod-keyvault -n student-XX -- cat /mnt/secrets/api-key

Expected output:

Secrets mounted:
api-key
db-password
---
db-password: SuperSecret123!
api-key: ak-training-key-2026

Each Key Vault secret becomes a file in the mounted directory. The filename matches the objectName from the SecretProviderClass.

Task 9: Challenge — Troubleshoot a Broken SecretProviderClass

Challenge: The following SecretProviderClass has three deliberate errors. Can you find them all without applying the manifest?

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: broken-spc
spec:
  provider: azurekeyvault
  parameters:
    usePodIdentity: "true"
    clientID: "<IDENTITY_CLIENT_ID>"
    keyvaultName: "<KEY_VAULT_NAME>"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: my-secret
          objectType: secrets

Clean Up

# Delete Kubernetes resources
kubectl delete pod pod-keyvault -n student-XX
kubectl delete secretproviderclass azure-kv-secrets -n student-XX --ignore-not-found
kubectl delete sa sa-keyvault-XX -n student-XX

# Delete Azure resources
az keyvault delete --name $KV_NAME --resource-group $RESOURCE_GROUP
az keyvault purge --name $KV_NAME --location $LOCATION 2>/dev/null
az identity delete --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP

Common Problems

Best Practices

1. Use Key Vault for production secrets — Keep secrets outside the cluster in a centrally managed, audited store.
2. Use Workload Identity — This is the recommended authentication method for AKS workloads accessing Azure resources. Avoid pod-managed identity (deprecated).
3. Set readOnly: true on CSI volume mounts — Prevent accidental writes.
4. Use separate Key Vaults per environment — Do not share a Key Vault between dev and production.
5. Enable Key Vault audit logging — Monitor who accesses secrets and when.
6. Use RBAC authorization on Key Vault — Prefer --enable-rbac-authorization over legacy access policies.
7. One identity per workload — Each application should have its own Managed Identity with minimal permissions.

Summary

In this exercise you learned how to:

• Create an Azure Key Vault and add secrets
• Create a Managed Identity and grant it Key Vault access via Azure RBAC
• Set up Workload Identity federation (OIDC issuer, federated credential, annotated ServiceAccount)
• Create a SecretProviderClass that references specific Key Vault secrets
• Mount Key Vault secrets as files in a Pod using the CSI volume driver
• Troubleshoot common issues with identity, federation, and permissions

This concludes the Configuration module. Proceed to Exercise 10 to learn about resource requests and limits.