09: Azure Key Vault Integration

09: Azure Key Vault Integration

Objective

• Understand why Azure Key Vault is preferred over Kubernetes Secrets for production workloads
• Learn how the Azure Key Vault Secrets Provider (CSI driver) works with AKS
• Create a SecretProviderClass to reference Key Vault secrets
• Mount Key Vault secrets as files in a Pod
• Verify secrets are accessible inside the container

Theory

Why Azure Key Vault?

Kubernetes Secrets have several limitations in production:

Azure Key Vault solves these problems by acting as an external, centralized secret store:

• Secrets are stored outside the cluster in a FIPS 140-2 compliant vault
• Fine-grained access policies and Azure RBAC control who can read or manage secrets
• Full audit logging via Azure Monitor and Microsoft Entra ID
• Built-in secret rotation and versioning
• Supports secrets, keys, and certificates

How It Works: Azure Key Vault Secrets Provider

The Azure Key Vault Provider for Secrets Store CSI Driver is an AKS add-on that makes Key Vault secrets available to Pods as mounted volumes.

flowchart LR
    KV["Azure Key Vault<br/>Stores secrets, keys, certs"]
    CSI["Secrets Store<br/>CSI Driver<br/>(runs on each node)"]
    SPC["SecretProviderClass<br/>(CRD in your namespace)"]
    POD["Pod<br/>mounts the volume"]
    FS["Container Filesystem<br/>/mnt/secrets/"]

    KV -->|"authenticated read"| CSI
    SPC -->|"defines which secrets"| CSI
    CSI -->|"provides volume"| POD
    POD -->|"volumeMount"| FS

Key Concepts

Authentication Flow

In production, the recommended authentication method is Workload Identity:

1. A Kubernetes ServiceAccount is annotated with an Azure Managed Identity client ID.
2. The Pod uses this ServiceAccount.
3. The CSI driver uses the identity to authenticate to Key Vault.
4. Key Vault checks its access policy and returns the secrets.

Instructor Setup (Reference Only)

Note: The following setup steps are performed by the instructor before the exercise. Participants do not need to run these commands. They are included here so you understand the full picture.

The instructor has:

1. Created an Azure Key Vault with secrets (db-password, api-key)
2. Enabled the AKS Key Vault Secrets Provider add-on (az aks enable-addons --addons azure-keyvault-secrets-provider)
3. Created a Managed Identity and configured it with Key Vault access policies
4. Created a Kubernetes ServiceAccount in each student namespace, linked to the Managed Identity via Workload Identity federation
5. Stored the Key Vault name, identity client ID, and tenant ID for participants to use in their SecretProviderClass manifests

These are cluster-level and Azure-level operations that require elevated permissions.

Practical Tasks

Task 1: Verify the CSI Driver Is Installed

The Secrets Store CSI Driver runs as a DaemonSet in kube-system. Check that it is running:

# Check CSI driver and Azure Key Vault provider pods
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)'

Expected output:

NAME                                     READY   STATUS    RESTARTS   AGE
aks-secrets-store-csi-driver-xxxxx       3/3     Running   0          ...
aks-secrets-store-csi-driver-yyyyy       3/3     Running   0          ...
aks-secrets-store-provider-azure-zzzzz   1/1     Running   0          ...
aks-secrets-store-provider-azure-wwwww   1/1     Running   0          ...

You should see one CSI driver pod and one provider pod per node.

Verify the SecretProviderClass CRD is available:

kubectl api-resources | grep SecretProviderClass

Expected output:

secretproviderclasses   ...   secrets-store.csi.x-k8s.io/v1   true    SecretProviderClass

If the CSI driver pods are not running or the CRD does not exist, inform the instructor — the AKS add-on may not be enabled.

Task 2: Create a SecretProviderClass

A SecretProviderClass defines which Key Vault to connect to and which secrets to retrieve.

Note: Replace <KEY_VAULT_NAME> with the Key Vault name provided by the instructor. Replace <IDENTITY_CLIENT_ID> and <TENANT_ID> with the values provided by the instructor.

Create the file secret-provider-class.yaml:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-secrets
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "<IDENTITY_CLIENT_ID>"
    keyvaultName: "<KEY_VAULT_NAME>"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
        - |
          objectName: api-key
          objectType: secret

kubectl apply -f secret-provider-class.yaml

# Verify it was created
kubectl get secretproviderclass
kubectl describe secretproviderclass azure-kv-secrets

The objects array lists the Key Vault secret names to fetch. Each objectName must match a secret that exists in the Key Vault.

Task 3: Create a Pod That Mounts Key Vault Secrets

Create the file pod-keyvault.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-keyvault
spec:
  serviceAccountName: <SERVICE_ACCOUNT_NAME>
  containers:
    - name: app
      image: busybox
      command: ["sh", "-c", "echo 'Secrets mounted:' && ls /mnt/secrets/ && echo '---' && cat /mnt/secrets/db-password && echo && cat /mnt/secrets/api-key && sleep 3600"]
      volumeMounts:
        - name: secrets-store
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets-store
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: azure-kv-secrets
  restartPolicy: Never

Note: Replace <SERVICE_ACCOUNT_NAME> with the ServiceAccount name provided by the instructor. This ServiceAccount is linked to the Azure Managed Identity that has access to Key Vault.

kubectl apply -f pod-keyvault.yaml

# Wait for the Pod to start
kubectl get pod pod-keyvault -w

If the Pod is stuck in ContainerCreating, check events:

kubectl describe pod pod-keyvault

Task 4: Verify Secrets Are Available

Once the Pod is running, verify the secrets are mounted:

# List the mounted secret files
kubectl exec pod-keyvault -- ls /mnt/secrets/

# Read the secret values
kubectl exec pod-keyvault -- cat /mnt/secrets/db-password
kubectl exec pod-keyvault -- cat /mnt/secrets/api-key

# Check the logs
kubectl logs pod-keyvault

Expected output:

Secrets mounted:
api-key
db-password
---
<value-of-db-password-from-keyvault>
<value-of-api-key-from-keyvault>

Each Key Vault secret becomes a file in the mounted directory. The filename matches the objectName from the SecretProviderClass.

Optional: Sync as Kubernetes Secret

The CSI driver can also sync Key Vault secrets as Kubernetes Secrets, making them available as environment variables. This is configured in the SecretProviderClass with secretObjects:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kv-secrets-sync
spec:
  provider: azure
  secretObjects:
    - secretName: db-secrets-k8s
      type: Opaque
      data:
        - objectName: db-password
          key: DB_PASSWORD
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "<IDENTITY_CLIENT_ID>"
    keyvaultName: "<KEY_VAULT_NAME>"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret

Note: The synced Kubernetes Secret is only created when a Pod mounts the SecretProviderClass volume. It does not exist before a Pod references it.

Task 5: Challenge — Troubleshoot a Broken SecretProviderClass

Challenge: The following SecretProviderClass has three deliberate errors. Can you find them all without applying the manifest?

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: broken-spc
spec:
  provider: azurekeyvault
  parameters:
    usePodIdentity: "true"
    clientID: "<IDENTITY_CLIENT_ID>"
    keyvaultName: "<KEY_VAULT_NAME>"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: my-secret
          objectType: secrets

Clean up:

kubectl delete pod pod-keyvault
kubectl delete secretproviderclass azure-kv-secrets --ignore-not-found

Common Problems

Best Practices

1. Use Key Vault for production secrets — Keep secrets outside the cluster in a centrally managed, audited store.
2. Use Workload Identity — This is the recommended authentication method for AKS workloads accessing Azure resources. Avoid pod-managed identity (deprecated).
3. Set readOnly: true on CSI volume mounts — Just like with regular Secret volumes.
4. Use separate Key Vaults per environment — Do not share a Key Vault between dev and production.
5. Enable Key Vault audit logging — Monitor who accesses secrets and when.
6. Use secret rotation — The CSI driver supports polling for updated secrets. Configure rotationPollInterval in the driver settings.
7. Test access before deploying — Verify the identity can read secrets using az keyvault secret show before creating Pods.

Summary

In this exercise you learned how to:

• Understand why Azure Key Vault is preferred over native Kubernetes Secrets for production
• Verify the Secrets Store CSI Driver is installed on the AKS cluster
• Create a SecretProviderClass that references specific Key Vault secrets
• Mount Key Vault secrets as files in a Pod using the CSI volume driver
• Troubleshoot common issues with identity permissions and secret name mismatches

This concludes the Configuration module. Proceed to Exercise 10 to learn about resource requests and limits.