08: Secrets

08: Secrets

Objective

• Understand the difference between Secrets and ConfigMaps
• Learn that base64 is encoding, not encryption
• Create Opaque and TLS Secrets using YAML and CLI
• Use Secrets as environment variables and volume mounts
• Generate a self-signed TLS certificate and store it as a Kubernetes Secret
• Combine multiple Secrets in a single Pod

Theory

Secrets vs ConfigMaps

Base64: Encoding, Not Encryption

Kubernetes Secrets store values as base64. This is not encryption — anyone with access can decode it:

# Encode
echo -n "my-password" | base64
# Output: bXktcGFzc3dvcmQ=

# Decode
echo "bXktcGFzc3dvcmQ=" | base64 -d
# Output: my-password

Base64 exists only because Secret values can contain binary data (TLS certificates, SSH keys) that cannot be stored as plain text in YAML.

Real Security for Secrets

Base64 alone provides no security. Real protection comes from:

1. RBAC — Restrict who can get, list, or watch Secrets in a namespace.
2. etcd encryption at rest — AKS supports encrypting etcd data with customer-managed keys (Azure Key Vault).
3. External secret stores — Azure Key Vault with the CSI driver (Exercise 09) keeps secrets outside the cluster entirely.
4. Audit logging — Track who accesses Secrets via Kubernetes audit logs.

Secret Types

Practical Tasks

Task 1: Create an Opaque Secret — YAML and CLI

Option A: Create from YAML

Secret values in YAML must be base64-encoded. Encode the values first:

echo -n "admin" | base64
# YWRtaW4=
echo -n "S3cr3tP@ss!" | base64
# UzNjcjN0UEBzcyE=

Create the file secret-db.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: student-XX
type: Opaque
data:
  username: YWRtaW4=
  password: UzNjcjN0UEBzcyE=

kubectl apply -f secret-db.yaml

Alternatively, use stringData to avoid manual base64 encoding:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials-plain
  namespace: student-XX
type: Opaque
stringData:
  username: admin
  password: "S3cr3tP@ss!"

Kubernetes will base64-encode the values for you when storing.

Option B: Create from CLI

kubectl create secret generic api-secret \
  --from-literal=API_KEY=abc123xyz \
  --from-literal=API_SECRET=super-secret-value \
  -n student-XX

Inspect the Secrets:

# List Secrets
kubectl get secrets -n student-XX

# View Secret details (values are base64-encoded)
kubectl get secret db-credentials -o yaml -n student-XX

# Decode a specific value
kubectl get secret db-credentials -n student-XX -o jsonpath='{.data.password}' | base64 -d

Clean up the extra Secrets:

kubectl delete secret db-credentials-plain api-secret --ignore-not-found -n student-XX

Task 2: Use Secret as Environment Variables with kuard

Create a Pod using the kuard application, which has a web UI where you can inspect environment variables.

Create the file pod-secret-env.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-secret-env
  namespace: student-XX
spec:
  containers:
    - name: kuard
      image: <ACR_NAME>.azurecr.io/kuard:1
      ports:
        - containerPort: 8080
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
  restartPolicy: Never

Apply and verify:

kubectl apply -f pod-secret-env.yaml

# Wait for the Pod to be running
kubectl get pod pod-secret-env -n student-XX -w

# Verify env vars from the command line
kubectl exec pod-secret-env -n student-XX -- env | grep DB_

# Port-forward to see the kuard UI
kubectl port-forward pod-secret-env 8080:8080 -n student-XX

Open http://localhost:8080 in your browser. Navigate to Server Env to see the DB_USERNAME and DB_PASSWORD variables.

Note: This demonstrates why Secrets as env vars can be risky — they are visible in the process environment and in UIs like kuard. Volume mounts (Task 3) are often preferred.

Clean up:

kubectl delete pod pod-secret-env -n student-XX

Task 3: Mount Secret as a Volume with kuard

Mounting a Secret as a volume creates files in the container filesystem. Each key becomes a file.

Create the file pod-secret-volume.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-secret-volume
  namespace: student-XX
spec:
  containers:
    - name: kuard
      image: <ACR_NAME>.azurecr.io/kuard:1
      ports:
        - containerPort: 8080
      volumeMounts:
        - name: secret-volume
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secret-volume
      secret:
        secretName: db-credentials
  restartPolicy: Never

Apply and verify:

kubectl apply -f pod-secret-volume.yaml

# List files in the secret mount
kubectl exec pod-secret-volume -n student-XX -- ls -la /etc/secrets/

# Read the secret files
kubectl exec pod-secret-volume -n student-XX -- cat /etc/secrets/username
kubectl exec pod-secret-volume -n student-XX -- cat /etc/secrets/password

# Port-forward to see in kuard UI
kubectl port-forward pod-secret-volume 8080:8080 -n student-XX

In the kuard UI, navigate to File system browser and browse to /etc/secrets/ to see the mounted files.

Using items for Custom Paths

You can control which keys are mounted and under what filenames:

Create the file pod-secret-items.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-secret-items
  namespace: student-XX
spec:
  containers:
    - name: app
      image: busybox
      command: ["sh", "-c", "cat /etc/db/db-user && echo && cat /etc/db/db-pass && sleep 3600"]
      volumeMounts:
        - name: secret-volume
          mountPath: /etc/db
          readOnly: true
  volumes:
    - name: secret-volume
      secret:
        secretName: db-credentials
        items:
          - key: username
            path: db-user
          - key: password
            path: db-pass
  restartPolicy: Never

kubectl apply -f pod-secret-items.yaml
kubectl logs pod-secret-items -n student-XX

# Verify the custom file names
kubectl exec pod-secret-items -n student-XX -- ls /etc/db/

Expected output:

admin
S3cr3tP@ss!

The items field gives you full control over file naming and which keys to include.

Clean up:

kubectl delete pod pod-secret-volume pod-secret-items -n student-XX

Task 4: TLS Secret

Kubernetes has a dedicated Secret type for TLS certificates. This task walks through generating a self-signed certificate and storing it.

Step 1: Generate a Self-Signed Certificate

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key \
  -out tls.crt \
  -subj "/CN=my-app.example.com/O=Training"

This creates two files:

• tls.crt — the certificate
• tls.key — the private key

Step 2: Create the TLS Secret

kubectl create secret tls my-tls-secret \
  --cert=tls.crt \
  --key=tls.key \
  -n student-XX

Inspect the Secret:

kubectl get secret my-tls-secret -o yaml -n student-XX

Notice:

• The type is kubernetes.io/tls
• It has two keys: tls.crt and tls.key

Step 3: Mount the TLS Secret in a Pod

Create the file pod-tls.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-tls
  namespace: student-XX
spec:
  containers:
    - name: app
      image: busybox
      command: ["sh", "-c", "ls -la /etc/tls/ && echo '--- Certificate ---' && head -3 /etc/tls/tls.crt && sleep 3600"]
      volumeMounts:
        - name: tls-volume
          mountPath: /etc/tls
          readOnly: true
  volumes:
    - name: tls-volume
      secret:
        secretName: my-tls-secret
  restartPolicy: Never

kubectl apply -f pod-tls.yaml
kubectl logs pod-tls -n student-XX

Expected output:

total 0
lrwxrwxrwx    1 root     root            14 ... tls.crt -> ..data/tls.crt
lrwxrwxrwx    1 root     root            14 ... tls.key -> ..data/tls.key
--- Certificate ---
-----BEGIN CERTIFICATE-----
MIIDazCCAlOgAwIBAgIUQ...
...

Clean up:

kubectl delete pod pod-tls -n student-XX
kubectl delete secret my-tls-secret -n student-XX
rm -f tls.crt tls.key

Task 5: Pod with Multiple Secrets (Env + Volume Combined)

In a real application, you often need both environment-variable Secrets and file-based Secrets. This task combines both approaches.

Create the Secrets:

kubectl create secret generic app-credentials \
  --from-literal=DB_USER=admin \
  --from-literal=DB_PASS=secret123 \
  -n student-XX

kubectl create secret generic app-cert \
  --from-literal=ca.crt="-----BEGIN CERTIFICATE-----
MIIBkTCB+wIJALEBo...fake...cert
-----END CERTIFICATE-----" \
  -n student-XX

Create the file pod-multi-secrets.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pod-multi-secrets
  namespace: student-XX
spec:
  containers:
    - name: app
      image: busybox
      command:
        - sh
        - -c
        - |
          echo "DB_USER=$DB_USER"
          echo "DB_PASS=$DB_PASS"
          echo "--- Certificate file ---"
          cat /etc/certs/ca.crt
          sleep 3600
      env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: DB_USER
        - name: DB_PASS
          valueFrom:
            secretKeyRef:
              name: app-credentials
              key: DB_PASS
      volumeMounts:
        - name: cert-volume
          mountPath: /etc/certs
          readOnly: true
  volumes:
    - name: cert-volume
      secret:
        secretName: app-cert
  restartPolicy: Never

kubectl apply -f pod-multi-secrets.yaml
kubectl logs pod-multi-secrets -n student-XX

Expected output:

DB_USER=admin
DB_PASS=secret123
--- Certificate file ---
-----BEGIN CERTIFICATE-----
MIIBkTCB+wIJALEBo...fake...cert
-----END CERTIFICATE-----

Clean up:

kubectl delete pod pod-multi-secrets -n student-XX
kubectl delete secret app-credentials app-cert db-credentials -n student-XX

Common Problems

Best Practices

1. Always use readOnly: true on Secret volume mounts — prevents accidental modification from inside the container.
2. Prefer stringData in YAML — Avoids manual base64 encoding errors. Kubernetes converts it to data automatically.
3. Use --dry-run=client -o yaml to generate Secret YAML — just like with ConfigMaps.
4. Do not commit Secrets to Git — Use sealed-secrets, external-secrets, or Azure Key Vault instead.
5. Restrict RBAC access — Not every ServiceAccount or user should be able to read Secrets. Limit with Roles and RoleBindings.
6. Prefer volume mounts over env vars for sensitive data — Env vars can leak through process listings, crash dumps, and logging. Files are easier to control access to.
7. Consider Azure Key Vault (Exercise 09) for production secrets — Keeps secrets out of etcd entirely.

Summary

In this exercise you learned how to:

• Create Opaque Secrets using YAML (data and stringData) and CLI (--from-literal)
• Inject Secrets as environment variables with secretKeyRef
• Mount Secrets as files using volumes with readOnly: true and custom paths via items
• Generate self-signed TLS certificates and store them as kubernetes.io/tls Secrets
• Combine multiple Secrets (env + volume) in a single Pod

Proceed to Exercise 09 to learn about Azure Key Vault integration for production-grade secret management.