05: Pod Operations

05: Pod Operations

Objective

Master day-to-day Pod operations: debugging with ephemeral containers, executing commands inside containers, copying files, and working with logs. Get introduced to AKS-specific monitoring and identity features.

Theory

Debugging Running Pods

In production, you often need to troubleshoot Pods without restarting them. Kubernetes provides several tools:

Ephemeral Debug Containers

Sometimes a container image is minimal (e.g., distroless) and does not include debugging tools like sh, curl, or nslookup. In this case, you can attach an ephemeral container to the Pod. This container:

• Is added to an already running Pod without restarting it
• Shares the same namespaces (network, PID) as the target container
• Is temporary and cannot be removed once added (it stays until the Pod is deleted)
• Does not appear in spec.containers — it is listed under spec.ephemeralContainers

Container Insights and Workload Identity (AKS)

Azure Kubernetes Service provides additional operational capabilities:

• Container Insights Live Logs — Stream logs from Pods directly in the Azure Portal, without needing kubectl access
• Workload Identity — Allows Pods to authenticate to Azure services (Key Vault, Storage, etc.) using a Microsoft Entra ID identity, replacing the deprecated Pod Identity v1

These are covered conceptually at the end of this exercise.

Prerequisites

Before starting, deploy a Pod to work with:

# debug-target.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kuard-debug
  namespace: student-XX       # Replace XX with your student number
  labels:
    app: kuard-debug
    team: teamXX
spec:
  containers:
    - name: kuard
      image: <ACR_NAME>.azurecr.io/kuard:1
      ports:
        - containerPort: 8080
          name: http

Note: Replace <ACR_NAME> with the actual ACR name and XX with your student number.

kubectl apply -f debug-target.yaml
kubectl wait --for=condition=Ready pod/kuard-debug -n student-XX --timeout=60s

Practical Task 1: Ephemeral Debug Containers

Scenario

The kuard container may not have all the networking tools you need for debugging. Use kubectl debug to attach a busybox container that shares the Pod’s network namespace.

Step 1: Attach an Ephemeral Debug Container

kubectl debug -it kuard-debug \
  --image=busybox:1.36 \
  --target=kuard \
  -n student-XX

Flags explained:

Step 2: Explore from the Debug Container

Once inside the debug container shell, run:

# Check the network — you share the Pod's IP
ip addr

# Verify you can reach the kuard process on localhost
wget -qO- http://localhost:8080

# Check DNS resolution
nslookup kubernetes.default.svc.cluster.local

# List processes (visible because of --target flag sharing PID namespace)
ps aux

# Exit the debug container
exit

Step 3: Verify the Ephemeral Container Exists

kubectl describe pod kuard-debug -n student-XX

Look for the Ephemeral Containers section in the output. The debug container will be listed there even after you exit.

Practical Task 2: Execute Commands in a Container

Run a Single Command

kubectl exec kuard-debug -n student-XX -- env

This prints the environment variables inside the container.

Run Multiple Commands

kubectl exec kuard-debug -n student-XX -- /bin/sh -c "hostname && cat /etc/os-release"

Interactive Shell

kubectl exec -it kuard-debug -n student-XX -- /bin/sh

Once inside:

# Check what user the process runs as
whoami

# Look at the filesystem
ls -la /

# Check mounted volumes
mount | head -10

# Check network configuration
cat /etc/resolv.conf

# Exit
exit

Exec into a Specific Container (Multi-Container Pods)

If the Pod has multiple containers, use the -c flag:

kubectl exec -it <pod-name> -c <container-name> -n student-XX -- /bin/sh

Practical Task 3: Copy Files To and From Containers

Copy a File from the Container to Your Machine

kubectl cp student-XX/kuard-debug:/etc/hostname ./hostname-from-pod
cat ./hostname-from-pod

The format is <namespace>/<pod>:<path>.

Copy a File from Your Machine to the Container

Create a test file locally:

echo "Hello from the host machine" > test-file.txt

Copy it into the container:

kubectl cp test-file.txt student-XX/kuard-debug:/tmp/test-file.txt

Verify it arrived:

kubectl exec kuard-debug -n student-XX -- cat /tmp/test-file.txt

Important Notes on kubectl cp

• Requires tar to be available in the container image
• Not suitable for large files or production data transfers
• Files are copied to/from a specific container (use -c flag for multi-container Pods)
• The namespace is part of the Pod path, not a separate flag

Practical Task 4: Working with Logs

View Current Logs

kubectl logs kuard-debug -n student-XX

Stream Logs in Real Time

kubectl logs kuard-debug -n student-XX --follow

Press Ctrl+C to stop streaming. Generate some traffic first using port-forward in another terminal to see live log output.

View Logs from a Previous (Crashed) Container

If a container has restarted, you can view logs from the previous instance:

kubectl logs kuard-debug -n student-XX --previous

Note: This only works if the container has restarted at least once. You may see an error if there is no previous instance.

View Logs with Timestamps

kubectl logs kuard-debug -n student-XX --timestamps

View Only Recent Logs

Last 10 lines:

kubectl logs kuard-debug -n student-XX --tail=10

Logs from the last 5 minutes:

kubectl logs kuard-debug -n student-XX --since=5m

Logs from All Containers in a Pod

kubectl logs kuard-debug -n student-XX --all-containers

AKS-Specific: Container Insights Live Logs

Container Insights is an Azure Monitor feature that collects container logs and metrics from AKS clusters.

What It Provides

• Live Logs — Real-time log streaming from Pods, directly in the Azure Portal
• Log Analytics — Query historical logs using KQL (Kusto Query Language)
• Metrics — CPU, memory, network, and disk usage per Pod and node
• Workbooks — Pre-built dashboards for cluster health

How to Access Live Logs (Instructor Demo)

The instructor will demonstrate the following path in the Azure Portal:

1. Navigate to Azure Portal > your AKS cluster
2. Go to Monitoring > Insights
3. Select the Containers tab
4. Find your Pod and click on it
5. Click Live Logs to see real-time log output

Key benefit: Live Logs allows team members without kubectl access to view Pod logs through Azure RBAC, which is useful for developers who do not have direct cluster access.

AKS-Specific: Workload Identity (Conceptual)

Workload Identity allows Pods to authenticate to Azure services (Key Vault, Storage, SQL Database) using a Microsoft Entra ID identity — no stored credentials needed. It replaces the deprecated Pod Identity v1.

How it works: A Kubernetes ServiceAccount is linked to an Azure Managed Identity via OIDC federation. Pods using that ServiceAccount can request tokens automatically.

Note: Hands-on Workload Identity setup is covered in Exercise 09 (Key Vault integration). This is just a brief introduction to the concept.

Clean Up

kubectl delete pod kuard-debug -n student-XX
rm -f hostname-from-pod test-file.txt

Useful Commands Reference

Common Problems

Best Practices

1. Use ephemeral debug containers instead of SSH — Never install SSH in your containers. Use kubectl debug for ad-hoc troubleshooting.
2. Prefer kubectl logs over exec — Always check logs first before exec-ing into a container. Logs often reveal the issue without the need for interactive access.
3. Use --since and --tail for large log volumes — Avoid dumping all logs. Narrow down the time window.
4. Adopt Workload Identity for Azure access — Never store Azure credentials in Kubernetes Secrets if Workload Identity is available.
5. Use Container Insights for team-wide observability — Not everyone needs kubectl access. Container Insights provides log access through Azure Portal with proper RBAC.

Summary

In this exercise you learned:

• How to attach ephemeral debug containers to running Pods with kubectl debug
• How to execute commands inside containers with kubectl exec
• How to copy files between your machine and containers with kubectl cp
• Advanced log viewing techniques: streaming, previous instance, timestamps, time-based filtering
• Container Insights Live Logs in AKS for portal-based log viewing
• The Workload Identity concept for credential-free access to Azure services

Review Questions

1. When would you use kubectl debug instead of kubectl exec?
2. What does the --target flag do in kubectl debug?
3. What prerequisite does kubectl cp have in the container image?
4. How do you view logs from a container that has crashed and restarted?
5. What problem does Workload Identity solve compared to storing credentials in Kubernetes Secrets?