29: Container Insights

29: Container Insights

Objective

Understand Azure Monitor Container Insights for AKS monitoring. Verify the monitoring agent is running, deploy a workload that generates logs, query logs using KQL, and explore Live Logs and node-level metrics in the Azure Portal.

Theory

What is Container Insights?

Azure Monitor Container Insights is a fully managed monitoring solution for AKS that collects metrics, logs, and inventory data from your cluster.

How It Works (Instructor Setup)

Note: Enabling Container Insights is a cluster-level operation performed by the instructor. The az aks commands below are included for educational reference only.

Container Insights is enabled when creating the AKS cluster:

# Instructor/admin command — do not run
az aks create \
  --resource-group <RG> \
  --name <CLUSTER> \
  --enable-addons monitoring \
  --workspace-resource-id <LOG_ANALYTICS_WORKSPACE_ID>

# Or enabled on an existing cluster:
az aks enable-addons --addon monitoring \
  --resource-group <RG> \
  --name <CLUSTER>

The monitoring agent runs as a DaemonSet on every node and:

• Collects stdout/stderr logs from all containers
• Collects Kubernetes metadata (Pod names, labels, namespaces)
• Collects node and container performance metrics (CPU, memory, network, disk)
• Sends everything to the Log Analytics workspace

Data Tables in Log Analytics

Practical Tasks

Task 1: Verify Monitoring Agent is Running

Check if the monitoring agent Pods are running on the cluster nodes.

For the legacy agent (omsagent):

kubectl get pods -n kube-system -l component=oms-agent

For the current agent (Azure Monitor Agent):

kubectl get pods -n kube-system | grep ama-

You should see one agent Pod per node (it runs as a DaemonSet). Check the DaemonSet directly:

kubectl get daemonsets -n kube-system | grep -E "oms|ama"

Task 2: Deploy a Pod and View Logs with kubectl (Participant Task)

Deploy a simple application that generates logs.

kubectl run log-demo-XX --image=gcr.io/kuar-demo/kuard-amd64:blue --port=8080 -n student-XX

Wait for the Pod to start, then generate some activity:

kubectl port-forward pod/log-demo-XX 8080:8080 -n student-XX &
curl http://localhost:8080

View the logs using kubectl:

kubectl logs log-demo-XX -n student-XX
kubectl logs log-demo-XX --follow -n student-XX

Task 2b: View Logs in Azure Portal (Instructor Demo)

Note: Azure Portal navigation and KQL queries are demonstrated by the instructor. Participants observe while the instructor shows how logs appear in Log Analytics.

The instructor will navigate in the Azure Portal:

1. Navigate to the AKS cluster in the Azure Portal
2. Go to Monitoring > Logs
3. Run the following KQL query:

ContainerLogV2
| where PodName startswith "log-demo"
| project TimeGenerated, PodName, LogMessage
| order by TimeGenerated desc
| take 50

Note: It may take 2-5 minutes for logs to appear in Log Analytics after they are generated.

Task 3: Live Logs (Instructor Demo)

Note: Live Logs in the Azure Portal is demonstrated by the instructor. Participants can use kubectl logs --follow as the equivalent command-line approach.

Live Logs provide real-time streaming of container logs directly in the Azure Portal, without the delay of Log Analytics ingestion. The instructor will demonstrate:

1. Navigate to the AKS cluster in the Azure Portal
2. Go to Workloads (under Kubernetes resources)
3. Find and select a Pod
4. Click on Live Logs tab
5. Observe log output appearing in real time

Live Logs connect directly to the Kubernetes API server and stream kubectl logs output through the Portal UI.

Key benefit: Live Logs allows team members without kubectl access to view Pod logs through Azure RBAC, which is useful for developers who do not have direct cluster access.

Task 4: Node-Level Metrics (Instructor Demo)

Note: Azure Portal Insights views are demonstrated by the instructor. Participants can use kubectl top nodes and kubectl top pods for command-line metrics.

Participant commands (run these in your terminal):

# View node resource usage
kubectl top nodes

# View pod resource usage in your namespace
kubectl top pods -n student-XX

Instructor demo in Azure Portal:

1. Navigate to the AKS cluster in the Azure Portal
2. Go to Monitoring > Insights
3. Select the Nodes tab
4. Review metrics for each node:
   • CPU utilization — percentage of CPU used vs allocated
   • Memory utilization — percentage of memory used vs allocated
   • Disk usage — node disk utilization
   • Network — bytes sent/received
5. Click on a specific node to drill down into container-level metrics

Useful KQL Queries (Instructor Reference)

Note: These KQL queries are run in the Azure Portal Log Analytics workspace by the instructor. They are included here so participants understand the types of queries available for production monitoring.

Container restarts in the last hour:

KubePodInventory
| where TimeGenerated > ago(1h)
| where ContainerRestartCount > 0
| project TimeGenerated, PodName, Namespace, ContainerRestartCount
| order by ContainerRestartCount desc

OOM Killed containers:

KubeEvents
| where TimeGenerated > ago(24h)
| where Reason == "OOMKilling"
| project TimeGenerated, Name, Namespace, Message
| order by TimeGenerated desc

Pod status changes:

KubePodInventory
| where TimeGenerated > ago(1h)
| summarize arg_max(TimeGenerated, *) by PodName, Namespace
| where PodStatus != "Running"
| project TimeGenerated, PodName, Namespace, PodStatus
| order by TimeGenerated desc

Top 10 Pods by CPU usage:

Perf
| where TimeGenerated > ago(1h)
| where ObjectName == "K8SContainer" and CounterName == "cpuUsageNanoCores"
| summarize AvgCPU = avg(CounterValue) by InstanceName
| top 10 by AvgCPU desc

Cleanup

kubectl delete pod log-demo-XX -n student-XX

Common Problems

Best Practices

• Enable Container Insights at cluster creation — use --enable-addons monitoring to start collecting data immediately
• Use ContainerLogV2 — the newer schema is more efficient and cost-effective than the legacy ContainerLog table
• Configure data collection rules — exclude noisy namespaces (e.g., kube-system) or specific log levels to reduce costs
• Use Live Logs for debugging — faster than querying Log Analytics for real-time troubleshooting
• Set up alerts on KQL queries — create alert rules for critical conditions (OOM kills, pod restarts, failed deployments)
• Retain logs appropriately — configure Log Analytics workspace retention based on compliance and cost requirements (default: 30 days)
• Combine with Prometheus — use Container Insights for logs and Prometheus for detailed metrics (covered in the next exercise)

Summary

In this exercise you learned:

• Azure Monitor Container Insights collects logs, metrics, and inventory data from AKS clusters
• The monitoring agent (ama-logs or omsagent) runs as a DaemonSet on every node
• KQL (Kusto Query Language) is used to query logs in Log Analytics
• Live Logs provide real-time container log streaming in the Azure Portal
• Container Insights provides pre-built dashboards for cluster, node, and container metrics
• Key log tables include ContainerLogV2, KubePodInventory, KubeEvents, and Perf