18: Readiness Probe & Service

18: Readiness Probe & Service

Objective

Understand how the Readiness Probe directly affects Service endpoints, observe the behavior in real time using kuard’s interactive UI, and learn the difference between Liveness and Readiness probes in the context of Service traffic routing.

Theory

How Readiness Probe Affects Service Endpoints

The Readiness Probe answers one question: “Is this Pod ready to receive traffic?”

• When a Pod’s readiness probe succeeds, the Pod’s IP is added to the Service’s Endpoints list and it starts receiving traffic.
• When a Pod’s readiness probe fails, the Pod’s IP is removed from the Service’s Endpoints list and it stops receiving traffic.
• The Pod is not restarted — it keeps running but is simply excluded from the load balancer pool.

This is fundamentally different from the Liveness Probe, which triggers a Pod restart on failure.

Liveness vs Readiness: Impact on Service

Endpoint Lifecycle Diagram

Normal state — all Pods pass readiness:

graph TB
    SVC["Service: kuard-svc-XX<br/>Endpoints: 3 of 3"]

    subgraph Healthy["All Pods Ready"]
        P1["Pod 1 ✓<br/>READY 1/1"]
        P2["Pod 2 ✓<br/>READY 1/1"]
        P3["Pod 3 ✓<br/>READY 1/1"]
    end

    SVC --> P1
    SVC --> P2
    SVC --> P3

    style SVC fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    style Healthy fill:#e8f5e9,stroke:#388e3c,stroke-width:1px

After readiness failure — Pod 2 removed from endpoints:

graph TB
    SVC2["Service: kuard-svc-XX<br/>Endpoints: 2 of 3"]

    subgraph Mixed["Pod 2 Not Ready"]
        P4["Pod 1 ✓<br/>READY 1/1"]
        P5["Pod 2 ✗<br/>READY 0/1"]
        P6["Pod 3 ✓<br/>READY 1/1"]
    end

    SVC2 --> P4
    SVC2 -.->|"removed"| P5
    SVC2 --> P6

    style SVC2 fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    style Mixed fill:#fff3e0,stroke:#f57c00,stroke-width:1px
    style P5 fill:#ffcdd2,stroke:#c62828,stroke-width:2px

Practical Task

In this exercise you will deploy kuard with readiness probes, expose it via a LoadBalancer Service, and use kuard’s built-in UI to manually trigger readiness failures and observe the effect on Service endpoints in real time.

Step 1: Create the Deployment and Service

Create a file called kuard-readiness.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuard-ready-XX                 # Replace XX with your student number
  namespace: student-XX
  labels:
    app: kuard-ready-XX
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kuard-ready-XX
  template:
    metadata:
      labels:
        app: kuard-ready-XX
    spec:
      containers:
        - name: kuard
          image: <ACR_NAME>.azurecr.io/kuard:1
          ports:
            - containerPort: 8080
              name: http
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
            successThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
  name: kuard-svc-XX                   # Replace XX with your student number
  namespace: student-XX
spec:
  type: LoadBalancer
  selector:
    app: kuard-ready-XX
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
      name: http

Deploy and wait for the external IP:

kubectl apply -f kuard-readiness.yaml
kubectl get svc kuard-svc-XX -n student-XX -w

Step 2: Verify All Pods Are Ready

Check that all 3 replicas are ready:

kubectl get pods -n student-XX -l app=kuard-ready-XX

All Pods should show 1/1 in the READY column.

Check the endpoints:

kubectl get endpoints kuard-svc-XX -n student-XX

You should see 3 IP addresses listed.

Step 3: Open kuard UI in Browser

Navigate to the external IP in your browser:

http://<EXTERNAL-IP>

You should see the kuard web interface. Click on the “Liveness Probe” tab in the left menu — kuard provides controls to manually trigger readiness and liveness probe failures.

Step 4: Watch Endpoints in Real Time

Open a terminal and watch the endpoints continuously:

kubectl get endpoints kuard-svc-XX -n student-XX -w

In a second terminal, watch the Pods:

kubectl get pods -n student-XX -l app=kuard-ready-XX -w

Step 5: Trigger a Readiness Failure

In the kuard UI in your browser:

1. Navigate to the “Liveness Probe” tab
2. In the Readiness section, click the “Fail” button

This causes the /ready endpoint on that specific Pod to return a failure response.

Step 6: Observe the Effect

Watch your terminals. After a few seconds (determined by periodSeconds and failureThreshold):

• The affected Pod’s READY status changes from 1/1 to 0/1
• The Pod’s IP is removed from the endpoints list
• The Pod is NOT restarted — it keeps running, it just stops receiving traffic
• The Deployment shows 2/3 ready replicas

Refresh the browser — your requests are now being served by the remaining 2 healthy Pods only.

Step 7: Restore Readiness

In the kuard UI:

1. Go back to the “Liveness Probe” tab
2. In the Readiness section, click the “Clear” button (or navigate to a healthy Pod instance)

Watch your terminals:

• The Pod’s READY status returns to 1/1
• The Pod’s IP is added back to the endpoints list
• The Deployment shows 3/3 ready replicas again

Step 8: Experiment Further

Try these additional experiments:

1. Fail readiness on multiple Pods — Open kuard in multiple browser tabs (each may hit a different Pod). Fail readiness on 2 out of 3 Pods and observe that only 1 endpoint remains.

2. Compare with liveness failure — Click “Fail” in the Liveness section instead. Notice that the Pod gets restarted (RESTARTS count increases) rather than just being removed from endpoints.

3. Scale during failure — While one Pod has failed readiness, scale the Deployment to 5 replicas. Observe that new Pods join the endpoints as they become ready, while the failed Pod remains excluded.

Clean Up

kubectl delete -f kuard-readiness.yaml

Useful Commands

Common Problems

Best Practices

1. Always use readiness probes for Services — Without a readiness probe, Pods receive traffic as soon as they start, even if the application is still initializing.
2. Set appropriate thresholds — Use failureThreshold to avoid removing Pods from endpoints due to transient issues. A value of 3 with a 5-second period means the Pod must fail for 15 seconds before being removed.
3. Use both liveness and readiness probes — Liveness detects crashed processes (restart), readiness detects temporary unavailability (stop traffic).
4. Do not use the same endpoint for liveness and readiness — A failing readiness probe should not trigger a restart. If you use the same endpoint, a temporarily overloaded Pod will be restarted instead of gracefully shedding traffic.
5. Monitor endpoint count — Alert when the number of ready endpoints drops below a threshold, as this indicates capacity reduction.

Summary

In this exercise you learned:

• Readiness probe failures remove a Pod from Service endpoints without restarting it
• Liveness probe failures restart the Pod, which is a fundamentally different behavior
• Service endpoints update dynamically as Pods pass or fail readiness checks
• kuard provides an interactive UI to manually trigger and clear probe failures
• The combination of readiness probes and Services provides graceful traffic management
• Both probes should be used together but with different endpoints to avoid unintended restarts