Zero downtime deployments

Search…
This is a list of things you should consider when dealing with a high traffic production environment if you want to minimise the impact of rolling updates and downscaling.
Deployment strategy
Limit the number of unavailable pods during a rolling update:
1
apiVersion: apps/v1
2
kind: Deployment
3
spec:
4
  progressDeadlineSeconds: 120
5
  strategy:
6
    type: RollingUpdate
7
    rollingUpdate:
8
      maxUnavailable: 0
Copied!
The default progress deadline for a deployment is ten minutes. You should consider adjusting this value to make the deployment process fail faster.
Liveness health check
You application should expose a HTTP endpoint that Kubernetes can call to determine if your app transitioned to a broken state from which it can't recover and needs to be restarted.
1
livenessProbe:
2
  exec:
3
    command:
4
    - wget
5
    - --quiet
6
    - --tries=1
7
    - --timeout=4
8
    - --spider
9
    - http://localhost:8080/healthz
10
  timeoutSeconds: 5
11
  initialDelaySeconds: 5
Copied!
If you've enabled mTLS, you'll have to use exec for liveness and readiness checks since kubelet is not part of the service mesh and doesn't have access to the TLS cert.
Readiness health check
You application should expose a HTTP endpoint that Kubernetes can call to determine if your app is ready to receive traffic.
1
readinessProbe:
2
  exec:
3
    command:
4
    - wget
5
    - --quiet
6
    - --tries=1
7
    - --timeout=4
8
    - --spider
9
    - http://localhost:8080/readyz
10
  timeoutSeconds: 5
11
  initialDelaySeconds: 5
12
  periodSeconds: 5
Copied!
If your app depends on external services, you should check if those services are available before allowing Kubernetes to route traffic to an app instance. Keep in mind that the Envoy sidecar can have a slower startup than your app. This means that on application start you should retry for at least a couple of seconds any external connection.
Graceful shutdown
Before a pod gets terminated, Kubernetes sends a SIGTERM signal to every container and waits for period of time (30s by default) for all containers to exit gracefully. If your app doesn't handle the SIGTERM signal or if it doesn't exit within the grace period, Kubernetes will kill the container and any inflight requests that your app is processing will fail.
1
apiVersion: apps/v1
2
kind: Deployment
3
spec:
4
  template:
5
    spec:
6
      terminationGracePeriodSeconds: 60
7
      containers:
8
      - name: app
9
        lifecycle:
10
          preStop:
11
            exec:
12
              command:
13
              - sleep
14
              - "10"
Copied!
Your app container should have a preStop hook that delays the container shutdown. This will allow the service mesh to drain the traffic and remove this pod from all other Envoy sidecars before your app becomes unavailable.
Delay Envoy shutdown
Even if your app reacts to SIGTERM and tries to complete the inflight requests before shutdown, that doesn't mean that the response will make it back to the caller. If the Envoy sidecar shuts down before your app, then the caller will receive a 503 error.
To mitigate this issue you can add a preStop hook to the Istio proxy and wait for the main app to exit before Envoy exits.
1
#!/bin/bash
2
set -e
3
if ! pidof envoy &>/dev/null; then
4
  exit 0
5
fi
6
​
7
if ! pidof pilot-agent &>/dev/null; then
8
  exit 0
9
fi
10
​
11
while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do
12
  sleep 1;
13
done
14
​
15
exit 0
Copied!
You'll have to build your own Envoy docker image with the above script and modify the Istio injection webhook with the preStop directive.
Thanks to Stono for his excellent tips on minimising 503s.
Resource requests and limits
Setting CPU and memory requests/limits for all workloads is a mandatory step if you're running a production system. Without limits your nodes could run out of memory or become unresponsive due to CPU exhausting. Without CPU and memory requests, the Kubernetes scheduler will not be able to make decisions about which nodes to place pods on.
1
apiVersion: apps/v1
2
kind: Deployment
3
spec:
4
  template:
5
    spec:
6
      containers:
7
      - name: app
8
        resources:
9
          limits:
10
            cpu: 1000m
11
            memory: 1Gi
12
          requests:
13
            cpu: 100m
14
            memory: 128Mi
Copied!
Note that without resource requests the horizontal pod autoscaler can't determine when to scale your app.
Autoscaling
A production environment should be able to handle traffic bursts without impacting the quality of service. This can be achieved with Kubernetes autoscaling capabilities. Autoscaling in Kubernetes has two dimensions: the Cluster Autoscaler that deals with node scaling operations and the Horizontal Pod Autoscaler that automatically scales the number of pods in a deployment.
1
apiVersion: autoscaling/v2beta2
2
kind: HorizontalPodAutoscaler
3
spec:
4
  scaleTargetRef:
5
    apiVersion: apps/v1
6
    kind: Deployment
7
    name: app
8
  minReplicas: 2
9
  maxReplicas: 4
10
  metrics:
11
  - type: Resource
12
    resource:
13
      name: cpu
14
      targetAverageValue: 900m
15
  - type: Resource
16
    resource:
17
      name: memory
18
      targetAverageValue: 768Mi
Copied!
The above HPA ensures your app will be scaled up before the pods reach the CPU or memory limits.
Ingress retries
To minimise the impact of downscaling operations you can make use of Envoy retry capabilities.
1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
spec:
4
  service:
5
    port: 9898
6
    gateways:
7
    - public-gateway.istio-system.svc.cluster.local
8
    hosts:
9
    - app.example.com
10
    retries:
11
      attempts: 10
12
      perTryTimeout: 5s
13
      retryOn: "gateway-error,connect-failure,refused-stream"
Copied!
When the HPA scales down your app, your users could run into 503 errors. The above configuration will make Envoy retry the HTTP requests that failed due to gateway errors.
Tutorials - Previous
Canary analysis with Prometheus Operator
Next - Dev
Development Guide
Last modified 10mo ago
Copy link
Contents
Deployment strategy
Liveness health check
Readiness health check
Graceful shutdown
Delay Envoy shutdown
Resource requests and limits
Autoscaling
Ingress retries