Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and canary ingress). These objects expose the application outside the cluster and drive the canary analysis and promotion.
Create a test namespace:
kubectlcreatenstest
Create a deployment and a horizontal pod autoscaler:
Save the above resource as podinfo-ingress.yaml and then apply it:
kubectlapply-f./podinfo-ingress.yaml
Create a canary custom resource (replace app.example.com with your own domain):
apiVersion:flagger.app/v1beta1kind:Canarymetadata:name:podinfonamespace:testspec:provider:skipper# deployment referencetargetRef:apiVersion:apps/v1kind:Deploymentname:podinfo# ingress referenceingressRef:apiVersion:networking.k8s.io/v1kind:Ingressname:podinfo# HPA reference (optional)autoscalerRef:apiVersion:autoscaling/v2kind:HorizontalPodAutoscalername:podinfo# the maximum time in seconds for the canary deployment# to make progress before it is rollback (default 600s)progressDeadlineSeconds:60service:# ClusterIP port numberport:80# container port number or nametargetPort:9898analysis:# schedule interval (default 60s)interval:10s# max number of failed metric checks before rollbackthreshold:10# max traffic percentage routed to canary# percentage (0-100)maxWeight:50# canary increment step# percentage (0-100)stepWeight:5# Skipper Prometheus checksmetrics: - name:request-success-rateinterval:1m# minimum req success rate (non 5xx responses)# percentage (0-100)thresholdRange:min:99 - name:request-durationinterval:1m# maximum req duration P99# millisecondsthresholdRange:max:500webhooks: - name:gatetype:confirm-rollouturl:http://flagger-loadtester.test/gate/approve - name:acceptance-testtype:pre-rollouturl:http://flagger-loadtester.test/timeout:10smetadata:type:bashcmd:"curl -sd 'test' http://podinfo-canary/token | grep token" - name:load-testtype:rollouturl:http://flagger-loadtester.test/timeout:5smetadata:type:cmdcmd:"hey -z 10m -q 10 -c 2 -host app.example.com http://skipper-ingress.kube-system"logCmdOutput:"true"
Save the above resource as podinfo-canary.yaml and then apply it:
kubectlapply-f./podinfo-canary.yaml
After a couple of seconds Flagger will create the canary objects:
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack or MS Teams.
Trigger a canary deployment by updating the container image:
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
kubectl -n flagger-system logs deploy/flagger -f | jq .msg
New revision detected! Scaling up podinfo.test
Canary deployment podinfo.test not ready: waiting for rollout to finish: 0 of 1 updated replicas are available
Starting canary analysis for podinfo.test
Pre-rollout check acceptance-test passed
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Advance podinfo.test canary weight 20
Halt podinfo.test advancement success rate 53.42% < 99%
Halt podinfo.test advancement success rate 53.19% < 99%
Halt podinfo.test advancement success rate 48.05% < 99%
Rolling back podinfo.test failed checks threshold reached 3
Canary failed! Scaling down podinfo.test
Custom metrics
The canary analysis can be extended with Prometheus queries.
Create a metric template and apply it on the cluster:
apiVersion:flagger.app/v1beta1kind:MetricTemplatemetadata:name:latencynamespace:testspec:provider:type:prometheusaddress:http://flagger-prometheus.flagger-system:9090query:| histogram_quantile(0.99, sum( rate( skipper_serve_route_duration_seconds_bucket{ route=~"{{ printf "kube(ew)?_%s__%s_canary__.*__%s_canary(_[0-9]+)?" namespace ingress service }}", le="+Inf" }[1m] ) ) by (le) )
Edit the canary analysis and add the latency check:
The threshold is set to 500ms so if the average request duration in the last minute goes over half a second then the analysis will fail and the canary will not be promoted.
Trigger a canary deployment by updating the container image: