Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services and canary ingress). These objects expose the application outside the cluster and drive the canary analysis and promotion.
Create a test namespace:
kubectlcreatenstest
Create a deployment and a horizontal pod autoscaler:
Save the above resource as podinfo-ingress.yaml and then apply it:
kubectlapply-f./podinfo-ingress.yaml
Create a canary custom resource (replace app.example.com with your own domain):
apiVersion:flagger.app/v1beta1kind:Canarymetadata:name:podinfonamespace:testspec:provider:nginx# deployment referencetargetRef:apiVersion:apps/v1kind:Deploymentname:podinfo# ingress referenceingressRef:apiVersion:networking.k8s.io/v1kind:Ingressname:podinfo# HPA reference (optional)autoscalerRef:apiVersion:autoscaling/v2kind:HorizontalPodAutoscalername:podinfo# the maximum time in seconds for the canary deployment# to make progress before it is rollback (default 600s)progressDeadlineSeconds:60service:# ClusterIP port numberport:80# container port number or nametargetPort:9898analysis:# schedule interval (default 60s)interval:10s# max number of failed metric checks before rollbackthreshold:10# max traffic percentage routed to canary# percentage (0-100)maxWeight:50# canary increment step# percentage (0-100)stepWeight:5# NGINX Prometheus checksmetrics: - name:request-success-rate# minimum req success rate (non 5xx responses)# percentage (0-100)thresholdRange:min:99interval:1m# testing (optional)webhooks: - name:acceptance-testtype:pre-rollouturl:http://flagger-loadtester.test/timeout:30smetadata:type:bashcmd:"curl -sd 'test' http://podinfo-canary/token | grep token" - name:load-testurl:http://flagger-loadtester.test/timeout:5smetadata:cmd:"hey -z 1m -q 10 -c 2 http://app.example.com/"
Save the above resource as podinfo-canary.yaml and then apply it:
kubectlapply-f./podinfo-canary.yaml
After a couple of seconds Flagger will create the canary objects:
Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack or MS Teams.
Flagger Canary Stages
Trigger a canary deployment by updating the container image:
When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.
kubectl -n test describe canary/podinfo
Status:
Canary Weight: 0
Failed Checks: 10
Phase: Failed
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Synced 3m flagger Starting canary deployment for podinfo.test
Normal Synced 3m flagger Advance podinfo.test canary weight 5
Normal Synced 3m flagger Advance podinfo.test canary weight 10
Normal Synced 3m flagger Advance podinfo.test canary weight 15
Normal Synced 3m flagger Halt podinfo.test advancement success rate 69.17% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 61.39% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 55.06% < 99%
Normal Synced 2m flagger Halt podinfo.test advancement success rate 47.00% < 99%
Normal Synced 2m flagger (combined from similar events): Halt podinfo.test advancement success rate 38.08% < 99%
Warning Synced 1m flagger Rolling back podinfo.test failed checks threshold reached 10
Warning Synced 1m flagger Canary failed! Scaling down podinfo.test
Custom metrics
The canary analysis can be extended with Prometheus queries.
The demo app is instrumented with Prometheus so you can create a custom check that will use the HTTP request duration histogram to validate the canary.
Create a metric template and apply it on the cluster:
The threshold is set to 500ms so if the average request duration in the last minute goes over half a second then the analysis will fail and the canary will not be promoted.
Trigger a canary deployment by updating the container image:
kubectl -n nginx-ingress logs deployment/flagger -f | jq .msg
Starting canary deployment for podinfo.test
Advance podinfo.test canary weight 5
Advance podinfo.test canary weight 10
Advance podinfo.test canary weight 15
Halt podinfo.test advancement latency 1.20 > 0.5
Halt podinfo.test advancement latency 1.45 > 0.5
Halt podinfo.test advancement latency 1.60 > 0.5
Halt podinfo.test advancement latency 1.69 > 0.5
Halt podinfo.test advancement latency 1.70 > 0.5
Rolling back podinfo.test failed checks threshold reached 5
Canary failed! Scaling down podinfo.test
If you have alerting configured, Flagger will send a notification with the reason why the canary failed.
A/B Testing
Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users. This is particularly useful for frontend applications that require session affinity.
Flagger A/B Testing Stages
Edit the canary analysis, remove the max/step weight and add the match conditions and iterations:
The above configuration will run an analysis for ten minutes targeting users that have a canary cookie set to always or those that call the service using the X-Canary: insider header.
Trigger a canary deployment by updating the container image: