Canary analysis with Prometheus Operator

This guide show you how to use Prometheus Operator for canary analysis.
Prerequisites
Install Prometheus Operator with Helm v3:
helm repo add stable https://kubernetes-charts.storage.googleapis.com
​
kubectl create ns monitoring
helm upgrade -i prometheus stable/prometheus-operator \
--namespace monitoring \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set fullnameOverride=prometheus
The prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false option allows Prometheus operator to watch serviceMonitors outside of his namespace.
Install Flagger by setting the metrics server to Prometheus:
helm repo add flagger https://flagger.app
​
kubectl create ns flagger-system
helm upgrade -i flagger flagger/flagger \
--namespace flagger-system \
--set metricsServer=http://prometheus-prometheus.monitoring:9090 \
--set meshProvider=kubernetes
Install Flagger's tester:
helm upgrade -i loadtester flagger/loadtester \
--namespace flagger-system
Install podinfo demo app:
helm repo add podinfo https://stefanprodan.github.io/podinfo
​
kubectl create ns test
helm upgrade -i podinfo podinfo/podinfo \
--namespace test \
--set service.enabled=false
Service monitors
The demo app is instrumented with Prometheus so you can create service monitors to scrape podinfo's metrics endpoint:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: podinfo-primary
  namespace: test
spec:
  endpoints:
  - path: /metrics
    port: http
    interval: 5s
  selector:
    matchLabels:
      app: podinfo
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: podinfo-canary
  namespace: test
spec:
  endpoints:
  - path: /metrics
    port: http
    interval: 5s
  selector:
    matchLabels:
      app: podinfo-canary
We are setting interval: 5s to have a more aggressive scraping. If you do not define it, you must to use a longer interval in the Canary object.
Metric templates
Create a metric template to measure the HTTP requests error rate:
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: error-rate
  namespace: test
spec:
  provider:
    address: http://prometheus-prometheus.monitoring:9090
    type: prometheus
  query: |
    100 - rate(
      http_requests_total{
        namespace="{{ namespace }}",
        job="{{ target }}-canary",
        status!~"5.*"
      }[{{ interval }}]) 
    / 
    rate(
      http_requests_total{
        namespace="{{ namespace }}",
        job="{{ target }}-canary"
      }[{{ interval }}]
    ) * 100
Create a metric template to measure the HTTP requests average duration:
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: latency
  namespace: test
spec:
  provider:
    address: http://prometheus-prometheus.monitoring:9090
    type: prometheus
  query: |
    histogram_quantile(0.99,
      sum(
        rate(
          http_request_duration_seconds_bucket{
            namespace="{{ namespace }}",
            job="{{ target }}-canary"
          }[{{ interval }}]
        )
      ) by (le)
    )
Canary analysis
Using the metrics template you can configure the canary analysis with HTTP error rate and latency checks:
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  provider: kubernetes
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 60
  service:
    port: 80
    targetPort: http
    name: podinfo
  analysis:
    interval: 30s
    iterations: 10
    threshold: 2
    metrics:
    - name: error-rate
      templateRef:
        name: error-rate
      thresholdRange:
        max: 1
      interval: 30s
    - name: latency
      templateRef:
        name: latency
      thresholdRange:
        max: 0.5
      interval: 30s
    webhooks:
      - name: load-test
        type: rollout
        url: "http://loadtester.flagger-system/"
        timeout: 5s
        metadata:
          type: cmd
          cmd: "hey -z 1m -q 10 -c 2 http://podinfo-canary.test/"
Based on the above specification, Flagger creates the primary and canary Kubernetes ClusterIP service.
During the canary analysis, Prometheus will scrape the canary service and Flagger will use the HTTP error rate and latency queries to determine if the release should be promoted or rolled back.
Tutorials - Previous
Crossover Canary Deployments
Next - Tutorials
Canaries with Helm charts and GitOps
Last updated 6 months ago