stable
Install
Tutorials
Monitoring
Grafana
Flagger comes with a Grafana dashboard made for canary analysis. Install Grafana with Helm:
1
helm upgrade -i flagger-grafana flagger/grafana \
2
--set url=http://prometheus:9090
Copied!
The dashboard shows the RED and USE metrics for the primary and canary workloads:
Canary Dashboard
Logging
The canary errors and latency spikes have been recorded as Kubernetes events and logged by Flagger in json format:
1
kubectl -n istio-system logs deployment/flagger --tail=100 | jq .msg
2
3
Starting canary deployment for podinfo.test
4
Advance podinfo.test canary weight 5
5
Advance podinfo.test canary weight 10
6
Advance podinfo.test canary weight 15
7
Advance podinfo.test canary weight 20
8
Advance podinfo.test canary weight 25
9
Advance podinfo.test canary weight 30
10
Advance podinfo.test canary weight 35
11
Halt podinfo.test advancement success rate 98.69% < 99%
12
Advance podinfo.test canary weight 40
13
Halt podinfo.test advancement request duration 1.515s > 500ms
14
Advance podinfo.test canary weight 45
15
Advance podinfo.test canary weight 50
16
Copying podinfo.test template spec to podinfo-primary.test
17
Halt podinfo-primary.test advancement waiting for rollout to finish: 1 old replicas are pending termination
18
Scaling down podinfo.test
19
Promotion completed! podinfo.test
Copied!
Event Webhook
Flagger can be configured to send event payloads to a specified webhook:
1
helm upgrade -i flagger flagger/flagger \
2
--set eventWebhook=https://example.com/flagger-canary-event-webhook
Copied!
The environment variable EVENT_WEBHOOK_URL can be used for activating the event-webhook, too. This is handy for using a secret to store a sensible value that could contain api keys for example.
When configured, every action that Flagger takes during a canary deployment will be sent as JSON via an HTTP POST request. The JSON payload has the following schema:
1
{
2
"name": "string (canary name)",
3
"namespace": "string (canary namespace)",
4
"phase": "string (canary phase)",
5
"metadata": {
6
"eventMessage": "string (canary event message)",
7
"eventType": "string (canary event type)",
8
"timestamp": "string (unix timestamp ms)"
9
}
10
}
Copied!
Example:
1
{
2
"name": "podinfo",
3
"namespace": "default",
4
"phase": "Progressing",
5
"metadata": {
6
"eventMessage": "New revision detected! Scaling up podinfo.default",
7
"eventType": "Normal",
8
"timestamp": "1578607635167"
9
}
10
}
Copied!
The event webhook can be overwritten at canary level with:
1
analysis:
2
webhooks:
3
- name: "send to Slack"
4
type: event
5
url: http://event-recevier.notifications/slack
Copied!
Metrics
Flagger exposes Prometheus metrics that can be used to determine the canary analysis status and the destination weight values:
1
# Flagger version and mesh provider gauge
2
flagger_info{version="0.10.0", mesh_provider="istio"} 1
3
4
# Canaries total gauge
5
flagger_canary_total{namespace="test"} 1
6
7
# Canary promotion last known status gauge
8
# 0 - running, 1 - successful, 2 - failed
9
flagger_canary_status{name="podinfo" namespace="test"} 1
10
11
# Canary traffic weight gauge
12
flagger_canary_weight{workload="podinfo-primary" namespace="test"} 95
13
flagger_canary_weight{workload="podinfo" namespace="test"} 5
14
15
# Seconds spent performing canary analysis histogram
16
flagger_canary_duration_seconds_bucket{name="podinfo",namespace="test",le="10"} 6
17
flagger_canary_duration_seconds_bucket{name="podinfo",namespace="test",le="+Inf"} 6
18
flagger_canary_duration_seconds_sum{name="podinfo",namespace="test"} 17.3561329
19
flagger_canary_duration_seconds_count{name="podinfo",namespace="test"} 6
20
21
# Last canary metric analysis result per different metrics
22
flagger_canary_metric_analysis{metric="podinfo-http-successful-rate",name="podinfo",namespace="test"} 1
23
flagger_canary_metric_analysis{metric="podinfo-custom-metric",name="podinfo",namespace="test"} 0.918223108974359
Copied!
Last modified 1mo ago
Copy link