stable
Install
Tutorials
FAQ
Deployment Strategies
Which deployment strategies are supported by Flagger?
Flagger implements the following deployment strategies:
When should I use A/B testing instead of progressive traffic shifting?
For frontend applications that require session affinity, you should use HTTP headers or cookie match conditions to ensure a set of users will stay on the same version for the whole duration of the canary analysis.
Can I use Flagger to manage applications that live outside of a service mesh?
For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments with Kubernetes L4 networking.
When can I use traffic mirroring?
Traffic mirroring can be used for Blue/Green deployment strategy or a pre-stage in a Canary release. Traffic mirroring will copy each incoming request, sending one request to the primary and one to the canary service. Mirroring should be used for requests that are idempotent or capable of being processed twice (once by the primary and once by the canary).
How to retry a failed release?
A canary analysis is triggered by changes in any of the following objects:
- Deployment/DaemonSet PodSpec (metadata, container image, command, ports, env, resources, etc)
- ConfigMaps mounted as volumes or mapped to environment variables
- Secrets mounted as volumes or mapped to environment variables
To retry a release you can add or change an annotation on the pod template:
1
apiVersion: apps/v1
2
kind: Deployment
3
spec:
4
template:
5
metadata:
6
annotations:
7
timestamp: "2020-03-10T14:24:48+0000"
Copied!
How to change replicas for a deployment when not using HPA?
To change replicas for a deployment when not using HPA, you have to update the canary deployment with the desired replica count and trigger an analysis by annotating the template. After the analysis finishes, Flagger will promote the
spec.replicas changes to the primary deployment.Example:
1
apiVersion: apps/v1
2
kind: Deployment
3
spec:
4
replicas: 4 #update replicas
5
template:
6
metadata:
7
annotations:
8
timestamp: "2022-02-10T14:24:48+0000" #add annotation to trigger analysis
Copied!
Why is there a window of downtime during the canary initializing process when analysis is disabled?
A window of downtime is the intended behavior when the analysis is disabled. This allows instant rollback and also mimics the way a Kubernetes deployment initialization works. To avoid this, enable the analysis (
skipAnalysis: true), wait for the initialization to finish, and disable it afterward (skipAnalysis: false).How to disable cross namespace references?
Flagger by default can access resources across namespaces (
AlertProivder, MetricProvider and Gloo Upsteream). If you're in a multi-tenant environment and wish to disable this, you can do so through the no-cross-namespace-refs flag.1
flagger \
2
-no-cross-namespace-refs=true \
3
...
Copied!
Kubernetes services
How is an application exposed inside the cluster?
Assuming the app name is
podinfo, you can define a canary like:1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
metadata:
4
name: podinfo
5
namespace: test
6
spec:
7
targetRef:
8
apiVersion: apps/v1
9
kind: Deployment
10
name: podinfo
11
service:
12
# service name (optional)
13
name: podinfo
14
# ClusterIP port number (required)
15
port: 9898
16
# container port name or number
17
targetPort: http
18
# port name can be http or grpc (default http)
19
portName: http
Copied!
If the
service.name is not specified, then targetRef.name is used for the apex domain and canary/primary services name prefix. You should treat the service name as an immutable field; changing its could result in routing conflicts.Based on the canary spec service, Flagger generates the following Kubernetes ClusterIP service:
<service.name>.<namespace>.svc.cluster.localselectorapp=<name>-primary<service.name>-primary.<namespace>.svc.cluster.localselectorapp=<name>-primary<service.name>-canary.<namespace>.svc.cluster.localselectorapp=<name>
This ensures that traffic coming from a namespace outside the mesh to
podinfo.test:9898 will be routed to the latest stable release of your app.1
apiVersion: v1
2
kind: Service
3
metadata:
4
name: podinfo
5
spec:
6
type: ClusterIP
7
selector:
8
app: podinfo-primary
9
ports:
10
- name: http
11
port: 9898
12
protocol: TCP
13
targetPort: http
14
---
15
apiVersion: v1
16
kind: Service
17
metadata:
18
name: podinfo-primary
19
spec:
20
type: ClusterIP
21
selector:
22
app: podinfo-primary
23
ports:
24
- name: http
25
port: 9898
26
protocol: TCP
27
targetPort: http
28
---
29
apiVersion: v1
30
kind: Service
31
metadata:
32
name: podinfo-canary
33
spec:
34
type: ClusterIP
35
selector:
36
app: podinfo
37
ports:
38
- name: http
39
port: 9898
40
protocol: TCP
41
targetPort: http
Copied!
The
podinfo-canary.test:9898 address is available only during the canary analysis and can be used for conformance testing or load testing.Multiple ports
My application listens on multiple ports. How can I expose them inside the cluster?
If port discovery is enabled, Flagger scans the deployment spec and extracts the containers ports excluding the port specified in the canary service and Envoy sidecar ports. These ports will be used when generating the ClusterIP services.
For a deployment that exposes two ports:
1
apiVersion: apps/v1
2
kind: Deployment
3
spec:
4
template:
5
metadata:
6
annotations:
7
prometheus.io/scrape: "true"
8
prometheus.io/port: "9899"
9
spec:
10
containers:
11
- name: app
12
ports:
13
- containerPort: 8080
14
- containerPort: 9090
Copied!
You can enable port discovery so that Prometheus will be able to reach port
9090 over mTLS:1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
spec:
4
service:
5
# container port used for canary analysis
6
port: 8080
7
# port name can be http or grpc (default http)
8
portName: http
9
# add all the other container ports
10
# to the ClusterIP services (default false)
11
portDiscovery: true
12
trafficPolicy:
13
tls:
14
mode: ISTIO_MUTUAL
Copied!
Both port
8080 and 9090 will be added to the ClusterIP services.Label selectors
What labels selectors are supported by Flagger?
The target deployment must have a single label selector in the format
app: <DEPLOYMENT-NAME>:1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: podinfo
5
spec:
6
selector:
7
matchLabels:
8
app: podinfo
9
template:
10
metadata:
11
labels:
12
app: podinfo
Copied!
Besides
app, Flagger supports name and app.kubernetes.io/name selectors. If you use a different convention, you can specify your label with the -selector-labels flag. For example:1
flagger \
2
-selector-labels=service,name,app.kubernetes.io/name \
3
...
Copied!
Is pod affinity and anti affinity supported?
Flagger will rewrite the first value in each match expression, defined in the target deployment's pod anti-affinity and topology spread constraints, satisfying the following two requirements when creating, or updating, the primary deployment:
- The key in the match expression must be one of the labels specified by the parameter selector-labels. The default labels are
app,name,app.kubernetes.io/name. - The value must match the name of the target deployment.
The rewrite done by Flagger in these cases is to suffix the value with
-primary. This rewrite can be used to spread the pods created by the canary and primary deployments across different availability zones.Example target deployment:
1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: podinfo
5
spec:
6
selector:
7
matchLabels:
8
app: podinfo
9
template:
10
metadata:
11
labels:
12
app: podinfo
13
spec:
14
affinity:
15
podAntiAffinity:
16
preferredDuringSchedulingIgnoredDuringExecution:
17
- weight: 100
18
podAffinityTerm:
19
labelSelector:
20
matchExpressions:
21
- key: app
22
operator: In
23
values:
24
- podinfo
25
topologyKey: topology.kubernetes.io/zone
Copied!
Example of generated primary deployment:
1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: podinfo-primary
5
spec:
6
selector:
7
matchLabels:
8
app: podinfo-primary
9
template:
10
metadata:
11
labels:
12
app: podinfo-primary
13
spec:
14
affinity:
15
podAntiAffinity:
16
preferredDuringSchedulingIgnoredDuringExecution:
17
- weight: 100
18
podAffinityTerm:
19
labelSelector:
20
matchExpressions:
21
- key: app
22
operator: In
23
values:
24
- podinfo-primary
25
topologyKey: topology.kubernetes.io/zone
Copied!
It is also possible to use a different label than the
app, name or app.kubernetes.io/name.Anti affinity example(using a different label):
1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: podinfo
5
spec:
6
selector:
7
matchLabels:
8
app: podinfo
9
affinity: podinfo
10
template:
11
metadata:
12
labels:
13
app: podinfo
14
affinity: podinfo
15
spec:
16
affinity:
17
podAntiAffinity:
18
preferredDuringSchedulingIgnoredDuringExecution:
19
- weight: 100
20
podAffinityTerm:
21
labelSelector:
22
matchLabels:
23
affinity: podinfo
24
topologyKey: topology.kubernetes.io/zone
Copied!
Metrics
How does Flagger measure the request success rate and duration?
By default, Flagger measures the request success rate and duration using Prometheus queries.
HTTP requests success rate percentage
Spec:
1
analysis:
2
metrics:
3
- name: request-success-rate
4
# minimum req success rate (non 5xx responses)
5
# percentage (0-100)
6
thresholdRange:
7
min: 99
8
interval: 1m
Copied!
Istio query:
1
sum(
2
rate(
3
istio_requests_total{
4
reporter="destination",
5
destination_workload_namespace=~"$namespace",
6
destination_workload=~"$workload",
7
response_code!~"5.*"
8
}[$interval]
9
)
10
)
11
/
12
sum(
13
rate(
14
istio_requests_total{
15
reporter="destination",
16
destination_workload_namespace=~"$namespace",
17
destination_workload=~"$workload"
18
}[$interval]
19
)
20
)
Copied!
Envoy query (App Mesh):
1
sum(
2
rate(
3
envoy_cluster_upstream_rq{
4
kubernetes_namespace="$namespace",
5
kubernetes_pod_name=~"$workload",
6
envoy_response_code!~"5.*"
7
}[$interval]
8
)
9
)
10
/
11
sum(
12
rate(
13
envoy_cluster_upstream_rq{
14
kubernetes_namespace="$namespace",
15
kubernetes_pod_name=~"$workload"
16
}[$interval]
17
)
18
)
Copied!
Envoy query (Contour and Gloo):
1
sum(
2
rate(
3
envoy_cluster_upstream_rq{
4
envoy_cluster_name=~"$namespace-$workload",
5
envoy_response_code!~"5.*"
6
}[$interval]
7
)
8
)
9
/
10
sum(
11
rate(
12
envoy_cluster_upstream_rq{
13
envoy_cluster_name=~"$namespace-$workload",
14
}[$interval]
15
)
16
)
Copied!
HTTP requests milliseconds duration P99
Spec:
1
analysis:
2
metrics:
3
- name: request-duration
4
# maximum req duration P99
5
# milliseconds
6
thresholdRange:
7
max: 500
8
interval: 1m
Copied!
Istio query:
1
histogram_quantile(0.99,
2
sum(
3
irate(
4
istio_request_duration_milliseconds_bucket{
5
reporter="destination",
6
destination_workload=~"$workload",
7
destination_workload_namespace=~"$namespace"
8
}[$interval]
9
)
10
) by (le)
11
)
Copied!
Envoy query (App Mesh, Contour and Gloo):
1
histogram_quantile(0.99,
2
sum(
3
irate(
4
envoy_cluster_upstream_rq_time_bucket{
5
kubernetes_pod_name=~"$workload",
6
kubernetes_namespace=~"$namespace"
7
}[$interval]
8
)
9
) by (le)
10
)
Copied!
Note that the metric interval should be lower or equal to the control loop interval.
Can I use custom metrics?
The analysis can be extended with metrics provided by Prometheus, Datadog, AWS CloudWatch, New Relic and Graphite. For more details on how custom metrics can be used, please read the metrics docs.
Istio Gateway API
If you're using Istio with Gateway API, the Prometheus query needs to include
reporter="source". For example, to calculate HTTP requests error percentage, the query would be:1
100 - sum(
2
rate(
3
istio_requests_total{
4
reporter="source",
5
destination_workload_namespace=~"$namespace",
6
destination_workload=~"$workload",
7
response_code!~"5.*"
8
}[$interval]
9
)
10
)
11
/
12
sum(
13
rate(
14
istio_requests_total{
15
reporter="source",
16
destination_workload_namespace=~"$namespace",
17
destination_workload=~"$workload"
18
}[$interval]
19
)
20
) * 100
Copied!
Istio routing
How does Flagger interact with Istio?
Flagger creates an Istio Virtual Service and Destination Rules based on the Canary service spec. The service configuration lets you expose an app inside or outside the mesh. You can also define traffic policies, HTTP match conditions, URI rewrite rules, CORS policies, timeout and retries.
The following spec exposes the
frontend workload inside the mesh on frontend.test.svc.cluster.local:9898 and outside the mesh on frontend.example.com. You'll have to specify an Istio ingress gateway for external hosts.1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
metadata:
4
name: frontend
5
namespace: test
6
spec:
7
service:
8
# container port
9
port: 9898
10
# service port name (optional, will default to "http")
11
portName: http-frontend
12
# Istio gateways (optional)
13
gateways:
14
- public-gateway.istio-system.svc.cluster.local
15
- mesh
16
# Istio virtual service host names (optional)
17
hosts:
18
- frontend.example.com
19
# Istio traffic policy
20
trafficPolicy:
21
tls:
22
# use ISTIO_MUTUAL when mTLS is enabled
23
mode: DISABLE
24
# HTTP match conditions (optional)
25
match:
26
- uri:
27
prefix: /
28
# HTTP rewrite (optional)
29
rewrite:
30
uri: /
31
# Istio retry policy (optional)
32
retries:
33
attempts: 3
34
perTryTimeout: 1s
35
retryOn: "gateway-error,connect-failure,refused-stream"
36
# Add headers (optional)
37
headers:
38
request:
39
add:
40
x-some-header: "value"
41
# cross-origin resource sharing policy (optional)
42
corsPolicy:
43
allowOrigin:
44
- example.com
45
allowMethods:
46
- GET
47
allowCredentials: false
48
allowHeaders:
49
- x-some-header
50
maxAge: 24h
Copied!
For the above spec Flagger will generate the following virtual service:
1
apiVersion: networking.istio.io/v1alpha3
2
kind: VirtualService
3
metadata:
4
name: frontend
5
namespace: test
6
ownerReferences:
7
- apiVersion: flagger.app/v1beta1
8
blockOwnerDeletion: true
9
controller: true
10
kind: Canary
11
name: podinfo
12
uid: 3a4a40dd-3875-11e9-8e1d-42010a9c0fd1
13
spec:
14
gateways:
15
- public-gateway.istio-system.svc.cluster.local
16
- mesh
17
hosts:
18
- frontend.example.com
19
- frontend
20
http:
21
- corsPolicy:
22
allowHeaders:
23
- x-some-header
24
allowMethods:
25
- GET
26
allowOrigin:
27
- example.com
28
maxAge: 24h
29
headers:
30
request:
31
add:
32
x-some-header: "value"
33
match:
34
- uri:
35
prefix: /
36
rewrite:
37
uri: /
38
route:
39
- destination:
40
host: podinfo-primary
41
weight: 100
42
- destination:
43
host: podinfo-canary
44
weight: 0
45
retries:
46
attempts: 3
47
perTryTimeout: 1s
48
retryOn: "gateway-error,connect-failure,refused-stream"
Copied!
For each destination in the virtual service a rule is generated:
1
apiVersion: networking.istio.io/v1alpha3
2
kind: DestinationRule
3
metadata:
4
name: frontend-primary
5
namespace: test
6
spec:
7
host: frontend-primary
8
trafficPolicy:
9
tls:
10
mode: DISABLE
11
---
12
apiVersion: networking.istio.io/v1alpha3
13
kind: DestinationRule
14
metadata:
15
name: frontend-canary
16
namespace: test
17
spec:
18
host: frontend-canary
19
trafficPolicy:
20
tls:
21
mode: DISABLE
Copied!
Flagger keeps in sync the virtual service and destination rules with the canary service spec. Any direct modification to the virtual service spec will be overwritten.
To expose a workload inside the mesh on
http://backend.test.svc.cluster.local:9898, the service spec can contain only the container port and the traffic policy:1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
metadata:
4
name: backend
5
namespace: test
6
spec:
7
service:
8
port: 9898
9
trafficPolicy:
10
tls:
11
mode: DISABLE
Copied!
Based on the above spec, Flagger will create several ClusterIP services like:
1
apiVersion: v1
2
kind: Service
3
metadata:
4
name: backend-primary
5
ownerReferences:
6
- apiVersion: flagger.app/v1beta1
7
blockOwnerDeletion: true
8
controller: true
9
kind: Canary
10
name: backend
11
uid: 2ca1a9c7-2ef6-11e9-bd01-42010a9c0145
12
spec:
13
type: ClusterIP
14
ports:
15
- name: http
16
port: 9898
17
protocol: TCP
18
targetPort: 9898
19
selector:
20
app: backend-primary
Copied!
Flagger works for user facing apps exposed outside the cluster via an ingress gateway and for backend HTTP APIs that are accessible only from inside the mesh.
If
Delegation is enabled, Flagger would generate Istio VirtualService without hosts and gateway, making the service compatible with Istio delegation.1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
metadata:
4
name: backend
5
namespace: test
6
spec:
7
service:
8
delegation: true
9
port: 9898
10
targetRef:
11
apiVersion: v1
12
kind: Deployment
13
name: podinfo
14
analysis:
15
interval: 15s
16
threshold: 15
17
maxWeight: 30
18
stepWeight: 10
Copied!
Based on the above spec, Flagger will create the following virtual service:
1
apiVersion: networking.istio.io/v1alpha3
2
kind: VirtualService
3
metadata:
4
name: backend
5
namespace: test
6
ownerReferences:
7
- apiVersion: flagger.app/v1beta1
8
blockOwnerDeletion: true
9
controller: true
10
kind: Canary
11
name: backend
12
uid: 58562662-5e10-4512-b269-2b789c1b30fe
13
spec:
14
http:
15
- route:
16
- destination:
17
host: podinfo-primary
18
weight: 100
19
- destination:
20
host: podinfo-canary
21
weight: 0
Copied!
Therefore, the following virtual service forwards the traffic to
/podinfo by the above delegate VirtualService.1
apiVersion: networking.istio.io/v1alpha3
2
kind: VirtualService
3
metadata:
4
name: frontend
5
namespace: test
6
spec:
7
gateways:
8
- public-gateway.istio-system.svc.cluster.local
9
- mesh
10
hosts:
11
- frontend.example.com
12
- frontend
13
http:
14
- match:
15
- uri:
16
prefix: /podinfo
17
rewrite:
18
uri: /
19
delegate:
20
name: backend
21
namespace: test
Copied!
Note that pilot env
PILOT_ENABLE_VIRTUAL_SERVICE_DELEGATE must also be set. For the use of Istio Delegation, you can refer to the documentation of Virtual Service and pilot environment variables.Istio Ingress Gateway
How can I expose multiple canaries on the same external domain?
Assuming you have two apps -- one that serves the main website and one that serves its REST API -- you can define a canary object for each app as:
1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
metadata:
4
name: website
5
spec:
6
service:
7
port: 8080
8
gateways:
9
- public-gateway.istio-system.svc.cluster.local
10
hosts:
11
- my-site.com
12
match:
13
- uri:
14
prefix: /
15
rewrite:
16
uri: /
17
---
18
apiVersion: flagger.app/v1beta1
19
kind: Canary
20
metadata:
21
name: webapi
22
spec:
23
service:
24
port: 8080
25
gateways:
26
- public-gateway.istio-system.svc.cluster.local
27
hosts:
28
- my-site.com
29
match:
30
- uri:
31
prefix: /api
32
rewrite:
33
uri: /
Copied!
Based on the above configuration, Flagger will create two virtual services bounded to the same ingress gateway and external host. Istio Pilot will merge the two services and the website rule will be moved to the end of the list in the merged configuration.
Note that host merging only works if the canaries are bounded to an ingress gateway other than the
mesh gateway.Istio Mutual TLS
How can I enable mTLS for a canary?
When deploying Istio with global mTLS enabled, you have to set the TLS mode to
ISTIO_MUTUAL:1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
spec:
4
service:
5
trafficPolicy:
6
tls:
7
mode: ISTIO_MUTUAL
Copied!
If you run Istio in permissive mode, you can disable TLS:
1
apiVersion: flagger.app/v1beta1
2
kind: Canary
3
spec:
4
service:
5
trafficPolicy:
6
tls:
7
mode: DISABLE
Copied!
If Flagger is outside of the mesh, how can it start the load test?
In order for Flagger to be able to call the load tester service from outside the mesh, you need to disable mTLS:
1
apiVersion: networking.istio.io/v1beta1
2
kind: DestinationRule
3
metadata:
4
name: flagger-loadtester
5
namespace: test
6
spec:
7
host: "flagger-loadtester.test.svc.cluster.local"
8
trafficPolicy:
9
tls:
10
mode: DISABLE
11
---
12
apiVersion: security.istio.io/v1beta1
13
kind: PeerAuthentication
14
metadata:
15
name: flagger-loadtester
16
namespace: test
17
spec:
18
selector:
19
matchLabels:
20
app: flagger-loadtester
21
mtls:
22
mode: DISABLE
Copied!
ExternalDNS
Can I use annotations?
Flagger propagates annotations (and labels) to all the generated apex, primary and canary objects. This allows using external-dns annotations.
You can configure Flagger to set annotations with:
1
spec:
2
service:
3
apex:
4
annotations:
5
external-dns.alpha.kubernetes.io/hostname: "mydomain.com"
6
primary:
7
annotations:
8
external-dns.alpha.kubernetes.io/hostname: "primary.mydomain.com"
9
canary:
10
annotations:
11
external-dns.alpha.kubernetes.io/hostname: "canary.mydomain.com"
Copied!
Multiple sources and Istio
/!\ The apex annotations are added to both the generated Kubernetes Services and the generated Istio VirtualServices objects. If you have configured external-dns to use both sources, this will create conflicts!
1
spec:
2
containers:
3
args:
4
- --source=service # choose only one
5
- --source=istio-virtualservice # of these two
Copied!
Last modified 10d ago