Which deployment strategies are supported by Flagger?
Flagger implements the following deployment strategies:
When should I use A/B testing instead of progressive traffic shifting?
For frontend applications that require session affinity, you should use HTTP headers or cookie match conditions to ensure a set of users will stay on the same version for the whole duration of the canary analysis.
Can I use Flagger to manage applications that live outside of a service mesh?
For applications that are not deployed on a service mesh, Flagger can orchestrate Blue/Green style deployments with Kubernetes L4 networking.
When can I use traffic mirroring?
Traffic mirroring can be used for Blue/Green deployment strategy or a pre-stage in a Canary release. Traffic mirroring will copy each incoming request, sending one request to the primary and one to the canary service. Mirroring should be used for requests that are idempotent or capable of being processed twice (once by the primary and once by the canary).
How to retry a failed release?
A canary analysis is triggered by changes in any of the following objects:
How to change replicas for a deployment when not using HPA?
To change replicas for a deployment when not using HPA, you have to update the canary deployment with the desired replica count and trigger an analysis by annotating the template. After the analysis finishes, Flagger will promote the spec.replicas changes to the primary deployment.
Why is there a window of downtime during the canary initializing process when analysis is disabled?
A window of downtime is the intended behavior when the analysis is disabled. This allows instant rollback and also mimics the way a Kubernetes deployment initialization works. To avoid this, enable the analysis (skipAnalysis: false), wait for the initialization to finish, and disable it afterward (skipAnalysis: true).
How to disable cross namespace references?
Flagger by default can access resources across namespaces (AlertProivder, MetricProvider and Gloo Upsteream). If you're in a multi-tenant environment and wish to disable this, you can do so through the no-cross-namespace-refs flag.
flagger \
-no-cross-namespace-refs=true \
...
Kubernetes services
How is an application exposed inside the cluster?
Assuming the app name is podinfo, you can define a canary like:
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: podinfo
namespace: test
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: podinfo
service:
# service name (optional)
name: podinfo
# ClusterIP port number (required)
port: 9898
# container port name or number
targetPort: http
# port name can be http or grpc (default http)
portName: http
If the service.name is not specified, then targetRef.name is used for the apex domain and canary/primary services name prefix. You should treat the service name as an immutable field; changing its could result in routing conflicts.
Based on the canary spec service, Flagger generates the following Kubernetes ClusterIP service:
The podinfo-canary.test:9898 address is available only during the canary analysis and can be used for conformance testing or load testing.
Multiple ports
My application listens on multiple ports. How can I expose them inside the cluster?
If port discovery is enabled, Flagger scans the deployment spec and extracts the containers ports excluding the port specified in the canary service and Envoy sidecar ports. These ports will be used when generating the ClusterIP services.
You can enable port discovery so that Prometheus will be able to reach port 9090 over mTLS:
apiVersion: flagger.app/v1beta1
kind: Canary
spec:
service:
# container port used for canary analysis
port: 8080
# port name can be http or grpc (default http)
portName: http
# add all the other container ports
# to the ClusterIP services (default false)
portDiscovery: true
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
Both port 8080 and 9090 will be added to the ClusterIP services.
Label selectors
What labels selectors are supported by Flagger?
The target deployment must have a single label selector in the format app: <DEPLOYMENT-NAME>:
Besides app, Flagger supports name and app.kubernetes.io/name selectors. If you use a different convention, you can specify your label with the -selector-labels flag. For example:
Flagger will rewrite the first value in each match expression, defined in the target deployment's pod anti-affinity and topology spread constraints, satisfying the following two requirements when creating, or updating, the primary deployment:
The key in the match expression must be one of the labels specified by the parameter selector-labels. The default labels are app,name,app.kubernetes.io/name.
The value must match the name of the target deployment.
The rewrite done by Flagger in these cases is to suffix the value with -primary. This rewrite can be used to spread the pods created by the canary and primary deployments across different availability zones.
Note that the metric interval should be lower or equal to the control loop interval.
Can I use custom metrics?
Istio Gateway API
If you're using Istio with Gateway API, the Prometheus query needs to include reporter="source". For example, to calculate HTTP requests error percentage, the query would be:
Flagger creates an Istio Virtual Service and Destination Rules based on the Canary service spec. The service configuration lets you expose an app inside or outside the mesh. You can also define traffic policies, HTTP match conditions, URI rewrite rules, CORS policies, timeout and retries.
The following spec exposes the frontend workload inside the mesh on frontend.test.svc.cluster.local:9898 and outside the mesh on frontend.example.com. You'll have to specify an Istio ingress gateway for external hosts.
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: frontend
namespace: test
spec:
service:
# container port
port: 9898
# service port name (optional, will default to "http")
portName: http-frontend
# Istio gateways (optional)
gateways:
- istio-system/public-gateway
- mesh
# Istio virtual service host names (optional)
hosts:
- frontend.example.com
# Istio traffic policy
trafficPolicy:
tls:
# use ISTIO_MUTUAL when mTLS is enabled
mode: DISABLE
# HTTP match conditions (optional)
match:
- uri:
prefix: /
# HTTP rewrite (optional)
rewrite:
uri: /
# Istio retry policy (optional)
retries:
attempts: 3
perTryTimeout: 1s
retryOn: "gateway-error,connect-failure,refused-stream"
# Add headers (optional)
headers:
request:
add:
x-some-header: "value"
# cross-origin resource sharing policy (optional)
corsPolicy:
allowOrigin:
- example.com
allowMethods:
- GET
allowCredentials: false
allowHeaders:
- x-some-header
maxAge: 24h
For the above spec Flagger will generate the following virtual service:
Flagger keeps in sync the virtual service and destination rules with the canary service spec. Any direct modification to the virtual service spec will be overwritten.
To expose a workload inside the mesh on http://backend.test.svc.cluster.local:9898, the service spec can contain only the container port and the traffic policy:
Flagger works for user facing apps exposed outside the cluster via an ingress gateway and for backend HTTP APIs that are accessible only from inside the mesh.
If Delegation is enabled, Flagger would generate Istio VirtualService without hosts and gateway, making the service compatible with Istio delegation.
/!\ The apex annotations are added to both the generated Kubernetes Services and the generated Istio VirtualServices objects. If you have configured external-dns to use both sources, this will create conflicts!
spec:
containers:
args:
- --source=service # choose only one
- --source=istio-virtualservice # of these two
The analysis can be extended with metrics provided by Prometheus, Datadog, AWS CloudWatch, New Relic and Graphite. For more details on how custom metrics can be used, please read the .
Note that pilot env PILOT_ENABLE_VIRTUAL_SERVICE_DELEGATE must also be set. For the use of Istio Delegation, you can refer to the documentation of and .
Based on the above configuration, Flagger will create two virtual services bounded to the same ingress gateway and external host. Istio Pilot will the two services and the website rule will be moved to the end of the list in the merged configuration.