As part of the analysis process, Flagger can validate service level objectives (SLOs) like availability, error rate percentage, average response time and any other objective based on app specific metrics. If a drop in performance is noticed during the SLOs analysis, the release will be automatically rolled back with minimum impact to end-users.
Builtin metrics
Flagger comes with two builtin metric checks: HTTP request success rate and duration.
For each metric you can specify a range of accepted values with thresholdRange and the window size or the time series with interval. The builtin checks are available for every service mesh / ingress controller and are implemented with .
Custom metrics
The canary analysis can be extended with custom metric checks. Using a MetricTemplate custom resource, you configure Flagger to connect to a metric provider and run a query that returns a float64 value. The query result is used to validate the canary based on the specified threshold range.
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
name: my-metric
spec:
provider:
type: # can be prometheus, datadog, etc
address: # API URL
insecureSkipVerify: # if set to true, disables the TLS cert validation
secretRef:
name: # name of the secret containing the API credentials
query: # metric query
The following variables are available in query templates:
A canary analysis metric can reference a template with templateRef:
analysis:
metrics:
- name: "my metric"
templateRef:
name: my-metric
# namespace is optional
# when not specified, the canary namespace will be used
namespace: flagger
# accepted values
thresholdRange:
min: 10
max: 1000
# metric query time window
interval: 1m
A canary analysis metric can reference a set of custom variables with templateVariables. These variables will be then injected into the query defined in the referred MetricTemplate object during canary analysis:
analysis:
metrics:
- name: "my metric"
templateRef:
name: my-metric
namespace: flagger
# accepted values
thresholdRange:
min: 10
max: 1000
# metric query time window
interval: 1m
# custom variables used within the referenced metric template
templateVariables:
direction: inbound
The above configuration validates the canary by checking if the HTTP 404 req/sec percentage is below 5 percent of the total traffic. If the 404s rate reaches the 5% threshold, then the canary fails.
If your Prometheus API requires basic authentication, you can create a secret in the same namespace as the MetricTemplate with the basic-auth credentials:
If your Graphite API requires basic authentication, you can create a secret in the same namespace as the MetricTemplate with the basic-auth credentials:
type (required): Must be either keptnmetric or analysis.
namespace (required): The namespace of the referenced KeptnMetric/AnalysisDefinition.
resource-name (required): The name of the referenced KeptnMetric/AnalysisDefinition.
timeframe (optional): The timeframe used for the Analysis. This will usually be set to the same value as the analysis interval of a Canary. Only relevant if the type is set to analysis.
arguments (optional): Arguments to be passed to an Analysis. Arguments are passed as a list of key value pairs, separated by ; characters, e.g. foo=bar;bar=foo. Only relevant if the type is set to analysis.
For the type analysis, the value returned by the provider is either 0 (if the analysis failed), or 1 (analysis passed).
Splunk
You can create custom metric checks using the Splunk provider.
Create a secret that contains your authentication token that can be found in the Splunk o11y UI.
The above template is for gRPC services instrumented with .
The query format documentation can be found .
Enable Workload Identity on your cluster, create a service account key that has read access to the Cloud Monitoring API and then create an IAM policy binding between the GCP service account and the Flagger service account on Kubernetes. You can take a look at this
Create a secret that contains your project-id (and, if workload identity is not enabled on your cluster, your ).
Then reference the secret in the metric template. Note: The particular MQL query used here works if .
The reference for the query language can be found
The InfluxDB provider uses the query language.
Note: The particular MQL query used here works if .
You can create custom metric checks using the Keptn provider. This Provider allows to verify either the value of a single , representing the value of a single metric, or of a , which provides a flexible grading logic for analysing and prioritising a number of different metric values coming from different data sources.
This provider requires to be installed in the cluster.