Flagger Install on GKE Istio

Search…
This guide walks you through setting up Flagger and Istio on Google Kubernetes Engine.
GKE Cluster Overview
Prerequisites
You will be creating a cluster on Google’s Kubernetes Engine (GKE), if you don’t have an account you can sign up here for free credits.
Login into Google Cloud, create a project and enable billing for it.
Install the gcloud command line utility and configure your project with gcloud init.
Set the default project (replace PROJECT_ID with your own project):
1
gcloud config set project PROJECT_ID
Copied!
Set the default compute region and zone:
1
gcloud config set compute/region us-central1
2
gcloud config set compute/zone us-central1-a
Copied!
Enable the Kubernetes and Cloud DNS services for your project:
1
gcloud services enable container.googleapis.com
2
gcloud services enable dns.googleapis.com
Copied!
Install the kubectl command-line tool:
1
gcloud components install kubectl
Copied!
GKE cluster setup
Create a cluster with the Istio add-on:
1
K8S_VERSION=$(gcloud container get-server-config --format=json \
2
| jq -r '.validMasterVersions[0]')
3
​
4
gcloud beta container clusters create istio \
5
--cluster-version=${K8S_VERSION} \
6
--zone=us-central1-a \
7
--num-nodes=2 \
8
--machine-type=n1-highcpu-4 \
9
--preemptible \
10
--no-enable-cloud-logging \
11
--no-enable-cloud-monitoring \
12
--disk-size=30 \
13
--enable-autorepair \
14
--addons=HorizontalPodAutoscaling,Istio \
15
--istio-config=auth=MTLS_PERMISSIVE
Copied!
The above command will create a default node pool consisting of two n1-highcpu-4 (vCPU: 4, RAM 3.60GB, DISK: 30GB) preemptible VMs. Preemptible VMs are up to 80% cheaper than regular instances and are terminated and replaced after a maximum of 24 hours.
Set up credentials for kubectl:
1
gcloud container clusters get-credentials istio
Copied!
Create a cluster admin role binding:
1
kubectl create clusterrolebinding "cluster-admin-$(whoami)" \
2
--clusterrole=cluster-admin \
3
--user="$(gcloud config get-value core/account)"
Copied!
Validate your setup with:
1
kubectl -n istio-system get svc
Copied!
In a couple of seconds GCP should allocate an external IP to the istio-ingressgateway service.
Cloud DNS setup
You will need an internet domain and access to the registrar to change the name servers to Google Cloud DNS.
Create a managed zone named istio in Cloud DNS (replace example.com with your domain):
1
gcloud dns managed-zones create \
2
--dns-name="example.com." \
3
--description="Istio zone" "istio"
Copied!
Look up your zone's name servers:
1
gcloud dns managed-zones describe istio
Copied!
Update your registrar's name server records with the records returned by the above command.
Wait for the name servers to change (replace example.com with your domain):
1
watch dig +short NS example.com
Copied!
Create a static IP address named istio-gateway using the Istio ingress IP:
1
export GATEWAY_IP=$(kubectl -n istio-system get svc/istio-ingressgateway -ojson \
2
| jq -r .status.loadBalancer.ingress[0].ip)
3
​
4
gcloud compute addresses create istio-gateway --addresses ${GATEWAY_IP} --region us-central1
Copied!
Create the following DNS records (replace example.com with your domain):
1
DOMAIN="example.com"
2
​
3
gcloud dns record-sets transaction start --zone=istio
4
​
5
gcloud dns record-sets transaction add --zone=istio \
6
--name="${DOMAIN}" --ttl=300 --type=A ${GATEWAY_IP}
7
​
8
gcloud dns record-sets transaction add --zone=istio \
9
--name="www.${DOMAIN}" --ttl=300 --type=A ${GATEWAY_IP}
10
​
11
gcloud dns record-sets transaction add --zone=istio \
12
--name="*.${DOMAIN}" --ttl=300 --type=A ${GATEWAY_IP}
13
​
14
gcloud dns record-sets transaction execute --zone istio
Copied!
Verify that the wildcard DNS is working (replace example.com with your domain):
1
watch host test.example.com
Copied!
Install Helm
Install the Helm command-line tool:
1
brew install kubernetes-helm
Copied!
Create a service account and a cluster role binding for Tiller:
1
kubectl -n kube-system create sa tiller
2
​
3
kubectl create clusterrolebinding tiller-cluster-rule \
4
--clusterrole=cluster-admin \
5
--serviceaccount=kube-system:tiller
Copied!
Deploy Tiller in the kube-system namespace:
1
helm init --service-account tiller
Copied!
You should consider using SSL between Helm and Tiller, for more information on securing your Helm installation see docs.helm.sh.
Install cert-manager
Jetstack's cert-manager is a Kubernetes operator that automatically creates and manages TLS certs issued by Let’s Encrypt.
You'll be using cert-manager to provision a wildcard certificate for the Istio ingress gateway.
Install cert-manager's CRDs:
1
CERT_REPO=https://raw.githubusercontent.com/jetstack/cert-manager
2
​
3
kubectl apply -f ${CERT_REPO}/release-0.10/deploy/manifests/00-crds.yaml
Copied!
Create the cert-manager namespace and disable resource validation:
1
kubectl create namespace cert-manager
2
​
3
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
Copied!
Install cert-manager with Helm:
1
helm repo add jetstack https://charts.jetstack.io && \
2
helm repo update && \
3
helm upgrade -i cert-manager \
4
--namespace cert-manager \
5
--version v0.10.0 \
6
jetstack/cert-manager
Copied!
Istio Gateway TLS setup
Istio Let's Encrypt
Create a generic Istio Gateway to expose services outside the mesh on HTTPS:
1
REPO=https://raw.githubusercontent.com/fluxcd/flagger/main
2
​
3
kubectl apply -f ${REPO}/artifacts/gke/istio-gateway.yaml
Copied!
Create a service account with Cloud DNS admin role (replace my-gcp-project with your project ID):
1
GCP_PROJECT=my-gcp-project
2
​
3
gcloud iam service-accounts create dns-admin \
4
--display-name=dns-admin \
5
--project=${GCP_PROJECT}
6
​
7
gcloud iam service-accounts keys create ./gcp-dns-admin.json \
8
--iam-account=[email protected]${GCP_PROJECT}.iam.gserviceaccount.com \
9
--project=${GCP_PROJECT}
10
​
11
gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
12
--member=serviceAccount:[email protected]${GCP_PROJECT}.iam.gserviceaccount.com \
13
--role=roles/dns.admin
Copied!
Create a Kubernetes secret with the GCP Cloud DNS admin key:
1
kubectl create secret generic cert-manager-credentials \
2
--from-file=./gcp-dns-admin.json \
3
--namespace=istio-system
Copied!
Create a letsencrypt issuer for CloudDNS (replace [email protected] with a valid email address and my-gcp-projectwith your project ID):
1
apiVersion: certmanager.k8s.io/v1alpha1
2
kind: Issuer
3
metadata:
4
  name: letsencrypt-prod
5
  namespace: istio-system
6
spec:
7
  acme:
8
    server: https://acme-v02.api.letsencrypt.org/directory
9
    email: [email protected]
10
    privateKeySecretRef:
11
      name: letsencrypt-prod
12
    dns01:
13
      providers:
14
      - name: cloud-dns
15
        clouddns:
16
          serviceAccountSecretRef:
17
            name: cert-manager-credentials
18
            key: gcp-dns-admin.json
19
          project: my-gcp-project
Copied!
Save the above resource as letsencrypt-issuer.yaml and then apply it:
1
kubectl apply -f ./letsencrypt-issuer.yaml
Copied!
Create a wildcard certificate (replace example.com with your domain):
1
apiVersion: certmanager.k8s.io/v1alpha1
2
kind: Certificate
3
metadata:
4
  name: istio-gateway
5
  namespace: istio-system
6
spec:
7
  secretName: istio-ingressgateway-certs
8
  issuerRef:
9
    name: letsencrypt-prod
10
  commonName: "*.example.com"
11
  acme:
12
    config:
13
    - dns01:
14
        provider: cloud-dns
15
      domains:
16
      - "*.example.com"
17
      - "example.com"
Copied!
Save the above resource as istio-gateway-cert.yaml and then apply it:
1
kubectl apply -f ./istio-gateway-cert.yaml
Copied!
In a couple of seconds cert-manager should fetch a wildcard certificate from letsencrypt.org:
1
kubectl -n istio-system describe certificate istio-gateway
2
​
3
Events:
4
  Type    Reason         Age    From          Message
5
  ----    ------         ----   ----          -------
6
  Normal  CertIssued     1m52s  cert-manager  Certificate issued successfully
Copied!
Recreate Istio ingress gateway pods:
1
kubectl -n istio-system get pods -l istio=ingressgateway
Copied!
Note that Istio gateway doesn't reload the certificates from the TLS secret on cert-manager renewal. Since the GKE cluster is made out of preemptible VMs the gateway pods will be replaced once every 24h, if your not using preemptible nodes then you need to manually delete the gateway pods every two months before the certificate expires.
Install Prometheus
The GKE Istio add-on does not include a Prometheus instance that scrapes the Istio telemetry service. Because Flagger uses the Istio HTTP metrics to run the canary analysis you have to deploy the following Prometheus configuration that's similar to the one that comes with the official Istio Helm chart.
Find the GKE Istio version with:
1
kubectl -n istio-system get deploy istio-pilot -oyaml | grep image:
Copied!
Install Prometheus in istio-system namespace:
1
kubectl -n istio-system apply -f \
2
https://storage.googleapis.com/gke-release/istio/release/1.0.6-gke.3/patches/install-prometheus.yaml
Copied!
Install Flagger and Grafana
Add Flagger Helm repository:
1
helm repo add flagger https://flagger.app
Copied!
Install Flagger's Canary CRD:
1
kubectl apply -f https://raw.githubusercontent.com/fluxcd/flagger/main/artifacts/flagger/crd.yaml
Copied!
Deploy Flagger in the istio-system namespace with Slack notifications enabled:
1
helm upgrade -i flagger flagger/flagger \
2
--namespace=istio-system \
3
--set crd.create=false \
4
--set metricsServer=http://prometheus.istio-system:9090 \
5
--set slack.url=https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK \
6
--set slack.channel=general \
7
--set slack.user=flagger
Copied!
Deploy Grafana in the istio-system namespace:
1
helm upgrade -i flagger-grafana flagger/grafana \
2
--namespace=istio-system \
3
--set url=http://prometheus.istio-system:9090 \
4
--set user=admin \
5
--set password=replace-me
Copied!
Expose Grafana through the public gateway by creating a virtual service (replace example.com with your domain):
1
apiVersion: networking.istio.io/v1alpha3
2
kind: VirtualService
3
metadata:
4
  name: grafana
5
  namespace: istio-system
6
spec:
7
  hosts:
8
  - "grafana.example.com"
9
  gateways:
10
  - public-gateway.istio-system.svc.cluster.local
11
  http:
12
  - route:
13
    - destination:
14
        host: flagger-grafana
Copied!
Save the above resource as grafana-virtual-service.yaml and then apply it:
1
kubectl apply -f ./grafana-virtual-service.yaml
Copied!
Navigate to http://grafana.example.com in your browser and you should be redirected to the HTTPS version.
Install - Previous
Flagger Install on Kubernetes
Next - Install
Flagger Install on EKS App Mesh
Last modified 8mo ago
Copy link
Contents
Istio Gateway TLS setup
Install Prometheus
Install Flagger and Grafana