FluxCD GitOps operator supports Kustomize for dynamically featuring Kubernetes resources includind HelmRelease. This post is about auto-deployment.
The daily mood
Another week is over and I have made good progress on this project around Helm chart deployment automation. As described in my last posts, we've been using Flux Helm Operator for abstracting Helm installation, and Kustomize for building custom configurations.
Now we will look at how Flux can integrate with Kustomize. At the end we'll actually get a picture which is not common, indeed we'll use FluxCD GitOps operator in conjonction with Helm operator, but disconnected from each other. Like for a connected mode, the solution should allow to automatically and bi-directionnaly sync a configuration change either in the Git repo (Flux HelmRelease + Kustomize), or on the Kubernetes cluster (Released Helm charts).
Requirements
- An empty Git repository
- Assets from my previous post on Flux Helm Operator + Kustomize
- A Kubernetes Cluster with Helm2/Tiller
- A Flux Helm Operator
- A Kustomize configuration of HelmRelease
Deployment automation
$ git clone http://www.github.com/<my_git_user>/<my_git_repo> $ git add * && git commit -m "initial commit" $ git push --set-upstream origin master
At this stage Flux could submit HelmRelease files pulled from the repository folders passed via --git-path, but does not know yet that those resource objects are actually incomplete since they are just Kustomize patches. This happens with a .flux.yaml resource used to generate and update manifests.$ kubectl delete secret flux-git-deploy # re-create secret from previous dummy$ export GHUSER="<your_github_username>" $ export GHREPO="<your_github_reponame>" $ fluxctl install \ --git-user=${GHUSER} \ --git-email=${GHUSER}@users.noreply.github.com \ --git-url=git@github.com:${GHUSER}/${GHREPO} \ --git-path=microk8s/arch/app \ --manifest-generation=true \ --namespace flux \ | kubectl apply -f - # wait for flux to start $ kubectl -n flux rollout status deployment/flux # generate ssh key $ fluxctl identity --k8s-fwd-ns flux
Note that FluxCD comes with a standalone Kustomize client.
<<TODO: Work in progress >>
$ vi .flux.yaml version: 1 patchUpdated: generators: - command: kustomize build . patchFile: flux-patch.yaml
Note that any further Kustomize patch folder needs to contain a flux-patch.yaml as per the root .flux.yaml definition.
$ vi microk8s/arch/app/flux-patch.yaml --- apiVersion: apps/v1 kind: Deployment metadata: annotations: flux.weave.works/locked: "true" name: ... namespace: arch
fluxctl sync --k8s-fwd-ns flux # sync with git kubectl -n flux logs deploy/flux # watch log fluxctl list-workloads --k8s-fwd-ns flux # list automated resources
Registry access keys
I need to provide my target namespace with a secret key for accessing our private Docker repository. Let us take a look at different scenarios like how to create, replicate or provision them.
Manual creation:
$ kubectl create secret docker-registry my-registry-key \
--docker-server=<docker_server> \
--docker-username=<docker_username> \
--docker-password=<docker_password> \
--docker-email=<docker_email>
$ kubectl patch serviceaccount default -p \
"{\"imagePullSecrets\": [{\"name\": \"my-registry-key\"}]}"Replication:
$ kubectl get secret my-registry-key -n default --export -o yaml \
| kubectl apply -n arch -f -
For provision, you don't want your keys to be stored on the repository in a format that can be easily hacked. There are handy tools like kubedecode and kubernetes-secret-decode available, but let us see how it actually looks like:# this is the yaml of a secret as you may want to store it on a repository $ kubectl get secret my-registry-key -o yaml apiVersion: v1 data: .dockerconfigjson: eyJhdXRocyI6eyJkb2NrZXJfc2VydmVyIjp7InVzZXJuYW1lIjoiZG9ja2VyX3VzZXJuYW1lIiwicGFzc3dvcmQiOiJkb2NrZXJfcGFzc3dvcmQiLCJlbWFpbCI6ImRvY2tlcl9lbWFpbCIsImF1dGgiOiJaRzlqYTJWeVgzVnpaWEp1WVcxbE9tUnZZMnRsY2w5d1lYTnpkMjl5WkE9PSJ9fX0= kind: Secret metadata: creationTimestamp: "2020-06-18T07:31:37Z" name: my-registry-key namespace: arch resourceVersion: "31339" selfLink: /api/v1/namespaces/arch/secrets/my-registry-key uid: 8c71a0d1-3342-44bb-a6b8-dce7474ff9d1 type: kubernetes.io/dockerconfigjson
# secret data is just base64 encoded
$ echo "eyJhdXRocyI6eyJkb2NrZXJfc2VydmVyIjp7InVzZXJuYW1lIjoiZG9ja2VyX3VzZXJuYW1lIiwicGFzc3dvcmQiOiJkb2NrZXJfcGFzc3dvcmQiLCJlbWFpbCI6ImRvY2tlcl9lbWFpbCIsImF1dGgiOiJaRzlqYTJWeVgzVnpaWEp1WVcxbE9tUnZZMnRsY2w5d1lYTnpkMjl5WkE9PSJ9fX0=" | base64 -d
{"auths":{"docker_server":{"username":"docker_username","password":"docker_password","email":"docker_email","auth":"ZG9ja2VyX3VzZXJuYW1lOmRvY2tlcl9wYXNzd29yZA=="}}}Kubernetes secrets are recommended over configmaps because they cannot be read like clear text, but we still need to protect them using a private key, or a seal in case of a shared key. A common solution to this is Bitnami Sealed Secrets.
Conclusion
We now have put the final pieces of our "minimal value puzzle" together. I am also closing this topic for now, although there would be still quite a lot to evaluate and improve.
It looks like Flux requires one FluxCD operator for each Kustomize application. Although this is acceptable for integration, pre-prod and prod clusters, we definitely do not want to maintain that in dev and qa clusters, so that we clearly covered two different use-cases in my previous post and this one.
Unfortunately Kustomize brings too many constraints as compared to a HelmRelease replication approach (without Kustomize). Not only we are replicating objects anyway (base, patches etc.), but we will also have to flag them for FluxCD to know what to install from the base, and what not. With this, we probably not yet have found the right solution to our requirements, so I will continue to look for alternatives.
Alternative approaches
Replicated maintains an open-source project called Kotsadm for packaging Helm charts as a Kubernetes-off-the-shelf (KOTS) application.
Weaveworks paves the link between GitOps and progressive delivery using Flagger for canary/feature-flag deployment automation, and Gatekeeper for namespace policy management.
References
Project sources
Comments
Post a Comment