Helmfile is a declarative spec and client tool that enables better composition, configuration and deployment of multiple Helm charts, than Umbrellas.
The daily mood
If find it a great practice that my peers often use our team call to share what they are about to present to our internal customers, that is to say our Product Management (PM) team or Chief Technology Officer (CTO). They usually show only 2-3 slides from a larger presentation for which they are unsure or required feedback. The occasion for me to get familiar with diverse topics e.g. API versioning lifecycle, Security, Observability. I am also impressed how fast the other members of the team can connect with a core problem or solution without a reminder on the context, and then ask important questions. In my experience of customer-facing, presentations always started with a hook and reference, in order to get attention and credibility on what you are going to present next (cf. Great Demo from Peter Cohan, Talk like Ted from Carmin Gallo). Now we are in a trusted group of people taking the context and approach for granted. As a consequence, I not only find it difficult extremely difficult to get the point, where it comes from and where it goes to, but also nearly impossible to engage actively. For now.
One day it will be my turn to be the presenter. I have installed LibreOffice 7.0.0 Beta1 which is finally available from developer releases. Beside a minimal performance improvement and rework on look-and-feel, I actually wanted to check if it better supports our latest corporate presentation template, which is obviously designed for Microsoft Powerpoint users. Font sizes look good, but there are still a couple of issues with block positions like titles and subtitles overlapping each other. I'll better go without using subtitles, otherwise edit and present via the Browser (Sharepoint Online) or a Windows VM.
Regarding my ramp-up, I am now looking at helmfile in continuation of my initiative to catch-up on fundamental tooling for Kubernetes.
Use-case
Suppose your application requires to deploy both a Helm chart "A" packaging resources "1-3" and a Helm chart "B" packaging resources "4-6". Especially, you want to control the chart/package version to deploy, the target namespaces and the replicaCounts per deployment. You might consider several options for configuration and deployment:
- Run each deployment step by hand, but then a third and a fourth one will come, so that it will get not only effort-intensive but also error-prone.
- Write a custom script (ex. Shell, Ansible), but it will never be 100% convenient and you are the only maintainer.
- Create a so called umbrella chart C listing dependencies A and B as part of its requirement.yaml (see example further below), but there is too much potential for bad practices and unexpected configuration behaviours, as we will show further below.
These are excatly the problems that helmfile (and by the way, our internal Helm custom plugin as well) try to solve. Through a central, semi-declarative spec, helm chart configurations and deployments can be automated at scale.
This post will compare the umbrella chart approach with helmfile.
Requirements
- Helm client with diff plugin (cf. https://github.com/roboll/helmfile/issues/391)
- Helm repository (cf. previous post)
- Helmfile client utility
Note: We are going to explore a bit further Helm template approach, therefore it might also be a good idea to setup IntelliJ IDEA with Helm support plugin.
Image source: Jetbrains blog
Preparation
Create a new project with the following structure:
$ tree
.
├── charts ├── config
├── commons
├── helmfile.yaml
└── releases
Now create some Helm charts under charts folder. Note that we'll make deployment namespace configurable.
$ helm create charts/chart-a
$ sed -i '3s/^/namespace: {{ .Values.namespace | default "default" }}\n/' charts/chart-a/templates/deployment.yaml# same for chart-b and chart-c.
Build 2 versions of chart-a and chart-b, then publish them to your local Helm registry
$ helm package charts/chart-a $ sed -i 's/version: 0.1.0/version: 0.1.1/' charts/chart-a/Chart.yaml $ helm package --app-version 1.1 charts/chart-a
# same for chart-b only (we don't need chart-c in repository).
$ mv *.tgz ~/.helm/repository/local/ $ helm repo index ~/.helm/repository/local # +diff local/index.yaml cache/local-index.yaml
$ helm search local/chart- NAME CHART VERSION APP VERSION DESCRIPTION local/chart-a 0.1.1 1.1 A Helm chart for Kubernetes local/chart-b 0.1.1 1.1 A Helm chart for Kubernetes
Umbrella chart
Configure umbrella chart dependencies:
$ cd charts/chart-c $ vi charts/chart-c/requirements.yaml
dependencies:
- name: chart-a
version: 0.1.0
repository: "@local"
- name: chart-b
version: 0.1.1
repository: "@local"
Observe behaviour:
$ helm dep up # sub chart-a and sub chart-b downloaded under charts
$ helm template .
manifest svc-a
..
manifest svc-b
..
manifest svc-c
..
manifest test-a
..
etc.As we can see, charts are rendered in a special order which might not be what we need for our application. The Helm Spray plugin helps controling in which order ressources are deployed, based on dedicated "weights" defined as part of the umbrella chart values.Re-write umbrella values and observe overriding behaviour:
A helmfile.yaml is basically an ordered list of Helm chart releases, just with more capabilities.
$ rm -rf templates/tests
$ rm values.yaml && vi values.yaml
replicaCount: 2 # set parent chart values templated as {{ .Values.replicaCount }}
chart-a:
replicaCount: 3 # set sub chart-a values templated as {{ .Values.replicaCount }}
global:
replicaCount: 4 # override any chart values templated as {{ .Values.global.replicaCount }}
image:
repository: nginx
tag: stable
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
ingress:
enabled: false
$ helm template . | grep "replicas:" replicas: 3 # chart-a
replicas: 1 # chart-b
replicas: 2 # chart-cNote that global values are ignored unless all chart templates are re-written accordingly. Also, it might not be clear for the developper which value is taken in case we have multiple levels of umbrellas, like in a multi-stage environment configuration.
Helmfile
Helmfile is a wrapper utility to helm. The project is still young but actively used and maintained by a few large organizations like Adobe, AmercianExpress and Reddit (watch this talk).
$ vi helmfile-1.yaml helmfiles: - "commons/repositories.yaml" - "releases/helmfile-1.1.yaml"
$ vi commons/repositories.yaml repositories: - name: local url: "http://127.0.0.1:8879/charts"
$ vi releases/helmfile-1.1.yaml releases:
- name: release-a
chart: "local/chart-a"
version: 0.1.0 namespace: default values: - replicaCount: 2 # same for release-b
Release values allow overriding chart values, e.g. number of replicas from 1 (Helm chart) to 2 (Helmfile).
$ helm template charts/chart-a | grep replicas:
replicas: 1 $ helmfile -f helmfile-1.yaml template | grep replicas: replicas: 2
With this, we have configured a Helmfile release on top of a Helm chart, which is again on top of over definitions. Short we have added a configuration layer, thus we cannot yet see any value.
| 4 | Helmfile release |
| 3 | Helm chart values |
| 2 | Kubernetes configmap/secrets |
| 1 | Docker image environment |
One great feature of helmfile is the support the comprehensive support for multi-stage environments
$ vi helmfile-2.yaml environments: default: production: --- helmfiles: - "commons/repositories.yaml" - "releases/helmfile-2.1.yaml"
$ vi helmfile-2.1.yaml
environments:
default:
values:
- ../config/values-default.yaml
production:
values:
- ../config/values-production.yaml
---
releases:
- name: {{ .Environment.Name }}-release-a
chart: "local/chart-a"
version: 0.1.0
namespace: {{ .Environment.Values.namespace }}
values:
- ../config/values-{{ .Environment.Name }}.yaml
# same for release-b
$ vi config/values-default.yaml
replicaCount: 4
namespace: default
$ vi config/values-production.yaml
replicaCount: 6
namespace: apps
$ helmfile -f helmfile-2.yaml template | grep "replicas:\|namespace:"
namespace: default
replicas: 4
$ helmfile -f helmfile-2.yaml -e production template | grep "replicas:\|namespace:"
namespace: apps
replicas: 6As we can see, Helm template syntax can be applied inside Helmfiles as well. With this, we can maintain different configurations depending on the environment. Note that the following variant is even better than the above, in case you handle multiple release files and need to add a new environment.
$ vi helmfile-3.yaml
environments:
default:
production:
---
helmfiles:
- "commons/repositories.yaml"
- path: "releases/helmfile-3.1.yaml"
$ vi releases/helmfile-3.1.yaml
bases:
- environments.yaml
---
releases:
- name: {{ .Environment.Name }}-release-a
chart: "local/chart-a" version: 0.1.0 namespace: {{ .Environment.Values.namespace }}
values:
- ../config/values-{{ .Environment.Name }}.yaml
# same for release-b
$ vi releases/environments.yaml
environments:
default:
values:
- ../config/values-default.yaml
production:
values:
- ../config/values-production.yaml
Now let us deploy the stack based on our helmfile spec.
helmfile -f helmfile-3.yaml apply # sync for install or upgrade
We need to make sure that no release with same name has failed before otherwise helmfile would run into error (cf. https://github.com/roboll/helmfile/issues/471). This can be avoided by hand via helm del --purge <release name> or automatically via helm native parameter --atomic.
Using Helmfile environments forces you to adopt a structure that is highly configurable and extensible, thus we want to avoid having to duplicate release patterns over and over again (cf. DRY). Here comes an advanced feature handy: Helmfile release templates. Like Helm chart templates, Helmfile release templates are based on Go template language. Note the embedded executions via reverse quotes.
Using Helmfile environments forces you to adopt a structure that is highly configurable and extensible, thus we want to avoid having to duplicate release patterns over and over again (cf. DRY). Here comes an advanced feature handy: Helmfile release templates. Like Helm chart templates, Helmfile release templates are based on Go template language. Note the embedded executions via reverse quotes.
$ vi helmfile-4.yaml environments: default: production: --- helmfiles: - "commons/helmDefaults.yaml" - "commons/repositories.yaml" - path: "releases/helmfile-4.1.yaml" $ vi commons/helmDefaults.yaml helmDefaults: atomic: true cleanupOnFail: true wait: true
$ vi releases/helmfile-4.1.yaml
bases:
- environments.yaml
templates:
default: &my-tmpl
chart: local/{{`{{ .Release.Name }}`}}
values:
- ../config/values-{{ .Environment.Name }}.yaml
releases:
- name: "chart-a"
version: 0.1.0
<<: *my-tmpl
- name: "chart-b"
version: 0.1.1
<<: *my-tmplNote the following 2 pitfalls:
- Triple dash is not supported as resource separator when using release templates. This is because the feature is based on YAML anchors, where &my-tmpl is the anchor and *my-tmpl is the alias. So instead of dashes, we leave an empty line between blocks.
- Map has no entry for key "namespace" is the error you get if including it to the template. You can actually leave it and it will be rendered correctly anyway
Last but not least, the actual recommendation is to define specific product/app/chart folders under releases and config, so that their application values do not mix up with infrastructure values.
$ vi helmfile-5.yaml
environments:
default:
production:
---
helmfiles:
- "commons/helmDefaults.yaml"
- "commons/repositories.yaml"
- path: releases/*/helmfile*.yaml
$ vi releases/nginx/helmfile-5.1.yaml
bases:
- ../environments.yaml
templates:
default: &my-tmpl
chart: local/{{`{{ .Release.Name }}`}}
values:
- ../../config/values-{{ .Environment.Name }}.yaml
- ../../config/{{`{{ .Release.Name }}`}}/values-{{ .Environment.Name }}.yaml
releases:
- name: "chart-a"
<<: *my-tmpl
- name: "chart-b"
<<: *my-tmpl
$ vi releases/nginx/environments.yaml
environments:
default:
values:
- ../../config/values-default.yaml
production:
values:
- ../../config/values-production.yaml
$ vi config/chart-a/values-default.yaml
version: 0.1.0
$ vi config/chart-b/values-default.yaml
version: 0.1.1
$ helmfile -f helmfile-5.yaml template | grep "version:\|replicas:"
app.kubernetes.io/version: "1.1"
replicas: 4
app.kubernetes.io/version: "1.1"
replicas: 4Well unfortunately, this is not quite what I expected. Indeed, I had to recreate the environments.yaml under my application folder and I was not able to create a release of chart-a in app v1.0 in parallel to a release of chart-b in app v1.1. I assume it is possible to fix this but I already have spent more time on Helmfile than I actually wanted. Therefore I will close this topic for now.
Conclusion
Helmfile has the potential to solve a lot of standard problems occuring when it comes to configure and deploy complex applications on a Kubernetes cluster. But developer support (documentation, completion, generation, syntax check, error-handling etc.) is very limited, so that I ended up doing quite a lot of try-and-error in order to find a working configuration. Also, I suspect it to not be covering all major use-cases as it actually should.
Next
I've seen another use-case with Helmfile on top of Kustomize instead of Helm. It could be worth looking at it for a better understanding and comparison.
Source code
References
- Helm subcharts and global values
- Best practice for writing helmfiles
- Standard list of helmfiles
- Documentation of Go template language
- User story from Dailymotion
Comments
Post a Comment