Take Your Helm Charts to the Next Level

Helm is the package manager for Kubernetes. It reduces the effort of deploying complex applications thanks to its templating approach and rich ecosystem of reusable and production-ready packages, also known as Helm charts. With Helm, you can deploy packaged applications as a collection of versioned, pre-configured Kubernetes resources.

Let's assume you’re deploying a database with Kubernetes—including multiple deployments, containers, secrets, volumes, and services. Helm allows you to install the same database with a single command and a single set of values. Its declarative and idempotent commands makes Helm an ideal tool for continuous delivery (CD).

Helm is a Cloud Native Computing Foundation (CNCF) project created in 2015 and graduated in April 2020. With the latest version of Helm 3, it has become even more integrated into the Kubernetes ecosystem.

This article features 13 best practices for creating Helm charts to manage your applications running in Kubernetes.

1. Take Advantage of the Helm Ecosystem

Helm gives you access to a wealth of community expertise—perhaps the tool’s greatest benefit. It collects charts from developers worldwide, which are then shared through chart repositories. You can check Artifact Hub for available Helm chart repositories.

Once you found a chart repository, you can add it to your local setup as follows:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

Then you can search for charts, for example, MySQL:

$ helm search hub mysql

URL                                                     CHART VERSION   APP VERSION     DESCRIPTION                                       
https://hub.helm.sh/charts/bitnami/mysql                8.6.3           8.0.25          Chart to create a Highly available MySQL cluster

2. Use Subcharts to Manage Your Dependencies

Because applications deployed to Kubernetes consist of fine-grained, interdependent pieces, their Helm charts have various resource templates and dependencies. For instance, let's assume your backend relies on a database and a message queue. The database and message queue are already standalone applications (e.g. PostgreSQL and RabbitMQ). Creating or using separate charts for the standalone applications and adding them to the parent charts is therefore recommended. The dependent applications are named as subcharts here.

There are three essential elements for creating and configuring subcharts:

1. Chart structure The folder structure should be in the following order:

backend-chart
  - Chart.yaml
  - charts
      - message-queue
          - Chart.yaml
          - templates
          - values.yaml
      - database
          - Chart.yaml
          - templates
          - values.yaml
  - values.yaml

2. Chart.yaml

Additionally, the chart.yaml in the parent chart should list any dependencies and conditions:

apiVersion: v2
name: backend-chart
description: A Helm chart for backend
...
dependencies:
  - name: message-queue
    condition: message-queue.enabled
  - name: database
    condition: database.enabled

3. values.yaml

Finally, you can set or override the values of subcharts in the parent chart with the following values.yaml file:

message-queue:
  enabled: true
  image:
    repository: acme/rabbitmq
    tag: latest
database:
  enabled: false

Creating and using subcharts establishes an abstraction layer between the parent and dependency applications. These separate charts make it easy to deploy, debug, and update applications in Kubernetes with their separate values and upgrade lifecycles. You can walk through the folder structure, dependencies, and value files in a sample chart like bitnami/wordpress.

3. Use Labels to Find Resources Easily

Labels are crucial to Kubernetes’ internal operations and the daily work of Kubernetes operators. Almost every resource in Kubernetes offers labels for different purposes such as grouping, resource allocation, load balancing, or scheduling.

A single Helm command will allow you to install multiple resources. But it’s vital to know where these resources originate. Labels enable you to find your resources created by Helm releases quickly.

The most common method is to define labels in helpers.tpl, like so:

{{/*
Common labels
*/}}

{{- define "common.labels" -}} 
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}

You then need to use the “include” function with labels in the resource templates:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-queue
  labels:
{{ include "common.labels" . | indent 4 }}
...

Now you should be able to list all resources with the label selectors. For example, you can list all the pods of my-queue deployment with the kubectl get pods -l app.kubernetes.io/instance=[Name of the Helm Release] command. This step is essential for locating and debugging those resources managed by Helm.

4. Document Your Charts

Documentation is essential for ensuring maintainable Helm charts. Adding comments in the resource templates and the README helps teams with the development and use of Helm charts.

You should use the following three options to document your charts:

Comments: Template and values files are YAML files. You can add comments and provide helpful information about the fields inside the YAML files.
README : A chart’s README is a markdown file that explains how to use the charts. You can check the contents of a README file with the following command: helm show readme [Name of the Chart]
NOTES.txt : This is a special file located at templates/NOTES.txt that provides helpful information about the deployment of releases. The content of the NOTES.txt file can also be templated with functions and values similar to resource templates:

You have deployed the following release: {{ .Release.Name }}.
To get further information, you can run the commands:
  $ helm status {{ .Release.Name }}
  $ helm get all {{ .Release.Name }}

At the end of the helm install or helm upgrade command, Helm prints out the content of the NOTES.txt like so:

RESOURCES:
==> v1/Secret
NAME        TYPE      DATA      AGE
my-secret   Opaque    1         0s

==> v1/ConfigMap
NAME           DATA      AGE
db-configmap   3         0s

NOTES:
You have deployed the following release: precious-db.
To get further information, you can run the commands:
  $ helm status precious-db
  $ helm get all precious-db

5. Test Your Charts

Helm charts consist of multiple resources that are to be deployed to the cluster. It is essential to check that all the resources are created in the cluster with the correct values. For instance, when deploying a database, you should check that the database passwords are set correctly.

Fortunately, Helm offers a test functionality to run some containers in the cluster in order to validate applications. For example, the resource templates annotated with "helm.sh/hook": test-success are run by Helm as test cases.

Let's assume you are deploying WordPress with the MariaDB database. The Helm chart maintained by Bitnami has a pod to validate the database connection with the following definition:

...
apiVersion: v1
kind: Pod
metadata:
  name: "{{ .Release.Name }}-credentials-test"
  annotations:
    "helm.sh/hook": test-success
...
      env:
        - name: MARIADB\_HOST
          value: {{ include "wordpress.databaseHost" . | quote }}
        - name: MARIADB\_PORT
          value: "3306"
        - name: WORDPRESS\_DATABASE\_NAME
          value: {{ default "" .Values.mariadb.auth.database | quote }}
        - name: WORDPRESS\_DATABASE\_USER
          value: {{ default "" .Values.mariadb.auth.username | quote }}
        - name: WORDPRESS\_DATABASE\_PASSWORD
          valueFrom:
            secretKeyRef:
              name: {{ include "wordpress.databaseSecretName" . }}
              key: mariadb-password
      command:
        - /bin/bash
        - -ec
        - |
          mysql --host=$MARIADB\_HOST --port=$MARIADB\_PORT --user=$WORDPRESS\_DATABASE\_USER --password=$WORDPRESS\_DATABASE\_PASSWORD
  restartPolicy: Never
{{- end }}
...

It is recommended to write tests for your charts and to run them after the installation. For example, you can use the helm test <RELEASE_NAME> command to run tests. The tests are a valuable asset for validating and finding issues in applications installed with Helm.

6. Ensure Secrets Are Secure

Sensitive data, such as keys or passwords, are stored as secrets in Kubernetes. Although it is possible to secure secrets on the Kubernetes side, they are mostly stored as text files as part of Helm templates and values.

The helm-secrets plugin offers secret management and protection for your critical information. It delegates the secret encryption to Mozilla SOPS, which supports AWS KMS, Cloud KMS on GCP, Azure Key Vault, and PGP.

Let's assume you’ve collected your sensitive data in a file named secrets.yaml as follows:

postgresql:
  postgresqlUsername: postgres
  postgresqlPassword: WoZpCAlBsg
  postgresqlDatabase: wp

You can encrypt the file with the plugin:

$ helm secrets enc secrets.yaml
Encrypting secrets.yaml
Encrypted secrets.yaml

Now, the file will be updated and all values will be encrypted:

postgresql:
    postgresqlUsername: ENC\[AES256\_GCM,data:D14/CcA3WjY=,iv...==,type:str\]
    postgresqlPassword: ENC\[AES256\_GCM,data:Wd7VEKSoqV...,type:str\]
    postgresqlDatabase: ENC\[AES256\_GCM,data:8ur9pqDxUA==,iv:R...,type:str\]
sops:
  ...

The data in secrets.yaml above was not secure and helm-secrets solves the problem of storing sensitive data as part of Helm charts.

7. Make Your Chart Reusable by Using Template Functions

Helm supports over 60 functions that can be used inside templates. The functions are defined in the Go template language and Sprig template library. Functions in template files significantly simplify Helm operations.

Let's look at the following template file as an example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  environment: {{ .Values.environment | default "dev" | quote }}
  region: {{ .Values.region | upper | quote }}

When the environment value is not provided, it will be defaulted by the template function. When you check the region field, you’ll see there is no default value defined in the template. However, the field has another function called upper to convert the provided value into uppercase.

Another essential and useful function is required. It enables you to set a value as required for template rendering. For instance, let's assume you need a name for your ConfigMap with the following template:

...
metadata:
  name: {{ required "Name is required" .Values.configName }}
...

If the entry is empty, the template rendering will fail with the error Name is required. Template functions are very useful when creating Helm charts. They can improve templating, reduce code duplication, and can be used to validate values before deploying your applications to Kubernetes.

8. Update Your Deployments When ConfigMaps or Secrets Change

It is common to have ConfigMaps or secrets mounted to containers. Although the deployments and container images change with new releases, the ConfigMaps or secrets do not change frequently. The following annotation makes it possible to roll out new deployments when the ConfigMap changes:

kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
...

Any change in the ConfigMap will calculate a new sha256sum and create new versions of deployment. This ensures the containers in the deployments will restart using the new ConfigMap.

9. Opt Out of Resource Deletion with Resource Policies

In a typical setup, after installing a Helm chart, Helm will create multiple resources in the cluster. You can then upgrade it by changing values and adding or removing resources. Once you no longer need the application, you can delete it, which removes all resources from the cluster. Some resources, however, should be kept in the cluster even after running Helm uninstall. Let's assume you’ve deployed a database with PersistentVolumeClaim and want to store the volumes even if you are deleting the database release. For such resources, you need to use the resource-policy annotations as follows:

kind: Secret
metadata:
  annotations:
    "helm.sh/resource-policy": keep
...

Helm commands, such as uninstall, upgrade, or rollback would result in the deletion of the above secret. But by using the resource policy as shown, Helm will skip the deletion of the secret and allow it to be orphaned. The annotation should therefore be used with great care, and only for the resources needed after Helm Releases has been deleted.

10. Useful Commands for Debugging Helm Charts

Helm template files come with many different functions and multiple sources of values for creating Kubernetes resources. It is an essential duty of the user to know what is deployed to the cluster. Therefore, you need to learn how to debug templates and verify charts. There are four essential commands to use for debugging:

helm lint: The linter tool conducts a series of tests to ensure your chart is correctly formed.
helm install --dry-run --debug: This function renders the templates and shows the resulting resource manifests. You can also check all the resources before deployment and ensure the values are set and the templating functions work as expected.
helm get manifest: This command retrieves the manifests of the resources that are installed to the cluster. If the release is not working as expected, this should be the first command you use to find out what is running in the cluster.
helm get values: This command is used to retrieve the release values installed to the cluster. If you have any doubts about computed or default values, this should definitely be in your toolbelt.

11. Use the lookup Function to Avoid Secret Regeneration

Helm functions are used to generate random data, such as passwords, keys, and certificates. Random generation creates new arbitrary values and updates the resources in the cluster with each deployment and upgrade. For example, it can replace your database password in the cluster with every version upgrade. This causes the clients to be unable to connect to the database after the password change.

To address this, it is recommended to randomly generate values and override those already in the cluster. For example:

{{- $rootPasswordValue := (randAlpha 16) | b64enc | quote }}
{{- $secret := (lookup "v1" "Secret" .Release.Namespace "db-keys") }}
{{- if $secret }}
{{- $rootPasswordValue = index $secret.data "root-password" }}
{{- end -}}
apiVersion: v1
kind: Secret
metadata:
  name: db-keys
  namespace: {{ .Release.Namespace }}
type: Opaque
data:
  root-password: {{ $rootPasswordValue}}

The template above first creates a 16-character randAlpha value, then checks the cluster for a secret and its corresponding field. If found, it overrides and reuses the rootPasswordValue as root-password.

12. Migrate to Helm 3 for Simpler and More Secure Kubernetes Applications

The latest Helm release, Helm 3, offers many new features to make it a lighter, more streamlined tool. Helm v3 is recommended for its enhanced security and simplicity. This includes:

Removal of Tiller: Tiller was the Helm server-side component but has been removed from v3 due to the exhaustive permissions required to make changes on the cluster in earlier versions. This also created a security risk, as anyone gaining access to Tiller would have excessive permissions to your cluster.
Improved chart upgrade strategy: Helm v2 relies on a two-way strategic merge patch. It compares the new release with the one in the ConfigMap storage and applies the changes. Conversely, Helm v3 compares the old manifest, the state in the cluster, and the new release. So your manual changes will not be lost while upgrading your Helm releases. This simplifies the upgrade process and enhances the reliability of the applications.

There is a helm-2to3 plugin, which you can install with the following command:

$ helm3 plugin install https://github.com/helm/helm-2to3

It is a small but helpful plugin with cleanup, convert, and move commands to help you migrate and clean up your v2 configuration and create releases for v3.

13. Keep Your Continuous Delivery Pipelines Idempotent

Kubernetes resources are declarative in the sense that their specification and status are stored in the cluster. Similarly, Helm is required to create declarative templates and releases. Therefore, you need to design your continuous delivery and release management to be idempotent while using Helm. An idempotent operation is one you can apply many times without changing the result following the first run.

There are two essential rules to follow:

Always use the helm upgrade --install command. It installs the charts if they are not already installed. If they are already installed, it upgrades them.
Use --atomic flag to rollback changes in the event of a failed operation during helm upgrade. This ensures the Helm releases are not stuck in the failed state.

Summary

Helm is an indispensable tool for deploying applications to Kubernetes clusters. But it is only by following best practices that you’ll truly reap the benefits of Helm.

The best practices covered in this article will help your teams create, operate, and upgrade production-grade distributed applications. From the development side, your Helm charts will be easier to maintain and secure. From the operational side, you’ll enjoy automatically updated deployments, save resources from deletion, and learn how to test and debug.

Helm’s official topics guide is another good resource for checking the Helm commands and understanding their design philosophy. With these resources as well as the best practices and examples outlined in this blog, you’ll surely be armed and ready to create and manage production-grade Helm applications running on Kubernetes.

13 Best Practices for using Helm