helmfile使用详解

1 介绍

Helm 作为 Kubernetes 的包管理工具和 CNCF 毕业项目,在业界被广泛使用。但在实际使用场景中的一些需求 helm 并不能很好的满足。

  1. Helm 不提供 apply 命令;因此在 CI/CD 场景中必须考虑到判断是 install 还是 upgrade。
  2. 不方便控制安装的 chart 版本;例如指定版本范围、锁定某一版本等。
  3. Values 必须是纯文本;不支持模板渲染、不方便区分环境。

因此我们需要进行一些修改和适配,如同时部署多个 chart、不同部署环境的区分以及 chart 的版本控制。Helmfile 就是一个能够很好解决这些问题的小工具。

Helmfile 通过 helmfile.yaml 文件帮助用户管理和维护众多 helm chart,其最主要作用是:

  • 集成在 CI/CD 系统中,提高部署的可观测性和可重复性,区分环境,免去各种 --set 造成的困扰。

  • 方便对 helm chart 进行版本控制,如指定版本范围、锁定版本等。

  • 定期同步,避免环境中出现不符合预期的配置。

一键apply

1
2
3
helm repo add ...
helm install ...
helm upgrade ...

可以直接被简化为:

1
helmfile apply

同时,如果你安装了 helm-diff 插件,Helmfile 还会在执行操作前输出清晰的 diff:

2 安装

2.1 MacOS

1
brew install helmfile

2.2 Windows

1
scoop install helmfile

2.3 ArchLinux

1
pacman -S helmfile

2.4 openSUSE

1
zypper in helmfile

3 依赖安装

除了安装 helmfile 以外,还需要安装 helmkubectl 以及 helm 插件 helm-diff

1
2
3
4
5
[mychart] helm plugin install https://github.com/databus23/helm-diff
Downloading https://github.com/databus23/helm-diff/releases/download/v3.1.3/helm-diff-macos.tgz
...
Use "diff [command] --help" for more information about a command.
Installed plugin: diff

4 使用

4.1 helmfile.yaml

helmfile.yaml 是 helmfile 的核心文件,其用来声明所有的配置。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
# Chart repositories used from within this state file
#
# Use `helm-s3` and `helm-git` and whatever Helm Downloader plugins
# to use repositories other than the official repository or one backend by chartmuseum.
repositories:
# To use official "stable" charts a.k.a https://github.com/helm/charts/tree/master/stable
- name: stable
url: https://charts.helm.sh/stable
# To use official "incubator" charts a.k.a https://github.com/helm/charts/tree/master/incubator
- name: incubator
url: https://charts.helm.sh/incubator
# helm-git powered repository: You can treat any Git repository as a charts repository
- name: polaris
url: git+https://github.com/reactiveops/polaris@deploy/helm?ref=master
# Advanced configuration: You can setup basic or tls auth and optionally enable helm OCI integration
- name: roboll
url: http://roboll.io/charts
certFile: optional_client_cert
keyFile: optional_client_key
username: optional_username
password: optional_password
oci: true
# Advanced configuration: You can use a ca bundle to use an https repo
# with a self-signed certificate
- name: insecure
url: https://charts.my-insecure-domain.com
caFile: optional_ca_crt

# context: kube-context # this directive is deprecated, please consider using helmDefaults.kubeContext

# Path to alternative helm binary (--helm-binary)
helmBinary: path/to/helm3

# Default values to set for args along with dedicated keys that can be set by contributors, cli args take precedence over these.
# In other words, unset values results in no flags passed to helm.
# See the helm usage (helm SUBCOMMAND -h) for more info on default values when those flags aren't provided.
helmDefaults:
tillerNamespace: tiller-namespace #dedicated default key for tiller-namespace
tillerless: false #dedicated default key for tillerless
kubeContext: kube-context #dedicated default key for kube-context (--kube-context)
cleanupOnFail: false #dedicated default key for helm flag --cleanup-on-fail
# additional and global args passed to helm (default "")
args:
- "--set k=v"
# verify the chart before upgrading (only works with packaged charts not directories) (default false)
verify: true
# wait for k8s resources via --wait. (default false)
wait: true
# if set and --wait enabled, will wait until all Jobs have been completed before marking the release as successful. It will wait for as long as --timeout (default false, Implemented in Helm3.5)
waitForJobs: true
# time in seconds to wait for any individual Kubernetes operation (like Jobs for hooks, and waits on pod/pvc/svc/deployment readiness) (default 300)
timeout: 600
# performs pods restart for the resource if applicable (default false)
recreatePods: true
# forces resource update through delete/recreate if needed (default false)
force: false
# enable TLS for request to Tiller (default false)
tls: true
# path to TLS CA certificate file (default "$HELM_HOME/ca.pem")
tlsCACert: "path/to/ca.pem"
# path to TLS certificate file (default "$HELM_HOME/cert.pem")
tlsCert: "path/to/cert.pem"
# path to TLS key file (default "$HELM_HOME/key.pem")
tlsKey: "path/to/key.pem"
# limit the maximum number of revisions saved per release. Use 0 for no limit. (default 10)
historyMax: 10
# when using helm 3.2+, automatically create release namespaces if they do not exist (default true)
createNamespace: true
# if used with charts museum allows to pull unstable charts for deployment, for example: if 1.2.3 and 1.2.4-dev versions exist and set to true, 1.2.4-dev will be pulled (default false)
devel: true
# When set to `true`, skips running `helm dep up` and `helm dep build` on this release's chart.
# Useful when the chart is broken, like seen in https://github.com/roboll/helmfile/issues/1547
skipDeps: false

# these labels will be applied to all releases in a Helmfile. Useful in templating if you have a helmfile per environment or customer and don't want to copy the same label to each release
commonLabels:
hello: world

# The desired states of Helm releases.
#
# Helmfile runs various helm commands to converge the current state in the live cluster to the desired state defined here.
releases:
# Published chart example
- name: vault # name of this release
namespace: vault # target namespace
createNamespace: true # helm 3.2+ automatically create release namespace (default true)
labels: # Arbitrary key value pairs for filtering releases
foo: bar
chart: roboll/vault-secret-manager # the chart being installed to create this release, referenced by `repository/chart` syntax
version: ~1.24.1 # the semver of the chart. range constraint is supported
condition: vault.enabled # The values lookup key for filtering releases. Corresponds to the boolean value of `vault.enabled`, where `vault` is an arbitrary value
missingFileHandler: Warn # set to either "Error" or "Warn". "Error" instructs helmfile to fail when unable to find a values or secrets file. When "Warn", it prints the file and continues.
# Values files used for rendering the chart
values:
# Value files passed via --values
- vault.yaml
# Inline values, passed via a temporary values file and --values, so that it doesn't suffer from type issues like --set
- address: https://vault.example.com
# Go template available in inline values and values files.
- image:
# The end result is more or less YAML. So do `quote` to prevent number-like strings from accidentally parsed into numbers!
# See https://github.com/roboll/helmfile/issues/608
tag: {{ requiredEnv "IMAGE_TAG" | quote }}
# Otherwise:
# tag: "{{ requiredEnv "IMAGE_TAG" }}"
# tag: !!string {{ requiredEnv "IMAGE_TAG" }}
db:
username: {{ requiredEnv "DB_USERNAME" }}
# value taken from environment variable. Quotes are necessary. Will throw an error if the environment variable is not set. $DB_PASSWORD needs to be set in the calling environment ex: export DB_PASSWORD='password1'
password: {{ requiredEnv "DB_PASSWORD" }}
proxy:
# Interpolate environment variable with a fixed string
domain: {{ requiredEnv "PLATFORM_ID" }}.my-domain.com
scheme: {{ env "SCHEME" | default "https" }}
# Use `values` whenever possible!
# `set` translates to helm's `--set key=val`, that is known to suffer from type issues like https://github.com/roboll/helmfile/issues/608
set:
# single value loaded from a local file, translates to --set-file foo.config=path/to/file
- name: foo.config
file: path/to/file
# set a single array value in an array, translates to --set bar[0]={1,2}
- name: bar[0]
values:
- 1
- 2
# set a templated value
- name: namespace
value: {{ .Namespace }}
# will attempt to decrypt it using helm-secrets plugin
secrets:
- vault_secret.yaml
# Override helmDefaults options for verify, wait, waitForJobs, timeout, recreatePods and force.
verify: true
wait: true
waitForJobs: true
timeout: 60
recreatePods: true
force: false
# set `false` to uninstall this release on sync. (default true)
installed: true
# restores previous state in case of failed release (default false)
atomic: true
# when true, cleans up any new resources created during a failed release (default false)
cleanupOnFail: false
# name of the tiller namespace (default "")
tillerNamespace: vault
# if true, will use the helm-tiller plugin (default false)
tillerless: false
# enable TLS for request to Tiller (default false)
tls: true
# path to TLS CA certificate file (default "$HELM_HOME/ca.pem")
tlsCACert: "path/to/ca.pem"
# path to TLS certificate file (default "$HELM_HOME/cert.pem")
tlsCert: "path/to/cert.pem"
# path to TLS key file (default "$HELM_HOME/key.pem")
tlsKey: "path/to/key.pem"
# --kube-context to be passed to helm commands
# CAUTION: this doesn't work as expected for `tilerless: true`.
# See https://github.com/roboll/helmfile/issues/642
# (default "", which means the standard kubeconfig, either ~/kubeconfig or the file pointed by $KUBECONFIG environment variable)
kubeContext: kube-context
# passes --disable-validation to helm 3 diff plugin, this requires diff plugin >= 3.1.2
# It may be helpful to deploy charts with helm api v1 CRDS
# https://github.com/roboll/helmfile/pull/1373
disableValidation: false
# passes --disable-validation to helm 3 diff plugin, this requires diff plugin >= 3.1.2
# It is useful when any release contains custom resources for CRDs that is not yet installed onto the cluster.
# https://github.com/roboll/helmfile/pull/1618
disableValidationOnInstall: false
# passes --disable-openapi-validation to helm 3 diff plugin, this requires diff plugin >= 3.1.2
# It may be helpful to deploy charts with helm api v1 CRDS
# https://github.com/roboll/helmfile/pull/1373
disableOpenApiValidation: false
# limit the maximum number of revisions saved per release. Use 0 for no limit (default 10)
historyMax: 10
# When set to `true`, skips running `helm dep up` and `helm dep build` on this release's chart.
# Useful when the chart is broken, like seen in https://github.com/roboll/helmfile/issues/1547
skipDeps: false

# Local chart example
- name: grafana # name of this release
namespace: another # target namespace
chart: ../my-charts/grafana # the chart being installed to create this release, referenced by relative path to local helmfile
values:
- "../../my-values/grafana/values.yaml" # Values file (relative path to manifest)
- ./values/{{ requiredEnv "PLATFORM_ENV" }}/config.yaml # Values file taken from path with environment variable. $PLATFORM_ENV must be set in the calling environment.
wait: true

#
# Advanced Configuration: Nested States
#
helmfiles:
- # Path to the helmfile state file being processed BEFORE releases in this state file
path: path/to/subhelmfile.yaml
# Label selector used for filtering releases in the nested state.
# For example, `name=prometheus` in this context is equivalent to processing the nested state like
# helmfile -f path/to/subhelmfile.yaml -l name=prometheus sync
selectors:
- name=prometheus
# Override state values
values:
# Values files merged into the nested state's values
- additional.values.yaml
# One important aspect of using values here is that they first need to be defined in the values section
# of the origin helmfile, so in this example key1 needs to be in the values or environments.NAME.values of path/to/subhelmfile.yaml
# Inline state values merged into the nested state's values
- key1: val1
- # All the nested state files under `helmfiles:` is processed in the order of definition.
# So it can be used for preparation for your main `releases`. An example would be creating CRDs required by `releases` in the parent state file.
path: path/to/mycrd.helmfile.yaml
- # Terraform-module-like URL for importing a remote directory and use a file in it as a nested-state file
# The nested-state file is locally checked-out along with the remote directory containing it.
# Therefore all the local paths in the file are resolved relative to the file
path: git::https://github.com/cloudposse/helmfiles.git@releases/kiam.yaml?ref=0.40.0
# If set to "Error", return an error when a subhelmfile points to a
# non-existent path. The default behavior is to print a warning and continue.
missingFileHandler: Error

#
# Advanced Configuration: Environments
#

# The list of environments managed by helmfile.
#
# The default is `environments: {"default": {}}` which implies:
#
# - `{{ .Environment.Name }}` evaluates to "default"
# - `{{ .Values }}` being empty
environments:
# The "default" environment is available and used when `helmfile` is run without `--environment NAME`.
default:
# Everything from the values.yaml is available via `{{ .Values.KEY }}`.
# Suppose `{"foo": {"bar": 1}}` contained in the values.yaml below,
# `{{ .Values.foo.bar }}` is evaluated to `1`.
values:
- environments/default/values.yaml
# Each entry in values can be either a file path or inline values.
# The below is an example of inline values, which is merged to the `.Values`
- myChartVer: 1.0.0-dev
# Any environment other than `default` is used only when `helmfile` is run with `--environment NAME`.
# That is, the "production" env below is used when and only when it is run like `helmfile --environment production sync`.
production:
values:
- environment/production/values.yaml
- myChartVer: 1.0.0
# disable vault release processing
- vault:
enabled: false
## `secrets.yaml` is decrypted by `helm-secrets` and available via `{{ .Environment.Values.KEY }}`
secrets:
- environment/production/secrets.yaml
# Instructs helmfile to fail when unable to find a environment values file listed under `environments.NAME.values`.
#
# Possible values are "Error", "Warn", "Info", "Debug". The default is "Error".
#
# Use "Warn", "Info", or "Debug" if you want helmfile to not fail when a values file is missing, while just leaving
# a message about the missing file at the log-level.
missingFileHandler: Error
# kubeContext to use for this environment
kubeContext: kube-context

#
# Advanced Configuration: Layering
#
# Helmfile merges all the "base" state files and this state file before processing.
#
# Assuming this state file is named `helmfile.yaml`, all the files are merged in the order of:
# environments.yaml <- defaults.yaml <- templates.yaml <- helmfile.yaml
bases:
- environments.yaml
- defaults.yaml
- templates.yaml

#
# Advanced Configuration: API Capabilities
#
# 'helmfile template' renders releases locally without querying an actual cluster,
# and in this case `.Capabilities.APIVersions` cannot be populated.
# When a chart queries for a specific CRD, this can lead to unexpected results.
#
# Configure a fixed list of api versions to pass to 'helm template' via the --api-versions flag:
apiVersions:
- example/v1

4.2 apply

helmfile apply 是 helmfile 中最常用命令,体验与 kubectl apply 类似,根据 helmfile.yaml 中声明的配置可以一键执行相应的动作,如:添加 repo、安装或更新 release 等。

helmfile.yaml 如下:

1
2
3
4
5
6
releases:
- name: mychart
namespace: prod
chart: ./mychart
values:
- ./mychart/values.yaml

执行 helmfile apply 之后,helmfile 会进行如下操作:

1.添加 repositories 中声明的 repo

2.运行 helm diff 进行对比

3.根据 release中声明的配置,安装或更新 chart

效果如下(由于输出内容过多,这里只节选了部分输出):

4.3 模板化

helmfile 和 helm templete 一样可以使用 Go templates,同时还有一个特殊的功能 requiredEnv,该函数允许声明模板渲染所需的特定环境变量,如果环境变量未设置或为空,则渲染失败返回错误信息。

查看当前目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[Helm] tree
.
├── helmfile.yaml
└── mychart
├── Chart.yaml
├── charts
├── config
│   ├── dev
│   │   └── values.yaml
│   ├── prod
│   │   └── values.yaml
│   └── test
│   └── values.yaml
├── templates
│   └── configmap.yaml
└── values.yaml

7 directories, 7 files

helmfile.yaml

1
2
3
4
5
6
7
8
9
releases:
- name: mychart-{{ requiredEnv "DEPLOY_ENV" }}
namespace: {{ requiredEnv "DEPLOY_ENV" }}
chart: ./mychart
values: ./mychart/config/{{ requiredEnv "DEPLOY_ENV" }}/values.yaml
environments:
prod:
test:
dev:

指定环境运行

修改更新

4.4 待更新…