第17关 ConfigMap和Secret
大约 9 分钟
深入理解K8s配置管理：ConfigMap和Secret的终极指南
对于容器而言，如果我们想修改一个容器镜像里面的配置，可以在Dockerfile这一步，将修改好的配置复制到镜像里面再重新打包，对于不用变动配置的镜像而言，这样做属于硬编码当然也可以，但一旦我们的镜像服务需要修改配置，那么就需要重新重新打包非常麻烦，对于K8s而言，对于配置这么重要的一个环节，自然有它的解决方案，那就是configmap（通常普通配置使用）和secret（对于一些机密配置信息使用），在上面的部分章节里面，有提前涉及到这部分内容，但没有进行仔细的讲解，这里就对它们作下详细的实践。
我这里会准备一个deployment的yaml配置，用busybox来作为服务镜像，通过一个完整的yaml就可以快速带大家理解并能熟练在K8s上使用configmap和secret，如果一下子理解不了，后面可以保存这份yaml来作来生产配置参考也是没问题的，用多了自然就熟了，yaml配置如下：
配置
configmap-secret-example-simple.yaml
生产是通过命令去创建ConfigMap
---
# configmap
# --from-literal 基于命令行
# kubectl create configmap localconfig-env --from-literal=log_level_test=TEST --from-literal=log_level_produce=PRODUCE
apiVersion: v1
kind: ConfigMap
metadata:
name: localconfig-env
data:
log_level_test: TEST
log_level_produce: PRODUCE
---
# configmap
# --from-file 基于文件
# kubectl create configmap localconfig-file --from-file=localconfig-test=localconfig-test.conf --from-file=localconfig-produce=localconfig-produce.conf
apiVersion: v1
kind: ConfigMap
metadata:
name: localconfig-file
data:
localconfig-produce: |
TEST_RELEASE = False
PORT = 80
PROCESSES = 0
MESSAGE = Produce
localconfig-test: |
TEST_RELEASE = True
PORT = 8080
PROCESSES = 1
MESSAGE = Test
---
# secret
# 公钥和私钥
# kubectl create secret generic mysecret --from-literal=mysql-root-password='BogeMysqlPassword' --from-literal=redis-root-password='BogeRedisPassword' --from-file=my_id_rsa=/root/.ssh/id_rsa --from-file=my_id_rsa_pub=/root/.ssh/id_rsa.pub
apiVersion: v1
kind: Secret
metadata:
name: mysecret
namespace: default
type: Opaque
data:
my_id_rsa: bXlfaWRfcnNhCg==
my_id_rsa_pub: bXlfaWRfcnNhX3B1Ygo=
mysql-root-password: Qm9nZU15c3FsUGFzc3dvcmQ=
redis-root-password: Qm9nZVJlZGlzUGFzc3dvcmQ=
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: test-busybox
name: test-busybox
namespace: default
spec:
replicas: 1
selector:
matchLabels:
run: test-busybox
template:
metadata:
labels:
run: test-busybox
spec:
containers:
- name: test-busybox
image: registry.cn-shanghai.aliyuncs.com/acs/busybox:v1.29.2
args:
- /bin/sh
- -c
- >
echo "-------------------------------------------------";
echo "TEST_ENV is:$(TEST_ENV)";
echo "-------------------------------------------------";
echo "PRODUCE_ENV is:$(PRODUCE_ENV)";
echo "-------------------------------------------------";
echo "secret MYSQL_ROOT_PASSWORD is:$(MYSQL_ROOT_PASSWORD)";
echo "-------------------------------------------------";
echo "secret REDIS_ROOT_PASSWORD is:$(REDIS_ROOT_PASSWORD)";
echo "-------------------------------------------------";
echo "/etc/local_config_test.py body is:";
cat /etc/local_config_test.py;
echo "-------------------------------------------------";
echo "/etc/local_config_produce.py body is:";
cat /etc/local_config_produce.py;
echo "-------------------------------------------------";
echo "/etc/id_rsa body is:";
cat /etc/id_rsa;
echo "-------------------------------------------------";
echo "/etc/id_rsa.pub body is:";
cat /etc/id_rsa.pub;
echo "-------------------------------------------------";
ls -ltr /etc;
sleep 30000;
env:
- name: TEST_ENV
valueFrom:
configMapKeyRef:
name: localconfig-env
key: log_level_test
- name: PRODUCE_ENV
valueFrom:
configMapKeyRef:
name: localconfig-env
key: log_level_produce
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: mysql-root-password
- name: REDIS_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: redis-root-password
volumeMounts:
- name: testconfig
mountPath: "/etc/local_config_test.py"
subPath: localconfig-test
- name: testconfig
mountPath: "/etc/local_config_produce.py"
subPath: localconfig-produce
readOnly: true
- name: testsecret
mountPath: "/etc/id_rsa"
subPath: my_id_rsa
readOnly: true
- name: testsecret
mountPath: "/etc/id_rsa.pub"
subPath: my_id_rsa_pub
readOnly: true
volumes:
- name: testconfig
configMap:
name: localconfig-file
defaultMode: 0660
- name: testsecret
secret:
secretName: mysecret
defaultMode: 0600kind: ConfigMap
metadata:
name: localconfig-file
data:
localconfig-test: |
TEST_RELEASE = True
PORT = 8080
PROCESSES = 1
MESSAGE = Test
volumeMounts:
- name: testconfig
mountPath: "/etc/local_config_test.py"
subPath: localconfig-test
volumes:
- name: testconfig
configMap:
name: localconfig-file
defaultMode: 0660在一个配置文件里选择不同的key的值
要挂载的目录只挂载这个文件，不影响其他的文件和目录
把testconfig卷对应名为localconfig-file的configMap的localconfig-test的条目挂载到容器的/etc/local_config_test.py
kind: ConfigMap: 指定这个资源是一个 ConfigMap。
metadata: name: localconfig-file: ConfigMap 的名称为
localconfig-file。data: 这个部分包含键值对，每个键对应一段配置文件或配置数据。
- localconfig-test: 这个键包含实际的配置数据，以多行字符串的形式出现。
|表示这是一个块标量，后面的多行都是数据的一部分。- 这段数据定义了一些类似 Python 配置的变量，如
TEST_RELEASE、PORT、PROCESSES和MESSAGE。volumeMounts: 这一部分通常是 Pod 规范的一部分，定义了如何在容器中挂载这些卷（存储）。
- name: testconfig: 指定要挂载的卷的名称。
- mountPath: "/etc/local_config_test.py": 指定这个卷在容器内的挂载路径。在这个例子中，
localconfig-test键的内容会被挂载为一个文件，路径为/etc/local_config_test.py。- subPath: localconfig-test: 表示只使用 ConfigMap 中的
localconfig-test键，而不是整个 ConfigMap。volumes: 这一部分定义了容器中可以挂载的卷。
name: testconfig: 卷的名称。
configMap
: 指定这个卷的数据来源于一个 ConfigMap。
- name: localconfig-file: 引用了 ConfigMap 的名称（即
localconfig-file）。- defaultMode: 0660: 指定从这个 ConfigMap 创建的文件的默认权限。
0660表示文件对用户和组可读写，但对其他人不可读写。
逐一命令创建
实战（生产命令创建）
# kubectl create ns test-configmap# kubectl -n test-configmap create configmap localconfig-env --from-literal=log_level_test=TEST --from-literal=log_level_produce=PRODUCE
# kubectl -n test-configmap get configmap localconfig-env -o yaml
apiVersion: v1
data:
log_level_produce: PRODUCE
log_level_test: TEST
kind: ConfigMap
metadata:
creationTimestamp: "2024-05-29T15:11:32Z"
name: localconfig-env
namespace: test-configmap
resourceVersion: "109103"
uid: 50cd7849-0d53-4b67-a9af-4a8c72f08388localconfig-test.conf
TEST_RELEASE = True
PORT = 8080
PROCESSES = 1
MESSAGE = Testlocalconfig-produce.conf
TEST_RELEASE = False
PORT = 80
PROCESSES = 0
MESSAGE = Produce# kubectl -n test-configmap create configmap localconfig-file --from-file=localconfig-test=localconfig-test.conf --from-file=localconfig-produce=localconfig-produce.conf
# kubectl -n test-configmap get configmap localconfig-file -o yaml
apiVersion: v1
data:
localconfig-produce: |
TEST_RELEASE = False
PORT = 80
PROCESSES = 0
MESSAGE = Produce
localconfig-test: |
TEST_RELEASE = True
PORT = 8080
PROCESSES = 1
MESSAGE = Test
kind: ConfigMap
metadata:
creationTimestamp: "2024-05-29T15:16:47Z"
name: localconfig-file
namespace: test-configmap
resourceVersion: "109741"
uid: 3268cb5e-5566-4d7c-866e-39944f809de1# secret
# 公钥和私钥
# kubectl -n test-configmap create secret generic mysecret --from-literal=mysql-root-password='BogeMysqlPassword' --from-literal=redis-root-password='BogeRedisPassword' --from-file=my_id_rsa=/root/.ssh/id_rsa --from-file=my_id_rsa_pub=/root/.ssh/id_rsa.pub
# kubectl -n test-configmap get secret mysecret -o yaml
apiVersion: v1
data:
my_id_rsa: LS0tLS1CRUdJTiBPUEVO....LS0tCg==
my_id_rsa_pub: c3NoLXJzYSB....Cg==
mysql-root-password: Qm9nZU15c3FsUGFzc3dvcmQ=
redis-root-password: Qm9nZVJlZGlzUGFzc3dvcmQ=
kind: Secret
metadata:
creationTimestamp: "2024-05-29T15:18:33Z"
name: mysecret
namespace: test-configmap
resourceVersion: "109951"
uid: 9c344278-897a-47bf-8e81-1d3ace59ff2f
type: Opaque创建Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: test-busybox
name: test-busybox
spec:
replicas: 1
selector:
matchLabels:
run: test-busybox
template:
metadata:
labels:
run: test-busybox
spec:
containers:
- name: test-busybox
image: registry.cn-shanghai.aliyuncs.com/acs/busybox:v1.29.2
args:
- /bin/sh
- -c
- >
echo "-------------------------------------------------";
echo "TEST_ENV is:$(TEST_ENV)";
echo "-------------------------------------------------";
echo "PRODUCE_ENV is:$(PRODUCE_ENV)";
echo "-------------------------------------------------";
echo "secret MYSQL_ROOT_PASSWORD is:$(MYSQL_ROOT_PASSWORD)";
echo "-------------------------------------------------";
echo "secret REDIS_ROOT_PASSWORD is:$(REDIS_ROOT_PASSWORD)";
echo "-------------------------------------------------";
echo "/etc/local_config_test.py body is:";
cat /etc/local_config_test.py;
echo "-------------------------------------------------";
echo "/etc/local_config_produce.py body is:";
cat /etc/local_config_produce.py;
echo "-------------------------------------------------";
echo "/etc/id_rsa body is:";
cat /etc/id_rsa;
echo "-------------------------------------------------";
echo "/etc/id_rsa.pub body is:";
cat /etc/id_rsa.pub;
echo "-------------------------------------------------";
ls -ltr /etc;
sleep 30000;
env:
- name: TEST_ENV
valueFrom:
configMapKeyRef:
name: localconfig-env
key: log_level_test
- name: PRODUCE_ENV
valueFrom:
configMapKeyRef:
name: localconfig-env
key: log_level_produce
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: mysql-root-password
- name: REDIS_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret
key: redis-root-password
volumeMounts:
- name: testconfig
mountPath: "/etc/local_config_test.py"
subPath: localconfig-test
- name: testconfig
mountPath: "/etc/local_config_produce.py"
subPath: localconfig-produce
readOnly: true
- name: testsecret
mountPath: "/etc/id_rsa"
subPath: my_id_rsa
readOnly: true
- name: testsecret
mountPath: "/etc/id_rsa.pub"
subPath: my_id_rsa_pub
readOnly: true
volumes:
- name: testconfig
configMap:
name: localconfig-file
defaultMode: 0660
- name: testsecret
secret:
secretName: mysecret
defaultMode: 0600# kubectl -n test-configmap apply -f test-busybox.yaml
deployment.apps/test-busybox created
# kubectl -n test-configmap logs test-busybox-6cf8bb847f-j4zsp
-------------------------------------------------
TEST_ENV is:TEST
-------------------------------------------------
PRODUCE_ENV is:PRODUCE
-------------------------------------------------
secret MYSQL_ROOT_PASSWORD is:BogeMysqlPassword
-------------------------------------------------
secret REDIS_ROOT_PASSWORD is:BogeRedisPassword
-------------------------------------------------
/etc/local_config_test.py body is:
TEST_RELEASE = True
PORT = 8080
PROCESSES = 1
MESSAGE = Test
-------------------------------------------------
/etc/local_config_produce.py body is:
TEST_RELEASE = False
PORT = 80
PROCESSES = 0
MESSAGE = Produce
-------------------------------------------------
/etc/id_rsa body is:
-----BEGIN OPENSSH PRIVATE KEY-----
b3Bl2R...lLTE=
-----END OPENSSH PRIVATE KEY-----
-------------------------------------------------
/etc/id_rsa.pub body is:
ssh-rsa AAAAB3...tIUrE= root@node-1
-------------------------------------------------
total 48
-rw-r--r-- 1 root root 127 May 4 2018 localtime
-rw------- 1 root root 243 Jul 19 2018 shadow
-rw-r--r-- 1 root root 340 Jul 19 2018 passwd
-rw-rw-r-- 1 root root 307 Jul 19 2018 group
drwxr-xr-x 6 root root 4096 Jul 31 2018 network
-rw-r--r-- 1 root root 113 May 29 15:27 resolv.conf
-rw-rw---- 1 root root 61 May 29 15:27 local_config_test.py
-rw-rw---- 1 root root 63 May 29 15:27 local_config_produce.py
-rw------- 1 root root 565 May 29 15:27 id_rsa.pub
-rw------- 1 root root 2590 May 29 15:27 id_rsa
-rw-r--r-- 1 root root 30 May 29 15:27 hostname
-rw-r--r-- 1 root root 228 May 29 15:27 hostsreloader 配置自动更新器
https://github.com/stakater/Reloader
Reloader 是一个 Kubernetes 控制器，用于监视 ConfigMap 和 Secrets 的变化，并在这些资源发生变更时重新启动 Deployment、StatefulSet 和 DaemonSet 中的 Pod。
监听控制器
敏感生产环境谨慎使用
## kind: Deployment
## metadata:
## annotations:
## #------ all(ConfigMap and/or Secret) 所有关联到的配置文件有改动
## reloader.stakater.com/auto: "true"
## #------ only configmap for name: "foo-configmap" 指定配置foo-configmap文件有改动
## configmap.reloader.stakater.com/reload: "foo-configmap"
## #------ many configmaps 指定多个配置文件有改动
## configmap.reloader.stakater.com/reload: "foo-configmap,bar-configmap,baz-configmap"
## #------ only secret for name: "foo-secret"
## secret.reloader.stakater.com/reload: "foo-secret"
## #------ many secrets
## secret.reloader.stakater.com/reload: "foo-secret,bar-secret,baz-secret"
## spec:
## template:
## metadata:部署yaml配置
---
# Source: reloader/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
meta.helm.sh/release-namespace: "default"
meta.helm.sh/release-name: "reloader"
labels:
app: reloader-reloader
chart: "reloader-1.0.51"
release: "reloader"
heritage: "Helm"
app.kubernetes.io/managed-by: "Helm"
name: reloader-reloader
---
# Source: reloader/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
meta.helm.sh/release-namespace: "default"
meta.helm.sh/release-name: "reloader"
labels:
app: reloader-reloader
chart: "reloader-1.0.51"
release: "reloader"
heritage: "Helm"
app.kubernetes.io/managed-by: "Helm"
name: reloader-reloader-role
rules:
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- list
- get
- watch
- apiGroups:
- "apps"
resources:
- deployments
- daemonsets
- statefulsets
verbs:
- list
- get
- update
- patch
- apiGroups:
- "extensions"
resources:
- deployments
- daemonsets
verbs:
- list
- get
- update
- patch
- apiGroups:
- "batch"
resources:
- cronjobs
verbs:
- list
- get
- apiGroups:
- "batch"
resources:
- jobs
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
---
# Source: reloader/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
meta.helm.sh/release-namespace: "default"
meta.helm.sh/release-name: "reloader"
labels:
app: reloader-reloader
chart: "reloader-1.0.51"
release: "reloader"
heritage: "Helm"
app.kubernetes.io/managed-by: "Helm"
name: reloader-reloader-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: reloader-reloader-role
subjects:
- kind: ServiceAccount
name: reloader-reloader
namespace: test-configmap
---
# Source: reloader/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
meta.helm.sh/release-namespace: "default"
meta.helm.sh/release-name: "reloader"
labels:
app: reloader-reloader
chart: "reloader-1.0.51"
release: "reloader"
heritage: "Helm"
app.kubernetes.io/managed-by: "Helm"
group: com.stakater.platform
provider: stakater
version: v1.0.51
name: reloader-reloader
spec:
replicas: 1
revisionHistoryLimit: 2
selector:
matchLabels:
app: reloader-reloader
release: "reloader"
template:
metadata:
labels:
app: reloader-reloader
chart: "reloader-1.0.51"
release: "reloader"
heritage: "Helm"
app.kubernetes.io/managed-by: "Helm"
group: com.stakater.platform
provider: stakater
version: v1.0.51
spec:
containers:
- image: "ghcr.io/stakater/reloader:v1.0.51"
imagePullPolicy: IfNotPresent
name: reloader-reloader
ports:
- name: http
containerPort: 9090
livenessProbe:
httpGet:
path: /live
port: http
timeoutSeconds: 5
failureThreshold: 5
periodSeconds: 10
successThreshold: 1
initialDelaySeconds: 10
readinessProbe:
httpGet:
path: /metrics
port: http
timeoutSeconds: 5
failureThreshold: 5
periodSeconds: 10
successThreshold: 1
initialDelaySeconds: 10
securityContext:
{}
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: reloader-reloader部署
kubectl -n test-configmap apply -f reloader.yamlkubectl -n test-configmap edit deployments.apps test-busybox
# 修改annotations时Pod不会重启# kubectl -n test-configmap edit configmaps localconfig-file
configmap/localconfig-file edited
# kubectl -n test-configmap get pod
NAME READY STATUS RESTARTS AGE
reloader-reloader-7568f766fb-4bw2s 1/1 Running 0 13m
test-busybox-6cf8bb847f-j4zsp 1/1 Terminating 0 20m
test-busybox-86c7595967-sxjh6 1/1 Running 0 7s