第25关 operator部署Elasticserach
大约 12 分钟
利用operator部署生产级别的Elasticserach集群和kibana
21年是用yaml配置部署单点Elasticserach和kibana用来测试
生产要维持稳定性
helm通过大量脚本维系Pod状态，经过测试，helm安装redis集群删掉其中一个节点不会自己恢复
用operator部署redis class (mongo\ mysql)
什么是kubernetes operator
Kubernetes Operator 是一种用于扩展 Kubernetes 功能的软件。它的主要功能包括:
自动化操作 - Operator 可以自动执行诸如部署、扩缩容、升级、备份等常见操作。用户不需要手动一个个 Pod 去执行这些任务。
无状态应用管理 - Operator 很适合管理那些无状态的应用,比如数据库、缓存等。它通过控制器模式让应用始终处于预期的状态。
减少重复工作 - Operator 让用户从重复的日常工作中解脱出来,这些工作可以交给 Operator 来自动完成。
基于 CRD 扩展 - Operator 通过自定义资源定义(CRD)扩展 Kubernetes 的 API,并基于这些 CRD 来实现自定义控制循环。
结合服务目录 - Operator 可以与服务目录(如 Service Catalog)集成,来启用基于权限的资源控制和分发。
总的来说,Kubernetes Operator 的核心思想是通过程序化和自动化的方式来管理和扩展 Kubernetes 集群。它极大地简化了在 Kubernetes 上安装和运行复杂应用的过程。
Operator 的工作原理基于 Kubernetes 的控制器模式。它会不断地监测 Kubernetes 集群的状态,一旦发现自定义资源(CR)的实际状态与预期状态不符,Operator 就会执行相应的操作以使其达到预期状态。这种模式使得 Operator 可以实现自我修复和自动恢复的功能。
常见的 Kubernetes Operator 包括 Rook 提供存储解决方案的 Operator,Prometheus Operator 用于监控集群的 Operator,Istio Operator 用于服务网格的 Operator 等。这些 Operator 为 Kubernetes 生态带来了很大的便利。
部署eck operator
kubectl create -f https://download.elastic.co/downloads/eck/2.10.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.10.0/operator.yaml
wget https://download.elastic.co/downloads/eck/2.10.0/crds.yaml
wget https://download.elastic.co/downloads/eck/2.10.0/operator.yaml
# kubectl create -f crds.yaml
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co created
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co created
# kubectl apply -f operator.yaml
namespace/elastic-system created
serviceaccount/elastic-operator created
secret/elastic-webhook-server-cert created
configmap/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator created
clusterrole.rbac.authorization.k8s.io/elastic-operator-view created
clusterrole.rbac.authorization.k8s.io/elastic-operator-edit created
clusterrolebinding.rbac.authorization.k8s.io/elastic-operator created
service/elastic-webhook-server created
statefulset.apps/elastic-operator created
validatingwebhookconfiguration.admissionregistration.k8s.io/elastic-webhook.k8s.elastic.co created
# 会创建有一个命名空间
# kubectl -n elastic-system get pod
NAME READY STATUS RESTARTS AGE
elastic-operator-0 1/1 Running 0 50s
# 是一个statefulset,
# kubectl -n elastic-system get sts
NAME READY AGE
elastic-operator 1/1 67s
# kubectl -n elastic-system describe pod elastic-operator-0
# kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
{"log.level":"info","@timestamp":"2024-06-08T08:31:54.045Z","log.logger":"resource-reporter","message":"Creating resource","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2024-06-08T08:31:54.058Z","log.logger":"manager","message":"Orphan secrets garbage collection complete","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0"}
kubectl create ns es部署 elasticsearch 集群
version: 8.11.3，后续结合CICD做日志系统，如果版本太高写入不了就换7版本的
最好用固态硬盘
# 测试的话记得把下面相关参数值调低
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
namespace: es
spec:
version: 8.11.3
nodeSets:
- name: default # sts名称就是quickstart-es-default
count: 1 # es集群数量,因为测试环境资源较少,生产可以部署2-3个
config:
node.store.allow_mmap: false
podTemplate:
spec:
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS # es用java开发的
value: -Xms1g -Xmx1g # 资源是分配的1/2,生产需要2G-4G
resources:
requests:
memory: 2Gi
cpu: 0.5
limits:
memory: 2Gi
cpu: 0.5
initContainers:
- name: sysctl
securityContext:
privileged: true
# 初始化容器,配置内核参数
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi # 50G可以自己适当分配
storageClassName: nfs-boge
http:
tls:
selfSignedCertificate:
disabled: true# kubectl create ns es
# kubectl -n es apply -f elasticsearch-statefulset.yaml
elasticsearch.elasticsearch.k8s.elastic.co/quickstart created
image: docker.elastic.co/elasticsearch/elasticsearch:8.11.3
imagePullPolicy: IfNotPresent # 或者 Never拉取镜像失败
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.3
# docker pull nginx:1.21.6 && docker save docker.elastic.co/elasticsearch/elasticsearch:8.11.3 > elasticsearch-8.11.3.tar
# ctr -n k8s.io images import elasticsearch-8.11.3.tar
unpacking docker.elastic.co/elasticsearch/elasticsearch:8.11.3 (sha256:42ea556b8dc4d40738fef9d034ef596692fc201d04114aecb14ec13304f87e5c)...done
# ctr -n k8s.io images ls | grep elasticsearch
docker.elastic.co/elasticsearch/elasticsearch:8.11.3 application/vnd.docker.distribution.manifest.v2+json sha256:42ea556b8dc4d40738fef9d034ef596692fc201d04114aecb14ec13304f87e5c 1.3 GiB linux/amd64 io.cri-containerd.image=managed
# kubectl -n es describe pod quickstart-es-default-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 19m kubelet Failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3": rpc error: code = Unknown desc = failed to pull and unpack image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3": failed to copy: read tcp 10.0.1.203:44702->13.32.54.127:443: read: connection reset by peer
Warning Failed 12m kubelet Failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3": rpc error: code = Unknown desc = failed to pull and unpack image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3": failed to copy: read tcp 10.0.1.203:59686->13.32.54.127:443: read: connection reset by peer
Warning Failed 6m5s (x3 over 19m) kubelet Error: ErrImagePull
Warning Failed 6m5s kubelet Failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3": rpc error: code = Unknown desc = failed to pull and unpack image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3": failed to copy: read tcp 10.0.1.203:60300->13.32.54.18:443: read: connection reset by peer
Normal BackOff 5m27s (x7 over 19m) kubelet Back-off pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3"
Warning Failed 5m27s (x7 over 19m) kubelet Error: ImagePullBackOff
Normal Pulling 5m14s (x4 over 25m) kubelet Pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3"# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-boge nfs-provisioner-01 Retain Immediate false 3h12m
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-5bc64039-9cf5-46cf-b27a-5c59d6bd512b 10Gi RWO Retain Released es/elasticsearch-data-quickstart-es-default-0 nfs-boge 178m
pvc-b706f5b1-508e-4d8f-a261-c2416672b703 10Gi RWO Retain Bound es/elasticsearch-data-quickstart-es-default-0 nfs-boge 15m
pvc-b77c3eaf-2b9d-4241-babb-ff6dba4bba03 1Mi RWX Retain Released test-pvc-sc/pvc-sc nfs-boge 3h12m
# kubectl get pvc -n es
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-data-quickstart-es-default-0 Bound pvc-b706f5b1-508e-4d8f-a261-c2416672b703 10Gi RWO nfs-boge 15m测试ES集群
# kubectl -n es get sts
NAME READY AGE
quickstart-es-default 0/1 91s
# kubectl -n es get pod
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 0/1 Pending 0 108s# kubectl -n es describe pod quickstart-es-default-0
Name: quickstart-es-default-0
Namespace: es
Priority: 0
Service Account: default
Node: <none>
Labels: common.k8s.elastic.co/type=elasticsearch
elasticsearch.k8s.elastic.co/version=8.11.3
statefulset.kubernetes.io/pod-name=quickstart-es-default-0
Annotations: co.elastic.logs/module: elasticsearch
elasticsearch.k8s.elastic.co/config-hash: 2778526996
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/quickstart-es-default
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m31s default-scheduler 0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod..
Warning FailedScheduling 2m30s default-scheduler 0/4 nodes are available: 2 Insufficient memory, 2 node(s) were unschedulable. preemption: 0/4 nodes are available: 2 No preemption victims found for incoming pod, 2 Preemption is not helpful for scheduling..
这个警告表明没有可用的节点来满足Pod的要求。可能的原因之一是Pod请求了持久卷（PersistentVolume），但没有可用的卷来满足需求。这可能是由于没有适当的存储资源可用，或者是由于配置问题而导致卷无法绑定。
关于 "preemption" 的部分表示调度器尝试对其他Pod进行预先调度以腾出资源，但没有找到适合的预先调度目标。
这个警告指出没有足够的内存来调度Pod，并且有两个节点被标记为无法调度。这可能是因为集群中的可用内存资源不足，或者是由于Pod请求的内存超出了节点的可用内存量。
同样，关于 "preemption" 的部分表示调度器试图通过预先调度其他Pod来腾出资源，但没有找到适合的预先调度目标。
给203节点内存调整为3072M
Normal Scheduled 2m45s default-scheduler Successfully assigned es/quickstart-es-default-0 to 10.0.1.203
Normal Pulling 2m34s kubelet Pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3"
Normal Pulled 70s kubelet Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3" in 1m23.685510783s (1m23.68552387s including waiting)
Normal Created 69s kubelet Created container elastic-internal-init-filesystem
Normal Started 69s kubelet Started container elastic-internal-init-filesystem
Normal Pulled 67s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3" already present on machine
Normal Created 67s kubelet Created container elastic-internal-suspend
Normal Started 67s kubelet Started container elastic-internal-suspend
Normal Pulled 66s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3" already present on machine
Normal Created 66s kubelet Created container sysctl
Normal Started 65s kubelet Started container sysctl
Normal Pulled 65s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.11.3" already present on machine
Normal Created 65s kubelet Created container elasticsearch
Normal Started 65s kubelet Started container elasticsearch
Warning Unhealthy 50s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 47s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:34+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 42s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:39+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 37s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:44+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 33s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:48+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 28s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:53+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 22s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:06:58+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 17s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:07:04+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 12s kubelet Readiness probe failed: {"timestamp": "2024-06-08T09:07:09+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 3s (x2 over 8s) kubelet (combined from similar events): Readiness probe failed: {"timestamp": "2024-06-08T09:07:18+00:00", "message": "readiness probe failed", "curl_rc": "7"}再等几分钟后可以了
# kubectl -n es get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
quickstart-es-default-0 1/1 Running 0 26m 172.20.139.84 10.0.1.203 <none> <none># kubectl -n es get secret quickstart-es-elastic-user -o yaml
apiVersion: v1
data:
elastic: SWQ2SjBhMDBuWjFIaTYzMnNDbTJST0Qy
kind: Secret
metadata:
creationTimestamp: "2024-06-08T08:45:22Z"
labels:
common.k8s.elastic.co/type: elasticsearch
eck.k8s.elastic.co/credentials: "true"
eck.k8s.elastic.co/owner-kind: Elasticsearch
eck.k8s.elastic.co/owner-name: quickstart
eck.k8s.elastic.co/owner-namespace: es
elasticsearch.k8s.elastic.co/cluster-name: quickstart
name: quickstart-es-elastic-user
namespace: es
resourceVersion: "271302"
uid: 60f59f3c-18c8-4776-9b8f-cf794b9280fd
type: Opaque# 获取密码
# PASSWORD=$(kubectl -n es get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
# echo $PASSWORD
Id6J0a00nZ1Hi632sCm2ROD2使用 -o go-template 标志和模板语法来自定义输出格式。在这个模板中，
{{.data.elastic | base64decode}}部分是一个 Go 模板，它实际上是在获取该 Secret 对象中名为 elastic 的字段，并使用 base64decode 函数对其进行解码。
# kubectl -n es get svc|grep es-http
quickstart-es-http ClusterIP 10.68.16.91 <none> 9200/TCP 34m
# 请求看能不能正常返回 -u 加账号(默认管理员账号elastic),返回一个json
# curl -u "elastic:$PASSWORD" http://$(kubectl -n es get svc|grep es-http|awk '{print $3}'):9200
{
"name" : "quickstart-es-default-0",
"cluster_name" : "quickstart",
"cluster_uuid" : "ELFJAqAVQfaXqB0R71MpqA",
"version" : {
"number" : "8.11.3",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "64cf052f3b56b1fd4449f5454cb88aca7e739d9a",
"build_date" : "2023-12-08T11:33:53.634979452Z",
"build_snapshot" : false,
"lucene_version" : "9.8.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}# echo "elastic:$PASSWORD"
elastic:Id6J0a00nZ1Hi632sCm2ROD2
elastic:542rk6HmCY8BR92b5yPL9r2b
# 检查ES创建结果(可以通过edit来修改、用delete来删除)
# kubectl -n es get es
NAME HEALTH NODES VERSION PHASE AGE
quickstart green 1 8.11.3 Ready 37m
# 查看索引列表
# curl -s --basic -u "elastic:$PASSWORD" http://$(kubectl -n es get svc|grep es-http|awk '{print $3}'):9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size
# 查看具体索引数据
# curl -s --basic -u "elastic:$PASSWORD" http://$(kubectl -n es get svc|grep es-http|awk '{print $3}'):9200/test-es-2021-02-02/_search?pretty
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [test-es-2021-02-02]",
"resource.type" : "index_or_alias",
"resource.id" : "test-es-2021-02-02",
"index_uuid" : "_na_",
"index" : "test-es-2021-02-02"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [test-es-2021-02-02]",
"resource.type" : "index_or_alias",
"resource.id" : "test-es-2021-02-02",
"index_uuid" : "_na_",
"index" : "test-es-2021-02-02"
},
"status" : 404
}
# 删除具体索引数据
curl -s --basic -u "elastic:$PASSWORD" -XDELETE http://$(kubectl -n es get svc|grep es-http|awk '{print $3}'):9200/test-es-2021-02-02部署 kibana
获取es的数据源用UI展示
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana # 自定义的资源类型
metadata:
name: quickstart
namespace: es
spec:
version: 8.11.3
count: 1 # kibana无所谓
elasticsearchRef:
name: quickstart #es名称,引用es的地址
http:
tls:
selfSignedCertificate:
disabled: true# kubectl -n es apply -f kibana-quickstart.yaml
# kubectl -n es get pod
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 1/1 Running 0 41m
quickstart-kb-5c77f999bd-9f9fj 0/1 Init:0/1 0 12s# kubectl -n es delete -f kibana-quickstart.yaml
~/boge/0608/1621#
kubectl -n es delete pod quickstart-kb-5c77f999bd-7nl57# kubectl describe pod quickstart-kb-5c77f999bd-9f9fj -n es
Name: quickstart-kb-5c77f999bd-9f9fj
Namespace: es
Priority: 0
Service Account: default
Node: 10.0.1.204/10.0.1.204
Start Time: Sat, 08 Jun 2024 17:26:48 +0800
Labels: common.k8s.elastic.co/type=kibana
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m9s default-scheduler Successfully assigned es/quickstart-kb-5c77f999bd-9f9fj to 10.0.1.204
Normal Pulling 3m8s kubelet Pulling image "docker.elastic.co/kibana/kibana:8.11.3"
Normal Pulled 52s kubelet Successfully pulled image "docker.elastic.co/kibana/kibana:8.11.3" in 2m15.423830148s (2m15.423841219s including waiting)
Normal Created 52s kubelet Created container elastic-internal-init-config
Normal Started 51s kubelet Started container elastic-internal-init-config
Normal Pulled 46s kubelet Container image "docker.elastic.co/kibana/kibana:8.11.3" already present on machine
Normal Created 46s kubelet Created container kibana
Normal Started 46s kubelet Started container kibana
Warning Unhealthy 9s (x3 over 29s) kubelet Readiness probe failed: Get "http://172.20.217.110:5601/login": dial tcp 172.20.217.110:5601: connect: connection refused
172.20.217.110正是他自己
# kubectl -n es get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
quickstart-es-default-0 1/1 Running 0 45m 172.20.139.84 10.0.1.203 <none> <none>
quickstart-kb-5c77f999bd-9f9fj 1/1 Running 0 3m42s 172.20.217.110 10.0.1.204 <none> <none>生成tls证书
# 这里我先自签一个https的证书
#1. 先生成私钥key
# openssl genrsa -out boge.key 2048
Generating RSA private key, 2048 bit long modulus
..............................................................................................+++
.....+++
e is 65537 (0x10001)
#2.再基于key生成tls证书(注意：这里我用的*.boge.com，这是生成泛域名的证书，后面所有新增加的三级域名都是可以用这个证书的)
# openssl req -new -x509 -key boge.key -out boge.csr -days 360 -subj /CN=*.boge.com
# 看下创建结果
# ll
total 8
-rw-r--r-- 1 root root 1099 Nov 27 11:44 boge.csr
-rw-r--r-- 1 root root 1679 Nov 27 11:43 boge.key
#3. 创建k8s上的tls secret
kubectl -n es create secret tls boge-com-tls --key boge.key --cert boge.csr
kubectl -n es create secret tls boge-com-tls --key boge.key --cert boge.csr
使用前面的证书
cp /root/boge/0522/boge-com.key /root/boge/0608/1621/
cp /root/boge/0522/boge-com.csr /root/boge/0608/1621/
# kubectl -n es create secret tls boge-com-tls --key boge-com.key --cert boge-com.csr
secret/boge-com-tls created
# kubectl -n es get secret
NAME TYPE DATA AGE
boge-com-tls kubernetes.io/tls 2 53s
es-quickstart-kibana-user Opaque 2 10m配置kibana的ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kibana
namespace: es
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- kibana.boge.com
secretName: boge-com-tls
rules:
- host: kibana.boge.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: quickstart-kb-http
port:
number: 5601# kubectl -n es apply -f ingress-kibana.yaml
Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
ingress.networking.k8s.io/kibana created
# echo "elastic:$PASSWORD"
elastic:Id6J0a00nZ1Hi632sCm2ROD2
elastic:542rk6HmCY8BR92b5yPL9r2b