【k8s部署elasticsearch】k8s环境下安装elasticsearch集群和kibana

文章目录

• • • 简介
    • 一.条件及环境说明
    • 二.需求说明
    • 三.实现原理及说明
    • 四.详细步骤
    • • 4.1.规划节点标签
      • 4.2.创建三个statefulset和service headless配置
      • 4.3.创建service配置
      • 五.安装kibana
      • 六.调整索引分区
      • 七.安装说明

        简介

        k8s集群中搭建有elasticsearch服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将elasticsearch集群的搭建采用hostpath+亲和性的权重+多个副本分区的方式来实现数据持久化和高可用。

        一.条件及环境说明

        k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator。

        二.需求说明

        • 搭建三个节点的elasticsearch集群，并配置三个主分区，2个副本分区。
        • 高可用：有一个k8s节点长期故障后或重启pod之后也不影响使用。
        • 高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
        • 数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
        • 安装简单、管理维护容易

          三.实现原理及说明

          四.详细步骤

          4.1.规划节点标签

          节点名称	角色	标签
          k8s-worker-120-81	主	storage-selector=node-a-master
          k8s-worker-120-82	主	storage-selector=node-b-master
          k8s-worker-120-83	主	storage-selector=node-c-master
          k8s-worker-120-84	备	storage-selector=node-a-slve
          k8s-worker-120-85	备	storage-selector=node-a-slve
          k8s-worker-120-86	备	storage-selector=node-a-slve

          注：这里的角色划分是指：redis由于是一主两从，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就切换到备节点，例如：名称为autopocket-es-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

          4.2.创建三个statefulset和service headless配置

          1.elasticsearch_a.yaml 节点a的statuflset配置和无头服务配置

          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: autopocket-es-a
            namespace: zx-app
            labels:
              app: autopocket-es-a 
              appname: pcauto-pocket
              appgroup: autopocket-es
          spec:
            serviceName: "autopocket-es-a"
            replicas: 1 
            selector:
              matchLabels:
                app: autopocket-es-a
                appgroup: autopocket-es
            template:
              metadata:
                labels:
                  app: autopocket-es-a
                  appgroup: autopocket-es
              spec:
                affinity:
                  nodeAffinity:
                    preferredDuringSchedulingIgnoredDuringExecution:
                    - weight: 80
                      preference:
                        matchExpressions:
                        - key: storage-selector 
                          operator: In
                          values:
                          - node-a-master
                    - weight: 20
                      preference:
                        matchExpressions:
                        - key: storage-selector 
                          operator: In
                          values:
                          - node-a-slave
                containers:
                - name: elasticsearch
                  image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
                  command: ["/bin/sh","-c"]
                  args:
                  - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
                  env:
                  - name: cluster.name
                    value: autopocket-es 
                  - name: K8S_SERVICE_NAME
                    value: autopocket-es-a 
                  - name: POD_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.name
                  - name: POD_NAMESPACE
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.namespace
                  - name: discovery.seed_hosts
                    value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
                  - name: cluster.initial_master_nodes
                    value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
                  - name: node.name
                    value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
                  - name: path.data
                    value: /home/elastic/$(POD_NAME)/data
                  - name: path.logs
                    value: /home/elastic/$(POD_NAME)/logs
                  - name: xpack.security.enabled
                    value: "false"
                  ports:
                  - name: rest
                    containerPort: 9200
                  - name: inter-node
                    containerPort: 9300
                  volumeMounts:
                  - name: elasticsearch-data
                    mountPath: /home/elastic
                restartPolicy: Always
                terminationGracePeriodSeconds: 30
                volumes:
                  - name: elasticsearch-data 
                    hostPath:
                      path: /data/es_data
                      type: ""
                tolerations:
                - key: node.kubernetes.io/not-ready
                  operator: Exists
                  effect: NoExecute
                  tolerationSeconds: 3600
                - key: node.kubernetes.io/unreachable
                  operator: Exists
                  effect: NoExecute
                  tolerationSeconds: 3600
          ---
          apiVersion: v1
          kind: Service
          metadata:
            name: autopocket-es-a
            namespace: zx-app
            labels:
              appname: pcauto-pocket
              app: autopocket-es-a
          spec:
            ports:
            - port: 9200
              name: rest
            - port: 9300
              name: inter-node
            clusterIP: None
            selector:
              app: autopocket-es-a
          

          2.elasticsearch_b.yaml 节点b的statuflset配置和无头服务配置

          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: autopocket-es-b
            namespace: zx-app
            labels:
              app: autopocket-es-b 
              appname: pcauto-pocket
              appgroup: autopocket-es
          spec:
            serviceName: "autopocket-es-b"
            replicas: 1 
            selector:
              matchLabels:
                app: autopocket-es-b
                appgroup: autopocket-es
            template:
              metadata:
                labels:
                  app: autopocket-es-b
                  appgroup: autopocket-es
              spec:
                affinity:
                  nodeAffinity:
                    preferredDuringSchedulingIgnoredDuringExecution:
                    - weight: 80
                      preference:
                        matchExpressions:
                        - key: storage-selector 
                          operator: In
                          values:
                          - node-b-master
                    - weight: 20
                      preference:
                        matchExpressions:
                        - key: storage-selector 
                          operator: In
                          values:
                          - node-b-slave
                containers:
                - name: elasticsearch
                  image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
                  command: ["/bin/sh","-c"]
                  args:
                  - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
                  env:
                  - name: cluster.name
                    value: autopocket-es 
                  - name: K8S_SERVICE_NAME
                    value: autopocket-es-b 
                  - name: POD_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.name
                  - name: POD_NAMESPACE
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.namespace
                  - name: discovery.seed_hosts
                    value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
                  - name: cluster.initial_master_nodes
                    value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
                  - name: node.name
                    value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
                  - name: path.data
                    value: /home/elastic/$(POD_NAME)/data
                  - name: path.logs
                    value: /home/elastic/$(POD_NAME)/logs
                  - name: xpack.security.enabled
                    value: "false"
                  ports:
                  - name: rest
                    containerPort: 9200
                  - name: inter-node
                    containerPort: 9300
                  volumeMounts:
                  - name: elasticsearch-data
                    mountPath: /home/elastic
                restartPolicy: Always
                terminationGracePeriodSeconds: 30
                volumes:
                  - name: elasticsearch-data 
                    hostPath:
                      path: /data/es_data
                      type: ""
                tolerations:
                - key: node.kubernetes.io/not-ready
                  operator: Exists
                  effect: NoExecute
                  tolerationSeconds: 3600
                - key: node.kubernetes.io/unreachable
                  operator: Exists
                  effect: NoExecute
                  tolerationSeconds: 3600
          ---
          apiVersion: v1
          kind: Service
          metadata:
            name: autopocket-es-b
            namespace: zx-app
            labels:
              appname: pcauto-pocket
              app: autopocket-es-b
          spec:
            ports:
            - port: 9200
              name: rest
            - port: 9300
              name: inter-node
            clusterIP: None
            selector:
              app: autopocket-es-b
          

          3.elasticsearch_c.yaml 节点c的statuflset配置和无头服务配置

          apiVersion: apps/v1
          kind: StatefulSet
          metadata:
            name: autopocket-es-c
            namespace: zx-app
            labels:
              app: autopocket-es-c 
              appname: pcauto-pocket
              appgroup: autopocket-es
          spec:
            serviceName: "autopocket-es-c"
            replicas: 1 
            selector:
              matchLabels:
                app: autopocket-es-c
                appgroup: autopocket-es
            template:
              metadata:
                labels:
                  app: autopocket-es-c
                  appgroup: autopocket-es
              spec:
                affinity:
                  nodeAffinity:
                    preferredDuringSchedulingIgnoredDuringExecution:
                    - weight: 80
                      preference:
                        matchExpressions:
                        - key: storage-selector 
                          operator: In
                          values:
                          - node-c-master
                    - weight: 20
                      preference:
                        matchExpressions:
                        - key: storage-selector 
                          operator: In
                          values:
                          - node-c-slave
                containers:
                - name: elasticsearch
                  image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"
                  command: ["/bin/sh","-c"]
                  args:
                  - mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"
                  env:
                  - name: cluster.name
                    value: autopocket-es 
                  - name: K8S_SERVICE_NAME
                    value: autopocket-es-c 
                  - name: POD_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.name
                  - name: POD_NAMESPACE
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.namespace
                  - name: discovery.seed_hosts
                    value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"
                  - name: cluster.initial_master_nodes
                    value: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"
                  - name: node.name
                    value: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local 
                  - name: path.data
                    value: /home/elastic/$(POD_NAME)/data
                  - name: path.logs
                    value: /home/elastic/$(POD_NAME)/logs
                  - name: xpack.security.enabled
                    value: "false"
                  ports:
                  - name: rest
                    containerPort: 9200
                  - name: inter-node
                    containerPort: 9300
                  volumeMounts:
                  - name: elasticsearch-data
                    mountPath: /home/elastic
                restartPolicy: Always
                terminationGracePeriodSeconds: 30
                volumes:
                  - name: elasticsearch-data 
                    hostPath:
                      path: /data/es_data
                      type: ""
                tolerations:
                - key: node.kubernetes.io/not-ready
                  operator: Exists
                  effect: NoExecute
                  tolerationSeconds: 3600
                - key: node.kubernetes.io/unreachable
                  operator: Exists
                  effect: NoExecute
                  tolerationSeconds: 3600
          ---
          apiVersion: v1
          kind: Service
          metadata:
            name: autopocket-es-c
            namespace: zx-app
            labels:
              appname: pcauto-pocket
              app: autopocket-es-c
          spec:
            ports:
            - port: 9200
              name: rest
            - port: 9300
              name: inter-node
            clusterIP: None
            selector:
              app: autopocket-es-c
          

          4.3.创建service配置

          给9200 9300配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

          apiVersion: v1
          kind: Service
          metadata:
            name: autopocket-es
            namespace: zx-app
            labels:
              app: autopocket-es
              appname: pcauto-zx
          spec:
            ports:
            - port: 9200
              name: rest
            - port: 9300
              name: inter-node
            selector:
              appgroup: autopocket-es
            type: LoadBalancer 
          

          五.安装kibana

          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: autopocket-es-kibana
            namespace: zx-app
            labels:
              app: autopocket-es-kibana
              appname: pcauto-pocket
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: autopocket-es-kibana
            template:
              metadata:
                labels:
                  app: autopocket-es-kibana
              spec:
                containers:
                - name: kibana
                  image: "xxx.cn-shenzhen.cr.aliyuncs.com/public/kibana:7.17.22"
                  ports:
                  - containerPort: 5601
                  env:
                  - name: ELASTICSEARCH_HOSTS
                    value: '["http://autopocket-es-a-0.autopocket-es-a:9200","http://autopocket-es-b-0.autopocket-es-b:9200","http://autopocket-es-c-0.autopocket-es-c:9200"]'
          ---
          apiVersion: v1
          kind: Service
          metadata:
            name: autopocket-es-kibana
            namespace: zx-app
            labels:
              app: autopocket-es-kibana
              appname: pcauto-pocket
          spec:
            selector:
              app: autopocket-es-kibana
            ports:
            - port: 5601
              targetPort: 5601
            type: LoadBalancer
          

          查看集群健康情况：http://10.16.xx.xx:9200/_cat/health?v

          elasticsearch集群状态查看：http://10.16.1xx.xx:9200/_cat/nodes?v

          

          索引状态查看：http://10.16.1xx.xx:9200/_cat/indices?v

          

          kibana地址登录：http://10.16.1xx.xx:5601/app/home#/

          

          六.调整索引分区

          调整已存在索引的副本分区：curl -XPUT -H "Content-Type:application/json" -d '{"settings":{"number_of_replicas":2}}' 'http://10.16.xx.xx:9200/testindex/_settings'

          调整集群最大分片数：curl -XPUT -H "Content-Type:application/json" http://10.16.xx.xx:9200/_cluster/settings -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}'

          调整默认主分片和副本分片数: curl -X PUT -H "Content-Type:application/json" -d '{"index_patterns":["*"],"settings":{"number_of_shards":3,"number_of_replicas":2}}' http://10.16.xx.xx:9200/_template/template_http_request_record

          七.安装说明

          • 需要先在k8s的节点创建/data/es_data的路径来存放数据。
          • 使用配置前将配置中的autopocket替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-pocket，修改成自己的应用名称，该标签也可以删除。
          • 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的elasticsearch:7.17.22 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用elasticsearch:7.17.22,kibana的镜像是kibana:7.17.22。否则配置一个可以通的镜像地址。
          • 默认没有配置用户名和密码。
          • 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。