在k8s集群部署Jenkins服务

第一次部署jenkins服务时，镜像中缺少一些Jenkins服务需要的依赖环境，在使用过程中有很多问题，后来选用其他镜像，并为其配置好相关环境后，打包成一个新的镜像，最后在rancher上替换掉当前镜像地址。下面会贴出整个服务的配置文件，是可移植的。以下操作分别是在Mac、Mac的虚拟机和容器之间切换操作。

具体操作

# 在虚拟机上的操作
# 使用一个相对稳定的镜像
docker pull jenkinsci/blueocean
# 查看pull的结果
docker images | grep jenkins
# 启动Jenkins
docker run -d  --name jenkins -u root -p 9090:8080  -v /var/jenkins_home:/var/jenkins_home  jenkinsci/blueocean
# 查看Jenkins容器
docker ps | grep jenkins
# 进到容器里面
docker exec -it 0f5832ffb8a0 bash

# 下面是在容器中的操作
# 查看apk命令帮助
apk -h
# 安装一些常用的工具
# 安装maven
apk add maven
# 查看maven相关信息
apk info -L maven
# 查看git信息
apk info -L git
# 查看是否有Java环境
echo $JAVA_HOME
# 安装一些常用的工具
apk add gcc
apk add yum
apk add wget
# 安装allure
cd /usr/local/
mkdir src
cd src/
# 容器内操作结束

# 在Mac电脑上下载allure-2.13.8.tgz
# 下载链接：https://github.com/allure-framework/allure2/releases
# 下载后，使用scp命令将其拷贝到虚拟机
scp allure-2.13.8.tgz root@172.16.26.101:/root
# 然后再从虚拟机拷贝到容器内部
docker cp allure-2.13.8.tgz 0f5832ffb8a0:/usr/local/src/allure-2.13.8.tgz

# 进到容器内继续操作
docker exec -it 0f5832ffb8a0 bash
cd /usr/local/src
# 解压
tar -zxvf allure-2.13.8.tgz
mv allure-2.13.8 allure
# 创建软连接
ln -s /usr/local/src/allure/bin/allure /usr/bin/allure
# 验证
allure
allure --version
# 退出容器

# 在虚拟机上将当前正在运行的容器打包成镜像，推送到自己docker-hub上
docker commit -a "hex" -m "hex-jenkins" 0f5832ffb8a0 jenkins:0318
docker images
docker tag jenkins:0318 131412127/jenkins:0318
docker login
docker push 131412127/jenkins:0318

# 推送到公司镜像仓库
docker tag jenkins:0318 registry.XXXXXXX.cn/jenkins:0318
docker login registry.XXXXXXX.cn
docker push registry.XXXXXXX.cn/jenkins:0318
# 到rancher上替换成这个镜像 

# 如果想要看自己虚拟机上运行的这个Jenkins
# 在容器内查看Jenkins登录密码
cat /var/jenkins_home/secrets/initialAdminPassword
# 有密码后，就可以在浏览器中访问：http://172.16.26.101:9090/

相关配置文件

以下是在k8s集群中搭建jenkins服务用到的相关配置文件

jenkins-service-account.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]  
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

jenkins-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: jenkins
  name: jenkins
  namespace: qa
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: jenkins
  serviceName: jenkins
  template:
    metadata:
      labels:
        name: jenkins
      name: jenkins
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: jenkins
                operator: In
                values:
                - "true"
      containers:
      - command:
        - /sbin/tini
        - /usr/local/bin/jenkins.sh
        - --prefix=/jenkins
        env:
        - name: ALLURE_HOME
          value: /var/jenkins_home/allure/package/
        - name: JAVA_OPTS
          value: -Xmx512m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0
            -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
            -D--prefix=/jenkins -Duser.timezone=Asia/Shanghai -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true
        - name: MAVEN_HOME
          value: /var/jenkins_home/maven/
        - name: PATH
          value: /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/jenkins_home/maven/bin
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              divisor: "0"
              resource: limits.memory
        image: registry.hexcloud.cn/jenkins:0318
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 12
          httpGet:
            path: /jenkins/login
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: jenkins
        ports:
        - containerPort: 8080
          name: 8080tcp02
          protocol: TCP
        - containerPort: 50000
          name: 50000tcp02
          protocol: TCP
        readinessProbe:
          failureThreshold: 12
          httpGet:
            path: /jenkins/login
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          requests:
            cpu: 500m
            memory: 500Mi
        securityContext:
          capabilities: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins-home
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      serviceAccount: jenkins
      serviceAccountName: jenkins
      terminationGracePeriodSeconds: 10
      volumes:
      - hostPath:
          path: /data/jenkins
          type: ""
        name: jenkins-home
  updateStrategy:
    type: RollingUpdate

jenkins-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  namespace: qa
spec:
  rules:
  - host: jenkins.XXXXXXX.cn
    http:
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 80
        path: /jenkins

jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: qa
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: agent
    port: 50000
    protocol: TCP
    targetPort: 50000
  selector:
    name: jenkins
  sessionAffinity: None
  type: ClusterIP

注意事项：

如果将上述文件部署在其他命名空间下，需要修改下上面文件的命名空间，修改下域名，给要部署的主机加上一个标签。
kubectl label nodes k8s-node01 jenkins=true
如在rancher上遇到：
Readiness probe failed: HTTP probe failed with statuscode: 503
原因可能是master节点的集群ip映射了5台机器，实际master只有3台，导致访问出错，解决方式，换一台主机，重新打标签。

赏