DevOps（3）--Jenkins
Jenkins
提到基于 Kubernete 的CI/CD，可以使用的工具有很多，比如 Jenkins、Gitlab CI 以及新兴的 drone 之类的， Jenkins在国内很早就使用了，也是大多数公司首选的工具，所以我们使用Jenkins来做 CI/CD 的工具。
1. Local PV
pv，pvc以及storageclass的知识我们前面学过了，这里我们学习一种特殊的pv，叫做Local PV。
Local PV：Kubernetes直接使用宿主机的本地磁盘目录 ，来持久化存储容器的数据。
1.1 使用场景
典型的应用包括：分布式数据存储比如 MongoDB，分布式文件系统比如 GlusterFS、Ceph 等，以及需要在本地磁盘上进行大量数据缓存的分布式应用，其次使用 Local Persistent Volume 的应用必须具备数据备份和恢复的能力，允许你把这些数据定时备份在其他位置。
比如专门挂载一个高性能的ssd磁盘，使用Local PV将应用数据持久化到这个磁盘上，满足高性能IO的需求。
1.2 如何实现？
在宿主机NodeA上提前创建好目录 ，然后在定义Pod时添加nodeAffinity=NodeA，指定Pod在我们提前创建好目录的主机上运行。
但是我们绝不应该把一个宿主机上的目录当作 PV 使用，因为本地目录的磁盘随时都可能被应用写满，甚至造成整个宿主机宕机。
而且，不同的本地目录之间也缺乏哪怕最基础的 I/O 隔离机制。
所以，一个 Local Persistent Volume 对应的存储介质，一定是一块额外挂载在宿主机的磁盘或者块设备（“额外”的意思是，它不应该是宿主机根目录所使用的主硬盘）。
这个原则，我们可以称为“一个 PV 一块盘”。
对于常规的 PV，Kubernetes 都是先调度 Pod 到某个节点上，然后再持久化”这台机器上的 Volume 目录。
而 Local PV，则需要运维人员提前准备好节点的磁盘。
它们在不同节点上的挂载情况可以完全不同，甚至有的节点可以没这种磁盘。
所以调度器就必须能够知道所有节点与 Local Persistent Volume 对应的磁盘的关联关系，然后根据这个信息来调度 Pod。
也就是在调度的时候考虑Volume 分布。
需要创建一个 StorageClass 来描述这个 PV:
local-storage.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: local
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
provisioner 字段定义为no-provisioner，需要提前手动创建PV。
volumeBindingMode字段定义为WaitForFirstConsumer，它是 Local Persistent Volume 里一个非常重要的特性，即：延迟绑定。延迟绑定就是在我们提交PVC文件时，StorageClass为我们延迟绑定PV与PVC的对应关系。
这样做的原因是：比如我们在当前集群上有两个相同属性的PV，它们分布在不同的节点Node1和Node2上，而我们定义的Pod需要运行在Node1节点上 ，但是StorageClass已经为Pod声明的PVC绑定了在Node2上的PV，这样的话，Pod调度就会失败，所以我们要延迟StorageClass的绑定操作。
也就是延迟到到第一个声明使用该 PVC 的 Pod 出现在调度器之后，调度器再综合考虑所有的调度规则，当然也包括每个 PV 所在的节点位置，来统一决定，这个 Pod 声明的 PVC，到底应该跟哪个 PV 进行绑定。
手动创建 PV 的方式，在删除 PV 时需要按如下流程执行操作：
删除使用这个 PV 的 Pod
从宿主机移除本地磁盘（比如，umount 它）
删除 PVC
删除 PV
1. 安装
既然要基于 Kubernetes 来做 CI/CD，我们这里最好还是将 Jenkins 安装到 Kubernetes 集群当中：
jenkins.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-pv
spec:
storageClassName: local # Local PV
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
local:
path: /data/k8s/jenkins
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
namespace: devops
spec:
storageClassName: local
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: devops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments", "ingresses"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log", "events"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: devops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: devops
---
apiVersion: v1
kind: ConfigMap
metadata:
name: jenkins-mirror-conf
namespace: devops
data:
nginx.conf: |
user nginx;
worker_processes 3;
error_log /dev/stderr;
events {
worker_connections 10240;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time';
access_log /dev/stdout main;
server {
listen 80;
server_name mirrors.jenkins-ci.org;
location / {
proxy_redirect off;
proxy_pass https://mirrors.tuna.tsinghua.edu.cn/jenkins/;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Accept-Encoding "";
proxy_set_header Accept-Language "zh-CN";
}
index index.html index.htm index.php;
location ~ /\. {
deny all;
}
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: devops
spec:
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
serviceAccount: jenkins
hostAliases:
- ip: "127.0.0.1"
hostnames:
- "mirrors.jenkins-ci.org"
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
securityContext:
privileged: true
volumeMounts:
- name: jenkinshome
mountPath: /var/jenkins_home
containers:
- name: mirror
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/nginx
readOnly: true
name: nginx-conf
- name: jenkins
image: jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1500m
memory: 2048Mi
requests:
cpu: 1500m
memory: 2048Mi
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
mountPath: /var/jenkins_home
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: jenkins-pvc
- name: nginx-conf
configMap:
name: jenkins-mirror-conf
items:
- key: nginx.conf
path: nginx.conf
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: devops
labels:
app: jenkins
spec:
selector:
app: jenkins
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jenkins
namespace: devops
labels:
name: jenkins
spec:
ingressClassName: nginx # ingress控制器的名称
rules:
- host: jenkins.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: jenkins
port:
number: 8080
我们这里使用一个名为 jenkins/jenkins:lts 的镜像，这是 jenkins 官方的 Docker 镜像，我们这里使用默认的官方镜像就行，另外一个还需要注意的数据的持久化，将容器的 /var/jenkins_home 目录持久化即可，为了性能考虑，我们这里使用 Local PV，将 Pod 调度到固定的节点上。
由于我们这里使用的镜像内部运行的用户 uid=1000，所以我们这里挂载出来后会出现权限问题，为解决这个问题，我们同样还是用一个简单的 initContainer 来修改下我们挂载的数据目录。
另外我们这里还需要使用到一个拥有相关权限的 serviceAccount：jenkins，我们这里只是给 jenkins 赋予了一些必要的权限，当然如果你对 serviceAccount 的权限不是很熟悉的话，我们给这个 sa 绑定一个 cluster-admin 的集群角色权限也是可以的，当然这样具有一定的安全风险。
除此之外，这里我们还添加了一个额外的名为 mirror 的容器，添加这个容器的目的是使用一个 nginx 容器来反向代理 Jenkins 插件的官方源到清华大学的源上面，因为官方源实在是太慢了，我们这里将官方的镜像地址 mirrors.jenkins-ci.org 通过 hostAlias 映射到了 127.0.0.1 这个地址上，而这个地址恰好就是 mirror 这个 nginx 容器，我们通过一个 ConfigMap 来配置 Nginx，将 mirros.jenkins-ci.org 反向代理到了 proxy_pass https://mirrors.tuna.tsinghua.edu.cn/jenkins/;，这样当我们在 Jenkins 中要下载插件的时候实际上会被代理到清华的源上面去，这样就大大加快了插件下载的速度。
最后就是通过 Ingress 来暴露我们的服务。
我们直接来创建 jenkins 的资源清单即可：
[root@master jekines]# kubectl apply -f jenkins.yaml
persistentvolume/jenkins-pv created
persistentvolumeclaim/jenkins-pvc created
serviceaccount/jenkins created
clusterrole.rbac.authorization.k8s.io/jenkins created
clusterrolebinding.rbac.authorization.k8s.io/jenkins created
configmap/jenkins-mirror-conf created
deployment.apps/jenkins created
service/jenkins created
ingress.networking.k8s.io/jenkins created
[root@master jekines]# kubectl get pods -n devops -l app=jenkins
NAME READY STATUS RESTARTS AGE
jenkins-675cc9b744-kg6fz 2/2 Running 0 90s
[root@master jekines]# kubectl logs -f jenkins-675cc9b744-kg6fz jenkins -n devops
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
2022-10-25 12:39:32.505+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
2022-10-25 12:39:35.020+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2022-10-25 12:39:35.091+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-10.0.11; built: 2022-06-21T21:12:44.640Z; git: d988aa016e0bb2de6fba84c1659049c72eae3e32; jvm 11.0.16.1+1
2022-10-25 12:39:35.729+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2022-10-25 12:39:35.977+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
2022-10-25 12:39:37.425+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2022-10-25 12:39:37.712+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@150ede8b{Jenkins v2.361.2,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
2022-10-25 12:39:37.731+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@e84a8e1{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2022-10-25 12:39:37.782+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started Server@32c8e539{STARTING}[10.0.11,sto=0] @6608ms
2022-10-25 12:39:37.791+0000 [id=23] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2022-10-25 12:39:38.028+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
2022-10-25 12:39:38.087+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2022-10-25 12:39:39.273+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2022-10-25 12:39:39.276+0000 [id=31] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
2022-10-25 12:39:39.280+0000 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2022-10-25 12:39:39.681+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
2022-10-25 12:39:39.681+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
2022-10-25 12:39:39.681+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2022-10-25 12:39:39.682+0000 [id=28] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2022-10-25 12:39:39.718+0000 [id=44] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$1: Started Download metadata
2022-10-25 12:39:39.730+0000 [id=44] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/jenkins_home/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-10-25 12:39:41.616+0000 [id=29] INFO jenkins.install.SetupWizard#init:
*************************************************************
*************************************************************
*************************************************************
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
05bddb460b3349ccb7cfd844ed2abd73
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
*************************************************************
*************************************************************
2022-10-25 12:40:10.203+0000 [id=22] INFO hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
[root@master jekines]# kubectl get ingress -n devops
NAME CLASS HOSTS ADDRESS PORTS AGE
jenkins nginx jenkins.test.com 80 7m6s
看到 Jenkins is fully up and running 信息就证明我们的 Jenkins 应用以前启动起来了。
然后我们可以通过 Ingress中定义的域名 jenkins.test.com(需要做 DNS 解析或者在本地 /etc/hosts 中添加映射)来访问 jenkins 服务：
然后可以执行下面的命令获取解锁的管理员密码：
[root@master jekines]# kubectl exec -it jenkins-675cc9b744-kg6fz -c jenkins -n devops -- cat /var/jenkins_home/secrets/initialAdminPassword
05bddb460b3349ccb7cfd844ed2abd73
然后选择安装推荐的插件即可，由于我们已经做了插件的反向代理了，所以理论上安装速度会比较快
安装完成后添加管理员帐号即可进入到 jenkins 主界面：
2. 架构
Jenkins 安装完成了，接下来基于 Kubernetes 搭建 Jenkins 集群：
从图上可以看到 Jenkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。
这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。
那么我们使用这种方式带来了哪些好处呢？
服务高可用，当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master 容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用。
动态伸缩，合理使用资源，每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave 自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配 Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。
扩展性好，当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个 Kubernetes Node 到集群中，从而实现扩展。 是不是以前我们面临的种种问题在 Kubernetes 集群环境下面是不是都没有了啊？看上去非常完美。
3. 配置
接下来我们就需要来配置 Jenkins，让他能够动态的生成 Slave 的 Pod。
第1步. 我们需要安装 kubernetes 插件， 点击 Manage Jenkins -> Manage Plugins -> Available -> Kubernetes 勾选安装即可。
第2步. 安装完毕后，点击 Manage Jenkins
进入到集群配置
在该页面我们可以点击 Add a new cloud -> 选择 Kubernetes，然后填写 Kubernetes 和 Jenkins 配置信息：
第3步，配置Pod Template
这个地方要填写一个标签
添加一个容器：
名称必须是jnlp，镜像必须是jenkins/inbound-agent，运行命令和参数必须为空。
下面增加了 Host Path Volume：/run/containerd/containerd.sock、/root/.kube/、/etc/kubernetes/pki，这边便是为了 jenkins-slave 下有足够的权限可以执行 containerd及 kubectl 部署到 k8s 集群的权限
最后配置sa即可
保存。
到这里我们的 Kubernetes 插件就算配置完成了。
4. 测试
Kubernetes 插件的配置工作完成了，接下来我们就来添加一个 Job 任务，看是否能够在 Slave Pod 中执行，任务执行完成后看 Pod 是否会被销毁。
在 Jenkins 首页点击 create new jobs，创建一个测试的任务，输入任务名称，然后我们选择 Freestyle project 类型的任务
然后往下拉，在 Build 区域选择Execute shell
然后输入我们测试命令
ls -l
最后点击保存。
现在我们直接在页面点击左侧的 Build now 触发构建即可，然后观察 Kubernetes 集群中 Pod 的变化：
[root@master jekines]# kubectl get pods -n devops -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
jenkins-675cc9b744-kg6fz 2/2 Running 1 (158m ago) 3h21m 10.244.104.12 node2 <none> <none>
jenkins-slave-60l0t 1/1 Running 0 27s 10.244.166.132 node1 <none> <none>
我们可以看到在我们点击立刻构建的时候可以看到一个新的 Pod：jenkins-slave-60l0t 被创建了，这就是我们的 Jenkins Slave。任务执行完成后我们可以看到任务信息
[test] $ /bin/sh -xe /tmp/jenkins12274971254587118578.sh
+ ls -l
total 0
Finished: SUCCESS
到这里证明我们的任务已经构建完成，然后这个时候我们再去集群查看我们的 Pod 列表，发现 devops这个 namespace 下面已经没有之前的 Slave 这个 Pod 了。
到这里我们就完成了使用 Kubernetes 动态生成 Jenkins Slave 的方法。
5. 制作自定义的Jenkins-slave
上面我们使用的是官方默认的jenkins/inbound-agent，其中没有kubectl的命令，如果想要执行kubectl命令的话，就会出现错误，如果想要根据自己的需求，我们可以自己制作：
Dockerfile
debain
FROM jenkins/inbound-agent
ARG MAVEN_VERSION=3.6.3
ARG KUBECTL_VERSION=v1.24.0
USER root
RUN apt-get update && \
apt-get install -y curl && \
apt-get clean
# maven
RUN curl -OL https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
tar -zxf ./apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
mv apache-maven-${MAVEN_VERSION} /usr/local && \
rm -f apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
ln -s /usr/local/apache-maven-${MAVEN_VERSION}/bin/mvn /usr/bin/mvn && \
ln -s /usr/local/apache-maven-${MAVEN_VERSION} /usr/local/apache-maven && \
mkdir -p /home/jenkins/.m2 && \
chown -R jenkins:jenkins /home/jenkins/.m2
RUN curl -OL https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl && \
chmod +x ./kubectl && \
mv ./kubectl /usr/local/bin/kubectl && \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
echo $TZ > /etc/timezone
VOLUME [ "/home/jenkins/.m2" ]
USER jenkins
ENTRYPOINT ["/usr/local/bin/jenkins-agent"]
[root@master jekines]# docker build -t jenkins-slave-kubectl .
[root@master jekines]# docker tag jenkins-slave-kubectl mszlu/jenkins-slave-kubectl:1.0
[root@master jekines]# docker push mszlu/jenkins-slave-kubectl:1.0
修改配置：
修改任务脚本
kubectl get pods
[test] $ /bin/sh -xe /tmp/jenkins1539373708302133389.sh
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-675cc9b744-kg6fz 2/2 Running 1 (4h31m ago) 5h14m
jenkins-slave-31rq1 1/1 Running 0 23s
Finished: SUCCESS
实际构建中，会涉及java，git，go，maven，docker，kubectl等等的，需要根据实际情况，进行Dockerfile的构建，做属于自己的镜像
6. 最新版本安装
在录制的时候 Jenkins已经更新了新版本，在安装的过程中和上面的教程可能不匹配，导致一些问题，按照最新的版本 重新说一下流程。
新版插件和更新文件 都是使用的https://updates.jenkins.io，所以需要生成证书去做nginx反向代理
[root@master jenkins]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
--2022-11-01 12:59:39-- https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
Resolving pkg.cfssl.org (pkg.cfssl.org)... 104.18.17.120, 104.18.16.120, 2606:4700::6812:1078, ...
Connecting to pkg.cfssl.org (pkg.cfssl.org)|104.18.17.120|:443... connected.
Unable to establish SSL connection.
[root@master jenkins]# export https_proxy=http://192.168.124.4:10809
[root@master jenkins]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
--2022-11-01 13:00:28-- https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl_linux-amd64 [following]
--2022-11-01 13:00:30-- https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl_linux-amd64
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/21591001/6deaa080-9ebe-11eb-919d-cbab8a7bb20b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221101T170032Z&X-Amz-Expires=300&X-Amz-Signature=39a1336907eb4b56e2c4db176dd5f13e52ddd44b59540be3e308e412181988c5&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=21591001&response-content-disposition=attachment%3B%20filename%3Dcfssl_linux-amd64&response-content-type=application%2Foctet-stream [following]
--2022-11-01 13:00:32-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/21591001/6deaa080-9ebe-11eb-919d-cbab8a7bb20b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221101T170032Z&X-Amz-Expires=300&X-Amz-Signature=39a1336907eb4b56e2c4db176dd5f13e52ddd44b59540be3e308e412181988c5&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=21591001&response-content-disposition=attachment%3B%20filename%3Dcfssl_linux-amd64&response-content-type=application%2Foctet-stream
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 200 OK
Length: 10376657 (9.9M) [application/octet-stream]
Saving to: ‘/usr/bin/cfssl’
100%[=========================================================================================>] 10,376,657 508KB/s in 20s
2022-11-01 13:00:54 (514 KB/s) - ‘/usr/bin/cfssl’ saved [10376657/10376657]
[root@master jenkins]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json
--2022-11-01 13:01:39-- https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssljson_linux-amd64 [following]
--2022-11-01 13:01:41-- https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssljson_linux-amd64
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/21591001/8a86d880-9ebe-11eb-9d16-2fd0c4fe9f34?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221101T170142Z&X-Amz-Expires=300&X-Amz-Signature=98fb65926f1a52ca1b4bd37416ddc29195e7d8a20ab1910ec0d640530d071b87&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=21591001&response-content-disposition=attachment%3B%20filename%3Dcfssljson_linux-amd64&response-content-type=application%2Foctet-stream [following]
--2022-11-01 13:01:43-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/21591001/8a86d880-9ebe-11eb-9d16-2fd0c4fe9f34?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221101T170142Z&X-Amz-Expires=300&X-Amz-Signature=98fb65926f1a52ca1b4bd37416ddc29195e7d8a20ab1910ec0d640530d071b87&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=21591001&response-content-disposition=attachment%3B%20filename%3Dcfssljson_linux-amd64&response-content-type=application%2Foctet-stream
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2277873 (2.2M) [application/octet-stream]
Saving to: ‘/usr/bin/cfssl-json’
100%[=========================================================================================>] 2,277,873 659KB/s in 3.4s
2022-11-01 13:01:49 (659 KB/s) - ‘/usr/bin/cfssl-json’ saved [2277873/2277873]
[root@master jenkins]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo
--2022-11-01 13:01:52-- https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl-certinfo_linux-amd64 [following]
--2022-11-01 13:01:54-- https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl-certinfo_linux-amd64
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/21591001/7b078f80-9ebe-11eb-8422-7005df0eb28f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221101T170155Z&X-Amz-Expires=300&X-Amz-Signature=d767a10f471892380f2927b954635269ddda08ad4345fda96b35686b5dcdc90e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=21591001&response-content-disposition=attachment%3B%20filename%3Dcfssl-certinfo_linux-amd64&response-content-type=application%2Foctet-stream [following]
--2022-11-01 13:01:56-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/21591001/7b078f80-9ebe-11eb-8422-7005df0eb28f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221101%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221101T170155Z&X-Amz-Expires=300&X-Amz-Signature=d767a10f471892380f2927b954635269ddda08ad4345fda96b35686b5dcdc90e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=21591001&response-content-disposition=attachment%3B%20filename%3Dcfssl-certinfo_linux-amd64&response-content-type=application%2Foctet-stream
Connecting to 192.168.124.4:10809... connected.
Proxy request sent, awaiting response... 200 OK
Length: 6595195 (6.3M) [application/octet-stream]
Saving to: ‘/usr/bin/cfssl-certinfo’
100%[=========================================================================================>] 6,595,195 2.62MB/s in 2.4s
2022-11-01 13:02:00 (2.62 MB/s) - ‘/usr/bin/cfssl-certinfo’ saved [6595195/6595195]
[root@master jenkins]# chmod +x /usr/bin/cfssl*
[root@master jenkins]# mkdir /opt/certs
[root@master jenkins]# cd /opt/certs
[root@master certs]# vim /opt/certs/ca-csr.json
{
"CN": "AuthEdu",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
],
"ca": {
"expiry": "175200h"
}
}
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"ca-csr.json" [New] 21L, 326C written
[root@master certs]# cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
2022/11/01 13:03:05 [INFO] generating a new CA key and certificate from CSR
2022/11/01 13:03:05 [INFO] generate received request
2022/11/01 13:03:05 [INFO] received CSR
2022/11/01 13:03:05 [INFO] generating key: rsa-2048
2022/11/01 13:03:05 [INFO] encoded CSR
2022/11/01 13:03:05 [INFO] signed certificate with serial number 46681810998533220466321279198559557108480038921
[root@master certs]# vim /opt/certs/ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
~
~
~
"ca-config.json" [New] 34L, 837C written
[root@master certs]# vim updates.jenkins.io-csr.json
{
"CN": "updates.jenkins.io",
"hosts": [
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
],
"Hosts": ["updates.jenkins.io"]
}
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"updates.jenkins.io-csr.json" [New] 19L, 327C written
[root@master certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server updates.jenkins.io-csr.json|cfssl-json -bare updates.jenkins.io
2022/11/01 13:03:57 [INFO] generate received request
2022/11/01 13:03:57 [INFO] received CSR
2022/11/01 13:03:57 [INFO] generating key: rsa-2048
2022/11/01 13:03:57 [INFO] encoded CSR
2022/11/01 13:03:57 [INFO] signed certificate with serial number 105858123765656509250033408948077023314843384086
2022/11/01 13:03:57 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master certs]# ll
total 36
-rw-r--r-- 1 root root 837 Nov 1 13:03 ca-config.json
-rw-r--r-- 1 root root 993 Nov 1 13:03 ca.csr
-rw-r--r-- 1 root root 326 Nov 1 13:02 ca-csr.json
-rw------- 1 root root 1675 Nov 1 13:03 ca-key.pem
-rw-r--r-- 1 root root 1338 Nov 1 13:03 ca.pem
-rw-r--r-- 1 root root 1070 Nov 1 13:03 updates.jenkins.io.csr
-rw-r--r-- 1 root root 327 Nov 1 13:03 updates.jenkins.io-csr.json
-rw------- 1 root root 1679 Nov 1 13:03 updates.jenkins.io-key.pem
-rw-r--r-- 1 root root 1419 Nov 1 13:03 updates.jenkins.io.pem
[root@master certs]# mkdir -p /data/nginx/conf/certs
[root@master certs]# mkdir -p /data/jenkins
[root@master certs]# vim /etc/exports
/data/harbor *(insecure,rw,sync,no_root_squash)
/data/mysql *(insecure,rw,sync,no_root_squash)
/data/wordpress *(insecure,rw,sync,no_root_squash)
/data/thanos *(insecure,rw,sync,no_root_squash)
/data/log *(insecure,rw,sync,no_root_squash)
/data/gitlab/config *(insecure,rw,sync,no_root_squash)
/data/gitlab/logs *(insecure,rw,sync,no_root_squash)
/data/gitlab/data *(insecure,rw,sync,no_root_squash)
/data/gitlab/redis *(insecure,rw,sync,no_root_squash)
/data/gitlab/postgresql *(insecure,rw,sync,no_root_squash)
/data/nginx/conf *(insecure,rw,sync,no_root_squash)
/data/jenkins *(insecure,rw,sync,no_root_squash)
[root@master certs]# systemctl restart nfs
[root@master certs]# cp updates.jenkins.io.pem /data/nginx/conf/certs/
[root@master certs]# cp updates.jenkins.io-key.pem /data/nginx/conf/certs/
[root@master certs]# ls /data/nginx/conf/certs/
updates.jenkins.io-key.pem updates.jenkins.io.pem
证书java是不会信任的，所以需要添加jdk的证书信任：
/opt/java/openjdk/lib/security
[root@master ca]# openssl x509 -in /opt/certs/ca.pem -out ca.crt
[root@master ca]# ls
ca.crt
[root@master ca]# cp ca.crt /data/jenkins/
[root@master ca]# ls /data/jenkins/
ca.crt
重新构建jenkins包，让其成为root用户
FROM jenkins/jenkins:2.375
USER root
RUN mkdir -p /data/jenkins
ADD ca.crt /data/jenkins
RUN ["sh","-c","/opt/java/openjdk/bin/keytool -import -trustcacerts -keystore /opt/java/openjdk/lib/security/cacerts -storepass changeit -noprompt -alias mycert -file /data/jenkins/ca.crt"]
jenkins.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins-pv
spec:
storageClassName: local # Local PV
capacity:
storage: 2Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
local:
path: /data/k8s/jenkins
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- node2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pvc
namespace: devops
spec:
storageClassName: local
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
namespace: devops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins
rules:
- apiGroups: ["extensions", "apps"]
resources: ["deployments", "ingresses"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["services"]
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log", "events"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins
namespace: devops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
namespace: devops
---
apiVersion: v1
kind: ConfigMap
metadata:
name: jenkins-mirror-conf
namespace: devops
data:
nginx.conf: |
user nginx;
worker_processes 3;
error_log /dev/stderr;
events {
worker_connections 10240;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time';
access_log /dev/stdout main;
server{
listen 80;
server_name updates.jenkins.io;
rewrite ^(.*)$ https://${server_name}$1 permanent;
}
server {
listen 443 ssl;
server_name updates.jenkins.io;
ssl_certificate /etc/nginx/conf/certs/updates.jenkins.io.pem;
ssl_certificate_key /etc/nginx/conf/certs/updates.jenkins.io-key.pem;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 10m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
proxy_redirect off;
proxy_pass https://mirrors.tuna.tsinghua.edu.cn/jenkins;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Accept-Encoding "";
proxy_set_header Accept-Language "zh-CN";
}
location ^~ /download/
{
proxy_pass https://mirrors.tuna.tsinghua.edu.cn/jenkins/;
}
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: devops
spec:
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
serviceAccount: jenkins
hostAliases:
- ip: "127.0.0.1"
hostnames:
- "updates.jenkins.io"
containers:
- name: mirror
image: nginx:1.7.9
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/nginx
readOnly: true
name: nginx-conf
- mountPath: /etc/nginx/conf
name: nginx-certs
- name: jenkins
image: mszlu/jenkins-root:2.375
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
ports:
- containerPort: 8080
name: web
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
resources:
limits:
cpu: 1500m
memory: 2048Mi
requests:
cpu: 1500m
memory: 2048Mi
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
volumeMounts:
- name: jenkinshome
mountPath: /var/jenkins_home
- mountPath: /data/jenkins
name: jenkins-certs
volumes:
- name: jenkinshome
persistentVolumeClaim:
claimName: jenkins-pvc
- name: emptydir
emptyDir: {}
- name: nginx-conf
configMap:
name: jenkins-mirror-conf
items:
- key: nginx.conf
path: nginx.conf
- name: jenkins-certs
nfs:
server: 192.168.200.101
path: /data/jenkins
- name: nginx-certs
nfs:
server: 192.168.200.101
path: /data/nginx/conf
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
namespace: devops
labels:
app: jenkins
spec:
selector:
app: jenkins
ports:
- name: web
port: 8080
targetPort: web
- name: agent
port: 50000
targetPort: agent
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jenkins
namespace: devops
labels:
name: jenkins
spec:
ingressClassName: nginx # ingress控制器的名称
rules:
- host: jenkins.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: jenkins
port:
number: 8080
启动完成进入容器，执行证书导入：
[root@master jenkins]# kubectl get pods -n devops
NAME READY STATUS RESTARTS AGE
jenkins-6649b98df4-vjm5c 2/2 Running 0 34sDevOps（3）--Jenkins
http://47.123.5.226:8090//archives/devops-3---jenkins