references:

podTempate

Jenkins Kubernetes Plugin: Running Agents In Other Clusters

Jenkins Kubernetes Plugin: Using the plugin in your pipelines

Create Kubernetes Service Accounts and Kubeconfigs

Setup Jenkins Pipeline and Blue Ocean in Kubernetes

* How to Setup Jenkins Build Agents on Kubernetes Pods

Pull an Image from a Private Registry

从私有仓库拉取镜像

How to access webSockets for Jenkins inbound agents?

namespace

$ cat << EOF | kubectl apply -f -
  ---
  apiVersion: v1
  kind: Namespace
  metadata:
    name: jenkins
EOF
namespace/jenkins created

quota

$ cat << EOF | kubectl apply -f -
  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: jenkins-quota
    namespace: jenkins
  spec:
    hard:
      requests.cpu: "16"
      requests.memory: 16Gi
      limits.cpu: "32"
      limits.memory: 32Gi
EOF
resourcequota/jenkins-quota edited

sa.yml

jenkins.io

  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: jenkins-admin
  rules:
    - apiGroups: [""]
      resources: ["*"]
      verbs: ["*"]
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: jenkins-admin
    namespace: devops-tools
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: jenkins-admin
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: jenkins-admin
  subjects:
  - kind: ServiceAccount
    name: jenkins-admin
    namespace: devops-tools

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: devops-tools
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins
  namespace: devops-tools
  labels:
    "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-role-binding
  namespace: devops-tools
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: devops-tools

grant more permissions via RoleBinding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: devops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: devops

generate credentials for pfx

ca.crt

$ grep certificate-authority-data ~/.kube/config | awkubectl -F': ' '{print $NF}' |  base64 -d > ca.crt
# OR
$ sudo cat /etc/kubernetes/pki/ca.crt

client.crt & client.key

$ grep client-certificate-data ~/.kube/config | awkubectl -F': ' '{print $NF}' |  base64 -d > client.crt
$ grep client-key-data ~/.kube/config | awkubectl -F': ' '{print $NF}' |  base64 -d > client.key

cert.pfx

$ openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
Enter Export Password:
Verifying - Enter Export Password:

$ ls
ca.crt  cert.pfx  client.crt  client.key

# or

# using password 'marslo'
$ openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt -password pass:marslo

full steps

$ cat ~/.kube/config \
      | grep certificate-authority-data \
      | awkubectl '{print $2}' \
      | base64 -d > ca.crt
$ cat ~/.kube/config \
      | grep client-certificate-data \
      | awkubectl '{print $2}' \
      | base64 -d > client.crt
$ cat ~/.kube/config \
      | grep client-key-data \
      | awkubectl '{print $2}' \
      | base64 -d > client.key
$ openssl pkcs12 -export \
                 -out cert.pfx \
                 -in client.crt \
                 -inkey client.key \
                 -certfile ca.crt \
                 -password pass:devops

configure in jenkins

[!NOTE]

url

by default: http://<service-name>.<namespace>.svc.cluster.local:8080

Manage Jenkins -> Configure System or Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds
Add a new Cloud -> Kuberentes
- Name:
- Kubernetes URL:
  - get from $ kubectl cluster-info
  - using https://kubernetes.default.svc.cluster.local
- Kubernetes server certificate key: content of ca.crt. ($ cat ca.crt)
- Credentials:
  - Add -> Jenkins
  - Kind: Certificate
1.5.8.1.1 -- plugin-1

1.5.8.1.2 -- plugin-2

1.5.8.1.3 -- plugin-3

1.5.8.1.4 -- plugin-4
setup in jenkins

1.5.8.1.5 -- plugin-5

using kubeconfig for remote cluster credential

reference:

jenkinsci/kubernetes-cli-plugin

Kubernetes Plugin: Authenticate with a ServiceAccount to a remote cluster

Creating a kubeconfig file for a self-hosted Kubernetes cluster

How to find your Jenkins admin password on Kubernetes

get Kubernetes URL

$ kubectl config view --minify | sed -n -re 's/^.*server: (https.*)$/\1/p'

$ kubectl config view --minify --raw --output 'jsonpath={..cluster.server}'

generate CA

the content can be also found in kubernetes-master:/etc/kubernetes/pki/ca.crt

$ kubectl -n jenkins get secret \
              $(kubectl -n jenkins get sa jenkins-admin -o jsonpath={.secrets[0].name}) \
              -o jsonpath={.data.'ca\.crt'} \
              | base64 --decode

$ cat ~/.kube/config \
      | grep certificate-authority-data \
      | awkubectl '{print $2}' \
      | base64 -d > ca.crt

Generate token in kubernetes

$ namespace='devops'
$ serviceAccount='jenkins-admin'
$ alias k='kubectl'

setup sa

$ kubectl -n jenkins create sa jenkins-admin
$ kubectl -n jenkins create rolebinding jenkins-admin-binding \
                     --clusterrole=cluster-admin \
                     --serviceaccount=devops:jenkins-admin

get token

$ kubectl -n jenkins \
       get sa jenkins-admin \
       -o go-template \
       --template='{{range .secrets}}{{.name}}{{"\n"}}{{end}}'
jenkins-admin-token-kshsh

$ kubectl -n jenkins \
       get secrets jenkins-admin-token-kshsh \
       -o go-template \
       --template '{{index .data "token"}}' \
       | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.***

more info

$ kubectl get secret -n jenkins
NAME                        TYPE                                  DATA   AGE
default-token-8k7vj         kubernetes.io/service-account-token   3      27m
jenkins-admin-token-9r8pt   kubernetes.io/service-account-token   3      21m

$ kubectl -n jenkins get rolebinding
NAME                    AGE
jenkins-admin-binding   15m
$ kubectl -n jenkins describe rolebinding jenkins-admin-binding
Name:         jenkins-admin-binding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name           Namespace
  ----            ----           ---------
  ServiceAccount  jenkins-admin  jenkins

$ kubectl describe clusterrolebindings jenkins-admin-cluster-binding
Name:         jenkins-admin-cluster-binding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name           Namespace
  ----            ----           ---------
  ServiceAccount  jenkins-admin  jenkins

$ kubectl -n jenkins \
    get secret \
    $(kubectl -n jenkins get sa jenkins-admin -o jsonpath={.secrets[0].name}) \
    -o jsonpath={.data.token} \
    | base64 --decode

$ kubectl -n jenkins \
       get sa jenkins-admin \
       --template='{{range .secrets}}{{ .name }} {{end}}' \
       | xargs -n 1 kubectl -n jenkins get secret \
                      --template='{{ if .data.token }}{{ .data.token }}{{end}}' \
                      | head -n 1 \
                      | base64 -d -

# sa='jenkins-admin'
$ kubectl -n jenkins \
             get secrets \
             $(kubectl -n jenkins get sa ${sa} -o=jsonpath='{.secrets[0].name}') \
             -o=jsonpath='{.data.token}' |
          base64 -d

setup in Jenkins

credential setup
cloud setup

using ClusterRoleBinding

references:

Configure Service Accounts for Pods

RoleBinding and ClusterRoleBinding

Role and ClusterRole

Jenkins minimal installation on Kubernetes

[!TIP] simplely it can be executed via commands:
$ kubectl -n kube-system create sa <service-account>
$ kubectl create clusterrolebinding <role-binding-name> --clusterrole cluster-admin --serviceaccount=<namespace>:<service-account>
[!NOTE]: <role-binding-name> can be the same as <service-account>

---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins-admin
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
  labels:
    k8s-app: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: jenkins

using RoleBinding

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
 name: jenkins-admin
 namespace: jenkins
rules:
- apiGroups: [""]
 resources: ["pods"]
 verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
 resources: ["pods/exec"]
 verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
 resources: ["pods/log"]
  verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: jenkins-admin
 namespace: jenkins
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: jenkins-admin
subjects:
- kind: ServiceAccount
 name: jenkins-admin

or grant more permissions via RoleBinding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: devops
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: devops

or get all permission via RoleBinding

# spinnaker-role-and-rolebinding-target.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spinnaker-role
  namespace: target                       # Should be namespace you are granting access to
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spinnaker-rolebinding
  namespace: target                       # Should be namespace you are granting access to
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: spinnaker-role                    # Should match name of Role
subjects:
- namespace: source                       # Should match namespace where SA lives
  kind: ServiceAccount
  name: spinnaker-service-account         # Should match service account name, above

or details rules in role binding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: devops-tools
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins
  namespace: devops-tools
  labels:
    "app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-role-binding
  namespace: devops-tools
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: devops-tools

pull with credentials

[!NOTE|label:references:]

Pull an Image from a Private Registry

从私有仓库拉取镜像

为特定名称空间创建 imagePullSecrets

in kubernetes

# create secrets in namespace

## via ~/.docker/config.json
$ kubectl -n devfops create secret generic image-pull-secrets \
          --from-file=.dockerconfigjson=.docker/config.json \
          --type=kubernetes.io/dockerconfigjson

## via cmd
$ kubectl -n devops create secret docker-registry image-pull-secrets \
          --docker-server=artifactory.example.com \
          --docker-username=devops \
          --docker-password=password \
          --docker-email=devops@example.com

# check result
$ kubectl -n devop get secret image-pull-secrets \
          -o jsonpath="{.data.\.dockerconfigjson}" |
  base64 -d

copy secrets to all namespaces

# copy
$ kubectl get ns -o custom-columns=":metadata.name" --no-headers |
          xargs -t -i bash -c "echo -e \"\\n-- {} --\"; kubectl -n devops get secrets image-pull-secrets -o yaml --export | kubectl apply -n {} -f -"

# check
$ kubectl get secrets --all-namespaces  | grep image-pull-secrets

in podTemplate

podTemplate( cloud: 'Staging Kubernetes',
             label: env.BUILD_TAG ,
             name: env.BUILD_TAG,
             showRawYaml: true,
             namespace: 'devops',
             yaml: """
               apiVersion: v1
               kind: Pod
               spec:
                 hostNetwork: true
                 nodeSelector:
                   kubernetes.io/hostname: "staging-node-1"
                 imagePullSecrets:
                 - name: "image-pull-secrets"             # secrets name here
                 containers:
                 - name: jnlp
                   image: 'artifactory.example.com/docker/sandbox:bionic'
                   workingDir: '/home/devops'
                   resources:
                     limits:
                       memory: "1024Mi"
                       cpu: "512m"
                     requests:
                       memory: "512Mi"
                       cpu: "256m"
             """
) { node( env.BUILD_TAG ) { container('jnlp') {
    stage('show info') {
      sh """
        id
        whoami
        echo ${WORKSPACE}
        realpath ${WORKSPACE}
      """
      println POD_CONTAINER
    }
  }}
}

Q&A

[!NOTE]

Kubernetes agents are failing with SocketTimeoutException: connect timed out

Message: pods is forbidden: User "system:serviceaccount:jenkins:jenkins-admin" cannot list resource "pods" in API group solution:

lachie83/jenkins-service-account

see also

chukaofili/k8s-dashboard-admin-user.yaml

$ kubectl -n <namespace> create rolebinding <sa>-binding \
                                --clusterrole=cluster-admin \
                                --serviceaccount=<namespace>:<sa>
$ kubectl create clusterrolebinding jenkins-admin-cluster-binding \
                                    --clusterrole cluster-admin \
                                    --serviceaccount=<namespace>:jenkins-admin

thinking

$ kubectl -n jenkins auth can-i list pods --as jenkins-admin
no

$ kubectl -n jenkins auth can-i list pods --as jenkins-admin-binding
no

$ kubectl -n jenkins auth can-i list pods --as jenkins-admin-cluster-binding
no

kubernetes

namespace

namespace

quota

sa.yml

generate credentials for pfx

ca.crt

client.crt & client.key

cert.pfx

full steps

configure in jenkins

using kubeconfig for remote cluster credential

get Kubernetes URL

generate CA

Generate token in kubernetes

setup in Jenkins

using ClusterRoleBinding

pull with credentials

Q&A

results matching ""

No results matching ""