• upgrading jenkins
• java configuration
• threadDump
• agent
• Mailing format
• Properties in Jenkins Core for JAVA_OPTS
• System Properties
• Configuring HTTP
• tips

upgrading jenkins

jenkins 2.357 requires Java 11

• get Java version from Jenkins master

  println " >> jenkins.rootUrl: ${Jenkins.instance.rootUrl} "
  println " >> jenkins.version: ${Jenkins.instance.version} "
  System.getProperties().findAll { k, v ->
    k.toLowerCase().contains( 'java' )
  }.each { k, v ->
    println " >>> ${k} ~> ${v} "
  }
  
  "DONE"
  

  • result

     >> jenkins.rootUrl: https://my-dev.jenkins.com/
     >> jenkins.version: 2.360
     >>> java.specification.version ~> 11
     >>> java.runtime.version ~> 11.0.15+10
     >>> java.class.path ~> /usr/share/jenkins/jenkins.war
     >>> java.vm.vendor ~> Eclipse Adoptium
     >>> javamelody.maven-repositories ~> /var/jenkins_home/.m2/repository,http://repo1.maven.org/maven2,http://repo.jenkins-ci.org/public
     >>> java.runtime.name ~> OpenJDK Runtime Environment
     >>> javamelody.gzip-compression-disabled ~> true
     >>> java.vendor.url ~> https://adoptium.net/
     >>> javamelody.About Monitoring ~> https://plugins.jenkins.io/monitoring/
     >>> java.vm.specification.version ~> 11
     >>> java.vm.name ~> OpenJDK 64-Bit Server VM
     >>> java.vendor.version ~> Temurin-11.0.15+10
     >>> javamelody.http-transform-pattern ~> /\d+/|(?<=/static/|/adjuncts/|/bound/)[\w\-]+|(?<=/ws/|/user/|/testReport/|/javadoc/|/site/|/violations/file/|/cobertura/).+|(?<=/job/).+(?=/descriptorByName/)
     >>> sun.java.launcher ~> SUN_STANDARD
     >>> sun.java.command ~> /usr/share/jenkins/jenkins.war
     >>> java.vendor.url.bug ~> https://github.com/adoptium/adoptium-support/issues
     >>> javamelody.system-actions-enabled ~> true
     >>> java.io.tmpdir ~> /tmp
     >>> javamelody.storage-directory ~> //var/jenkins_home/monitoring
     >>> java.version ~> 11.0.15
     >>> javamelody.custom-reports ~> Jenkins Info,About Monitoring
     >>> java.specification.vendor ~> Oracle Corporation
     >>> java.vm.specification.name ~> Java Virtual Machine Specification
     >>> java.awt.printerjob ~> sun.print.PSPrinterJob
     >>> java.version.date ~> 2022-04-19
     >>> java.home ~> /opt/java/openjdk
     >>> java.vm.compressedOopsMode ~> Zero based
     >>> java.library.path ~> /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
     >>> java.specification.name ~> Java Platform API Specification
     >>> java.vm.specification.vendor ~> Oracle Corporation
     >>> java.vm.info ~> mixed mode
     >>> java.vendor ~> Eclipse Adoptium
     >>> java.vm.version ~> 11.0.15+10
     >>> java.awt.graphicsenv ~> sun.awt.X11GraphicsEnvironment
     >>> java.awt.headless ~> true
     >>> javamelody.csrf-protection-enabled ~> true
     >>> java.class.version ~> 55.0
     >>> javamelody.Jenkins Info ~> /systemInfo
     >>> javamelody.no-database ~> true
    Result: DONE
    

• get Java version from Jenkins agent

  println " >> jenkins.rootUrl: ${Jenkins.instance.rootUrl}"
  println " >> jenkins.version: ${Jenkins.instance.version}"
  
  Jenkins.instance.nodes.each { agent ->
    println " >> agent: ${agent.displayName}"
    agent.computer.getSystemProperties().findAll{ k, v ->
      k.toLowerCase().contains( 'java' )
    }.each { k, v ->
      println " ~~> ${k} -> ${v}"
    }
  }
  

  • result

    >> jenkins.rootUrl: https://my-dev.jenkins.com/
    >> jenkins.version: 2.360
    >> agent: yaml-15-x946w-s48xb-jqkn0
    ~~> java.awt.graphicsenv -> sun.awt.X11GraphicsEnvironment
    ~~> java.awt.printerjob -> sun.print.PSPrinterJob
    ~~> java.class.path -> /usr/share/jenkins/slave.jar
    ~~> java.class.version -> 55.0
    ~~> java.home -> /usr/lib/jvm/java-11-openjdk-amd64
    ~~> java.io.tmpdir -> /tmp
    ~~> java.library.path -> /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
    ~~> java.runtime.name -> OpenJDK Runtime Environment
    ~~> java.runtime.version -> 11.0.15+10-Ubuntu-0ubuntu0.18.04.1
    ~~> java.specification.name -> Java Platform API Specification
    ~~> java.specification.vendor -> Oracle Corporation
    ~~> java.specification.version -> 11
    ~~> java.vendor -> Private Build
    ~~> java.vendor.url -> Unknown
    ~~> java.vendor.url.bug -> Unknown
    ~~> java.version -> 11.0.15
    ~~> java.version.date -> 2022-04-19
    ~~> java.vm.compressedOopsMode -> Zero based
    ~~> java.vm.info -> mixed mode, sharing
    ~~> java.vm.name -> OpenJDK 64-Bit Server VM
    ~~> java.vm.specification.name -> Java Virtual Machine Specification
    ~~> java.vm.specification.vendor -> Oracle Corporation
    ~~> java.vm.specification.version -> 11
    ~~> java.vm.vendor -> Private Build
    ~~> java.vm.version -> 11.0.15+10-Ubuntu-0ubuntu0.18.04.1
    ~~> javamelody.no-database -> true
    ~~> sun.java.command -> hudson.remoting.jnlp.Main -headless -tunnel 1.2.3.4:12345 -url https://my-dev.jenkins.com/ -workDir /home/devops ae62043877285d6ba763f254ce041f64674ce2c4768d9872621af0ea65c07b2d yaml-15-x946w-s48xb-jqkn0
    ~~> sun.java.launcher -> SUN_STANDARD
    Result: [KubernetesSlave name: yaml-15-x946w-s48xb-jqkn0]
    

• or via RemotingDiagnostics

  import hudson.util.RemotingDiagnostics
  import jenkins.model.Jenkins
  
  println " >> jenkins.rootUrl: ${Jenkins.instance.rootUrl}"
  println " >> jenkins.version: ${Jenkins.instance.version}"
  
  String result
  String javaVersion = """
    def stdout = new StringBuffer()
    def stderr = new StringBuffer()
    "java -version".execute().waitForProcessOutput( stdout, stderr )
    println stderr
  """
  
  Jenkins.instance.nodes.each { agent ->
    println ">> agent: ${agent.displayName}"
    result = RemotingDiagnostics.executeGroovy( javaVersion.trim(), agent.channel )
                    .tokenize( '\n' )
                    .collect{ "\t${it}" }
                    .join( '\n' )
  }
  println ">>> java version: \n${result}"
  

  • result

    >> jenkins.rootUrl: https://my-dev.jenkins.com/
    >> jenkins.version: 2.360
    >> agent: yaml-15-x946w-s48xb-jqkn0
    >>> java version:
      openjdk version "11.0.15" 2022-04-19
      OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1)
      OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1, mixed mode, sharing)
    

java configuration

[!TIP|label:official recommended]

• ** A. Java Configuration
• ** Memory problem: 'unable to create new native thread'
• * Too many open files
• * Supported Java 8 arguments
• * Supported Java 11 arguments
• ** Java Heap settings Best Practice
  • Minimum and maximum heap sizes
• Enterprise JVM Administration and Jenkins Performance
• CloudBees Jenkins JVM troubleshooting
• How to Troubleshoot and Address Jenkins Startup Performances
• How to generate a thread dump?
• Jenkins : Obtaining a thread dump
• Required Data: Jenkins Hang Issue On Linux
• Upgrading to Jenkins 2.176.2 : Improved CSRF protection
• JENKINS-71273: Gerrit Trigger fails to connect Gerrit 2.14 - com.jcraft.jsch.JSchException: verify: false

  -Djsch.client_pubkey="ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa"
  -Djsch.server_host_key="ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa"
  

• java 11

  [!NOTE]

  • Omitting -XX:+UnlockDiagnosticVMOptions or -XX:+UnlockExperimentalVMOptions might cause your instance to fail to startup.
  • To increase GC logs to a longer period of time, we suggest
    • increasing the value of the arguments -Xlog option filecount=2 and/or filesize=100M
    • and as ultimate option use file=${LOGDIR}/gc-%t.log instead of file=${LOGDIR}/gc.log.
    • With the parameter %t, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.

  -XX:+AlwaysPreTouch
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=${LOGDIR}
  -XX:+UseG1GC
  -XX:+UseStringDeduplication
  -XX:+ParallelRefProcEnabled
  -XX:+DisableExplicitGC
  -XX:+UnlockDiagnosticVMOptions
  -XX:+UnlockExperimentalVMOptions
  -Xlog:gc*=info,gc+heap=debug,gc+ref*=debug,gc+ergo*=trace,gc+age*=trace:file=${LOGDIR}/gc.log:utctime,pid,level,tags:filecount=2,filesize=100M
  -XX:ErrorFile=${LOGDIR}/hs_err_%p.log
  -XX:+LogVMOutput
  -XX:LogFile=${LOGDIR}/jvm.log
  # Heap setting for CloudBees CI on modern cloud platforms:
  #   -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
  # Heap setting for CloudBees CI on traditional platforms:
  #   Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"
  

• java 8

  [!NOTE]

  • To increase GC logs to a longer period of time, we suggest
    • increasing the value of the arguments -XX:GCLogFileSize and -XX:NumberOfGCLogFiles
    • and as ultimate option use -Xloggc:${LOGDIR}/gc-%t.log instead of -Xloggc:${LOGDIR}/gc.log.
    • With the parameter %t, the JVM create a new set of GC files each time that the instance is restarted. It is well known that when the GC log folder gets big enough in terms of size, the support bundle might produce performance issues in the instance given that it needs to compress all of them.

  -Dhudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID=true
  -Djenkins.model.Jenkins.logStartupPerformance=true
  -Xms192G
  -Xmx192G
  -XX:+AlwaysPreTouch
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=/var/jenkins_home/logs
  -XX:+UseG1GC
  -XX:+UseStringDeduplication
  -XX:+ParallelRefProcEnabled
  -XX:+DisableExplicitGC
  -XX:+UnlockDiagnosticVMOptions
  -XX:+UnlockExperimentalVMOptions
  -verbose:gc
  -XX:+PrintGC
  -XX:+PrintGCDetails
  -XX:ErrorFile=/var/jenkins_home/logs/hs_err_%p.log
  -XX:+LogVMOutput
  -XX:LogFile=/var/jenkins_home/logs/jvm.log
  -XX:InitialRAMPercentage=50.0
  -XX:MaxRAMPercentage=50.0
  -Xlog:gc*=info,gc+heap=debug,gc+ref*=debug,gc+ergo*=trace,gc+age*=trace:file=/var/jenkins_home/logs/gc-%t.log:utctime,pid,level,tags:filecount=2,filesize=100M
  

  • more info

    $ docker run --rm openjdk:8-jre java -XX:MaxRAMPercentage=75 -help 2>&1 | head
    Improperly specified VM option 'MaxRAMPercentage=75'
    Error: Could not create the Java Virtual Machine.
    Error: A fatal exception has occurred. Program will exit.
    $ docker run --rm openjdk:8-jre java -XX:MaxRAMPercentage=75.0 -help 2>&1 | head
    Usage: java [-options] class [args...]
               (to execute a class)
       or  java [-options] -jar jarfile [args...]
               (to execute a jar file)
    where options include:
        -d32    use a 32-bit data model if available
        -d64    use a 64-bit data model if available
        -server   to select the "server" VM
                      The default VM is server,
                      because you are running on a server-class machine.
    

  doc for JAVA_OPT

  -XX:+AlwaysPreTouch
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=${LOGDIR}
  -XX:+UseG1GC
  -XX:+UseStringDeduplication
  -XX:+ParallelRefProcEnabled
  -XX:+DisableExplicitGC
  -XX:+UnlockDiagnosticVMOptions
  -XX:+UnlockExperimentalVMOptions
  -verbose:gc
  -Xloggc:${LOGDIR}/gc.log
  -XX:NumberOfGCLogFiles=2
  -XX:+UseGCLogFileRotation
  -XX:GCLogFileSize=100m
  -XX:+PrintGC
  -XX:+PrintGCDateStamps
  -XX:+PrintGCDetails
  -XX:+PrintHeapAtGC
  -XX:+PrintGCCause
  -XX:+PrintTenuringDistribution
  -XX:+PrintReferenceGC
  -XX:+PrintAdaptiveSizePolicy
  -XX:ErrorFile=${LOGDIR}/hs_err_%p.log
  -XX:+LogVMOutput
  -XX:LogFile=${LOGDIR}/jvm.log
  # Heap setting for CloudBees CI on modern cloud platforms:
  #   -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=50.0
  # Heap setting for CloudBees CI on traditional platforms:
  #   Heap Size `-Xmx` and `-Xms` should be set to the same value, and determined by following the above section "JVM Heap Size"
  

threadDump

[!NOTE]

• visit via GUI: https://.domain.com/threadDump
• Using Thread Dumps
• How to generate a thread dump?
• Jenkins : Obtaining a thread dump

agent

$ sudo update-alternatives --remove java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java

Mailing format

• Show the logs after building
  • Format:

    ${BUILD_LOG, maxLines, escapeHtml}
    maxLines: 250
    

  • For example:

    ${BUILD_LOG, maxLines=8000, escapeHtml=true}
    

Properties in Jenkins Core for JAVA_OPTS

disable the

<useSecurity>true</useSecurity>
<authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
<securityRealm class="hudson.security.SecurityRealm$None"/>

Disabling CSRF Protection

reference:

• CSRF Protection

-Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true

CSRF protection tokens did not expire

SECURITY-626 :

Scripts that obtain a crumb using the /crumbIssuer/api URL will now fail to perform actions protected from CSRF unless the scripts retain the web session ID in subsequent requests.

-Dhudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID=true

enable crumb proxy compatibility

-Djenkins.model.Jenkins.crumbIssuerProxyCompatibility=true

change workspace name

# default
-Djenkins.model.Jenkins.workspaceDirName='workspace'

workspace path

# default
-Djenkins.model.Jenkins.workspacesDir="${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}"

a cache for UserDetails should be valid

# default 2 mins
-Djenkins.security.UserDetailsCache.EXPIRE_AFTER_WRITE_SEC=120

copyArtifacts v1.29 : JENKINS-14999 : Support for QueueItemAuthenticator

-Dhudson.security.ArtifactsPermission=true

Unauthorized view fragment access

# to disable the feature
-Djenkins.security.stapler.StaplerDispatchValidator.disabled=false

System Properties

• java.lang.System
• java.util.Properties
• jenkins.util.SystemProperties
• Controlling features of Hudson using system properties

set property

System.setProperty('org.apache.commons.jelly.tags.fmt.timeZone', 'Asia/Shanghai')
System.setProperty('user.timezone', 'Asia/Shanghai')

• setup user.timezone in Jenkins

  println( System.getProperty('user.timezone') );
  System.setProperty('user.timezone', 'Asia/Shanghai');
  println( System.getProperty('user.timezone') )
  

• enable crumb proxy compatibility

  System.setProperty('jenkins.model.Jenkins.crumbIssuerProxyCompatibility', 'true')
  System.getProperty('jenkins.model.Jenkins.crumbIssuerProxyCompatibility')
  

• setup CSRF protection tokens did not expire for SECURITY-626

  System.setProperty('hudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID', 'true')
  System.getProperty('hudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID')
  

get property

• get all properties

  System.getProperties()
  

• or

  System.getProperties().each { k, v ->
    println " >>> ${k} ~> ${v} "
  }
  

Configuring HTTP

remote configuration

information:

• ${PROTOCOL_FULLY_QUALIFIED_NAME}.disabled:
  where PROTOCOL_FULLY_QUALIFIED_NAME equals PROTOCOL_HANDLER_CLASSNAME without the Handler suffix.

• description
  • hudson.remoting.FlightRecorderInputStream.BUFFER_SIZE :
    Size (in bytes) of the flight recorder ring buffer used for debugging remoting issues
  • hudson.remoting.Launcher.pingIntervalSec :
    Seconds between ping checks to monitor health of agent nodes; 0 to disable ping |
  • hudson.remoting.Launcher.pingTimeoutSec :
    If ping of agent node takes longer than this, consider it dead; 0 to disable ping |
  • hudson.remoting.RemoteClassLoader.force :
    Class name String. Forces loading of the specified class name on incoming requests. Works around issues like JENKINS-19445 |
  • hudson.remoting.Engine.socketTimeout :
    Socket read timeout in milliseconds. If timeout happens and the failOnSocketTimeoutInReader property is true, the channel will be interrupted. |
  • hudson.remoting.SynchronousCommandTransport.failOnSocketTimeoutInReader :
    Boolean flag. Enables the original aggressive behavior, when the channel reader gets interrupted by any SocketTimeoutException |
  • hudson.remoting.ExportTable.unexportLogSize :
    Defines number of entries to be stored in the unexport history, which is being analyzed during the invalid object ID analysis. |
  • ${PROTOCOL_FULLY_QUALIFIED_NAME}.disabled :
    Boolean flag, which allows disabling particular protocols in remoting.
    Property example:
    org.jenkinsci.remoting.engine.JnlpProtocol3.disabled |
  • org.jenkinsci.remoting.nio.NioChannelHub.disabled :
    Boolean flag to disable NIO-based socket connection handling, and switch back to classic IO. Used to isolate the problem. |
  • org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.protocolNamesToTry :
    If specified, only the protocols from the list will be tried during the connection. The option provides protocol names, but the order of the check is defined internally and cannot be changed. |

tips

• Add Users to Jenkins with "Allow users to sign up" Disabled

  $JENKINS_URL/securityRealm/addUser
  

• sa.yml

  [!NOTE|label:references:]

  • How To Setup Jenkins On Kubernetes Cluster – Beginners Guide
  • How to Setup Jenkins Build Agents on Kubernetes Pods

  • sample 1

    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: jenkins-admin
    rules:
      - apiGroups: [""]
        resources: ["*"]
        verbs: ["*"]
    
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins-admin
      namespace: devops-tools
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: jenkins-admin
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: jenkins-admin
    subjects:
    - kind: ServiceAccount
      name: jenkins-admin
      namespace: devops-tools
    

  • sample 2

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins-admin
      namespace: devops-tools
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: jenkins
      namespace: devops-tools
      labels:
        "app.kubernetes.io/name": 'jenkins'
    rules:
    - apiGroups: [""]
      resources: ["pods"]
      verbs: ["create","delete","get","list","patch","update","watch"]
    - apiGroups: [""]
      resources: ["pods/exec"]
      verbs: ["create","delete","get","list","patch","update","watch"]
    - apiGroups: [""]
      resources: ["pods/log"]
      verbs: ["get","list","watch"]
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: jenkins-role-binding
      namespace: devops-tools
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: jenkins
    subjects:
    - kind: ServiceAccount
      name: jenkins-admin
      namespace: devops-tools
    

• ssl

  [!NOTE|label:references:]

  • SSL Certificates Troubleshooting
  • How to install a new SSL certificate on Traditional Platforms?
  • jenkins-scripts/scriptler/checkSSLConnection.groovy
  • JAVA_OPT: Jenkins Features Controlled with System Properties

  $ kubectl exec -it devops-jenkins-0 -- /bin/bash
  $ jrunscript -Djavax.net.ssl.trustStore=/opt/java/openjdk/lib/security/cacerts -Djavax.net.ssl.trustStorePassword=changeit -e "println(new java.net.URL(\"https://k8s-api.example.com:16443\").openConnection().getResponseCode())"
  Warning: Nashorn engine is planned to be removed from a future JDK release
  java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:531)
    ...

  $ mkdir -p /var/jenkins_home/cacerts-share
  $ cp /opt/java/openjdk/lib/security/cacerts /var/jenkins_home/cacerts-share/cacerts
  $ chmod 644 /var/jenkins_home/cacerts-share/cacerts
  $ keytool -import -noprompt -trustcacerts -alias k8s-api -keystore /var/jenkins_home/cacerts-share/cacerts -file /run/secrets/additional/devops_kubernetes_api.crt -storepass changeit
  Certificate was added to keystore
  $ jrunscript -Djavax.net.ssl.trustStore=/var/jenkins_home/cacerts-share/cacerts -Djavax.net.ssl.trustStorePassword=changeit -e "println(new java.net.URL(\"https://k8s-api.example.com:16443\").openConnection().getResponseCode())"
  Warning: Nashorn engine is planned to be removed from a future JDK release
  403

• full step

  $ keytool -printcert -rfc -sslserver ssdfw-k8s-api.sample.com:16443 > /run/secrets/additional/devops_kubernetes_api.crt
  # or
  $ echo -n | openssl s_client -connect ssdfw-k8s-api.sample.com:16443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /run/secrets/additional/devops_kubernetes_api.crt