• troubleshooting
  • tools
  • thread dump
• other tools

[!TIP|label:official recommended]

• Prepare Jenkins for Support
• CloudBees Jenkins JVM troubleshooting
  • * Supported Java 8 arguments
  • * Supported Java 11 arguments
  • * collectPerformanceData Script
• Diagnosing Errors
• Tuning Jenkins GC For Responsiveness and Stability with Large Instances
• ** Java Heap settings Best Practice
  • Minimum and maximum heap sizes
• How to Troubleshoot and Address Jenkins Startup Performances

troubleshooting

[!NOTE|label:references:]

• How to Troubleshoot and Address Jenkins Startup Performances
• Required Data: Jenkins Hang Issue On Linux
• collectPerformanceData Script
  • collectPerformanceData.sh
• Understanding Thread Dumps
  • fastthread.io
• gceasy.io

• What causes high CPU usage and how can I reduce it

• Unrecognized VM Option

  • UseGCLogFileRotation
  • GCLogFileSize=100m
  • PrintGCDateStamps
  • PrintGCCause
  • PrintTenuringDistribution
  • PrintReferenceGC
  • PrintAdaptiveSizePolicy

tools

[!NOTE|label:tools]

• iostat
• nfsiostat
• nfsstat
• vmstat - understanding Thread DumpUnderstanding Thread Dumpss
• top
  • top -H
• netstat
• jstack
• stap

• prepare

  $ apt update
  $ apt install sudo vim netstat net-tools sysstat nfs-common
  $ sudo systemctl start sysstat
  $ sudo systemctl enable sysstat
  $ cat /etc/cron.d/sysstat
  

  [!NOTE|label:tips for sar]

  • if you wanted to check your memory usage instead, you could use the -r argument rather than -u

    $ sar -r 2 30
    

• collectPerformanceData.sh

  $ curl -sO https://s3.amazonaws.com/cloudbees-jenkins-scripts/e206a5-linux/collectPerformanceData.sh
  $ chmod +x collectPerformanceData.sh
  
  $ sudo -u $JENKINS_USER sh collectPerformanceData.sh $JENKINS_PID 300 5
  $ or
  <jenkins> $ bash collectPerformanceData.sh $JENKINS_PID 300 5
  [INFO] Collected a threadDump for PID 8.
  [INFO] A new collection will start in 5 seconds.
  [INFO] Taking top data collection.
  [INFO] Taking TopdashH data collection.
  [INFO] Taking vmstat data collection.
  [INFO] Taking netstat collection.
  [INFO] Taking iostat data collection.
  [INFO] Taking nfsiostat data collection.
  [INFO] Taking nfsstat data collection.
  

  [!NOTE]

  • 300: "Length to run the script in seconds"
  • 5: "Intervals to execute commands in seconds"

thread dump

• generated via monitor plugin

  

• generated via jmap

  [!NOTE|label:references:]

  • List All the Classes Loaded in the JVM

  $ pid=$(ps auxfww | grep devops-jenkins | awk '{print $2}')
  $ jmap -dump:format=b,file=/opt/tmp/heapdump.bin ${pid}
  

• analysis via

  • fastthread.io

  • MAT: Eclipse Memory Analyzer

    

  • VisualVM

    • i.e.: visualvm.exe --jdkhome "C:\Software\Java\jdk1.6.0" --userdir "C:\Temp\visualvm_userdir"

    

other tools

• Decimal to Hexadecimal Converter
• GC Log Analyzer