Jenkins Installation and First Pipeline

Jenkins is still the workhorse for teams that want a self-hosted build server. It is not glamorous, but it runs on a small VM, it has a plugin for everything, and a Jenkinsfile checked into the repository makes the build definition reviewable like any other code. I run one for database migration scripts and a couple of internal tools, and the setup below is what I do on a fresh Ubuntu 18.04 or CentOS 7 machine.

Plan for 2 GB of RAM, since the master alone wants about 1 GB. I install Java 8; Jenkins 2.164 LTS and later also runs on Java 11, but the plugins are still more comfortable on 8. Open port 8080 or put nginx in front.

Install Java and the LTS repository

Ubuntu 18.04:

sudo apt update
sudo apt install openjdk-8-jdk
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
echo "deb https://pkg.jenkins.io/debian-stable binary/" | sudo tee /etc/apt/sources.list.d/jenkins.list
sudo apt update
sudo apt install jenkins

CentOS 7:

sudo yum install java-1.8.0-openjdk-devel
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
sudo yum install jenkins
sudo firewall-cmd --permanent --add-port=8080/tcp
sudo firewall-cmd --reload

The -stable repositories deliver the LTS line (2.164.x or 2.176.x at the moment) rather than the weekly builds. Then:

sudo systemctl enable --now jenkins
sudo systemctl status jenkins

Jenkins stores everything under /var/lib/jenkins, runs as the jenkins user, and reads startup options from /etc/default/jenkins (Ubuntu) or /etc/sysconfig/jenkins (CentOS). If you need more heap, that is where JAVA_ARGS="-Xmx2g" goes.

The setup wizard

Open http://your-server:8080/. It asks for the initial admin password, which is in a file only root can read:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Choose "Install suggested plugins". That set includes Git, Pipeline, Credentials, and the Timestamper, which is everything the example below needs. Create the first admin user when prompted, and leave the Jenkins URL as the address you connect with; it is used for links in notifications.

Git credential

For a private repository, Jenkins needs a deploy key. I use a read-only one per repository, so a leaked key can't push. Generate it on the Jenkins host as the jenkins user:

sudo -u jenkins ssh-keygen -t ed25519 -f /var/lib/jenkins/.ssh/id_ed25519 -N ''
sudo -u jenkins ssh-keyscan github.com >> /var/lib/jenkins/.ssh/known_hosts
sudo cat /var/lib/jenkins/.ssh/id_ed25519

Add the public key as a read-only deploy key on the repository. Then in Jenkins: Credentials → System → Global credentials → Add Credentials → Kind: "SSH Username with private key", username git, paste the private key, ID repo-deploy-key. The ssh-keyscan line matters: the Git plugin refuses hosts that are not in known_hosts.

The Jenkinsfile

In the root of the repository, commit a Jenkinsfile using declarative syntax:

pipeline {
    agent any

    options {
        timestamps()
        buildDiscarder(logRotator(numToKeepStr: '20'))
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Lint SQL') {
            steps {
                sh 'find migrations -name "*.sql" -print0 | xargs -0 -n1 sh -c "head -c 1 \\"\\$0\\" > /dev/null"'
            }
        }
        stage('Test') {
            steps {
                sh './run-tests.sh'
            }
        }
    }

    post {
        failure {
            echo "Build failed on ${env.BRANCH_NAME}"
        }
        always {
            cleanWs()
        }
    }
}

checkout scm uses whatever repository and branch the job was configured with. cleanWs() comes from the Workspace Cleanup plugin, which is in the suggested set.

The pipeline job

New Item → name it → "Multibranch Pipeline" → OK. Under Branch Sources add Git, enter the SSH clone URL (git@github.com:org/repo.git) and select the repo-deploy-key credential. Save. Jenkins scans the repository, finds every branch with a Jenkinsfile, and creates a job for each. Click "Scan Multibranch Pipeline Now" if it does not start immediately.

A plain "Pipeline" job with "Pipeline script from SCM" works too if you only ever build one branch.

First build

Open the branch job and click "Build Now". The stage view should show Checkout, Lint SQL and Test in sequence, each green. Click into the console output; with timestamps() each line is prefixed with the time. On the server:

ls /var/lib/jenkins/workspace/
sudo tail -f /var/log/jenkins/jenkins.log

The workspace directory for the job should contain a checkout of the repository. Push a commit that breaks run-tests.sh and confirm the build goes red and the post { failure } block runs.

When it goes wrong

• stderr: Host key verification failed. during checkout: the known_hosts step was skipped, or the key was added under root's home instead of /var/lib/jenkins/.ssh.
• Permission denied (publickey) from Git: the deploy key was not added to the repository, or the credential in Jenkins holds the public key instead of the private one.
• Jenkins is going to shut down banner and a hung upgrade: a plugin update is waiting for running builds. sudo systemctl restart jenkins after they finish.
• java.lang.OutOfMemoryError: Java heap space in jenkins.log: raise -Xmx in /etc/default/jenkins or /etc/sysconfig/jenkins and restart.
• Builds queue forever with "Waiting for next available executor": the master has 2 executors by default (Manage Jenkins → Configure System → # of executors). Raise it, or add an agent node.

Back up /var/lib/jenkins. I learned that one the annoying way; the job history and credentials live there and are painful to recreate. Next on my list is a webhook so pushes trigger builds instead of polling.