Building an End-to-End CI/CD Pipeline for Java Applications - Part 5: Using Separate CI and CD Jobs

Introduction

In this tutorial, we will learn how to set up a CI/CD pipeline for a Java application using two different Jenkins jobs, one for Continuous Integration (CI) and one for Continuous Deployment (CD). This separation provides better control over the pipeline stages and allows for more flexibility in managing deployments.

Code Repository

The sample code for this project is available at Spring Boot CI/CD Docker Jenkins GitHub Repository.

Prerequisites

Before starting this project, make sure you have the following prerequisites:

1. Jenkins: Ensure Jenkins is installed and running on a server.
2. SonarQube: Ensure SonarQube is installed and running on a server.
3. Docker: Install Docker on your server.
4. Java: Ensure your Java application is built and ready for deployment.
5. Maven: Install Maven or use the Maven version bundled with Jenkins.
6. OWASP Dependency Check Plugin: Install the OWASP Dependency Check Plugin in Jenkins.

Setting up Jenkins

For this we need to use an EC2 instance of type t2.large so that we have enough resources to run the Jenkins server and also to install other services like Docker, SonarQube and Trivy.

Steps to install Jenkins on an EC2 instance:

1. SSH into the EC2 instance and first install Java:

sudo apt update -y
sudo apt install openjdk-11-jdk -y
java -version

2. Download and install Jenkins:

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

Don’t forget to open port 8080 in security group to access Jenkins.

3. Open the Jenkins URL in the browser and get the initial admin password by following the instructions in the browser.

4. Install suggested plugins.

5. Set up the admin user.

6. Once Jenkins is set up, you can access the Jenkins dashboard.

Installing Required Jenkins Plugins

You need to install the following plugins:

1. JDK –> Eclipse Temurin
2. SonarQube Scanner
3. OWASP Dependency Check Plugin
4. All Docker plugins
5. Maven

To install these plugins:

1. Go to Manage Jenkins → Manage Plugins → Available
2. Search for each plugin and install it
3. Restart Jenkins after installation

Configuring Jenkins Global Tools

Configure the global tools by going to Manage Jenkins → Global Tool Configuration:

1. JDK: Add JDK → Name it as jdk11 → Install automatically → Install from Adoptium → Click on 11 version.
2. Maven: Add Maven → Name it as maven3 → Install automatically → Use version 3.8.6
3. SonarQube Scanner: Add SonarQube Scanner → Name it as sonarqube scanner → Install automatically

Creating the CI Pipeline Job

1. Go to Dashboard → New Item → Enter the name of the project (e.g., “Java-App-CI”) → Select Pipeline → OK
2. First choose the discard old builds → And then choose the number of builds to keep. For this example, we’ll use 2.
3. Next choose the SCM as Git.
4. Add the repository URL: https://github.com/darey-devops/spring-boot-cicd-docker-jenkins.git
5. Choose the branch as main.
6. In the pipeline section, add the following pipeline script:

pipeline {
    agent any

    tools {
        maven 'maven3'
        jdk 'jdk11'
    }

    environment {
        SONARQUBE_HOME = tool name: 'sonarqube'
    }

    stages {
        stage('Checkout Code') {
            steps {
               git url: 'https://github.com/darey-devops/spring-boot-cicd-docker-jenkins.git', branch: 'main'
            }
        }

        stage('Compile Code') {
            steps {
                sh 'mvn clean compile'
            }
        }

        stage('Run Tests') {
            steps {
                sh 'mvn test'
            }
        }

        stage('Sonar Analysis') {
            steps {
                withSonarQubeEnv('sonarqube') {
                    sh ''' $SONARQUBE_HOME/bin/sonar-scanner -Dsonar.projectKey=spring-boot-cicd-docker-jenkins -Dsonar.qualitygate.wait=true -Dsonar.qualitygate.timeout=600000 -Dsonar.java.binaries=.
                    '''
                }
            }
        }

        stage('OWASP Dependency Check') {
            steps {
                dependencyCheck additionalArgs: '-DskipTests --scan ./', odcInstallation: 'DP'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }

        stage('Maven Install') {
            steps {
                sh 'mvn clean install -DskipTests=true'
            }
        }

        stage('Build Docker Image') {
            steps {
                script {
                    withDockerRegistry([credentialsId: 'docker-hub-credentials', toolName: 'docker']) {
                        sh 'docker build -t darey-devops/spring-boot-cicd-docker-jenkins:latest .'
                        sh 'docker push darey-devops/spring-boot-cicd-docker-jenkins:latest'
                    }
                }
            }
        }

        stage('Trigger CD Pipeline') {
            steps {
                build job: 'Java-App-CD', wait: true
            }
        }
    }

    post {
        success {
            echo 'CI Pipeline completed successfully!'
        }
        failure {
            echo 'CI Pipeline failed!'
        }
    }
}

Creating the CD Pipeline Job

1. Go to Dashboard → New Item → Enter the name of the project (e.g., “Java-App-CD”) → Select Pipeline → OK
2. First choose the discard old builds → And then choose the number of builds to keep. For this example, we’ll use 2.
3. In the pipeline section, add the following pipeline script:

pipeline {
    agent any

    stages {
        stage('Deploy the image to a container') {
            steps {
                script {
                    withDockerRegistry([credentialsId: 'docker-hub-credentials', toolName: 'docker']) {
                        sh 'docker stop spring-boot-app || true'
                        sh 'docker rm spring-boot-app || true'
                        sh 'docker run -d --name spring-boot-app -p 8080:8080 darey-devops/spring-boot-cicd-docker-jenkins:latest'
                    }
                }
            }
        }
    }

    post {
        success {
            echo 'CD Pipeline completed successfully!'
        }
        failure {
            echo 'CD Pipeline failed!'
        }
    }
}

Understanding the Pipeline Structure

Let’s examine the components of our pipeline:

Continuous Integration (CI) Pipeline

1. Tools and Environment Setup:

   • Tools: We define Maven and JDK.
   • Environment: We set up SonarQube path.

2. Checkout Code:

   • Pulls the latest code from the Git repository.

3. Compile Code:

   • Compiles the Java application to catch syntax errors.

4. Run Tests:

   • Executes unit and integration tests.

5. SonarQube Analysis:

   • Performs code quality and security analysis.

6. OWASP Dependency Check:

   • Scans for vulnerabilities in dependencies.

7. Maven Install:

   • Builds the application and creates the JAR file.

8. Build Docker Image:

   • Creates a Docker image from the application.
   • Pushes the image to Docker Hub.

9. Trigger CD Pipeline:

   • Automatically triggers the CD job after successful CI.

Continuous Deployment (CD) Pipeline

1. Deploy the Docker Container:
   • Stops and removes any existing container.
   • Runs a new container with the latest image.

Adding Docker Hub Credentials

Before running the pipeline, add Docker Hub credentials to Jenkins:

1. Go to Manage Jenkins → Manage Credentials → Global
2. Click on Add Credentials
3. Choose Username with password
4. Enter your Docker Hub username and password
5. Add ID as docker-hub-credentials and description as Docker Hub

Setting Up Docker

Install Docker on the Jenkins server/EC2 instance:

sudo apt-get update
sudo apt-get install docker.io -y
sudo systemctl start docker
sudo docker run hello-world
sudo systemctl enable docker
docker --version
sudo usermod -a -G docker $(whoami)
sudo usermod -a -G docker jenkins
newgrp docker
sudo systemctl restart jenkins

Setting up SonarQube

Set up SonarQube on the Jenkins server using Docker:

docker run -d --name sonarqube -p 9000:9000 sonarqube:lts-community

Access SonarQube by navigating to http://<jenkins-server-ip>:9000 in your browser.

The default username and password are both: admin

To create a token for connecting Jenkins with SonarQube:

1. Go to Administration tab
2. Then go to security tab
3. Click on users
4. Click on update token
5. Create a token named jenkins and save it

Now add this token in Jenkins credentials:

1. Go to Manage Jenkins → Manage Credentials → Global
2. Click on Add Credentials
3. Select Secret text as the type
4. Add the token you created in SonarQube
5. Add ID as sonar and description as sonar

Benefits of Separate CI/CD Jobs

1. Separation of Concerns: CI and CD processes have different responsibilities and can be managed separately.

2. Independent Deployment: You can trigger deployments manually or on different schedules from the build process.

3. Security: You can restrict access to the CD job to specific team members.

4. Visibility: Clearer view of where a failure occurs - either in the build/test phase or in deployment.

5. Easier Debugging: When issues arise, it’s clearer whether they’re related to building the application or deploying it.

Pipeline Visualization

Using separate jobs creates two distinct pipeline visualizations in Jenkins:

1. CI Pipeline: Shows all build, test, and quality assurance steps.
2. CD Pipeline: Shows deployment steps only.

This makes it easier to monitor and troubleshoot each phase of your software delivery process.

Conclusion

In this tutorial, we’ve created a comprehensive CI/CD pipeline using separate jobs for CI and CD. This approach:

1. Improves separation of concerns
2. Provides more control over deployments
3. Enhances pipeline visualization
4. Makes troubleshooting easier

By splitting your pipeline into CI and CD jobs, you gain more flexibility in how you manage your software delivery process, particularly for complex applications or when you need to implement approval gates between build and deployment.

In the next part of this series, we’ll explore how to extend this approach for a full-stack application with frontend, backend, and database components.