Building an End-to-End CI/CD Pipeline for Java Applications - Part 3: Using Azure DevOps

Introduction

In this project, we will create an end-to-end Continuous Integration and Continuous Deployment (CI/CD) pipeline for a Java application using Azure DevOps. This is the third part of our series, and we’ll leverage Azure DevOps’ powerful CI/CD capabilities to build, test, and deploy our Spring Boot application.

Code Repository

The sample code for this project is available at: Spring PetClinic GitHub Repository.

Prerequisites

Before starting this project, make sure you have the following prerequisites:

1. Azure DevOps: Set up an Azure DevOps account and organization.
2. Docker: Install Docker on your server.
3. Java: Ensure your Java application is built and ready for deployment.
4. Maven: Install Maven or use the Maven version bundled with Azure DevOps.
5. SonarQube: Install SonarQube on your server.
6. Tomcat: Install Tomcat on your server.
7. EC2 instance: Create an EC2 instance of type t2.medium.

Setting up Prerequisites

1. Install Java on the EC2 instance:

   sudo apt install openjdk-17-jdk -y
   

2. Install Maven on the EC2 instance:

   sudo apt install maven -y
   

3. SonarQube connection in Azure DevOps:

   • Go to project settings → Service connections → New service connection
   • Provide the server URL which is the IP address of agent with 9000 port
   • Enter the token which you generated in SonarQube
   • Give a name to the service connection
   • Save

4. SSH connection in Azure DevOps:

   • Go to project settings → Service connections → New service connection
   • Choose SSH
   • Provide the private key of the server
   • Enter username as ubuntu
   • Enter host as the IP address of the server
   • Give a name to the service connection
   • Save

Setting up Tomcat

Install this after installing Java:

1. Download the Tomcat9 package:

   sudo apt install tomcat9 -y
   

2. Start the Tomcat service:

   sudo systemctl start tomcat9
   

3. Enable the Tomcat service:

   sudo systemctl enable tomcat9
   

4. Check the status of the Tomcat service:

   sudo systemctl status tomcat9
   

Setting up Docker

In the agent of Azure DevOps, install Docker using these commands:

sudo apt-get update
sudo apt-get install docker.io -y
sudo systemctl start docker
sudo docker run hello-world
sudo systemctl enable docker
docker --version
sudo usermod -a -G docker $(whoami)
newgrp docker

Setting up SonarQube on Azure DevOps server

Run this Docker command to set up SonarQube:

docker run -d --name sonarqube -p 9000:9000 sonarqube:lts-community

Access SonarQube by navigating to http://<server-ip>:9000 in your browser.

The default username and password are both: admin

To create a token for connecting Azure DevOps with SonarQube:

1. Go to Administration tab
2. Then go to security tab
3. Click on users
4. Click on update token
5. Create a token and copy it

Setting up Azure DevOps

1. Go to Azure DevOps and create a project
2. Add the project name and description
3. Create the project
4. Click on Azure Repos → Repos
5. Click on Clone
6. Copy the repository URL

Setting up Agent in Azure DevOps

1. Go to Azure DevOps
2. Click on your project
3. Click on Agent pools
4. Click on Add pool
   • Choose self-hosted agent
   • Give a name to the agent
   • Choose the option “grant the agent access to all pipelines”
5. Click on Create
6. Now download the agent from Azure DevOps:
   • Click on Agents
   • Click on Download
   • Download the agent in the EC2 instance using the command:

     wget https://vstsagentpackage.azureedge.net/agent/2.204.2/vsts-agent-linux-x64-2.204.2.tar.gz
     

7. Extract the agent:

   tar xzvf vsts-agent-linux-x64-2.204.2.tar.gz
   

8. Configure the agent:

   ./config.sh
   

   You will be prompted with some questions. Answer these questions to complete the setup.
9. Open the agent folder and run the following command to start the agent:

   ./run.sh
   

Note: In the agent, add User defined capabilities. Give it a name (e.g., Maven) and value as true.

Setting up Pipeline in Azure DevOps

1. Create a pipeline
2. Choose classic editor
3. Choose the source as Azure Repos Git
4. Choose the repository
5. Choose build and test maven project
6. Next choose the agent

Creating the Build Pipeline

Task 1: Compile the Code

First task is to compile the code. Change the command to “compile” in classic editor.

Task 2: SonarQube Analysis

For this, first install the SonarQube scanner extension:

1. In the top right corner, click on Browse marketplace
2. Search for SonarQube
3. Click on “Get it free”
4. Then search for SonarQube scanner task in Add tasks

Add the following tasks:

1. Prepare Analysis Configuration:

   • Choose the service connection as SonarQube
   • Choose standalone server mode
   • Choose manual configuration
   • Provide project key, project name, project version, and project description
   • In advanced options add: sonar.java.binaries=.

2. Run Code Analysis

3. Publish Quality Gate Result

Task 3: OWASP Dependency Check

First, install the OWASP dependency check extension:

1. In the top right corner, click on Browse marketplace
2. Search for OWASP dependency check
3. Click on “Get it free”
4. Then search for OWASP dependency check task in Add tasks

Add an OWASP Dependency Check task:

• Give it a name
• Set scan path as ./
• Set report format as XML
• In additional arguments add: --disableYarnAudit

Task 4: Build the Artifact

1. Use Maven task
2. Set Goals to “clean install”

Task 5: Copy the Artifact to the Tomcat Server

Use the SSH task:

1. Choose the service connection as SSH
2. Run the command:

   scp ./target/my-app-1.0-SNAPSHOT.jar ubuntu@<ec2-ip-address>:/var/lib/tomcat9/webapps
   

Complete Pipeline in YAML

Here’s what the complete pipeline would look like in YAML format:

trigger:
  - main

pool:
  name: "your-agent-pool"

variables:
  buildConfiguration: "Release"
  projectName: "petclinic"

steps:
  - task: Maven@3
    displayName: "Compile Java Code"
    inputs:
      mavenPomFile: "pom.xml"
      goals: "compile"

  - task: SonarQubePrepare@4
    displayName: "Prepare SonarQube Analysis"
    inputs:
      SonarQube: "SonarQube Connection"
      scannerMode: "Other"
      extraProperties: |
        sonar.projectKey=$(projectName)
        sonar.projectName=$(projectName)
        sonar.projectVersion=1.0
        sonar.sources=src
        sonar.java.binaries=.        

  - task: SonarQubeAnalyze@4
    displayName: "Run SonarQube Analysis"

  - task: SonarQubePublish@4
    displayName: "Publish SonarQube Results"
    inputs:
      pollingTimeoutSec: "300"

  - task: dependency-check-build-task@5
    displayName: "Run OWASP Dependency Check"
    inputs:
      projectName: "$(projectName)"
      scanPath: "./"
      format: "XML"
      additionalArguments: "--disableYarnAudit"

  - task: Maven@3
    displayName: "Build Java Artifact"
    inputs:
      mavenPomFile: "pom.xml"
      goals: "clean install"

  - task: CopyFilesOverSSH@0
    displayName: "Deploy to Tomcat"
    inputs:
      sshEndpoint: "EC2 SSH Connection"
      sourceFolder: "target"
      contents: "*.jar"
      targetFolder: "/var/lib/tomcat9/webapps"

Understanding the Azure DevOps Pipeline

Pipeline Trigger

The pipeline is triggered whenever a change is pushed to the main branch. This ensures continuous integration.

Agent Pool

We use our custom agent pool with the agent we set up earlier. This agent runs on our EC2 instance.

Variables

We define variables to be used across tasks:

• buildConfiguration: Determines the build configuration (Release)
• projectName: Name of our project, used in SonarQube and OWASP tasks

Tasks

1. Maven Compile: Compiles the Java code, checking for syntax errors.
2. SonarQube Prepare: Sets up parameters for the SonarQube analysis.
3. SonarQube Analyze: Runs the actual code analysis.
4. SonarQube Publish: Publishes the analysis results and waits for the quality gate.
5. OWASP Dependency Check: Scans for vulnerabilities in dependencies.
6. Maven Build: Creates the JAR artifact.
7. SSH Copy: Deploys the JAR to the Tomcat server.

Extending the Pipeline with Docker

As an alternative to deploying to Tomcat directly, you can modify the pipeline to build and deploy using Docker:

- task: Docker@2
  displayName: "Build Docker Image"
  inputs:
    command: build
    Dockerfile: "**/Dockerfile"
    repository: "$(dockerRegistryServiceConnection)/$(projectName)"
    tags: |
      $(Build.BuildId)
      latest      

- task: Docker@2
  displayName: "Push Docker Image"
  inputs:
    command: push
    containerRegistry: "$(dockerRegistryServiceConnection)"
    repository: "$(projectName)"
    tags: |
      $(Build.BuildId)
      latest      

- task: SSH@0
  displayName: "Deploy Docker Container"
  inputs:
    sshEndpoint: "EC2 SSH Connection"
    runOptions: "inline"
    inline: |
      docker pull $(dockerRegistryServiceConnection)/$(projectName):latest
      docker stop $(projectName) || true
      docker rm $(projectName) || true
      docker run -d -p 8080:8080 --name $(projectName) $(dockerRegistryServiceConnection)/$(projectName):latest      

This extended pipeline would:

1. Build a Docker image from your application
2. Push the image to a Docker registry
3. SSH into the server and deploy the image as a container

Monitoring the Pipeline

Azure DevOps provides several ways to monitor your pipeline:

1. Pipeline Runs: Go to Pipelines → Pipelines to see all runs
2. Build Summary: Click on a specific run to see detailed logs
3. Tests: View test results under the “Tests” tab
4. Extensions: Add extensions for enhanced monitoring

Setting Up Approval Gates

For better control over deployments, you can set up approval gates:

1. Go to Pipelines → Environments
2. Create a new environment (e.g., “Production”)
3. Add approval checks:
   • Click on your environment
   • Click on “Checks” and “Add check”
   • Choose “Approvals” and add approvers

Then modify your pipeline to use this environment:

- deployment: Deploy
  displayName: "Deploy to Production"
  environment: "Production"
  strategy:
    runOnce:
      deploy:
        steps:
          - task: CopyFilesOverSSH@0
            inputs:
              sshEndpoint: "EC2 SSH Connection"
              sourceFolder: "target"
              contents: "*.jar"
              targetFolder: "/var/lib/tomcat9/webapps"

Conclusion

In this tutorial, we’ve set up a complete CI/CD pipeline for a Java application using Azure DevOps. We’ve configured:

1. Self-hosted agents on EC2
2. SonarQube integration for code quality
3. OWASP Dependency Check for security
4. Deployment to Tomcat
5. Optional Docker-based deployment

Azure DevOps provides a robust platform for implementing DevOps practices, with features like:

• Version control
• Continuous integration and delivery
• Automated testing
• Approvals and gates
• Extensibility through marketplace extensions

In the next part of this series, we’ll explore enhancing our CI/CD pipeline with webhooks and additional security features.