GitLab CI/CD Ultimate Guide - Part 1: Fundamentals

Introduction

GitLab serves as an all-in-one DevOps platform with robust CI/CD capabilities. This guide will help you understand what GitLab is, why it’s ideal for CI/CD, its key features, and how its CI/CD pipelines work. In this first part of our GitLab CI/CD Ultimate Guide, we’ll cover the fundamentals you need to get started with GitLab CI/CD.

1. What is GitLab?

GitLab is an all-in-one DevOps platform that integrates tools for version control, CI/CD pipelines, project management, and monitoring into a single interface. It enables teams to collaborate efficiently, manage their codebase, and automate workflows seamlessly.

Key Highlights of GitLab:

• Version Control System: GitLab uses Git for tracking code changes, allowing developers to work collaboratively.
• Integrated DevOps Tools: It provides built-in CI/CD, issue tracking, code review, and monitoring tools.
• Self-Hosted and Cloud Options: Users can choose between the SaaS (GitLab.com) version or host their own instance for more control.

2. Why GitLab for CI/CD?

GitLab stands out as a CI/CD tool because of its seamless integration, powerful automation capabilities, and ease of use.

Advantages of GitLab CI/CD:

• Built-in CI/CD: Unlike other tools requiring plugins, GitLab CI/CD is integrated directly into the platform.

  Why does built-in CI/CD matter? It simplifies setup and reduces dependency on third-party tools. This integration means you don’t need to configure and maintain separate systems for source control and CI/CD, leading to a more streamlined workflow.

• Single Platform for DevOps: GitLab eliminates the need for separate tools, as it combines source code management, CI/CD, and project management.

• Flexibility: Supports a wide range of environments, including Kubernetes, Docker, and multi-cloud setups.

• Scalability: GitLab CI/CD can handle everything from small personal projects to large enterprise applications with ease.

Example Use Case:

• A developer commits code to a GitLab repository.
• The GitLab CI/CD pipeline automatically triggers, runs tests, builds the application, and deploys it to production—all without manual intervention.

3. Key Features of GitLab

GitLab offers a robust feature set, making it ideal for modern software development and delivery workflows.

Version Control:

GitLab’s Git-based version control system helps teams collaborate efficiently, track changes, and manage branches.

Example:

git clone https://gitlab.com/<username>/<project>.git

• What it does:
  Clones a GitLab repository to the local system.
• Outcome:
  The repository’s files are available locally for development.

Merge Requests (MRs):

Merge Requests streamline code reviews, enabling teams to collaborate on changes before merging them into the main branch.

Issue Tracking and Boards:

GitLab provides a powerful issue tracker to manage tasks, bugs, and features. Its Kanban-style boards make workflow visualization easy.

CI/CD Pipelines:

The CI/CD system automates testing, building, and deployment.

What are pipelines? Pipelines consist of stages (like build, test, and deploy) that execute defined tasks in a sequential or parallel manner. They automate the process of taking code from version control to production.

Integrated Security:

Built-in tools like SAST (Static Application Security Testing) and DAST (Dynamic Application Security Testing) help identify vulnerabilities early.

Container Registry:

GitLab provides a built-in Docker Container Registry, enabling seamless containerization workflows.

Monitoring and Metrics:

GitLab integrates with monitoring tools like Prometheus and Grafana to track performance and logs.

4. How GitLab CI/CD Works

GitLab CI/CD automates the software delivery pipeline, enabling teams to ship code faster and more reliably. The core workflow involves the following steps:

Step 1: Define a .gitlab-ci.yml File

The .gitlab-ci.yml file is at the heart of GitLab CI/CD. It defines the pipeline’s stages, jobs, and scripts.

Example .gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - echo "Building the application"
    - npm install

test-job:
  stage: test
  script:
    - echo "Running tests"
    - npm test

deploy-job:
  stage: deploy
  script:
    - echo "Deploying the application"
    - npm run deploy

What does this file do?

• Defines a three-stage pipeline: build, test, and deploy.
• Specifies the scripts to run at each stage, such as installing dependencies, running tests, and deploying the application.

Step 2: Commit Code to a Repository

• When code is committed to the GitLab repository, the pipeline defined in .gitlab-ci.yml automatically triggers.

Command:

git commit -m "Add new feature"
git push origin main

• What it does:
  Commits code changes locally and pushes them to the main branch on GitLab.
• Outcome:
  Triggers the CI/CD pipeline.

Step 3: Pipeline Execution

• GitLab Runner, an agent that executes the pipeline’s jobs, picks up the tasks defined in .gitlab-ci.yml.

What is a GitLab Runner?

• A GitLab Runner is a lightweight agent that executes jobs in the pipeline. It can run in Docker containers, virtual machines, or on physical servers.

Step 4: Feedback and Logs

• Developers get real-time feedback on pipeline status (success or failure) and logs for debugging.

Command:

gitlab-runner run

• What it does:
  Starts a GitLab Runner locally for debugging pipeline jobs.

Step 5: Deployment

• If all pipeline stages pass, the final stage deploys the application to the target environment (e.g., a server or Kubernetes cluster).

High-Level Workflow Summary

1. Developer pushes code to GitLab.
2. The .gitlab-ci.yml pipeline triggers automatically.
3. GitLab Runner executes jobs like building, testing, and deploying.
4. Developers monitor pipeline logs and resolve issues.
5. The pipeline deploys the application if all jobs succeed.

5. Setting Up GitLab CI/CD

This section walks you through the initial steps to get started with GitLab CI/CD, from creating an account to setting up a GitLab Runner for pipeline execution.

1. Creating a GitLab Account

To begin using GitLab CI/CD, you need a GitLab account. Here’s how to create one:

1. Visit GitLab and click Sign Up.
2. Fill in the required details:
   • Email address
   • Username
   • Password
3. Verify your email and log in.

Pro Tip: If you’re working in an organization, sign up with your company email to access group-specific features and repositories.

Why do you need a GitLab account?

• A GitLab account allows you to manage repositories, set up CI/CD pipelines, and collaborate with your team.

2. Setting Up Your First GitLab Project

A GitLab Project is where your code, issues, CI/CD configuration, and pipeline results live.

Steps to Create a GitLab Project:

1. Log in to GitLab.
2. Click New Project on the dashboard.
3. Choose one of the following:
   • Create from Scratch: Start with an empty repository.
   • Import Existing Repository: Import from GitHub, Bitbucket, or another source.
   • Use a Template: Leverage predefined project templates for quicker setup.
4. Name your project, provide a description (optional), and set visibility to Private, Internal, or Public.
5. Click Create Project.

Command to Push Local Code to GitLab:

git init
git add .
git commit -m "Initial commit"
git branch -M main
git remote add origin https://gitlab.com/<username>/<project>.git
git push -u origin main

• What it does:
  Links your local repository to the GitLab project and pushes your code to the main branch.
• Outcome:
  Your code is now hosted on GitLab and ready for CI/CD integration.

3. Integrating GitLab with Your Code Repository

To enable GitLab CI/CD, you must link your code repository to GitLab.

Steps to Integrate Your Repository:

1. Navigate to your GitLab project.
2. Go to Settings > Repository.
3. Configure Push Rules to ensure consistency (e.g., requiring commit messages to follow specific guidelines).
4. Use Webhooks for real-time integration with other tools (like Slack for notifications or Jenkins for advanced tasks).

Using SSH for Integration:

Set up SSH to securely connect your local machine with GitLab:

ssh-keygen -t rsa -b 4096 -C "[email protected]"

• What it does:
  Generates an SSH key pair to authenticate your local machine with GitLab.
• Outcome:
  You can push/pull code without entering your password each time.

4. GitLab Runner Setup

A GitLab Runner is an agent that executes CI/CD jobs. Runners can be hosted on your local machine, a cloud provider, or GitLab’s shared runners.

Types of Runners:

• Shared Runners: Provided by GitLab; ideal for small projects.
• Specific Runners: Dedicated to a single project or group.
• Group Runners: Shared across multiple projects within a group.

Installing a GitLab Runner:

1. Download and install the GitLab Runner:

   curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
   sudo apt-get install gitlab-runner
   

   • What it does:
     Downloads and installs the GitLab Runner on your machine.
   • Outcome:
     Your machine is ready to execute pipeline jobs.

2. Register the Runner:

   sudo gitlab-runner register
   

   • What it asks for:
     • GitLab URL: URL of your GitLab instance (e.g., https://gitlab.com/<username>).
     • Token: Found under Settings > CI/CD > Runners in your project.
     • Description: Name your runner (e.g., my-runner).
     • Executor: Choose the execution environment (shell, docker, kubernetes, etc.). Docker is recommended for most cases.
   • Outcome:
     The Runner is now registered and linked to your GitLab project.

3. Test the Runner:

   gitlab-runner verify
   

   • What it does:
     Confirms that the runner is active and ready to execute jobs.
   • Outcome:
     Ensures your setup is complete and functional.

Example Workflow: Putting It All Together

1. Create a .gitlab-ci.yml File: Add the following to your project to define a basic pipeline:

   stages:
     - test
   
   test-job:
     stage: test
     script:
       - echo "Running tests..."
   

   • What it does:
     Defines a pipeline with a test stage and a job (test-job) that runs echo "Running tests...".
   • Outcome:
     Executes the echo "Running tests..." command whenever you push code to GitLab.

2. Push Your Code:

   git add .gitlab-ci.yml
   git commit -m "Add CI/CD pipeline"
   git push origin main
   

   • What it does:
     Pushes the pipeline configuration to GitLab.
   • Outcome:
     The pipeline triggers and runs the test-job.

3. Monitor the Pipeline:

   • Navigate to CI/CD > Pipelines in your GitLab project.
   • View logs, status, and results of the pipeline.

Troubleshooting Tips

1. Pipeline Not Triggering?

   • Ensure the .gitlab-ci.yml file is in the root directory of your repository.
   • Verify the file syntax using GitLab CI Lint.

2. Runner Offline?

   • Recheck the runner registration and verify the token.
   • Restart the runner service:

     sudo gitlab-runner restart
     

6. Key GitLab CI/CD Concepts

Understanding GitLab CI/CD concepts is crucial for leveraging its full potential. In this section, we’ll break down the building blocks of GitLab CI/CD, from pipelines and configuration files to artifacts and secrets.

1. GitLab CI/CD Pipeline Overview

A pipeline in GitLab is a series of stages and jobs that automate tasks like building, testing, and deploying code.

• Pipeline Flow:
  Pipelines are triggered by specific events (e.g., code push, merge request).
  A pipeline consists of:
  1. Stages: Logical groupings of tasks like build, test, and deploy.
  2. Jobs: Specific tasks within each stage (e.g., running unit tests).
  3. Artifacts: Files generated by jobs (e.g., build outputs, test results).

Example Pipeline Visualization:

Pipeline: [ Build Stage ] → [ Test Stage ] → [ Deploy Stage ]
          | job-1       |   | job-2     |    | job-3        |

Why Use Pipelines?

• Ensures a consistent workflow for CI/CD.
• Automates repetitive tasks, reducing human error.

2. GitLab CI/CD Configuration Files (.gitlab-ci.yml)

The .gitlab-ci.yml file is the backbone of GitLab CI/CD. It defines your pipeline’s structure, including stages, jobs, and execution rules.

Key Elements of .gitlab-ci.yml:

1. Stages: Define the pipeline flow (e.g., build, test, deploy).
2. Jobs: Specify tasks in each stage.
3. Scripts: Commands to execute within a job.
4. Rules: Determine when jobs or pipelines should run.
5. Artifacts and Caches: Store and reuse data across jobs.

Basic Example:

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - echo "Building the application..."

test-job:
  stage: test
  script:
    - echo "Running tests..."

deploy-job:
  stage: deploy
  script:
    - echo "Deploying the application..."

• What it does:
  Defines a pipeline with three stages and corresponding jobs.
• Outcome:
  Each stage is executed sequentially when the pipeline is triggered.

Validation:

Use the GitLab CI Lint Tool to validate your .gitlab-ci.yml.

3. Stages in GitLab Pipelines

Stages are the logical steps of your CI/CD process. Each stage contains one or more jobs.

Common Stages:

• Build: Compiles the code.
• Test: Runs tests to ensure code quality.
• Deploy: Deploys the application to a target environment.

Note: You can give any name to the stage

Key Points:

• Jobs in the same stage run in parallel.
• Stages run sequentially.

Example:

stages:
  - build
  - test
  - deploy

test-job:
  stage: test
  script:
    - echo "Running parallel tests..."

What if a stage fails?
If a stage fails, subsequent stages are skipped unless specified otherwise. This helps prevent deploying broken code to production environments.

4. Jobs and Runners

A job is a single task in a pipeline, and a runner is the agent that executes these jobs.

Job Configuration:

Jobs include:

• Name: Identifier for the job.
• Stage: Stage the job belongs to.
• Script: Commands to run.
• Tags: Link jobs to specific runners.

Example Job:

build-job:
  stage: build
  script:
    - echo "Building the app..."
  tags:
    - docker

What if we don’t specify the stage?
If we don’t specify the stage, the job will run in the test stage by default.

What are Runners?

• Shared Runners: Provided by GitLab; suitable for small projects.
• Specific Runners: Dedicated to your project or group in which the nodes are like EC2 instances or any other cloud instances.
• Runners support multiple executors like shell, docker, and kubernetes.

Example Command to List Runners:

gitlab-runner list

• Outcome: Displays all active runners linked to your project.

5. GitLab Artifacts and Caching

Artifacts: Temporary files created by a job that can be passed to subsequent jobs.
Caching: Reuses dependencies or build outputs to speed up pipelines.

Artifacts Example:

build-job:
  stage: build
  script:
    - mkdir build
    - echo "Build files" > build/output.txt
  artifacts:
    paths:
      - build/

• What it does:
  Saves the build/ folder for later stages.
• Outcome:
  Subsequent jobs can access the build/ folder.

Caching Example:

cache:
  paths:
    - node_modules/

• What it does:
  Caches the node_modules/ directory to speed up future pipeline runs.

How are artifacts and caches different?

• Artifacts: Shared across stages within the same pipeline run, typically used to pass build outputs between jobs.
• Caches: Persist between pipeline runs, ideal for dependencies like npm packages that rarely change.

6. GitLab Secrets and Environment Variables

Sensitive data like API keys, passwords, or configurations should not be hardcoded. Use environment variables for secure management.

What Are Environment Variables?

Environment variables are a way to store sensitive data, such as API keys, passwords, and configuration settings, outside of your codebase. This approach enhances security by preventing sensitive information from being hardcoded in your repositories.

Defining Secrets in GitLab

To set up environment variables for your GitLab CI/CD pipelines, follow these steps:

1. Navigate to the Settings:

   • Go to your specific project in GitLab.
   • Click on Settings in the left sidebar.
   • Select CI/CD from the dropdown menu.

2. Access Variables Section:

   • Scroll down to the Variables section in the CI/CD settings.

3. Add Variables:

   • Click on the Add Variable button.
   • Enter the variable name (e.g., API_KEY) in the Key field.
   • Enter the value (e.g., your-secret-api-key) in the Value field.

4. Configure Options:

   When creating a variable, you have several options to choose from:

   • Protected:

     • If enabled, this variable will only be available to protected branches or tags. This is useful if you want to restrict access to certain sensitive data.

   • Masked:

     • If enabled, the value of the variable will be hidden in job logs. This means that when the variable is echoed in a job (e.g., echo $API_KEY), it won’t show the actual value in the output logs, providing an additional layer of security.

   • Environment Scope:

     • You can limit the variable’s availability to specific environments (e.g., development, staging, production). By default, the variable will be available to all environments if this option is not specified.

   • Value Type:

     • You can define whether the value is a plain text string or a file variable, which will be based on the content type used in your pipeline.

5. Save the Variable:

   • Once you have filled in the details and configured the options, click on the Add Variable button at the bottom.

Using Secrets in Pipelines:

Once you have defined your environment variables, you can easily use them in your .gitlab-ci.yml file.

Example of Using an Environment Variable:

test-job:
  script:
    - echo "API_KEY is $API_KEY"

• Outcome:
  • The job will replace $API_KEY with the value of the API_KEY environment variable you defined in the GitLab CI/CD settings.
  • If masked, it will not show the actual API key in the job logs, instead displaying asterisks (e.g., API_KEY is ********).

Why use environment variables?

• Enhanced Security: Sensitive data remains out of your codebase, reducing the risk of accidental exposure.
• Easier Configuration Management: Allows for easy updates without code changes. If you need to change a key or password, you can do it in the UI without editing source files.
• Environment-Specific Values: Different environments (development, testing, production) can use different variables without modifying code or configuration files.

Summary of Options When Creating Variables

• Key: The name of the variable (e.g., API_KEY).
• Value: The actual sensitive value (e.g., your-secret-api-key).
• Protected: Restricts the variable to protected branches/tags.
• Masked: Hides the variable value in job logs.
• Environment Scope: Limits where the variable can be used (e.g., specific environments).

7. Manual and Triggered Pipelines

Not all pipelines should run automatically. GitLab allows manual and triggered pipelines for more control.

Manual Pipelines:

Use the when: manual keyword to specify jobs that require manual approval.

deploy-job:
  stage: deploy
  script:
    - echo "Deploying..."
  when: manual

• Outcome:
  The deploy job runs only after manual approval. This is particularly useful for production deployments where human verification is desired.

Triggered Pipelines:

Trigger pipelines from external systems or another pipeline.

trigger-job:
  stage: trigger
  trigger:
    project: path/to/project
    branch: main

• Outcome:
  Starts a pipeline in another project. This is useful for complex workflows spanning multiple repositories.

Summary

In this first part of our GitLab CI/CD Ultimate Guide, we’ve covered the fundamentals of GitLab CI/CD including:

• What GitLab is and why it’s an excellent choice for CI/CD
• Key features of GitLab and how its CI/CD pipelines work
• Step-by-step setup instructions for GitLab CI/CD
• Essential concepts like pipelines, jobs, runners, artifacts, and environment variables

In the second part of this guide, we’ll explore advanced GitLab CI/CD features, integrations with other tools, best practices, and real-world examples to help you take your CI/CD pipelines to the next level.

By understanding these fundamentals, you’re now equipped to start building basic CI/CD pipelines with GitLab and take advantage of its powerful automation capabilities.