GitHub Actions Guide Part 2: Building Advanced Workflows

Welcome to the second part of our GitHub Actions guide! In Part 1, we covered the fundamentals of GitHub Actions, including key concepts and setting up your first workflow. Now, let’s dive deeper into building more sophisticated workflows and customizing them to meet your specific project needs.

Building a Basic Workflow

In this section, we’ll create and understand more practical GitHub Actions workflows. We’ll build upon the basic concepts and explore how to construct workflows that can run jobs both sequentially and in parallel.

Writing Your First Workflow

Let’s automate a simple “Hello World” script. We’ll create two examples: one where jobs run sequentially and another where jobs run in parallel.

Example 1: Sequential Jobs

name: Sequential Workflow
on:
  push:
    branches:
      - main
jobs:
  setup:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Setup Environment
        run: echo "Setting up environment..."

  execute:
    needs: setup
    runs-on: ubuntu-latest
    steps:
      - name: Run Hello World
        run: echo "Hello, World!"

Explanation:

1. on: push: The workflow triggers when you push code to the main branch.
2. jobs:
   • setup:
     • Checks out the repository and sets up the environment.
   • execute:
     • Depends on the setup job (needs: setup) and runs only after it completes.
     • Prints “Hello, World!” to the console.

Outcome:

• Tasks are executed sequentially: the setup job completes first, then the execute job runs.

Example 2: Parallel Jobs

name: Parallel Workflow
on:
  push:
    branches:
      - main
jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
      - name: Run First Task
        run: echo "Running job 1..."

  job2:
    runs-on: ubuntu-latest
    steps:
      - name: Run Second Task
        run: echo "Running job 2..."

Explanation:

1. Both job1 and job2 run independently and simultaneously.
2. Each job performs its own task without depending on the other.

Outcome:

• Tasks execute in parallel, reducing the total runtime for independent operations.

Running and Reviewing the Workflow Output

1. Triggering the Workflow:

   • Push changes to the main branch to start the workflow.
   • Example command:

     git add .
     git commit -m "Add basic workflow"
     git push origin main
     

2. Reviewing the Output:

   • Navigate to the Actions tab in your GitHub repository.
   • Select the workflow run to view detailed logs for each job and step.

Outcome: You’ll see logs like:

Running job 1...
Running job 2...

This confirms that the jobs ran successfully.

Using Pre-built Actions

GitHub Actions allows you to use pre-built actions to simplify workflows. These actions are reusable tasks provided by GitHub or the community.

Introduction to GitHub Marketplace

The GitHub Marketplace is a repository of pre-built actions. These actions can automate tasks like:

• Running tests.
• Linting code.
• Deploying applications.

When to Use Pre-built Actions

1. Use When:

   • A task is repetitive and commonly used (e.g., linting, testing, deployment).
   • The action is well-maintained and has good reviews.

2. Avoid When:

   • The task is highly specific to your project.
   • You need full control over the task’s execution.

Example: Using a Pre-built Action for Linting

Let’s use the github/super-linter action to lint code.

name: Lint Code
on: push
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Run Linter
        uses: github/super-linter@v4
        env:
          VALIDATE_ALL_CODEBASE: true

Explanation:

1. uses: actions/checkout@v3:
   • Checks out your repository so the linter can access the code.
2. uses: github/super-linter@v4:
   • Runs the Super Linter to check your code for style and syntax issues.
3. Environment Variables:
   • VALIDATE_ALL_CODEBASE: true: Ensures the linter checks all files in your codebase.

Outcome:

• The workflow validates your code for common issues.
• You’ll see a report in the Actions tab detailing errors or warnings.

Example: Using a Pre-built Action for Testing

Suppose you’re testing a Node.js application. You can use the actions/setup-node action.

name: Test Node.js App
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - name: Install Dependencies
        run: npm install

      - name: Run Tests
        run: npm test

Explanation:

1. actions/setup-node@v3:
   • Sets up Node.js version 18 for the environment.
2. npm install:
   • Installs project dependencies.
3. npm test:
   • Runs the test suite.

Outcome:

• The workflow installs dependencies and runs tests automatically when code is pushed.

Customizing Workflows

Customizing workflows is about tailoring them to your project’s needs, whether it’s by reusing values, handling sensitive information securely, or running tasks across different configurations.

Environment Variables

Environment variables are reusable values that you can define once and use throughout your workflow. This is helpful for storing common configurations, like API URLs or project names, that might change depending on the environment (e.g., development or production).

Using Environment Variables in Workflows

Here’s an example of using env to define variables directly in a workflow:

name: Use Environment Variables
on:
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    env:
      API_URL: https://api.example.com
      PROJECT_NAME: MyCoolProject
    steps:
      - name: Print Environment Variables
        run: |
          echo "The API URL is $API_URL"
          echo "The project name is $PROJECT_NAME"          

Explanation:

• env Block:
  • Defines environment variables (API_URL and PROJECT_NAME) that are available to all steps in the job.
• run Command:
  • Prints the variables to the console using echo.

Outcome:

• Outputs:

  The API URL is https://api.example.com
  The project name is MyCoolProject
  

Secrets Management

Secrets are used to store sensitive information like API keys, tokens, or passwords securely. Unlike environment variables, secrets are encrypted and cannot be accessed directly in plain text.

How to Set Up Secrets in GitHub

1. Go to your repository on GitHub.
2. Click on the Settings tab.
3. Under the Secrets and variables section, click Actions.
4. Select Secrets and add your secret name and value (e.g., API_TOKEN = abc123xyz).

Using Secrets in Workflows

name: Use Secrets
on:
  push:
    branches:
      - main
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Use Secret API Token
        env:
          API_TOKEN: ${{ secrets.API_TOKEN }}
        run: |
          echo "Using secret API token to make requests..."
          curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com          

Explanation:

• secrets:
  • Accesses the encrypted secret (API_TOKEN) defined in repository settings.
• curl Command:
  • Makes a secure request to an API using the secret token in the authorization header.

Outcome:

• The API token is securely used to authenticate the request without exposing its value.

Matrix Builds

Matrix builds allow you to test or run workflows across multiple configurations, like different operating systems, Node.js versions, or Python versions. This ensures compatibility across various environments.

Example: Testing on Multiple Node.js Versions

name: Matrix Build Example
on:
  push:
    branches:
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [12, 14, 16]
    steps:
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}

      - name: Checking the node version
        run: node --version

      - name: Install Dependencies
        run: npm install

      - name: Run Tests
        run: npm test

Explanation:

1. matrix.node-version:
   • Defines a matrix of Node.js versions (12, 14, 16) to test against.
2. actions/setup-node:
   • Configures the Node.js environment for the current version in the matrix.
3. Steps:
   • Installs dependencies and runs tests for each Node.js version.

Outcome:

• The workflow runs three separate jobs, one for each Node.js version:

  Testing with Node.js 12
  Testing with Node.js 14
  Testing with Node.js 16
  

Simplified Explanation for Matrix Builds

Imagine you’re a chef testing a recipe with three different ingredients (e.g., sugar, honey, and maple syrup). Instead of making one dish and swapping the ingredients manually, you have three chefs who each make the dish with one ingredient at the same time. This way, you can see which one tastes best, faster!

When to Use These Customizations

1. Environment Variables:

   • Use for values that are not sensitive but may change based on the environment (e.g., database URLs or app names).

2. Secrets:

   • Use for sensitive values like passwords or API tokens. Always keep them secure using GitHub’s secrets management.

3. Matrix Builds:

   • Use when you need to ensure compatibility across multiple configurations or environments. Ideal for testing software or building platform-specific binaries.

Advanced Features

This section explores powerful GitHub Actions features to enhance workflow efficiency and flexibility.

Creating Reusable Workflows

Reusable workflows save time by centralizing common tasks. Instead of duplicating code across multiple workflows, you can call a pre-defined workflow.

Defining Reusable Workflows with workflow_call

Reusable workflows are stored in a repository and can be invoked from other workflows. To make a workflow reusable, you define on: workflow_call.

Example: Creating a Shared Deployment Workflow

In ./.github/workflows/deploy.yml (Reusable Workflow):

name: Reusable Deployment Workflow
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Deploy to ${{ inputs.environment }}
        run: echo "Deploying to ${{ inputs.environment }}"

In another workflow that calls this reusable one:

name: Main Workflow
on:
  push:
    branches:
      - main
jobs:
  invoke-deploy:
    uses: org/repo/.github/workflows/deploy.yml@main
    with:
      environment: production

Explanation:

• Reusable Workflow (deploy.yml):
  • Defines an input environment to specify the deployment environment.
• Calling Workflow:
  • Uses the reusable workflow with specific inputs (production).

Outcome:

• Streamlines deployment by sharing a standard deployment process.

Writing Custom Actions

Custom actions let you define unique tasks that aren’t covered by existing pre-built actions. You can create these actions using JavaScript or Docker.

Step-by-step Guide to Creating a JavaScript Action

1. Create a Repository: For your custom action.

2. Add an action.yml File:

   name: My Custom Action
   description: Prints a greeting message.
   inputs:
     name:
       description: "Name to greet"
       required: true
   runs:
     using: "node16"
     main: "index.js"
   

3. Write the JavaScript Logic: In index.js:

   const core = require("@actions/core");
   
   try {
     const name = core.getInput("name");
     console.log(`Hello, ${name}!`);
   } catch (error) {
     core.setFailed(error.message);
   }
   

4. Publish to GitHub Marketplace:

   • Push your code to a public repository.
   • Tag a release, and GitHub will automatically prompt you to publish it.

Outcome:

• A reusable custom action that can greet users by name.

Caching Dependencies

Caching improves build speeds by reusing previously downloaded dependencies.

Example: Caching npm Dependencies

name: Cache npm Dependencies
on:
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Cache npm
        uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-            
      - name: Install Dependencies
        run: npm install

Explanation:

• actions/cache:
  • Caches the ~/.npm directory to reuse dependencies between builds.
• key:
  • Uniquely identifies the cache based on the OS and package-lock.json file changes.

Outcome:

• Faster builds by avoiding re-downloading npm packages.

Self-hosted Runners

A self-hosted runner is a machine that you manage yourself to run GitHub Actions. It allows for more customization, such as using specific hardware or software configurations.

What is a Self-hosted Runner?

• A server or VM you control.
• Used when GitHub-hosted runners don’t meet your requirements (e.g., private network access, special dependencies).

Setting Up and Using a Self-hosted Runner

1. Install Runner Software:

   • On your server, download the runner software from your repository’s Settings > Actions > Runners page.

2. Configure the Runner:

   ./config.sh --url https://github.com/your-repo --token YOUR_TOKEN
   

3. Start the Runner:

   ./run.sh
   

4. Use the Runner in a Workflow:

   name: Use Self-hosted Runner
   on:
     push:
       branches:
         - main
   jobs:
     build:
       runs-on: self-hosted
       steps:
         - name: Checkout Code
           uses: actions/checkout@v3
         - name: Run a Script
           run: echo "Running on a self-hosted runner!"
   

When to Use Self-hosted Runners:

• Special hardware requirements (e.g., GPUs for machine learning).
• Running workflows within a private network.
• Cost optimization for large workflows.

Outcome:

• Greater control over the execution environment.

When to Use These Advanced Features

1. Reusable Workflows:

   • When multiple repositories share the same process (e.g., deployment or testing).

2. Custom Actions:

   • When pre-built actions don’t meet specific needs.

3. Caching:

   • When builds are repetitive and involve downloading the same dependencies.

4. Self-hosted Runners:

   • When you need specialized hardware or private network access.

Best Practices

Adhering to best practices ensures that your GitHub Actions workflows are efficient, maintainable, and easy to debug.

Optimizing Workflows

Optimizing workflows can save time and reduce costs by ensuring jobs run only when needed and making efficient use of resources.

Reducing Execution Time and Costs

GitHub Actions workflows consume resources, and unnecessary executions can be costly. You can optimize them using conditionals and caching.

Example: Running Jobs Only When Necessary

name: Conditional Workflow
on:
  push:
    branches:
      - main
jobs:
  test:
    if: github.event_name == 'push' && github.event.head_commit.message != 'skip-ci'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Run Tests
        run: npm test

Explanation:

• if Condition:
  • Checks if the event is a push and the commit message doesn’t include skip-ci.
• Outcome:
  • Jobs are skipped for commits labeled skip-ci, saving execution time and resources.

Organizing Workflow Files

Well-organized workflows improve readability, maintainability, and collaboration.

Using Descriptive Names and Clear Comments

1. Descriptive Names:
   • Use meaningful file names such as build-and-deploy.yml instead of generic ones like main.yml.
2. Add Comments:

   # This workflow builds and deploys the application
   name: Build and Deploy
   on:
     push:
       branches:
         - main
   jobs:
     build:
       # Build the application
       runs-on: ubuntu-latest
       steps:
         - name: Checkout Code
           uses: actions/checkout@v3
   

Keeping Workflows Modular

Break down workflows into smaller, modular files for specific tasks. For example:

• build.yml for building the app.
• deploy.yml for deployment.

You can call reusable workflows to combine these smaller files when needed.

Outcome:

• Easier debugging and updates when workflows are modular and self-explanatory.

Monitoring and Debugging

Efficient debugging and monitoring of workflows ensure smooth execution and quick resolution of issues.

Using Workflow Logs

GitHub Actions provide logs for each step in a workflow. Use them to identify and fix issues.

Example: Debugging with debug Logs

name: Debugging Example
on:
  push:
    branches:
      - main
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v3
      - name: Debugging Step
        run: echo "Current branch: ${{ github.ref }}"
      - name: Run Tests
        run: npm test
        env:
          CI: true

Explanation:

• echo Statement:
  • Prints the current branch name to the logs.
• env: CI:
  • Ensures a CI-friendly environment for test commands.

Common Errors and Fixes

1. Error: Workflow not triggering.

   • Cause: Incorrect on event configuration.
   • Fix: Double-check the event name (e.g., push, pull_request) and branch filters.

2. Error: Missing or invalid secrets.

   • Cause: Secrets not set up in the repository.
   • Fix: Go to Settings > Secrets in your repository, and add the required secret.

3. Error: Job fails due to missing dependencies.

   • Cause: Dependencies not installed correctly.
   • Fix: Add an installation step, such as npm install or pip install.

When to Use These Best Practices

1. Optimize Workflows:
   • For projects with frequent changes to save on time and cost.
2. Organize Workflow Files:
   • For large projects with multiple contributors.
3. Monitor and Debug:
   • When troubleshooting failed workflows or unexpected behavior.

Conclusion and Next Steps

In this second part of our GitHub Actions guide, we’ve covered:

• Building practical workflows with sequential and parallel jobs
• Using pre-built actions from the GitHub Marketplace
• Customizing workflows with environment variables, secrets, and matrix builds
• Implementing advanced features like reusable workflows, custom actions, caching, and self-hosted runners
• Following best practices for optimizing, organizing, and debugging your workflows

By now, you should have a solid understanding of how to create sophisticated GitHub Actions workflows tailored to your project’s needs. These automated workflows can significantly improve your development process, ensuring code quality and streamlining repetitive tasks.

In Part 3, we’ll explore real-world use cases for GitHub Actions, including deploying applications, running CI/CD pipelines, automating documentation, and scheduling periodic tasks. We’ll also cover integrations with other tools and troubleshooting techniques to help you address common issues.

Continue to Part 3: Real-world Applications and Troubleshooting or go back to Part 1: Introduction and Setup