AWS CodePipeline: A Comprehensive Guide - Part 1

Introduction

What is AWS CodePipeline?

AWS CodePipeline is a fully managed continuous integration and continuous delivery (CI/CD) service that automates your software release processes, enabling fast and reliable updates.

Imagine you’re building a house. Instead of manually handling every task—like laying bricks, painting walls, or installing doors—you create an assembly line where each task happens in sequence or parallel without delays. Similarly, AWS CodePipeline automates the stages of software development, such as pulling code, building it, testing it, and deploying it, ensuring everything runs smoothly without manual intervention.

Key Features:

• Automates the entire software release process.
• Integrates seamlessly with AWS services and third-party tools.
• Enables fast iterations and faster delivery of new features or fixes.

Why Use AWS CodePipeline?

AWS CodePipeline offers several benefits, making it an essential tool for modern software development workflows:

1. Automation:

• With CodePipeline, you no longer need to manually execute commands for building, testing, or deploying your application.
• Example: When a developer pushes code to a repository (like GitHub or AWS CodeCommit), CodePipeline can automatically build and deploy the application to a staging or production environment.

2. Integration:

• It works out of the box with popular source control systems (e.g., GitHub, Bitbucket, CodeCommit) and AWS services (like CodeBuild, CodeDeploy, CloudFormation).
• It also integrates with third-party tools like Jenkins, Selenium, and Slack.

3. Scalability:

• CodePipeline handles large-scale operations and can run multiple pipelines in parallel without impacting performance.

Traditional Deployment vs. CodePipeline:

How Does AWS CodePipeline Work?

AWS CodePipeline orchestrates the software release process by breaking it down into stages. Each stage represents a step in the CI/CD process.

High-Level Overview of Stages:

1. Source:

   • This is where CodePipeline pulls the code from a repository like AWS CodeCommit, GitHub, or Bitbucket.
   • Example: Suppose you’ve written a web application and stored the code in GitHub. CodePipeline will automatically detect changes and fetch the updated code for the next stages.

2. Build:

   • This stage compiles the code, runs scripts, and generates build artifacts.

   • Tool: AWS CodeBuild or third-party tools like Jenkins.

   • Example: For a Node.js application, this stage could run npm install and npm build commands to package your application.

   • What is an artifact?

     • Answer: Artifacts are the output of the build stage, like compiled code or packaged files, which are passed to the next stages. Think of it as delivering raw ingredients to a chef, who turns them into a ready-to-serve dish.

3. Test:

   • Executes automated tests to validate the application.

   • Example: Running unit tests to check if individual parts of your application work as expected.

   • Why is this stage important?

     • Answer: It ensures your code is error-free before moving to production, reducing downtime and customer impact.

4. Deploy:

   • Deploys the application to environments like development, staging, or production.
   • Example: If you’re hosting your app on Amazon EC2, this stage will launch the latest version of the app on your instances.

Example Walkthrough: A Simple Web Application

Scenario:

Let’s say you’re building a blog website and want to automate its release process.

Steps:

1. Source Stage:

   • Use GitHub to store your code.
   • CodePipeline pulls the latest code every time you push changes.

   Command Example (GitHub integration):

   aws codepipeline create-pipeline --cli-input-json file://pipeline-config.json
   

   • This command creates a new pipeline using a JSON configuration file.
   • Outcome: CodePipeline fetches your code from GitHub whenever you push updates.

2. Build Stage:

   • Use AWS CodeBuild to package your blog.
   • Command Example:

     npm install && npm build
     

     • Installs dependencies and builds your project.
     • Outcome: Creates a ready-to-deploy build artifact.

3. Deploy Stage:

   • Use AWS Elastic Beanstalk to host your blog.
   • CodePipeline deploys the latest version automatically.

   Outcome: Your blog is live with the new updates, and you didn’t have to lift a finger after pushing your code!

Getting Started with AWS CodePipeline

Prerequisites

Before diving into AWS CodePipeline, make sure you have the following ready:

1. AWS Account Setup

   • Sign up for an AWS account if you don’t already have one.
   • Enable billing alerts to keep track of your usage costs, especially if you’re just experimenting.

2. Basic Knowledge of Key Concepts

   • Git: Familiarity with Git commands like git clone, git push, and git pull.
     • Example: Git is like a notebook where you write (commit) your code, and GitHub/AWS CodeCommit is like the online storage for sharing it with your team.
   • CI/CD: Understand the basics of Continuous Integration (CI) and Continuous Deployment (CD).
     • CI: Automates building and testing your code.
     • CD: Automates deploying your code to production.

3. Tools Required

   • AWS CLI: A command-line interface for interacting with AWS services.
     • Installation Command:

       curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
       sudo installer -pkg AWSCLIV2.pkg -target /
       

       • Outcome: AWS CLI installed, allowing you to manage AWS services directly from your terminal.
   • Git: Install Git to manage your source code.
     • Installation links: Git Downloads.
   • Code Editor (Optional): Tools like Visual Studio Code or IntelliJ for writing and editing your code.

Core Concepts

Pipeline

• A pipeline in AWS CodePipeline is the workflow that automates your software release process.
• Think of it as an assembly line where each stage represents a step like coding, testing, or deploying.

Why is it important?

• It removes manual intervention, ensures consistent quality, and speeds up deployment.

Stages and Actions

• A stage is a group of sequential or parallel actions in your pipeline.
  • Example: The “Source” stage pulls your code, while the “Build” stage compiles it.
• Actions are tasks performed in each stage.
  • Example: In the “Test” stage, running automated tests is an action.

Question: Why break the process into stages?

• Answer: Breaking it into stages makes it easier to debug and monitor specific parts of the release process.

Visualizing Stages:

• Source Stage: Gets code from a repository.
• Build Stage: Compiles the code and packages it.
• Deploy Stage: Uploads the application to a server or service like Amazon S3.

Artifacts

• Artifacts are the files or data produced during the pipeline’s execution, like a compiled codebase or packaged app.
• Example: If you’re building a React app, the build stage generates index.html and bundle.js files, which are artifacts sent to the deploy stage.

Question: Why are artifacts important?

• Answer: They ensure the correct version of your app gets passed between stages, maintaining consistency.

First Pipeline: Step-by-Step

Scenario:

We’ll create a basic pipeline to automate the deployment of a static website hosted on S3.

Step 1: Create an S3 Bucket

1. Open the AWS Management Console and navigate to S3.
2. Click Create bucket, give it a name (e.g., my-static-sit-unique-name), and enable public access for hosting.

Command Example:

aws s3 mb s3://my-static-sit-unique-name

By running the above command it automatically opens the public access for the bucket.

• What this does: Creates a new S3 bucket named my-static-sit-unique-name.
• Outcome: A storage space for your static website files.

Step 2: Update the Bucket Policy and Allow Public Access

1. Update the bucket policy to allow public access.

Command Example:

aws s3api put-bucket-policy --bucket my-static-sit-unique-name --policy file://bucket-policy.json

Example Bucket Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Principal": {
        "AWS": "arn:aws:iam::935389764279:user/testing-user"
      },
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": [
        "arn:aws:s3:::my-static-sit-unique-name",
        "arn:aws:s3:::my-static-sit-unique-name/*"
      ]
    }
  ]
}

Allow Public Access:

• Go to the S3 bucket and click on the bucket name my-static-sit-unique-name.
• Click on the Permissions tab.
• Scroll down to the Bucket Policy section.
• Click on the Edit button.
• Choose the Allow public access option.
• Click on the Save changes button.

Step 3: Upload Static Website Files

1. Write a simple index.html file like this:

   <!DOCTYPE html>
   <html>
     <head>
       <title>My Static Website</title>
     </head>
     <body>
       <h1>Hello, AWS CodePipeline!</h1>
     </body>
   </html>
   

2. Upload it to your S3 bucket using the console or CLI.

Command Example:

aws s3 cp index.html s3://my-static-sit-unique-name

• Outcome: Your index.html is now hosted in the S3 bucket.

Step 4: Enable static website hosting

1. Go to the S3 bucket and click on the bucket name my-static-sit-unique-name.
2. Click on the Properties tab.
3. Scroll down to the Static website hosting section.
4. Enable the Static website hosting by clicking on the Edit button.
5. Choose the Host a static website option.
6. Specify the Index document as index.html.
7. Click on the Save changes button.

Step 5: Create the Pipeline

1. Go to the AWS CodePipeline Console and click Create pipeline.
2. And choose the Custom pipeline option.
3. Name your pipeline (e.g., StaticWebsitePipeline) and leave the rest of the fields as default in choose pipeline setting page.

Step 6: Add the Source Stage

1. Choose your source provider (e.g., GitHub or AWS CodeCommit).
2. Connect your repository where the static website code is stored and keep the rest of the fields as default.

Command Example:

aws codepipeline create-pipeline --cli-input-json file://pipeline.json

• What this does: Creates a new pipeline based on a JSON configuration file.
• Outcome: The pipeline is ready to pull code from your repository.
• Example of pipeline.json: Mentioned below go through it.

Step 7: Skip Build Stage (Optional for Static Websites)

• Since it’s a static website, you can skip the build stage or add one for custom actions (like minifying CSS/JS).

Step 8: Add the Deploy Stage

1. Choose S3 as the deployment provider.
2. Choose the region where you want to deploy the website.
3. Keep the Input Artifacts as default.
4. Specify the S3 bucket name (my-static-sit-unique-name).
5. Specify the S3ObjectKey as index.html.
6. Rest of the fields are optional.
7. Click on Create pipeline button.

Step 9: Trigger the Pipeline

• Push changes to your repository and watch the pipeline automatically pull the code, process it, and deploy it to S3.

Outcome:

• Open your S3 bucket’s URL in a browser to see your deployed static website.

Example Pipeline JSON (For CLI Users)

Here’s a JSON example for creating the above pipeline:

{
  "pipeline": {
    "name": "StaticWebsitePipeline",
    "roleArn": "arn:aws:iam::123456789012:role/service-role/AWSCodePipelineServiceRole",
    "artifactStore": {
      "type": "S3",
      "location": "codepipeline-artifacts"
    },
    "stages": [
      {
        "name": "Source",
        "actions": [
          {
            "name": "SourceAction",
            "actionTypeId": {
              "category": "Source",
              "owner": "AWS",
              "provider": "S3",
              "version": "1"
            },
            "outputArtifacts": [{ "name": "SourceOutput" }],
            "configuration": {
              "S3Bucket": "my-static-sit-unique-name",
              "S3ObjectKey": "index.html"
            }
          }
        ]
      },
      {
        "name": "Deploy",
        "actions": [
          {
            "name": "DeployAction",
            "actionTypeId": {
              "category": "Deploy",
              "owner": "AWS",
              "provider": "S3",
              "version": "1"
            },
            "inputArtifacts": [{ "name": "SourceOutput" }],
            "configuration": { "BucketName": "my-static-sit-unique-name" }
          }
        ]
      }
    ]
  }
}

• What this does: Defines a simple pipeline with Source and Deploy stages.
• Outcome: Creates and configures the pipeline to automate static website deployment.

Intermediate Concepts

Pipeline Stages and Actions

AWS CodePipeline is divided into stages, with each stage comprising actions that perform specific tasks.

Stages in Detail:

1. Source Stage:
   • This is where the pipeline retrieves the source code.
   • Supported source providers:
     • AWS CodeCommit: AWS’s version control service.
     • GitHub/Bitbucket: External repositories.
     • S3: For static files or zip packages.
   • Example Action:
     Retrieving code from GitHub.

     {
       "name": "SourceAction",
       "actionTypeId": {
         "category": "Source",
         "owner": "ThirdParty",
         "provider": "GitHub",
         "version": "1"
       },
       "configuration": {
         "Owner": "your-github-username",
         "Repo": "your-repository-name",
         "Branch": "main",
         "OAuthToken": "your-github-oauth-token"
       },
       "outputArtifacts": [{ "name": "SourceOutput" }]
     }
     

     • What this does: Sets up GitHub as the source.
     • Outcome: The pipeline automatically pulls code whenever you push changes to the repository.

In AWS Console

• Go to the AWS CodePipeline Console and click on the Create pipeline button.
• Choose the Custom pipeline option.
• Name your pipeline (e.g., StaticWebsitePipeline) and leave the rest of the fields as default in choose pipeline setting page.
• Click on Next button.
• Add the source stage and choose the GitHub option or any other source provider and keep the rest of the fields as default.
• Click on Next button.

2. Build Stage:

   • This is where the application is compiled, dependencies are installed, or custom scripts are executed.

   • Common actions:

     • AWS CodeBuild: Compiles source code, runs tests, and produces deployable artifacts.

   • Example: Running a build process for a Node.js app.

     • Install dependencies using npm install and run tests using npm test.

     buildspec.yml (used by CodeBuild):

     version: 0.2
     phases:
       install:
         runtime-versions:
           nodejs: 14
         commands:
           - echo Installing dependencies
           - npm install
       build:
         commands:
           - echo Building the project
           - npm run build
       post_build:
         commands:
           - echo Build complete
     artifacts:
       files:
         - "**/*"
     

     • Outcome: The build stage compiles the app and packages it for deployment.

(For more details go through the AWS CodeBuild Documentation)

3. Test Stage:
   • Ensures that the application works as expected before deployment.
   • Common actions:
     • Running automated tests (e.g., unit tests, integration tests).
   • Example: Running unit tests using a custom test script in Python:

     phases:
       build:
         commands:
           - echo Running tests
           - pytest
     

4. Deploy Stage:

   • Deploys the build artifacts to the target environment.

   • Common destinations:

     • Amazon S3: For static websites or packaged files.
     • AWS Elastic Beanstalk: For web apps.
     • AWS Lambda: For serverless functions.

   • Example Action: Deploying to S3.

     {
       "name": "DeployAction",
       "actionTypeId": {
         "category": "Deploy",
         "owner": "AWS",
         "provider": "S3",
         "version": "1"
       },
       "inputArtifacts": [{ "name": "BuildOutput" }],
       "configuration": {
         "BucketName": "my-deployment-bucket",
         "Extract": "true"
       }
     }
     

     • Outcome: Deploys build artifacts to the S3 bucket for hosting.

Integrating with Source Repositories

CodePipeline can integrate seamlessly with various source control systems.

1. Supported Repositories:

   • AWS CodeCommit: AWS-hosted Git repositories.
   • GitHub/Bitbucket: For public or private repositories.
   • S3: For directly uploading source files.

2. Example: Connecting a CodeCommit Repository

   • Step 1: Create a repository in AWS CodeCommit.

     aws codecommit create-repository --repository-name MyDemoRepo
     

     • Outcome: Creates a new Git repository in AWS.

   • Step 2: Clone the repository locally.

     git clone https://git-codecommit.<region>.amazonaws.com/v1/repos/MyDemoRepo
     cd MyDemoRepo
     

     • Outcome: The repository is ready for local development.

   • Step 3: Add the repository to the pipeline as the source stage.

Pipeline Notifications

AWS CodePipeline supports notifications for key events like pipeline failures, stage completion, or success.

1. Using Amazon SNS for Notifications:

   • SNS (Simple Notification Service) sends alerts to email, SMS, or other systems.

2. Steps to Set Up Notifications:

   • Step 1: Create an SNS topic.

     aws sns create-topic --name PipelineAlerts
     

     • Outcome: Creates a topic for sending alerts.

   • Step 2: Subscribe to the topic.

     aws sns subscribe --topic-arn arn:aws:sns:<region>:<account-id>:PipelineAlerts --protocol email --notification-endpoint [email protected]
     

     • Outcome: Adds your email as a subscriber.

   • Step 3: Link the topic to CodePipeline using CloudWatch Events.

     {
       "source": ["aws.codepipeline"],
       "detail-type": ["CodePipeline Pipeline Execution State Change"],
       "resources": ["arn:aws:codepipeline:<region>:<account-id>:PipelineName"],
       "detail": {
         "state": ["FAILED"]
       }
     }
     

     • Outcome: Alerts you via email if the pipeline fails.

Example Scenario: Building a Node.js App

We’ll build and deploy a Node.js app to AWS Elastic Beanstalk using CodePipeline.

1. Create the Node.js App:

   • index.js:

     const express = require("express");
     const app = express();
     
     app.get("/", (req, res) => {
       res.send("Hello, AWS CodePipeline!");
     });
     
     const port = process.env.PORT || 3000;
     app.listen(port, () => console.log(`App running on port ${port}`));
     

   • package.json:

     {
       "name": "nodejs-app",
       "version": "1.0.0",
       "main": "index.js",
       "scripts": {
         "start": "node index.js"
       },
       "dependencies": {
         "express": "^4.17.1"
       }
     }
     

2. Push the App to CodeCommit:

   git add .
   git commit -m "Initial commit"
   git push origin main
   

3. Create the Pipeline:

   • Source Stage: CodeCommit.
   • Build Stage: CodeBuild with the following buildspec.yml:

     version: 0.2
     phases:
       install:
         runtime-versions:
           nodejs: 14
         commands:
           - npm install
       build:
         commands:
           - npm run build
     artifacts:
       files:
         - "**/*"
     

   • Deploy Stage: Deploy to Elastic Beanstalk.

4. Outcome:

   • Access the Node.js app via the Elastic Beanstalk endpoint.