Concourse CI CD pipeline

In this blog, we are going to set up a typical CI/CD pipeline using concourse to make the development process more agile and deployment more visible.

The concourse is an open-source continuous thing-doer.

Built on the simple mechanics of resources, tasks, and jobs, Concourse presents a general approach to automation that makes it great for CI/CD.

Quick Start Concourse CI

There are many ways to setup concourse but in this blog, we are going to use the easiest way to setup concourse that is using docker-compose.

Download the docker-compose file

It contains a concourse server and PostgreSQL database for concourse backend.

wget https://concourse-ci.org/docker-compose.yml

Run that docker-compose file

docker-compose up -d

The concourse will be running at localhost:8080. You can log in with the username/password as a test/test.

fly CLI command-line tool

fly CLI is a command-line tool that you need to use to set up and manage pipelines on Concourse. We need to install fly CLI by downloading it from the web UI (localhost:8080). You can see in the above welcome page to download the CLI for windows, mac, and Linux.

After download the CLI tool we need to give execute permission to that and copy to the /usr/bin/

chmod +x fly

sudo cp fly /usr/bin/

Next, set target your local Concourse as the test user:

fly -t tutorial login -c http://localhost:8080 -u test -p test

Now you have successfully setup concourse on your local machine, let’s move to the CI/CD part.

Concourse Pipeline Directory structure

Concourse recomended directory structure.

Set up CI/CD

In this practical, we are going to deploy a spring boot application to the Kubernetes, Lets Start.

Clone the below repository

git clone https://github.com/azmathasan92/concourse-ci-cd.git

Go the to project root directory

cd concourse-ci-cd

Create a pipeline using a fly tool

Note: Before creating the pipeline we need to update the config.yml file to provide the docker and Kubernetes credentials.

Run the below command

fly -t tutorial set-pipeline -p spring-boot-service -c concourse_ci/pipeline.yml -l concourse_ci/config.yml

The pipeline is configured now, and you can see that in the concourse UI.

When you create a new pipeline by default it is paused, so we need to unpause the pipeline, Run the below command to unpause the pipeline

fly -t tutorial unpause-pipeline -p spring-boot-service

Now, you have successfully created a CI/CD pipeline to the kubernetes.

Lets Understand pipeline.yml file

resource_types:
  - name: kubectl-resource
    type: docker-image
    source: 
      repository: jmkarthik/concourse-kubectl-resource
      tag: latest

resources:
  - name: spring-boot-service
    type: git
    source:
      uri: https://github.com/azmathasan92/concourse-ci-cd.git
      branch: master

  - name: spring-boot-service-docker-repository
    type: docker-image
    source:
      email: ((docker-hub-email))
      username: ((docker-hub-username))
      password: ((docker-hub-password))
      repository: ((docker-hub-username))/((docker-hub-repo-name))

  - name: kubectl
    type: kubectl-resource
    source:
      api_server_uri: ((server))
      namespace: ((namespace))
      certificate_authority_data: ((cad))
      token: ((token))

jobs:
  - name: test
    public: true

    plan:
      - get: spring-boot-service
        trigger: true
      - task: mvn-test
        file: "spring-boot-service/concourse_ci/tasks/maven-test.yml"

  - name: package
    public: true
    serial: true
    plan:
      - get: spring-boot-service
        trigger: true
        passed: [test]
      - task: mvn-package
        file: "spring-boot-service/concourse_ci/tasks/maven-package.yml"
      - put: spring-boot-service-docker-repository
        params:
          build: spring-boot-service-out

  - name: deploy
    public: true
    serial: true
    plan:
      - get: spring-boot-service
        trigger: false
        passed: [package]

      - put: kubectl
        params:
          file: "spring-boot-service/spring-boot-deploy.yaml"

In the above CI/CD pipeline, we are seeing the following components:

1. Resource Type
   Each resource in a pipeline has a type. The resource’s type determines what versions are detected, the bits that are fetched when the resource’s get step runs, and the side effect that occurs when the resource’s put step runs.
   An exhaustive list of all resource types is available in the Resource Types catalog.
2. Resource
   Resources are the heart and soul of Concourse. They represent all external inputs to and outputs of jobs in the pipeline.
3. Job
   Jobs determine the actions of your pipeline. They determine how resources progress through it, and how the pipeline is visualized.
   The most important attribute of a job is its build plan, configured as a job.plan. This determines the sequence of steps to execute in any builds of the job.
4. Task
   The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from task.inputs to task.outputs that can either succeed or fail.

Resources:

• https://concourse-ci.org/docs.html
• https://ci.concourse-ci.org/teams/main/pipelines/concourse

Thank you for sticking to the end. If you like this blog, please do show your appreciation by giving thumbs-ups and share this blog and give me suggestions on how I can improve my future posts to suit your needs. Follow me to get updates on different technologies