In this blog, we are going to set up a typical CI/CD pipeline using concourse to make the development process more agile and deployment more visible.
The concourse is an open-source continuous thing-doer.
Quick Start Concourse CI
There are many ways to setup concourse but in this blog, we are going to use the easiest way to setup concourse that is using docker-compose.
Download the docker-compose file
It contains a concourse server and PostgreSQL database for concourse backend.
Run that docker-compose file
docker-compose up -d
The concourse will be running at localhost:8080. You can log in with the username/password as a test/test.
fly CLI command-line tool
fly CLI is a command-line tool that you need to use to set up and manage pipelines on Concourse. We need to install fly CLI by downloading it from the web UI (localhost:8080). You can see in the above welcome page to download the CLI for windows, mac, and Linux.
After download the CLI tool we need to give execute permission to that and copy to the /usr/bin/
chmod +x fly
sudo cp fly /usr/bin/
Next, set target your local Concourse as the test user:
fly -t tutorial login -c http://localhost:8080 -u test -p test
Now you have successfully setup concourse on your local machine, let’s move to the CI/CD part.
Concourse Pipeline Directory structure
Concourse recomended directory structure.
Set up CI/CD
In this practical, we are going to deploy a spring boot application to the Kubernetes, Lets Start.
Clone the below repository
Go the to project root directory
Create a pipeline using a fly tool
Note: Before creating the pipeline we need to update the config.yml file to provide the docker and Kubernetes credentials.
Run the below command
fly -t tutorial set-pipeline -p spring-boot-service -c concourse_ci/pipeline.yml -l concourse_ci/config.yml
The pipeline is configured now, and you can see that in the concourse UI.
When you create a new pipeline by default it is paused, so we need to unpause the pipeline, Run the below command to unpause the pipeline
fly -t tutorial unpause-pipeline -p spring-boot-service
Now, you have successfully created a CI/CD pipeline to the kubernetes.
Lets Understand pipeline.yml file
resource_types: - name: kubectl-resource type: docker-image source: repository: jmkarthik/concourse-kubectl-resource tag: latest resources: - name: spring-boot-service type: git source: uri: https://github.com/azmathasan92/concourse-ci-cd.git branch: master - name: spring-boot-service-docker-repository type: docker-image source: email: ((docker-hub-email)) username: ((docker-hub-username)) password: ((docker-hub-password)) repository: ((docker-hub-username))/((docker-hub-repo-name)) - name: kubectl type: kubectl-resource source: api_server_uri: ((server)) namespace: ((namespace)) certificate_authority_data: ((cad)) token: ((token)) jobs: - name: test public: true plan: - get: spring-boot-service trigger: true - task: mvn-test file: "spring-boot-service/concourse_ci/tasks/maven-test.yml" - name: package public: true serial: true plan: - get: spring-boot-service trigger: true passed: [test] - task: mvn-package file: "spring-boot-service/concourse_ci/tasks/maven-package.yml" - put: spring-boot-service-docker-repository params: build: spring-boot-service-out - name: deploy public: true serial: true plan: - get: spring-boot-service trigger: false passed: [package] - put: kubectl params: file: "spring-boot-service/spring-boot-deploy.yaml"
In the above CI/CD pipeline, we are seeing the following components:
- Resource Type
Each resource in a pipeline has a type. The resource’s type determines what versions are detected, the bits that are fetched when the resource’s get step runs, and the side effect that occurs when the resource’s put step runs.
An exhaustive list of all resource types is available in the Resource Types catalog.
Resources are the heart and soul of Concourse. They represent all external inputs to and outputs of jobs in the pipeline.
Jobs determine the actions of your pipeline. They determine how resources progress through it, and how the pipeline is visualized.
The most important attribute of a job is its build plan, configured as a job.plan. This determines the sequence of steps to execute in any builds of the job.
The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from task.inputs to task.outputs that can either succeed or fail.
Thank you for sticking to the end. If you like this blog, please do show your appreciation by giving thumbs-ups and share this blog and give me suggestions on how I can improve my future posts to suit your needs. Follow me to get updates on different technologies