Continuous Integration : Integrating BuildKite with Your Scala Project

Table of contents

Reading Time: 3 minutes

In a very broad sense, BuildKite (earlier called BuildBox) is a continuous integration server that allows you to keep working on your code while there is a CI box which is reporting about any issues. The problem with most web based CI servers is that they have to do a lot of magic beneath the hood to let you build your custom system on their environment.

What does this mean? This means that if your product uses the Riak database and mine uses PostgreSQL then a web based CI system would have to give me default installations of both of them so that both our products can be supported. Remember this is just 2 databases that we talked about. Now, bring in more databases, more external integration’s and to cap it, different versions of all of these for different products. Suddenly having a hosted CI is not really the space you want to be 🙂

Contrary to this, BuildKite seems to have pushed down these problems back to the development team. Ok, this is not necessarily as bad as it sounds. What this means is that now you have control over your build environment. You know exactly what instances and what versions you would be working with.

Conceptually, it is quite similar to any other CI environment that you would have worked with.
You have a Project on which you want to run CI. This could be your GitHub, Bitbucket or any other repository project. You need to specify a file which would get executed when the CI would trigger. For example in our case the file is build.sh It is IMPORTANT to note that this should be an executable script and buildkite should have rights to execute this script.

Now comes the interesting part. Which is the Agents. Agent is responsible for running the CI job on your CI server. The agent polls Buildkite looking for work. When a new job is ready to run, the agent will run the bootstrap.sh script with all the environment variables required to run the job.
This script is responsible for creating the build directory, cloning the repo, running the build script, and uploading artifacts.

Agent would do the following for any repository that it works with as a part of setting up the environment.

$ mkdir -p "/home/vikas/.buildbox/builds/vikas-dev-agent/knoldus/akka-roller"
$ cd "/home/vikas/.buildbox/builds/vikas-dev-agent/knoldus/akka-roller"
$ git clean -fdq
$ git fetch -q
$ git reset --hard origin/master
HEAD is now at 87763ca Update README.md
$ git checkout -qf "87763ca0bd0bdf3e905d6bd545693a14350779fd"

Once it has downloaded the latest code onto your CI box then it would run the script that you have mentioned. In our case, it ends up being src/main/resources/build.sh

For example, refer to the diagram below. We have 2 projects and 2 CI servers. On one CI server we have the agent running for Project 1 and on the other, we have the agent running for Project 2. Now when BuildKite gets the trigger for building then it invokes the running agent on either of the CI servers as necessary.