Continuous Integration : Integrating BuildKite with Your Scala Project

In a very broad sense, BuildKite (earlier called BuildBox) is a continuous integration server that allows you to keep working on your code while there is a CI box which is reporting about any issues. The problem with most web based CI servers is that they have to do a lot of magic beneath the hood to let you build your custom system on their environment.

What does this mean? This means that if your product uses the Riak database and mine uses PostgreSQL then a web based CI system would have to give me default installations of both of them so that both our products can be supported. Remember this is just 2 databases that we talked about. Now, bring in more databases, more external integration’s and to cap it, different versions of all of these for different products. Suddenly having a hosted CI is not really the space you want to be 🙂

Contrary to this, BuildKite seems to have pushed down these problems back to the development team. Ok, this is not necessarily as bad as it sounds. What this means is that now you have control over your build environment. You know exactly what instances and what versions you would be working with.

Conceptually, it is quite similar to any other CI environment that you would have worked with.
You have a Project on which you want to run CI. This could be your GitHub, Bitbucket or any other repository project. You need to specify a file which would get executed when the CI would trigger. For example in our case the file is It is IMPORTANT to note that this should be an executable script and buildkite should have rights to execute this script.

Now comes the interesting part. Which is the Agents. Agent is responsible for running the CI job on your CI server. The agent polls Buildkite looking for work. When a new job is ready to run, the agent will run the script with all the environment variables required to run the job.
This script is responsible for creating the build directory, cloning the repo, running the build script, and uploading artifacts.

Agent would do the following for any repository that it works with as a part of setting up the environment.

$ mkdir -p "/home/vikas/.buildbox/builds/vikas-dev-agent/knoldus/akka-roller"
$ cd "/home/vikas/.buildbox/builds/vikas-dev-agent/knoldus/akka-roller"
$ git clean -fdq
$ git fetch -q
$ git reset --hard origin/master
HEAD is now at 87763ca Update
$ git checkout -qf "87763ca0bd0bdf3e905d6bd545693a14350779fd"

Once it has downloaded the latest code onto your CI box then it would run the script that you have mentioned. In our case, it ends up being src/main/resources/

For example, refer to the diagram below. We have 2 projects and 2 CI servers. On one CI server we have the agent running for Project 1 and on the other, we have the agent running for Project 2. Now when BuildKite gets the trigger for building then it invokes the running agent on either of the CI servers as necessary.

What I like?

  1. Controlled environment – We can set up our environment
  2. Ease of use
  3. Has Bells and whistles like build passing badges etc

What I dont like?

  1. It should be free for public projects on Github

Our public project which has been integrated with BuildBox is present here.

About Vikas Hazrati

Vikas is the Founding Partner @ Knoldus which is a group of software industry veterans who have joined hands to add value to the art of software development. Knoldus does niche Reactive and Big Data product development on Scala, Spark and Functional Java. Knoldus has a strong focus on software craftsmanship which ensures high-quality software development. It partners with the best in the industry like Lightbend (Scala Ecosystem), Databricks (Spark Ecosystem), Confluent (Kafka) and Datastax (Cassandra). To know more, send a mail to or visit
This entry was posted in Architecture, Devops. Bookmark the permalink.

2 Responses to Continuous Integration : Integrating BuildKite with Your Scala Project

  1. Reblogged this on Play!ng with Scala and commented:

    Integrating BuildKite with Your Scala Project

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s