In this going to be a long detailed blog we will optically discern how one can transform its long running SBT build project and can convert it into Bazel. So bear with me and lets start by introducing an incipient build giant in tech industry called “BAZEL“
What is Bazel?
Bazel is an open-source build and test tool similar to Maven, and Gradle, which uses a human-readable, high-level build language. The key difference here in Bazel is that it supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large code-bases across multiple repositories, and large numbers of users.
For more advantage of Bazel one can refer to – WHY SHOULD I USE BAZEL?.
So in this blog we will focus on how to convert a SBT multi module project into Bazel multi module project. I have created a sample sbt multi module project and also its final converted Bazel project, both can be dowloaded from links below –
Sample Scala/SBT Multi Module Example –
https://techhub.knoldus.com/dashboard/projects/scala/5f0218800b31c2f100025449
A sample example to provide a multi-module SBT build conversion to BAZEL build –
https://techhub.knoldus.com/dashboard/projects/scala/5f021a7d0b31c2f10002544a
The final Project structure will looks like –
├── BUILD
├── WORKSPACE
├── users
│ ├── BUILD
│ └── src
│ ├── main
│ │ └── scala
│ │ └── io.knoldus
│ │ └── userInfo.scala
│ └── test
│ └── scala
│ └── io.knoldus
│ └── userInfoSpec.scala
├── dependencies.yaml
└── common
├── BUILD
└── src
├── main
│ └── scala
│ └── util
│ └── Utils.scala
└── test
└── scala
└── util
└── UtilsSpec.scala
So we have two modules called: users and common.
users’ modules will depend on the common module.
STEP 1 – Setup Bazel
Set up Bazel. Download and install Bazel.
STEP 2 – The WORKSPACE File
Set up a project workspace, which is a directory where Bazel looks for build inputs and BUILD files, and where it stores build outputs.
As the rule of thumb, each project has one WORKSPACE file, where we will define things like Scala version and dependencies.
Note* – If in the project directory there is a sub-directory with a WORKSPACE file, then while doing our builds this sub-directory will be omitted.
To add the WORKSPACE one can use the build file example from –
https://github.com/bazelbuild/rules_scala#getting-started
In the provided example I have added the modified and comment provided WORKSPACE file, with the latest versions of Scala in it.
Be aware of the change in rules_scala_version. Rules_scala_version is commit’s sha. So if you want to use the newest version of the rules, check GitHub repository and copy-paste commit’s sha.
You will also notice that we have added a specific Scala version, using the code from https://github.com/bazelbuild/rules_scala#getting-started
By default Scala 2.12.10 is used and to use another version you need to specify it when calling scala_repositories.
scala_repositories takes a tuple (scala_version, scala_version_jar_shas) as a parameter where scala_version is the scala version and scala_version_jar_shas is a dict with sha256 hashes for the maven artifacts scala_compiler, scala_library, and scala_reflect:
#########################################
# Scala Toolchain #
#########################################
SCALA_MAJOR_VERSION = “2.12”
SCALA_VERSION = SCALA_MAJOR_VERSION + “.10”
scala_repositories((
SCALA_VERSION,
{
“scala_compiler”: “cedc3b9c39d215a9a3ffc0cc75a1d784b51e9edc7f13051a1b4ad5ae22cfbc0c”,
“scala_library”: “0a57044d10895f8d3dd66ad4286891f607169d948845ac51e17b4c1cf0ab569d”,
“scala_reflect”: “56b609e1bab9144fb51525bfa01ccd72028154fc40a58685a1e9adcbe7835730”,
},
))
In this file, we will set up the Scala rules and everything else that is needed to compile the Scala project.
We also have added at the end of the file:
#########################################
# External Libraries #
#########################################
load(“//3rdparty:workspace.bzl”, “maven_dependencies”)
maven_dependencies()
load(“//3rdparty:target_file.bzl”, “build_external_workspace”)
build_external_workspace(name = “external”)
//This will be used by a third-party tool called bazel-deps. We will discuss about this in upcoming sections, So skip this for now. 🙂
Step 3: A BUILD File
We need to Write a BUILD file, which tells Bazel what to build and how to build it. So lets start adding the build file for our modules – “users” and “common”
To write BUILD files we will use the following methods:
load – which loads the Bazel Scala rules, and extensions
scala_binary – generates a Scala executable
scala_library – generates a .jar file from Scala source files.
scala_test – generates a Scala executable that runs unit test suites written using the scala test library.
Let define our BUILD file for 1st Module “users” –
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "users",
srcs = ["src/main/scala/io/knoldus/UserInfo.scala"],
visibility = ["//:__pkg__"],
deps = [
"//3rdparty/com/github/julien-truffaut/monocle-core",
"//3rdparty/com/github/julien-truffaut/monocle-macro",
"//common",
],
)
scala_test(
name = "test-main",
srcs = ["src/test/scala/io/knoldus/UserInfoSpec.scala"],
deps = [":users"],
)
Here say our UserInfo.scala file will use some external third party dependency such as
val monocleCore = “com.github.julien-truffaut” %% “monocle-core” % monocleV
val monocleMacro = “com.github.julien-truffaut” %% “monocle-macro” % monocleV
,
and classes from the common module.
– In srcs we set our UserInfo.scala file, but it could be a list of files, listed one by one or a matching path pattern for example:
glob([“src/main/scala/users/*.scala”])
( then we use glob ), could even be a package with all the subpackages, such as:
srcs = glob(["src/main/scala/users/**/*..scala"])
and in
– deps all the necessary dependencies, so for this example our own sub pack package plus third part joda date time.
For now, it points to the 3rdparty folder which does not exist yet, this will be done at one of the next steps so don’t worry.
– Visibility is used to define which other targets can use this target as a dependency, in this example, we define a project folder containing the main BUILD file.
Now lets first define the BUILD file for common module –
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "common",
srcs = glob(["src/main/scala/util/*.scala"]),
visibility = ["//bazeltest:__pkg__"],
deps = [],
)
scala_test(
name = "test-common",
srcs = ["src/test/scala/util/UtilsSpec.scala"],
deps = [":common"],
)
Step 4 : Configure the Dependencies
Remember we added few lines in our WORKSPACE file –
“””
load(“//3rdparty:workspace.bzl”, “maven_dependencies”)
maven_dependencies()
“””
We will use a third-party tool for this: https://github.com/johnynek/bazel-deps
Sample Dependency file – https://github.com/johnynek/bazel-deps/blob/master/dependencies.yaml
So now one can copy from the above link or can create one dependencies.yaml file and add only required used dependencies.
Now we need to generate bazel dependencies transitively for Maven artifacts, with scala support.
Clone https://github.com/johnynek/bazel-deps and enter the bazel-deps folder. Ensure that this tool uses the same rules_scala commit sha.
Open the WORKSPACE file inside the bazel-deps and look for this:
git_repository(
name = “io_bazel_rules_scala”,
remote = “https://github.com/bazelbuild/rules_scala“,
commit = “0f89c210ade8f4320017daf718a61de3c1ac4773” # HEAD as of 2019-10-17, update this as needed
)
Now run the follow command pointing the path to your repo –
bazel run //:parse generate -- --repo-root "PATH_TO_MY_SBT_PROJECT_DIR" --sha-file 3rdparty/workspace.bzl --deps dependencies.yaml
e.g - ~/Downloads/bazel-deps$ bazel run //:parse generate -- --repo-root "/home/piyush/Downloads/Sbt2BazelBuild" --sha-file 3rdparty/workspace.bzl --deps dependencies.yaml
This will download dependencies into a 3rdparty folder into your project directory.
Step 5 : Run the Bazel
Now you are done with all the one time configuration, now lets add some code and import the code in intellij. Here are few screen shot that will help you load the project.
Run project `bazel run //:sbt2bazel` To build `bazel build //...` To Test `bazel test //...` Test individual modules Common Module `bazel test //common:test-common` User Module `bazel test //users:test-main` To check dependency graph `bazel query --noimplicit_deps "deps(//:sbt2bazel)" --output graph` We can easily visualize our dependency graph: In the command line run: `bazel query --noimplicit_deps "deps(//:App)" --output graph` To check coverage `bazel coverage \ --extra_toolchains="@io_bazel_rules_scala//scala:code_coverage_toolchain" \ --combined_report=lcov \ --coverage_report_generator="@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main" \ //... ` To view report - `lcov --list PAHT_TO_GENERATED_COVERAGE.DAT`
So Bazel is fun and it can speed up your development/ deployment like super fast, a few hand-on will make you expertise.