Using Spark , Spray and Couchbase for lightening fast REST Api’s

As we all know Apache Spark™ is a fast and general engine for large-scale data processing, and Couchbase is in-memory no-sql database. So by connecting these two we can get a lightening fast speed.

In this blog we are focusing on how to make CRUD operations on couchbase with Spark’s speed. I am assuming that you have a basic Spark’s installation and couchbase installation on your system. If you do not have do not worry you can easily look up the steps to download and install spark here , and couchbase here

So for using Couchbase API’s with RDD’s we need to make a build.sbt file and add this line to it.

"com.couchbase.client" %% "spark-connector" % "1.1.0"

So basically your build.sbt should look like.

Now you can access the Couchbase’s API’s for RDD’s in your code.

So now the next part is making building a service and binding it to the port.

Here we are just using the spray server for building the REST Api and binding it to the 8080 port.

Now comes the main part of implementing  the couchbase CRUD operation using Spark and spray. So for performing this we need to set the spark conf so that it may know where to insert the data.

val sparkConf: SparkConf = new SparkConf().setAppName("couchbase-spark-spray-starter-kit").setMaster("local")
  .set("com.couchbase.nodes", "127.0.0.1").set("com.couchbase.bucket.userBucket", "")
val sc: SparkContext = new SparkContext(sparkConf)

Here we are giving the node 127.0.0.1  and the bucket name here is userBucket. Using this configuration we are making the Spark Context(sc).

Now  we are going to implement the CRUD operations using the Couchbase’s API for RDD’s.

The methods saveToCouchbase(), couchbaseGet(), couchbaseView(),couchbaseQuery() are provided by couchbase so that we can perform the functionality on RDD’s . This is a basic implementation of how to perform CRUD operation on couchbase using the Spark.

We can also use the power of N1QL in this too by using the SQlContext, and they are said to be highly compatible too.

If you want you can check the code , it is in this repository.

References:
1. Spark with Spray Starter Kit by Himanshu Gupta.

2. Spark with Couchbase to Electrify Your Data Processing: Couchbase Connect 2015 by Michael Nitschinger

 

1 thought on “Using Spark , Spray and Couchbase for lightening fast REST Api’s

Leave a Reply

%d bloggers like this: