Working With Elasticsearch In Scala

Elasticsearch is an open-source, restful, distributed, search engine built on top of apache-lucene. You can read it more on their website.

We have an application built in scala with sbt as our build tool. Now we required to have a search capability on the output created by the application. We chose to use elasticsearch for our application.

In this post, we will learn to use elasticsearch java api in Scala. The scenario will be that we will index a json to elasticsearch. Then search the json document in elasticsearch in order to validate that it got indexed.

Let’s start by installing it. Installing Elasicsearch is simple, download it from link and unzip it. To run elasticsearch descend to unzipped directory and execute:

elasticsearch -f

“-f” tells elasticsearch to run in foreground. So, we will be able to see logs here.

Here is how it appears when it runs in foreground.

To start with the coding part, begin with adding dependency of elasticsearch in the project. At the time 0.19.8 was the latest. The artifact is available on typesafe repository. Here is the snippet in build.sbt file.

name := "elasticsearchproject"

version := "0.1.0"

scalaVersion := "2.9.2"

resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/"

libraryDependencies += "org.elasticsearch" % "elasticsearch" % "0.19.8"

Elasticsearch is schemaless. We can index any json to it. We have a bulk json file, each line is a json. For our implementation: Application reads file line by line and index json to the elasticsearch.

Here is the bulk json. Each line is a json.

{ "id": 1, "source": "twitter", "data": "first sample data" }
{ "id": 2, "source": "twitter", "data": "second sample data" }
{ "id": 3, "source": "twitter", "data": "third sample data" }
{ "id": 4, "source": "twitter", "data": "fourth sample data" }
{ "id": 5, "source": "twitter", "data": "fifth sample data" }
{ "id": 6, "source": "twitter", "data": "sixth sample data" }

Here is the complete application.

package com.meetu.elasticsearch

import org.elasticsearch.node.NodeBuilder.nodeBuilder
import scala.io.Source

object ElasticSearchApplication extends App {
  val node = nodeBuilder().client(true).node()
  val client = node.client()

  val jsons = Source.fromFile("src/main/resources/sample.json").getLines().toList
  for (i <- 1 to jsons.length) {
    indexJson(jsons(i - 1), i.toString)
  }

  def indexJson(json: String, id: String) = {
    client.prepareIndex("esa", "activityStream", id).setSource(json).execute().actionGet()
  }

}

We create a node using nodeBuilder. We then create a client from node created. There is a method indexJson that accepts json string and an id which is like a primary key.

in the prepareIndex call on client we pass three parameters
1) name of the index: “esa” in our case
2) document type: “activityStream” in our case
3) unique id: an unique value. We can use it for searching

We then set the source as json passed in the method and we are done. We now expect that json will be indexed by elasticsearch. Let’s search it.

Elasticsearch has rest api through which we can search documents on an index. Since it is rest api we can search by a GET request. Let’s use curl utility.

curl -XGET http://localhost:9200/esa/activityStream/1

here “http://localhost:9200” is url:port of elasticsearch. “esa” is name of index, recall that in the api call in scala code we used “esa” as index. “activityStream” is document type and last one is the id.

And here is the output of the curl request.

{"_index":"esa","_type":"activityStream","_id":"1","_version":3,"exists":true, "_source" : { "id": 1, "source": "twitter", "data": "first sample data" }}

We are done with a quick-start scala project that uses elasticsearch java api in Scala. It is a simple api and in a few lines of code we can index a json from our application in elasticsearch. You can explore more on the api here.